Table of Contents
I recently had a look at the security assessments of different well-known cloud providers and found some interesting discrepancies. It shows at least four things in my opinion:
- There are indeed differences in how security is handled across cloud providers and for customers it is well worth to compare as part of their due diligence.
- Even though questions in an assessment may be standardized, they are subject to interpretation by the assessed provider.
- Before even looking at an assessment, check the scope: Are scopes across assessments comparable? Is what you are actually interested in also in scope? Different answers on the same questions in the assessment also can stem from differences in scope.
- For the abovementioned reasons, just looking at certifications and counting how many times a provider crossed the “yes, we can” answer in a self-assessed security questionnaire will not tell you much about the real security posture of a provider, but only lure you into a false sense of security.
Before looking at the specific differences, let’s first understand how cloud provider security can be assessed and compared.
Cloud Security Assessments
The Cloud Security Alliance (CSA) is a well-known not-for-profit organization that helps companies to build secure cloud environments and to choose safe cloud providers. Apart from certifications for individuals like CCSK (Certified Cloud Security Knowledge), it has several free tools on offer to facilitate provider selection:
- Cloud Controls Matrix (CCM): Contains information on over 130 best-practices in 16 security domains and maps these to different security and compliance standards. It is meant to be used as starting point for a company’s own cloud assessment program, especially to ensure that compliance requirements are met.
- Consensus Assessments Initiative Questionnaire (CAIQ): Template that can be used by cloud providers to document security and compliance controls. The questions are very similar to specifications in CCM, but in very direct language to avoid ambiguity.
A whole lot of companies – including Alibaba, Amazon, Google, IBM and Microsoft – have answered the many questions in the CAI Questionnaire. Thanks to this standardized format, security and compliance controls across these (and many other) providers can easily be compared. CSA has published these assessments as in their “Security Trust Assurance and Risk (STAR) Registry“.
Companies participating in the STAR program can reach different levels of assurance. Most companies are on “STAR Level 1”. This means that they filled out the CAIQ to (hopefully) their best knowledge and belief, but without verification on behalf of a third party. If another party assesses the cloud provider’s environment either regarding ISO/IEC 27001 certification or regarding SOC 2 attestation, the cloud provider reaches STAR Level 2. Theoretically there is also a STAR Level 3 that is about continuous auditing; CSA is still defining the requirements for this level, but once it is finished, it will be a very useful one, relieving the problem that current audit results are just observations at one specific point in time. As Kim Zetter writes in her great book “Countdown to Zero Day“:
“[…] compliance doesn’t guarantee a system won’t get hacked. Security is an evolving condition, not a static one, and can change anytime new equipment is installed or configurations are changed.”Zetter, Kim. Countdown to Zero Day (p. 178).
I had a look at five of the most well-known cloud providers and discovered quite a few differences in their respective answers. I picked three examples, but it is well worth to have a look by yourself. Interestingly, Microsoft registered in STAR even before market-leader Amazon, as you can see from the registration dates below.
|Cloud Provider||Products||STAR Registration Since||STAR Level||Certification Body||Self-Assessment|
|Alibaba||Alibaba Cloud||April 23, 2014||Level 2 Certification and Attestation||British Standards Institution (BSI)||Link|
|Amazon||Amazon Web Services (AWS)||July 20, 2012||Level 2 Certification||Ernst & Young (EY)||Link|
|-Google Cloud Platform (GCP)|
|February 21, 2017||Level 2 Attestation||Ernst & Young (EY)||Link|
|IBM||-IBM Cloud Infrastructure|
-IBM Cloud Services
|October 16, 2017||Level 1, but ISO27001 & SOC 2 Type 2 reports available (necessary for Level 2)||N/A|
(SOC reports by PricewaterhouseCoopers (PwC) )
|Microsoft||-Azure -Microsoft Cloud App Security|
|March 30, 2012||Level 2 Certification and Attestation||British Standards Institution (BSI)||Link|
After this general description of CSA STAR and information on the covered cloud providers, let’s have a look at some differences.
What I should mention here is that I did not do a full comparison between the abovementioned providers. Instead, I was looking at a few different of the 16 domains in the STAR Level 1 self assessment (CAIQ) for one product of each provider. Obviously, to make things more comparable, I tried to look at similar products, namely the IaaS offering from each provider.
All of the questions can be answered with “yes”, “no”, or “n/a”. Since “yes” for most questions is the “preferable” answer (meaning “yes, we have this security measure in place”), I was especially interested in such questions where a provider answered with “no”.
There is no clear-cut definition of what comprises IaaS, PaaS and SaaS – and what was actually assessed might well differ among each provider. Moreover, if an offering works with tightly interconnected components from IaaS, PaaS and SaaS, an individual review might not be possible.
There are service model-specific questions in the CAIQ stating something along the lines of “(SaaS only)” (e.g. AIS-01.5), so you would assume that in an IaaS assessment each provider answered with “n/a”. This is however not the case: Among the five compared providers, only IBM ticked “n/a” for their Cloud Infrastructure. All others responded with “yes”, implying that their assessed services contain SaaS components. This difference in scope obviously needs to be taken into consideration when comparing answers.
In the following I will present examples from different domains where I found the differences in answers especially noteworthy.
Application & Interface Security (AIS)
The very first question in the CSA CIAQ is about application security:
|Applications and programming interfaces (APIs) shall be designed, developed, deployed, and tested in accordance with leading industry standards (e.g., OWASP for web applications) and adhere to applicable legal, statutory, or regulatory compliance obligations.|
The third question in that category is about manual code review:
|Do you use manual source-code analysis to detect security defects in code prior to production?|
This is in fact an interesting question, because it refers to two contrary imperatives for cloud providers:
- Need for maximal standardization of contracts and offerings in order to cut costs and benefit from economies of scale.
- Need for maximal security to avoid loss of reputation; to reduce this substantial business risk, considerable investments in security are necessary.
Of course, investments in security can also benefit from economies of scale if performed by a cloud provider instead of by each individual tenant. Nevertheless, whenever there is a “manual” component (as in “manual source-code analysis”), costs in this cost-driven business rise. Let’s see what the different CSPs answered:
|Product||Answer||Notes (as provided in the respective CAIQ)|
|Alibaba Cloud||Yes||Yes, Alibaba Cloud follows the rules in the SPLC process and scans the code base through an automated code security scanning tool to ensure that products that fail the code scan and product that fail the manual code security audit cannot be officially launched. Detailed control measures can be found in the SOC2/3 report.|
|Amazon AWS||No||Manual source-code analysis is not employed. Automated code analysis tools are run as a part of the AWS Software Development Lifecycle.|
|Google Cloud Platform||Yes||Google follows a structured code development and release process. As part of this process, all code is peer reviewed. Google makes purpose built code analysis tools available for engineers to deploy against application code. Google also performs continuous post-production tests based on real-time threats.|
|IBM Cloud Infrastructure||Yes||The IBM Cloud IaaS system development lifecycle is built on the foundation of NIST 800-53 with special consideration of other relevant standards such as ISO27001. Including best practices of rule implementation for the development of software and systems, formal change control, utilization of test environments to conduct rigorous testing prior to release.|
NIST 800-53 control group SC
|Microsoft Azure||No||A Final Security Review (FSR) is performed for software releases prior to production deployment by a designated Security Advisor outside of the Microsoft Azure development team. Web applications are scanned with the PortSwigger Burp Suite scanning solution.|
Amazon and Microsoft answered with “no”, just as you could expect from a large provider operating at scale. However, Microsoft’s “Final Security Review […] performed […] by a designated Security Advisor” could be interpreted as manual review – I wonder, what that advisor is otherwise doing after all.
Alibaba refers to “SPLC”, without going into more detail. However, in security whitepapers for Alibaba Cloud we can find:
In the secure product lifecycle (SPLC) of cloud products, Alibaba Cloud security experts strictly review and evaluate the code security on each development node to guarantee the code security of Apsara Stack products.Alibaba CloudApsara Stack EnterpriseSecurity Whitepaper, p. 20
It should not come as a surprise that solely looking at the CAIQ alone will not suffice to fully understand a CSP’s security posture.
Microsoft goes into detail and even mentions specific software, while IBM seems to remain on the most abstract level, using one general answer for several questions combined. From a security perspective, I consider Google’s response to be the most comprehensible one, explaining both manual and automatic reviews, including continuous monitoring after deployment.
Infrastructure and Virtualization Security (IVS)
CSA recommends the following measures for “Change Detection” as part of “Infrastructure and Virtualization Security”:
|The provider shall ensure the integrity of all virtual machine images at all times. Any changes made to virtual machine images must be logged and an alert raised regardless of their running state (e.g., dormant, off, or running). The results of a change or move of an image and the subsequent validation of the image’s integrity must be immediately available to customers through electronic methods (e.g., portals or alerts).|
Question 1 in this section reads as follows:
|Do you log and alert any changes made to virtual machine images regardless of their running state (e.g., dormant, off or running)?|
And here are the different answers given by the cloud providers:
|Product||Answer||Notes (as provided in the respective CAIQ)|
|Alibaba Cloud||Yes||Yes, Alibaba Cloud records all changes, and the change log will be included in the scope of log monitoring.|
|Amazon AWS||No||Virtual Machines are assigned to customers as a part of the EC2 service. Customers retain control over what resources are being used and where resources reside. Refer to the AWS website for additional details – http://aws.amazon.com|
|Google Cloud Platform||Yes||Google machine configuration changes are continuously monitored when online.|
|IBM Cloud Infrastructure||No||Once a customer takes control of their virtual machine they are able to control what resources are being used.|
|Microsoft Azure||Yes||Read and write operations to virtual machines are logged via storage analytics, which the customer can view within their own storage account.|
Azure Virtual Machines staged in the Azure Gallery are maintained according to established software asset management procedures, which includes update logs to stored VHDs.
Several things stand out here: Amazon and IBM are the only ones who answer with “No”. They both use a very similar wording which – to my understanding – indicates that the customer itself will be required to monitor changes in a VM.
Amazon and IBM talk about VMs, while the question is also about VM images, i.e. the template from which a VM will be launched. While the customer can of course install whichever software in a running VM, it is another question if changes to a golden image are being logged, since this requires logging technology enabled by the IaaS provider. This is what I mentioned in my second observation at the beginning: Different interpretations of the same question.
It is also noteworthy that Google answers “yes”, but then explains that changes are monitored while the machine is “online”. The CSA question is explicitly about “dormant, off or running” images. I think this is another example for what I was mentioning in my second observation above, as well as a question of scope, which I mentioned in the third observation. On the other hand, Google does recommend and support the usage and monitoring of immutable images, which are a good idea for both stability and security.
Among the five providers, only Microsoft’s answer seems to specifically address the different dimensions of the question, distinguishing the VMs running state and providing details on what is logged and where it is stored.
Business Continuity Management & Operational Resilience (BCR)
CSA recommends the following precaution regarding “Equipment Location” as part of “Business Continuity Management & Operational Resilience”:
|To reduce the risks from environmental threats, hazards, and opportunities for unauthorized access, equipment shall be kept away from locations subject to high probability environmental risks and supplemented by redundant equipment located at a reasonable distance.|
Question 1 in this section wants to know:
|Are any of your data centers located in places that have a high probability/occurrence of high-impact environmental risks (floods, tornadoes, earthquakes, hurricanes, etc.)?|
As you can see, the way the question is stated, our preferred answer in this case would obviously be a “no”. Let’s see what the providers had to say:
|Product||Answer||Notes (as provided in the respective CAIQ)|
|Alibaba Cloud||No||No, during the data center site selection phase, Alibaba Cloud will assess the high-risk environment to ensure that the data center is not built in a high-risk environment|
|Amazon AWS||No||Each AWS data center is evaluated to determine the controls that must be implemented to mitigate, prepare, monitor, and respond to natural disasters or malicious acts that may occur. Refer to ISO 27001 standard, Annex A domain 11 and link below for Data center controls overview: https://aws.amazon.com/compliance/data-center/controls/|
|Google Cloud Platform||Yes||Google carefully selects the locations of its datacenters to avoid exposure to high-impact environmental risks to the extent possible.|
|IBM Cloud Infrastructure||Yes||IBM Cloud IaaS’s Physical and Environmental Protection controls are in place in all data centers and are validated frequently through internal audits and by independent auditors.|
NIST 800-53 control group PE
|Microsoft Azure||No||Microsoft data center site selection is performed using a number of criteria, including mitigation of environmental risks. In areas where the exists a higher probability of earthquakes, seismic bracing of the facility is employed. Data centers are built as redundant, highly-available components of the Azure platform.|
Environmental controls have been implemented to protect systems inside the facility, including temperature and heating, ventilation, and air conditioning (HVAC) controls, fire detection and suppression systems, and power management systems.
This is a question where I think it very much depends on how you want to define “high probability of occurrence”. As you can see from the answers, Alibaba, Amazon and Microsoft say that their DCs are only built in safe areas, while Google and IBM see a real risk of natural disaster.
However, when you look at the notes, also Google and IBM state that they carefully select each location to “avoid exposure to high-impact environmental risks” (Google) – very similar to Alibaba who “ensure that the data center is not built in a high-risk environment”.
Microsoft even somehow negates their own answer, by admitting that they do build in risky areas: “In areas where the [sic] exists a higher probability of earthquakes, seismic bracing of the facility is employed.”. Although the Microsoft description is again the most detailed (and the one with most typos / spelling inconsistencies), in this case it appears artificially inflated, as it basically describes a standard setup of a professional data center: Heating, cooling, power management, etc. This is not really different from what IBM – who instead answered with “yes” – describe as “Protection controls are in place in all data centers”.
So, again there is a lot of room for interpretation due to the lack of specific measurements. Only by reading the notes we discover that the opposite answer in fact means the same.
Audit Assurance & Compliance (AAC)
In the “Independent Audits” part of the Audit Assurance & Compliance domain, the CSA best-practice reads as follows:
|Independent reviews and assessments shall be performed at least annually to ensure that the organization addresses nonconformities of established policies, standards, procedures, and compliance obligations.|
The sixth question is about the publication of test results:
|Are the results of the penetration tests available to tenants at their request?|
The differences in answers here seem to not solely come from differences in interpretation, but indeed from differences in policies and processes:
|Product||Answer||Notes (as provided in the respective CAIQ)|
|Alibaba Cloud||No||No, the penetration test results on the side of Alibaba Cloud platform will be followed up by the cloud platform itself.|
|Amazon AWS||No||Although AWS Security regularly engages carefully selected industry experts and independent security firms to perform recurring penetration testing, we do not share the results directly with customers. Instead, the results are reviewed and validated by our auditors.|
|Google Cloud Platform||No||Google’s Security Policy prohibits sharing this information but customers may conduct their own testing of our products and services.|
|IBM Cloud Infrastructure||No||[…]|
IBM Cloud IaaS cannot risk the security of systems and all customers by releasing the results of penetration tests however customers should leverage IBM Cloud IaaS’s third party certifications as external auditors do review samples of penetration testing.
|Microsoft Azure||Yes||Yes, summary penetration testing reports are available to customers under NDA. For more information, visit the Microsoft Service Trust Portal, or contact your Microsoft account representative.|
Interestingly, only Microsoft provides penetration test results to customers. All other providers argue that releasing vulnerability findings would jeopardize cloud security. However, I do not think that Microsoft would release pentest results right away, but only after having fixed the issues. Considering the advantages of responsible disclosure, I find it unfortunate that most providers are unwilling to disclose anything, and instead following the idea of security by obscurity. However, even though the providers will have to fix security issues by themselves, tenants might still be able to learn from penetration test reports. Microsoft sets itself apart with a reasonable policy that equally highlights the mutual trust relationship between provider and customer.
Google highlights that customers can perform their own pentests at any time. This is of course a double-egded sword:
- Many providers do not freely allow their customers to perform penetration tests for several reasons: In a multi-tenant environment such tests would obviously affect other customers, but even in a dedicated space it would still be an “attack” against infrastructure owned by the CSP. In this sense, the Google policy gives freedom to its customers which is not standard.
- With freedom comes responsibility; since customers are allowed to perform pentests, they will have a hard time arguing that Google did not properly protect them, because in the Shared Responsibility Model, Google is shifting some responsibility to the customer.
Threat and Vulnerability Management (TVM)
CSA recommends the following for Vulnerability / Patch Management:
|Policies and procedures shall be established, and supporting processes and technical measures implemented, for timely detection of vulnerabilities within organizationally-owned or managed applications, infrastructure network and system components (e.g., network vulnerability assessment, penetration testing) to ensure the efficiency of implemented security controls. A risk-based model for prioritizing remediation of identified vulnerabilities shall be used. Changes shall be managed through a change management process for all vendor-supplied patches, configuration changes, or changes to the organization’s internally developed software. Upon request, the provider informs customer (tenant) of policies and procedures and identified weaknesses especially if customer (tenant) data is used as part the service and/or customer (tenant) has some shared responsibility over implementation of control.|
The fourth question in the CSA CAIQ is as follows:
|Will you make the results of vulnerability scans available to tenants at their request?|
Note the similarity of this question to the previous question above (AAC-02.6). However, the previous question was on penetration tests, the question now is on vulnerability scans. The latter is often done using automatic scanning tools to quickly discover known vulnerabilities in commonly used software. A penetration test instead usually is a highly manual activity where ethical hackers try to enter a client’s custom infrastructure.
Obviously, a penetration test report will contain much more data on internal infrastructure and processes than an automated vulnerability scan report. Surprisingly, providers are even more restrictive when it comes to the release of vulnerability scans – none of the providers does it:
|Product||Answer||Notes (as provided in the respective CAIQ)|
|Alibaba Cloud||No||No, Alibaba Cloud doesn’t provide the scan results to customers.|
|Amazon AWS||No||AWS does not share the results directly with customers. AWS third-party auditors review the results to verify frequency of penetration testing and remediation of findings.|
|Google Cloud Platform||No||Google does not make internal vulnerability scan results available to customers. Customers may perform their own scans on their own projects or public address space at any time.|
|IBM Cloud Infrastructure||No||IBM Cloud IaaS vulnerability and patch management procedures and processes are part of the core controls that are maintained internally and validated by external auditors.|
IBM Cloud IaaS cannot release the results of these scans, as it would put systems and customers at risk. Customers should leverage the SOC report and ISO27001 certificate as these controls are tested within both of these assessments by third party auditors.
Customers are permitted to scan their environment at any time.
|Microsoft Azure||No||Azure does not provide scans of customer VMs or any customer applications running on the VMs. Patching customer VMs is the responsibility of customers.|
Again, looking at the notes shows some differences in how the questions were interpreted. It seems that Microsoft is understanding the question in such a way that the vulnerability scan would be on applications managed and run by the customer itself. They argue that they do not scan customer applications and that patching these is the responsibility of each customer.
On the other hand, all other four providers do state that they perform vulnerability scans, but they are not providing results. I do not think that providers are actually scanning customer applications, because as Microsoft states, it is indeed the responsibility of the client to keep these safe. However, if the question is understood in such a way that the CSP’s own internal infrastructure should be subject to these scans, it would only be consequent if Microsoft also released these vulnerability scan reports, considering that they do provide (usually much more detailed) penetration test reports.
Amazon seems to have completely missed the intention of the question, as they continue talking about “penetration tests”, while the question is explicitly about vulnerability tests. Then again, it is not unheard of that security service providers market a standard vulnerability scan as penetration test, so I do not think that circumventing a real answer was ill intent.
Although the question was not about the customer’s rights to perform own tests, looking at the notes does bring some further insights here. Only Google explicitly allowed customers to perform their own penetration tests; when it comes to customers running (usually less intrusive) vulnerability scans, both IBM and Google specifically grant that permission. As explained for the previous question, this of course comes with greater responsibility on behalf of the client, but it is an important right that apparently not all providers readily grant.
What we need to be aware when reading these answers is again that everyone could have a slightly different understanding. One main component of the cloud is Software-Defined Networking, which means that tenants will be working in a virtualized network. Even if an answer in the CAIQ states that customers can perform their own tests, it is highly unlikely that the CSP allows tests against the CSP’s physical perimeter. Instead, they are probably talking about the customer’s responsibility to perform security testing of the applications which they manage. If CSPs state that they do not publish VA / penetration test results, they are most likely talking about tests on the underlying infrastructure which the client cannot easily penetrate from within an SDN.
Even if a CSP allows a client to perform security testing in the cloud, it should be clarified what kind of tests are ok. A test can basically performed from two locations:
- Penetrate system from an external location, with all security controls like WAF, SDN firewalls (aka security groups), IDS/IPS etc. in between.
- Install an agent on or directly in front of the system that shall be tested.
While some people say that the first option is the more realistic one to simulate the attack of an external hacker, the CSA recommends the second option. The idea behind this is that in case a system is accidentally exposed to the internet, attackers could directly penetrate that system, without having to worry about additional security controls.
Finding the right level of detail
Having a look into these self-assessments also provides some interesting insights into what technology and is used and in which way. Microsoft for example explains that it uses Burp Suite for penetration testing (see AIS-01.3 above). For question AIS-01.5 regarding vulnerability testing, Microsoft outlines the usage of Qualys:
Procedures have been established and implemented to scan for vulnerabilities on hosts in the scope boundary. Vulnerability scanning is performed on server operating systems, databases, and network devices with the Qualys Guard vulnerability scanning tool. The vulnerability scans are performed on a quarterly basis at minimum, but Azure security teams employ continuous monitoring processes to detect potential issues on an ongoing basis. Microsoft Azure contracts with independent assessors to perform penetration testing of the Microsoft Azure boundary. Red-Team / Blue-Team exercises (See AIS-01.1) are also routinely performed and results used to make security improvements.Answer on AIS-01.5; Source: Microsoft
While such insights are interesting from a technical standpoint, the question arises how sensible it is to put rather short-lived details into an assessment questionnaire. Microsoft might for example change its vulnerability scanning solution – or the solution itself changes: QualysGuard is now called Qualys Cloud Platform. The assessment itself of course is still valuable, but if too much detail is put into the answers, information might be outdated very quickly.
A better approach could be to keep the answers in the assessment at a higher level (but detailed enough to remain meaningful), while putting technical details into periodically released whitepapers. For example, in Alibaba’s April 2020 “CloudSecurity White Paper”, there are almost 200 pages filled with technical details (e.g. usage of reviewable open-source software Nginx Tengine for load balancing, p. 77) that allow to deepen the understanding gained from CAIQ.