An “interesting” part when working with APIs or runtime environments like containers is of course credential management. Somehow you have to let the script know the credentials to use the API or provide the container with the password to use a service. A straightforward approach would be to hardcode the credentials into the source code, but that is obviously also the least secure. Well, actually it’s not only not secure, but really dangerous – there are automated malicious scanners that find cleartext passwords in source code within minutes and immediately exploit these.

Hardcoded plaintext credentials

Hardcoded credentials in source code are a prevalent problem. Fortunately, there are also many tools – such as detect-secrets and truffleHog – that help to find and alert on cleartext credentials, before they make their way into a source code control system like GitHub.

Amazon Web Services (AWS) as the world’s largest cloud provider also has an interest in protecting clients from working with clear text credentials. If threat actors detect keys that provide access to a cloud user’s account, the bad guys can log in and do harm to the running infrastructure or incur high costs by mining cryptocurrencies. While this is not an immediate security threat for AWS itself, it certainly damages their reputation when suddenly a considerable part of their cloud workloads is due to malicious actors exploiting client environments. To help clients mitigate this risk, they have their own credential detection tool named git-secrets.

Ironically, even in some security-focused posts I saw the bad practice of hardcoding credentials in the clear, like in a post that explains how to securely store parameters:

“First we setup our AWS credentials. The AWS access key and AWS secret keys seem like the perfect things to store in Parameter Store. Trouble is you kinda need them off the bat to even connect to AWS.

You can setup the shell environment your Ruby code will be running in to get around this, but for now hard-coding them will do.“

WebDriver and AWS: Use AWS Parameter Store to Guard Your Secrets.

I think the author is well aware that this should never be used in production, but it would not be the first time that code that was intended for some quick tests makes it into production. In every case, the provided code encourages users to simply type their access key and secret key directly into the code:

credentials = Aws::Credentials.new('YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY')

Bad idea!

Credential injection via environment variables

Many developers know of course that hardcoded credentials are not the way to go and hence they choose another solution: Passing credentials in the form of environment variables.

When playing around with the AWS Rekognition API, I found an example in the AWS documentation that provides code to detect an analyze faces in an image. To use the API, you have to somehow authenticate. The exemplary script contains the following part:

   credentials = Aws::Credentials.new(
      ENV['AWS_ACCESS_KEY_ID'],
      ENV['AWS_SECRET_ACCESS_KEY']
   )

Now, the positive side of this code is that it does not directly encourage to type cleartext credentials directly into the script. Instead, you would pass environment variables. There are however several problems with that:

• The history of your CLI would save the credentials in cleartext. The example below shows the AWS Cloud9 IDE, which stores the command history across sessions:

• When the application crashes, it might dump the environment variables into a log file, for example CloudWatch.
• Equally, if your application is configured in a way that it logs a lot of information into the system logs, environment variables might well be included.

As should be clear by now, having these credentials somewhere in cleartext – even if they are not hardcoded in the script itself – is a real security threat:

“The access_key and secret_key associated with your AWS account are the keys to your castle. Anyone who gains access to these can stop all of your instances, delete all of your EBS volumes and snapshots, and wipe out all of your S3 objects. You need to manage those credentials very carefully and, somehow, storing them in clear text on a server that is potentially accessible to the Internet doesn’t sound like a good way to manage them.”

Mitch Garnaat – Python and AWS Cookbook, p. 40

The problem is especially prevalent if it is unclear who can access these system logs. An application itself might be well protected from being accessed by other accounts, but totally unrelated users might well have access to the log stream the application is writing to. Amazon explains in detail how for example environment variables in AWS Lambda serverless functions can be hidden.

To make things yet more complicated, in some cases there also are bugs in applications that lead to cleartext credential leakage. For this reason, it’s always a good idea to monitor application logs and alert on cleartext credentials.

Stealing container credentials

Especially in “lightweight” scenarios where code is not running in an own VM, but simply serverless/in a container, a curious admin or an attacker with access to the host system can read out such environment variables extremely easily. You can quickly try this out in your own Linux environment:

1. Start up a docker instance, passing a “secret” environment variable, e.g. (run as sudo if your user is not part of the Docker group):

docker run -it --rm -e AWS_SECRET_ACCESS_KEY=VERY_SECRET_KEY ubuntu sh

2. In a second terminal, check for the process ID of the sh process which we just spawned. This is possible because the container shares the same host environment, so – different from a VM – the process inside the container is visible from the host itself!

ps -C sh

3. As admin on the host you can easily dump the environment variables of the running container:

sudo cat /proc/*PID*/environ

As you can see, data passed as environment variable is also not safe. In fact, AWS provides guidance on how to securely pass secrets to containers – environment variables are not part of the description. There are several techniques for injecting credentials (1, 2) that might be suitable depending on the specific use case. As Jens Eickmeyer explains in his very well-written blog post:

“AWS offers different ways for managing secrets in a secure way […]. The approach which matches the requirements best can be used. There’s no excuse to store secrets in cleartext somewhere in the source code or configuration.”

Jens Eickmeyer – Managing Secrets for AWS Lambda

All this however does not help if a bug in some part of the management infrastructure allows to steal authentication tokens. While such vulnerabilities are dangerous, they are of course not an excuse to go without proper credentials management. Therefore, in my opinion it is not a good practice that AWS provides sample code that encourages to pass cleartext credentials via environment variables. Sure, the idea is to give an immediate sense of achievement to people interested in the AWS SDK. I just hope that these people will then also understand how to change their code later on in production.