Maintain AWS Security with AWS Keys Rotation

As part of my new role as the Head of Security, I am responsible for maintaining the credentials and AWS security.

Today I will walk you through one of the best practices for AWS security called the “Key rotation”. According to this practice, you need to rotate your keys at least every 90 days, and more often if you can, so that in case of a data leak the credential would already be expired.

As much as you can you are supposed to use Roles on your AWS instances and reduce the usage of API keys as much as possible. Nevertheless, at Nuxeo, some Jenkins are outside AWS and it is still useful to be able to launch some AWS APIs from your laptop from time to time.

Before enforcing any content security policy at Nuxeo, I like to put myself in everyone’s shoes and try to make things easy for everyone. So for the above scenario, I decided to find a solution for both Shell and Jenkins key rotation.

The key rotation by itself is a simple process. It includes the following steps:

1. Create a new AWS key 2. Store the new AWS key 3. Remove the old AWS key

AWS CLI

It is always nice to be able to check some Object ACL on your computer when someone asks you to help him/her understand what the issue is with his/her S3 Object

aws --profile=nuxeo s3api get-object-acl --bucket MyBucket --key MyObject

It would be very cool if I could rotate the access key using just one command like:

aws --profile=nuxeo configure rotate-keys

This command, unfortunately, does not exist yet. The Pull request has been done here and awaiting the merge (maybe if you all add a +1, it will speed up the merge process :) )

While I am waiting, I created Yet Another AWS utils: yaws

It is available on pip

pip install yaws

It allows you to do the key rotation by typing:

yaws --profile=nuxeo rotate-keys

And just like that the keys have been updated! You can continue to use your AWS and just need to add a crontab entry.

0 0 * * * yaws --profile=nuxeo rotate-keys

There you go - you are now rotating your keys every hour without even noticing it.

Jenkins

Jenkins usually needs to do some operation on the cloud, such as push an artifact to an S3 bucket or launch an Ansible script.

There is a plugin aws-credentials-plugin that handles the storage of keys for you and also does the assumeRole operation if you specify a roleArn.

So you can have basically one AccessKey that is mapped to several AWS Jenkins credentials with different roles.

But we still need to rotate those AccessKeys, and update all the credentials that use these accessKeys.

For that we can use a Groovy script as follows:

import com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl;
import com.cloudbees.plugins.credentials.domains.*;

node {
    // Load all the AWSCredentials from Jenkins
    def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
        com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl.class,
        jenkins.model.Jenkins.instance
    )
    // Create a map of AccessKey to Credentials attached to this accessKeys
    def rotate = [:]
    // Building the map
    creds.each {
        if (!rotate[it.accessKey]) {
            rotate[it.accessKey] = []
        }
        rotate[it.accessKey].add(it)
    }
    // For each accessKey
    rotate.each {
       // Rotate the access key
        accessKey = it.value[0].accessKey
        secret = it.value[0].secretKey.getPlainText()
        withEnv(["AWS_ACCESS_KEY_ID=${accessKey}","AWS_SECRET_ACCESS_KEY=${secret}"]) {
            newKeyRaw = sh(returnStdout: true, script: "${aws_bin} iam create-access-key").trim()
            newKey = readJSON(text: newKeyRaw)
            sh(script: "${aws_bin} iam delete-access-key --access-key-id=${accessKey}")
            println("Rotate key ${accessKey} to ${newKey.AccessKey.AccessKeyId}")
        }
        // Now that the accessKey has been rotated on AWS, we need to update all credentials using it
        def credentials_store = jenkins.model.Jenkins.instance.getExtensionList(
            'com.cloudbees.plugins.credentials.SystemCredentialsProvider'
            )[0].getStore()
        it.value.each {
            cred = it
            domain = null
            // Search domain for this credentials : not optimal
            credentials_store.getDomains().each {
                curDomain = it
                credentials_store.getCredentials(it).each {
                    if (it == cred) {
                        domain = curDomain
                    }
                }
            }
            if (curDomain) {
                newIt = new AWSCredentialsImpl(it.scope, it.id, newKey.AccessKey.AccessKeyId, newKey.AccessKey.SecretAccessKey, it.description, it.iamRoleArn, it.iamMfaSerialNumber)
                credentials_store.updateCredentials(curDomain, it, newIt)
            } else {
                println("WARNING: key ${it.id} (${it.AccessKey} failed to be update with ${newKey.AccessKey.AccessKeyId}")
            }
        }
    }
}

The script requires you to be an administrator of Jenkins and not to run it in the Groovy sandbox.

Now all you need to do is create a scheduled job on Jenkins and you are all set!

HashiCorp Vault

Besides what we discussed today, there are other approaches, such as HashiCorp Vault, that can generate throwable credentials for you. It will probably be my subject for a later blog.

Read more about Nuxeo on AWS in this whitepaper.