You might have read about some of the recent data leaks which came from unsecured ElasticSearch servers directly connected on the internet.
This is why you get a thorough firewall review In security norms & regulations like PCI/DSS. In a public Cloud, updating security groups while testing and then forgetting about it, can happen really fast. Separation of duties is a way to prevent this from happening, but as you probably know, problems usually don’t have a unique cause.
The Nuxeo security team motto is: “automation, automation, automation”. And so, it seems obvious for us to face this new security challenge in an automated way. In this blog, I’m going to cover how we automated our firewall reviews and what we think about doing next in terms of automated security.
We already process all cloudtrail events in real time, so we added a token at the end of every security group rule that is a proof of acceptance from the security team. In our production environment, we enforce these tokens; any rule that does not contain or match a token authorizing the rule is removed by our automation.
A security group rule has several attributes: Protocol, Type, Target, Port, Description.
We decided to use the Description to add the token so if the rule is:
| Type | Protocol | Target | Port | Description | |
|---|---|---|---|---|---|
| Rule | INGRESS | TCP | 1.1.1.1 | 22 | NYC Office |
| Rule with Token | INGRESS | TCP | 1.1.1.1 | 22 | NYC Office$TK123 |
| Token (TK123) | INGRESS | TCP | 1.1.1.1 | 22 | NYC Office |
As we are deploying each customer in its own separate VPC, we came up with regex to validate the full deployment:
| Type | Protocol | Target | Port | Description | |
|---|---|---|---|---|---|
| Rule | INGRESS | TCP | My-core-app-sg | 22 | NYC Office |
| Rule with Token | INGRESS | TCP | My-core-app-sg | 22 | NYC Office$TK124 |
| Token (TK124) | INGRESS | TCP | .*-core-app-sg | 22 | NYC Office |
We use the regex to allow as target all the core-app-sg.
The tokens also have regex to validate on which security group they are applied at.
In our custom Security Operations Center (SOC), people can request new tokens and once the request is validated by a Security team member, the token is delivered.
We have then the capabilities to see where a token is used within our AWS environments.
Those tokens are digitally signed to make them impossible to alter from database (DynamoDB), here is an scraped export of one:
{
"_creationDate": "2020-06-17T16:37:42.056Z",
"_lastUpdate": "2020-06-17T16:37:42.056Z",
"data": "{\"GroupName\":\".*-alb-sg\",\"TokenType\":\"SG_RULE\",\"Target\":\"140.82.112.0/20\",\"Type\":\"INGRESS\",\"Description\":\"GitHub webhooks\",\"ToPort\":\"443\",\"IpProtocol\":\"TCP\",\"login\”:\”uanon\”,\”ts\":1592301098594}",
"publicKey": "-----BEGIN PUBLIC KEY-----\r\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA33UkG8UoMs+t4iw8/Th/\r\nFFpziS8MUxgKV/Trcs6kqRxz8Ytfj+F30+ylJ4uuiMxlPBZLAGPCCKOCIXUKJNvG\r\nH3AvW9zxf19+OLC0z/qdkCpf5x4LwBy9XCxwy7997YT/429Ah3RZF0GHaAR1Fwbp\r\nOkj8AlX2zv0FV9zx6i2k9eKmbsoD0a0dM7wCJq1p1So9Rj63aJXba4lDu44jLvuq\r\nZJu9Ty0cnDmsXqcR57ULByERqnx8nMCbvPWTq/Ya+UyGQuhbE7Lce3gJ5Wv3m9qU\r\npuVVJn4zGnksIl4mJZyShqDM7t4ETYstkghfOpJDI5ubuidWGOPXJIhmQ9DMtwFp\r\nox7T/q1MdosKej8VZanXQ9ELVemXhfpnnOjQK88Pc0HxbJK5nZAVaiXrb5Audi1R\r\nWRqdXeBMt9d/v9Ugl85VrOtbrlbbF7hK2L/UvxFaI1aoeXvHSkrwFtXDeNkvM8nf\r\nZp7Z0WEwjrpF7zcHE7mneZhmNHUxOQGBSbNHsQM6HNNOcczgx0RqcfbN63YA0gh6\r\nob6lbkU11qa6rYpWmwmKYj2GVEQ5TOJwFbDSH7+BwLhxRqGpro2Zx3AGXMxv/ECe\r\nlZaKHPAHllwoVrAuPS2VoqhqczkMoatouIzuMnbpBo9S3kohZVGmvmtqIghWPiW1\r\nilD0AF5Xf8BiDTNKZi2i8qECAwEAAQ==\r\n-----END PUBLIC KEY-----\r\n",
"signature": "028a286c6e81b5f74f0397e30f1fcc9e9a7b2281148eceb85d67ddc83ba8e13059c8b6835cf7c23b247afd67613fa2de892f5a282f3a27ab5c21cafcdbe30d7679b4a2ba4b77bef6342c1cb5c027ef1a3b0f376c0e147564f3bb272b6086edaf6162f7fa83febe793c9fdea3b3babf34455a6c0e206dc965ffccccfe72e39a5ec633d6f7d695da1740fea1a0e1ecb30169e5a1168f4dc41d02b7fba3d46948d68f8c0425e25429a27da52da5acf8ed07a8aaf36fbc912d99c9b1aba2c4aeb994dd50c4f190144d0a69269a0c09fb5425547e1c19afa74d2c523fc8178262dbad4f379b09036bed70469becce171548d20d051616a78b9539eb2999039332fe989511530d5b53765e350587fc0b0a9ffc513c2a85c634da64f0fdaacdb82ba1d2658dcd6d2cb4c629e75cdec74ac457ac8bd0abc1e6c2ee0821c3b5a4f150f8b7b00dcd4bdb11255ecb90063396a793c6040fd5c5b6f8b223ae8457c15e918c0ec4722cc0acf5567e1abaa7cbb2c72f722d4dd10b122b1085886b3e4d398eea2f251a2857a208c34ba4b042d427444b3bfe6d2bc8548b7d179f793762f61de3298e079767eaab2fbffccebb36d5a5d271603e962408eb088648cc6ccdcc509650ff53c3d63ce3077906a40037642ae1c70c811b58d60a0ab24362e69a07ef8b36b0dbf0719615bf1c7fa1f39d4c8177fef3159fdbddc3f19c2d3dbc95a0d24b4f",
"signedDate": "2020-06-17T16:37:39.515Z",
"signer": “usecurity”,
"type": "SG_RULE",
"uuid": "75_paKzzzeWSjq.h81cjWA"
}
So whenever a SecurityGroup is modified, we do analyze each rule to verify they contain a token, if they don’t,we delete the rule to avoid any invalidated traffic. Also, as soon as we revoke an approval, it will remove all matching rules within the environments.
One of the goals of our team is to simplify as much as we can those processes, so if we identify a rule that matches a token but doesn’t contain one, we will add it on behalf of the user. We can then let them know that the process wasn’t respected, but work can continue seamlessly.
The process has been in place for almost 2 years now. We’re very happy with its implementation and the requirements are better understood by our teams.
But I have to admit, I was also responsible for an outage of an hour when we deployed the first time. Let me tell you what happened: in one of the first deployments, a bug sneaked in, letting the automation think the tokens were not legitimate, which led to a removal of all security group rules. We quickly identified the issues and our production team recovered within an hour. I could try to pretend it was some chaos engineering to ensure the disaster recovery process was working well but I’m too honest for that. We learn from our mistakes.
What’s next? We will soon be adding some DNS resolutions within token to allow some rules to include DNS name, so our system will help our teams to allow our office provider inside their system.We will also backport it as an IvoryShield module, let us know if you want to know more about it.
I hope this helped you understand how we manage our firewall review and I hope to share more of our security processes with you in the future, to continue showing you the Nuxeo Platform is best suited for organizations who have deep and complex regulations environments.
Read more about Nuxeo on AWS in this whitepaper.
Learn everything about Nuxeo Secure Content Management Systems