If you are an AWS nerd, s3 encryption probably does work the way you think it works. But for most folks who have a casual or non-technical relationship with AWS, the way it works is likely a bit of a surprise.
Why does it matter?
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) recently warned organisations about a breach at business intelligence company Sisense. Brian Krebs did some Krebsing (yes it’s a word, and a very good one at that), and found sources that said, “the attackers used the S3 access to copy and exfiltrate several terabytes worth of Sisense customer data, which apparently included millions of access tokens, email account passwords, and even SSL certificates.”
This sparked lots of gossiping and discussion on social media and private communities, speculating about what might have happened, and what controls might have failed or been missing. One such line of speculation was that s3 bucket encryption was not enabled, and how it would have saved Sisense if it had been enabled.
I have no idea if bucket encryption was enabled or what the attackers did, but I do know that if they downloaded the data from S3 as is claimed, it’s likely completely irrelevant if bucket encryption was enabled. Every cloud standard on earth requires encryption of all the things and every auditor will check for it, often because of the belief it will protect against data exfiltration.
Let’s explore why that belief is wrong.
Basic S3 encryption
It all starts with this little nugget, “all new object uploads to Amazon S3 are automatically encrypted”.
Okay let’s give it a try by creating a bucket.
% aws s3api create-bucket --bucket s3abovetheline
Then checking its encryption status.
% aws s3api get-bucket-encryption --bucket s3abovetheline --profile awseye-admin
{
"ServerSideEncryptionConfiguration": {
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256"
},
"BucketKeyEnabled": false
}
]
}
}
Since ApplyServerSideEncryptionByDefault “specifies the default server-side encryption to apply to new objects in the bucket”, we have confirmed AES256 is used to encrypt objects in this bucket. Awesome, encryption is great!
Let’s try to upload a file.
% echo "neverbelow" > neverbelow.txt
% aws s3 cp ./neverbelow.txt s3://s3abovetheline/neverbelow.txt
upload: ./neverbelow.txt to s3://s3abovetheline/neverbelow.txt
I guess the encryption happened?
% aws s3api head-object --bucket s3abovetheline --key neverbelow.txt
{
"AcceptRanges": "bytes",
"LastModified": "2024-04-18T06:22:29+00:00",
"ContentLength": 11,
"ETag": "\"092bd32a0cbb8e481dabab53dec6b671\"",
"ContentType": "text/plain",
"ServerSideEncryption": "AES256",
"Metadata": {}
}
Perfect. Now, let’s download it and see if it’s encrypted.
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow-downloaded.txt
download: s3://s3abovetheline/neverbelow.txt to ./neverbelow-downloaded.txt
% cat ./neverbelow-downloaded.txt
neverbelow
Wait… Why can we read the contents of the file if it’s encrypted?
The answer is in the type of encryption used – server-side encryption with Amazon S3 managed keys (SSE-S3). All of the encryption operations happen transparently, without the need for the user to perform encryption or decryption operations explicitly.
This is where common understanding starts to diverge from reality. If an attacker has access to an s3 bucket that’s using the default encryption, they won’t ever see or interact with the encrypted version of the file. They can just smash the download button, and get the clear-text copy.
Compliance ✅
Customer managed key with KMS
What happens if an organisation does something more sophisticated than just the default?
Much of the encryption magic inside AWS happens around the Key Management Service (KMS). AWS services that use KMS keys to encrypt resources often create keys for you. KMS keys that AWS services create in an AWS account are AWS managed keys.
The KMS keys that you create are customer managed keys. Customer managed keys are KMS keys in your AWS account that you create, own, and manage. You have full control over these KMS keys, including establishing and maintaining their key policies, IAM policies, and grants, enabling and disabling them, rotating their cryptographic material, adding tags, creating aliases that refer to the KMS keys, and scheduling the KMS keys for deletion.
Customer managed keys (CMK) is what we want! Let’s make a new CMK.
% aws kms create-key --description "Best key ever"
{
"KeyMetadata": {
"AWSAccountId": "0123456789012",
"KeyId": "ad1c83a3-c780-45eb-bccc-5c933abb8729",
"Arn": "arn:aws:kms:us-east-1:0123456789012:key/ad1c83a3-c780-45eb-bccc-5c933abb8729",
"CreationDate": "2024-04-18T16:53:31.916000+10:00",
"Enabled": true,
"Description": "Best key ever",
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"Origin": "AWS_KMS",
"KeyManager": "CUSTOMER",
"CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
"KeySpec": "SYMMETRIC_DEFAULT",
"EncryptionAlgorithms": [
"SYMMETRIC_DEFAULT"
],
"MultiRegion": false
}
}
The output shows the key has been created, is enabled, is managed by the customer (us) and can be used for symmetric encryption.
Next, we upload the file again, this time using our newly minted key.
% aws s3 cp ./neverbelow.txt s3://s3abovetheline/neverbelow.txt \
--sse aws:kms --sse-kms-key-id ad1c83a3-c780-45eb-bccc-5c933abb8729
upload: ./neverbelow.txt to s3://s3abovetheline/neverbelow.txt
It looks like it worked. Did it work?
% aws s3api head-object --bucket s3abovetheline --key neverbelow.txt
{
"AcceptRanges": "bytes",
"LastModified": "2024-04-18T06:55:43+00:00",
"ContentLength": 11,
"ETag": "\"3d56bbcb4a13138189ce3e2f7a7e8d22\"",
"ContentType": "text/plain",
"ServerSideEncryption": "aws:kms",
"Metadata": {},
"SSEKMSKeyId": "arn:aws:kms:us-east-1:0123456789012:key/ad1c83a3-c780-45eb-bccc-5c933abb8729"
}
The output tells us it worked. The file is encrypted with our key, not an Amazon key.
Let’s download it.
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow-downloaded.txt
download: s3://s3abovetheline/neverbelow.txt to ./neverbelow-downloaded.txt
% cat neverbelow-downloaded.txt
neverbelow
Damnit… we didn’t even have to specify the key. It’s still all magic. This is awkward.
What’s missing? Well, we can decrypt the file because we have access to the KMS key we created. Duh! Let’s add a key policy (because we can do that with CMKs) to make sure no one can use it to decrypt anything with it.
{
"Version": "2012-10-17",
"Id": "key-default-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::0123456789012:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Deny Decrypt Operation",
"Effect": "Deny",
"Principal": "*",
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
% aws kms put-key-policy --key-id ad1c83a3-c780-45eb-bccc-5c933abb8729 \
--policy-name default \
--policy file://./deny-decrypt.json
Now that we’ve explicitly disallowed anyone from using the key for decryption, we’ll have no choice but to download the encrypted file.
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow-downloaded.txt
download failed: s3://s3abovetheline/neverbelow.txt to ./neverbelow-downloaded.txt An error occurred (AccessDenied) when calling the GetObject operation: User: arn:aws:sts::0123456789012:assumed-role/attacker is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:0123456789012:key/ad1c83a3-c780-45eb-bccc-5c933abb8729 with an explicit deny in a resource-based policy
Nope. No file for us. 😭
The way AWS has chosen to implement encryption often makes the interaction between the service (e.g. S3) and KMS feel like access control rather than encryption. If an object in S3 is encrypted with a customer managed key than the person trying to access the contents of that object needs to have access to both the object AND the key. If they don’t have access to either, the access doesn’t even start.
Compliance ✅
Customer provided key with KMS
In case a customer doesn’t want AWS to manage their keys but still wants AWS to do the encryption and decryption work, there’s the server-side encryption with customer-provided keys (SSE-C) option. In this flow, the key needs to be included in every upload and download request. Instead of storing the key, AWS only stores a Hash-based Message Authentication Code (HMAC), which is a just a fancy hash.
Let’s try to upload a file with our very hard to guess secret key 12345678901234567890123456789012.
% aws s3 cp ./neverbelow.txt s3://s3abovetheline/neverbelow.txt \
--sse-c AES256 \
--sse-c-key 12345678901234567890123456789012 \
It’s no longer possible to even get information about the object.
% aws s3api head-object --bucket s3abovetheline --key neverbelow.txt
An error occurred (400) when calling the HeadObject operation: Bad Request
Nor download it without providing an decryption key.
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow-downloaded.txt
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request
What if we give it an intentionally incorrect key? Surely we’ll just get garbage content.
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow_downloaded.txt \
--sse-c AES256 \
--sse-c-key 00000000000000000000000000000000 \
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
No dice. We have to provide the correct key, one that results in a matching Hash-based Message Authentication Code (HMAC).
% aws s3 cp s3://s3abovetheline/neverbelow.txt ./neverbelow_downloaded.txt \
--sse-c AES256 \
--sse-c-key 12345678901234567890123456789012 \
download: s3://s3abovetheline/neverbelow.txt to ./neverbelow_downloaded.txt
Again, even though KMS is not involved, access to the decryption key looks and feels like access control because the cryptographic operations have to happen server-side.
Compliance ✅
Client-side encryption
There’s one other option, client-side encryption. The idea is that an application encrypts the data before sending it to s3, and then decrypts it after downloading it from S3. S3 never even knows an object is encrypted and AWS never touches the keys.
This perfectly reasonable in some situations but kind of defeats the purpose of having a cloud do the heavy lifting. It’s extremely rare to see this approach, relative to all the other magic AWS options.
Ironically, no AWS security tool (and most auditors) can identify that this option is being used.
Compliance ❌
What have we learned?
Here’s the deal:
- S3 decryption works more like access control than decryption;
- Because of this, the thing downloading the data has to have access to the decryption key; and
- Therefore s3 encryption can prevent data exfiltration but is irrelevant after exfiltration.
To finish off the story, if data was in fact exfiltrated from Sisense using S3, we can’t infer anything about whether encryption was enabled. And it doesn’t matter, even if we could.
Want to know whether cloud encryption is a worthwhile security control? Chris Farris makes a strong case against it. Also, if you want to nerd out over a KMS threat model, Costas Kourmpoglou has done the work.
