Debugging AccessDenied in AWS IAM

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Ugh… that looks like it could be the start of a two hour or two week long goose chase.

Understanding why access was denied and implementing a secure solution can be complicated. Sometimes it’s not even clear where to start and what to do when you get stuck.

Update: An improved version of this Debugging AccessDenied in AWS IAM is now maintained by k9 Security. k9 helps Cloud teams improve security policies and accelerate delivery.

Here’s how I usually approach debugging AWS access control problems, a specialized form of The Debugging Rules:

Read logs, guess, and check by using application
CloudTrail
Finer-grained test events
Policy Simulator
Tool to test the app activity in a quick, repeatable fashion

I used this approach to dive into and climb out of a deep access control rabbit hole of cross-account access involving: IAM, Lambda, S3 bucket policy, and KMS encryption key policy.

The tricky bit is depicted in the following diagram where a customer’s report is stored in an internal ‘Reports’ bucket and then published to their ‘Secure Inbox‘ bucket.

The challenge was nailing down the security for this ‘Secure Inbox’ pattern, where:

a service in Account A (orange), does work on behalf of a customer
the service must store an object in an internal S3 bucket, encrypted with a customer-managed KMS key
the service must deliver data to a customer-managed S3 bucket in Account B
the data must also be encrypted with the customer-managed KMS key in Account B

I’ll share details of how to implement this pattern soon, but wanted to share insights of the debugging process while they are still fresh.

The critical API actions are s3:PutObject to an ‘internal’ S3 bucket managed by the service and s3:CopyObject to deliver the object to the customer. Both actions use the customer-managed key to encrypt the customer’s data and keep them in control of it.

Read logs, guess, and check by using application

Let’s start with the fact that I’m primarily in application development mode, developing an application. Secondarily, I’m creating some CloudFormation templates that customers will be able to use to configure resources in their accounts.

My mindset isn’t “Oh, I’m looking forward to digging deep into IAM, S3, and KMS policy!”

So, I try to solve the problem using what’s at hand: reading the AWS SDK (boto3) and S3 API docs, AWS security policy docs, S3 API responses, and application log messages logged into CloudWatch Logs.

I also thought I started off pretty close to the target.

The s3:PutObject action to internal storage is simple enough:

# _s3_client is a boto3.client('s3')
response = self._s3_client.put_object(ACL='private',
                                      ServerSideEncryption='aws:kms',
                                      SSEKMSKeyId=kms_encryption_key_id,
                                      Bucket=bucket_name,
                                      Key=key,
                                      Body=body_bytes)

The ‘obvious’ part is to specify server-side encryption withaws:kms and the customer’s KMS encryption key ARN with the S3 PUT API action.

AWS KMS provides customer managed encryption keys and an api. The really neat thing about the KMS API is that you can Allow use of:

particular api actions like kms:Encrypt and kms:GenerateDataKey
for particular encryption keys
for particular AWS principals: IAM roles and users or entire AWS accounts

But… what you don’t see and isn’t documented directly in the s3:PutObject API docs are the specific permissions you need for the KMS key policy in order to PUT the object. To be fair, the KMS policy generated by the AWS wizard when you provision a key would allow the necessary actions, but I think that policy is a bit broad.

Further, from the application perspective, if you you go ahead and catch the ClientError from the s3 client’s put_object method, and print out the error response, you’ll still only see something like:

Unexpected error putting object into bucket: { 
  'Error': {
    'Code': 'AccessDenied',
    'Message': 'Access Denied'
  },
... snip ...
}

Still nothing more than AccessDenied. AWS doesn’t give detailed information about how to debug permissions problems in API responses.

This is really important to understand if you’re a security or platform engineer — application engineers often cannot debug permissions problems from their application code, even if they want to.

We need a better tool for understanding the system. The next tool I use is CloudTrail.

CloudTrail

CloudTrail is an AWS service that provides an audit log of important events that occur in your account. The logs, called trails, record most AWS API usage including important request parameters and the principal (user or role) they were executed with.

If you have CloudTrail enabled in your account (you definitely should) and access to view the trail, you may be able to find valuable clues as to why access was denied, and which object it was denied to.

Here’s an example of a CloudTrail event in the service account A that told me why I couldn’t store an object in S3:

Getting closer! I found an event with an Error Code of AccessDenied. The skuenzli user was denied access to use the kms:GenerateDataKey api. Now, I’m running through this example from my laptop with nearly Admin permissions. I know I have access to invoke kms:GenerateDataKey. The real issue is hidden inside the errorMessage of the event:

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDAJREII7F7Q2K7QMCLE",
        "arn": "arn:aws:iam::account_A:user/skuenzli",
        "accountId": "account_A",
        "accessKeyId": "ASIAJVOBWCCQR3OTPSUA",
        "userName": "skuenzli",
        "sessionContext": {
            "sessionIssuer": {},
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-12-03T15:55:35Z"
            }
        },
        "invokedBy": "AWS Internal"
    },
    "eventTime": "2019-12-03T15:55:35Z",
    "eventSource": "kms.amazonaws.com",
    "eventName": "GenerateDataKey",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "AWS Internal",
    "userAgent": "AWS Internal",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:iam::account_A:user/skuenzli is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:account_B:key/e9d04e90-8148-45fe-9a75-411650eea80f",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "722abc0e-77c2-42e0-8448-4f0469420f3a",
    "eventID": "92edcefe-6d12-4357-85f4-20f709f3e413",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "recipientAccountId": "account_A"
}

Aha:

"User: arn:aws:iam::account_A:user/skuenzli is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:account_B:key/e9d04e90-8148-45fe-9a75-411650eea80f"

I’m not permitted to invoke kms:GenerateDataKey with Account B’s encryption key. This is what I was really being denied access to.

Note that the s3:PutObject action invoked kms:GenerateDataKey on my behalf. s3:PutObject will do the same for kms:EncryptData.

Time to update the KMS encryption key policy.

It’s also time for a new tactic: finer-grained test events

Fine-grained test events

You may start off investigating and debugging access problems like this through the application.

But if this process is expensive, consider creating ‘fine-grained’ integration tests or Lambda test events that isolate one aspect of the integration. My application normally takes several minutes to run, which is certainly expensive to me. So it was time to focus in on the access control problem.

In the Secure Inbox use case, there are two integrations at work:

store the object in s3
copy the object

I created a number of integration tests that verified the expected behavior of key processes, including those above as part of the normal development process. And those tests were and still are very useful.

So why was I having trouble?

Well, those tests were all operating with resources and roles within a single AWS account I use for development. And cross-account access is very different.

The full cross-account test setup was planned and near the top of the backlog, but we hadn’t done it yet. A realistic multiple account setup is critical to testing cross-account scenarios accurately and quickly. So, we knocked that out and got back on track.

Once you have a quick and easy way to ‘make it fail,’ you can iterate and learn much quicker.

Policy simulator

Possibly the quickest IAM testing tool of all is to use the IAM policy simulator to help you narrow in on the IAM policy. No application deployments needed!

With the policy simulator you can simulate AWS API actions with all of the contextual information we’ve been talking about here:

the actual IAM user or role
current or proposed policies
one or more api actions, like s3:PutObject
specific resources: buckets, KMS keys and their policies

The simulator will tell you if an action is allowed and tell you which policy allowed it. The simulator also provides basic diagnostic information about why an action was not permitted.

That said, the simulator is a little clunky to use. You may find this tutorial on Testing an S3 policy using the IAM simulator a helpful introduction to the mechanics.

If your application and access control problems are bounded by a single account, these debugging tools and approaches may be sufficient.

If you’re integrating applications across AWS accounts, I think you’ll need a cross-account integration check.

Cross-Account Integration Check

If you have a cross-account integration scenario, I recommend a quick check of that integration.

Once you think you have the plethora of resources and policies in place and are using the right principals, it’s time to verify that it all works together.

Combine the fine-grained test events and actions into a function or end-to-end functional test that checks all of the cross-account integration in one go.

I created a quick-and-focused customer_configuration_check Lambda function that exercises the minimal path through all the integration to create and deliver a test (report) object. This function can be invoked with test events in AWS or locally and executes in less than 10 seconds. This tool’s super quick and accurate feedback has proved extremely valuable in application development.

This customer_configuration_check function will have lasting value to our customers. We will use this tool to validate customers’ configurations during onboarding in addition to developing and testing new policy configurations.

What I’ve describe here is one of the ways you “DevOps” in a Serverless world.

Through this IAM debugging exercise we’ve:

shortened feedback loops
shown how we (still) need to collaborate across ‘development’ and ‘security’ job functions
improved daily work
repositioned to deliver value to customers quicker and with greater reliability.

Stephen

#NoDrama