How to troubleshoot Resource Based Policies Scan Failures?

[iankowsk]

After running the Resource-Based Policies scan for all regions and services, the job has been failed with the status "Failed".

How to troubleshoot this error?

[morjoan]

Hi, please see our guide for troubleshooting that issue here:
https://docs.aws.amazon.com/solutions/latest/account-assessment-for-aws-organizations/troubleshooting.html

[iankowsk]

After troubleshooting, we have found the following error:

After adding the following trust policy, the job started with no errors, but it still running for more than 48 hours.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

We are unable to take any action.

Not able to delete it.

Not able to restart it.

There is no lambda function or step functions running.

Questions:

1. How can we stop/cancel the current job?
2. How can we fix the error "The principal states.amazonaws.com it not authorized to assume the provided role"?

[tbelmega]

Hi @iankowsk ,
which version of the solution are you running?

The "Scan running" message you see on the UI is related to the "JobStatus" field on the "lastJobMarker" item in DynamoDB table JobHistory. The "FailJob" step you see on the step function diagram is basically the "catch" clause of the step function, supposed to update the JobTable and and set the "lastJobMarker" from "RUNNING" to "FAILED". Since this update failed, the solution treats the job as still running and can't start another one.
This is obviously not supposed to happen. The immediate solution is to manually update the JobStatus field in DynamoDB from "RUNNING" to "FAILED" (see screenshot).

To be clear, no compute is technically "running", so the customer isn't billed for anything. It's just that the database entry representing the Job hasn't been updated from "RUNNING" to "FAILED" as it should have, so the solution logic treats the state as if a job was running and doesn't let you start a new one.

Regarding the actual cause for the failure, I can't tell why it fails to assume the role. Please contact me directly internally, I can help troubleshoot on Thursday or Friday if that helps.

[iankowsk]

Thanks for the quick reply, @tbelmega.

I will reach out to you to discuss how we can solve the role issue.

[tbelmega]

Scheduled a troubleshooting session internally, closing this ticket.