Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Crossplane AWS provider unexpectedly deletes inline policy on update failure #1633

Open
1 task done
Darwin014 opened this issue Jan 15, 2025 · 1 comment
Open
1 task done

[Bug]: Crossplane AWS provider unexpectedly deletes inline policy on update failure #1633

Darwin014 opened this issue Jan 15, 2025 · 1 comment
Labels
bug Something isn't working needs:triage

Comments

@Darwin014
Copy link

Darwin014 commented Jan 15, 2025 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Affected Resource(s)

  • iam.aws.upbound.io/v1beta1 - Role

Resource MRs required to reproduce the bug

apiVersion: iam.aws.upbound.io/v1beta1
kind: Role
metadata:
  annotations:
    meta.upbound.io/example-id: iam/v1beta1/role
  labels:
    testing.upbound.io/example-name: role
  name: role-with-inline-policy
spec:
  forProvider:
    assumeRolePolicy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "eks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }
    inlinePolicy:
      - name: my_inline_policy
        policy: |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Resource": "*",
                "Action": "ec2:Describe*"
              },
              {
                "Effect": "Allow",
                "Resource": "*",
                "Action": "ec2:testtest*"
              }
            ]
          }

Steps to Reproduce

  • Create a Role with an inline policy using the sample code provided, and ensure it is created successfully.
  • Modify the inline policy by changing the Action to an invalid one that is not provided by AWS.
  • Apply the updated Role with the invalid policy.

What happened?

In Crossplane, a 400 error occurs, but when checking AWS CloudTrail, a delete event is triggered, and the inline policy is actually deleted.

Events:
  Type     Reason                        Age    From                                           Message
  ----     ------                        ----   ----                                           -------
  Normal   CreatedExternalResource       4m30s  managed/iam.aws.upbound.io/v1beta1, kind=role  Successfully requested creation of external resource
  Normal   UpdatedExternalResource       6s     managed/iam.aws.upbound.io/v1beta1, kind=role  Successfully requested update of external resource
  Warning  CannotUpdateExternalResource  5s     managed/iam.aws.upbound.io/v1beta1, kind=role  async update failed: failed to update the resource: [{0 updating IAM Role (role-with-inline-policy): adding inline policy (my_inline_policy): operation error IAM: PutRolePolicy, https response error StatusCode: 400, RequestID: 050269ac-8d88-455c-810c-b1670e739695, MalformedPolicyDocument: Syntax errors in policy.  []}]
  Warning  CannotUpdateExternalResource  3s     managed/iam.aws.upbound.io/v1beta1, kind=role  async update failed: failed to update the resource: [{0 updating IAM Role (role-with-inline-policy): adding inline policy (my_inline_policy): operation error IAM: PutRolePolicy, https response error StatusCode: 400, RequestID: 57cc57f2-6285-4ef0-b076-cdb52c15b622, MalformedPolicyDocument: Syntax errors in policy.  []}]
  Warning  CannotUpdateExternalResource  2s     managed/iam.aws.upbound.io/v1beta1, kind=role  async update failed: failed to update the resource: [{0 updating IAM Role (role-with-inline-policy): adding inline policy (my_inline_policy): operation error IAM: PutRolePolicy, https response error StatusCode: 400, RequestID: 47573685-c4f3-4353-a793-0600a7326722, MalformedPolicyDocument: Syntax errors in policy.  []}]

AWS Cloudtrail Log

2025-01-15T07:20:28Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:19:27Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:18:26Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:17:43Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:17:22Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:17:10Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:17:03Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:59Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:57Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:55Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:53Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:51Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:50Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:48Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:47Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:46Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:44Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	PutRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0	MalformedPolicyDocumentException	[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]
2025-01-15T07:16:44Z	[iam.amazonaws.com](http://iam.amazonaws.com/)	DeleteRolePolicy	APN/1.0 HashiCorp/1.0 Terraform (+https://www.terraform.io/) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.32.6 ua/2.1 os/linux lang/go#1.23.3 md/GOOS#linux md/GOARCH#amd64 api/iam#1.38.2 crossplane-provider-aws/v1.19.0 upbound-provider-aws/v1.19.0		[{"resourceType":"AWS::IAM::Policy","resourceName":"my_inline_policy"},{"resourceType":"AWS::IAM::Role","resourceName":"role-with-inline-policy"}]

Relevant Error Output Snippet

No response

Crossplane Version

1.18.2

Provider Version

xpkg.upbound.io/upbound/provider-family-aws:v1

Kubernetes Version

1.30

Kubernetes Distribution

EKS

Additional Info

No response
@Darwin014 Darwin014 added bug Something isn't working needs:triage labels Jan 15, 2025
@mergenci
Copy link
Collaborator

Short discussion on the topic: https://crossplane.slack.com/archives/CEG3T90A1/p1736921595719059

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs:triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mergenci @Darwin014