[Bug]: Garbled user data on EC2 instance

[dhaines-quera]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

• ec2.aws.upbound.io/v1beta1 - Instance

Resource MRs required to reproduce the bug

---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: clusters.compute.example.com
spec:
  group: compute.example.com
  names:
    kind: Cluster
    plural: cluster
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties: {}
            required: []
    served: true
    referenceable: true
  claimNames:
    kind: ClusterClaim
    plural: clusterclaims
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: test-cluster
spec:
  compositeTypeRef:
    apiVersion: compute.example.com/v1alpha1
    kind: Cluster
  resources:
  ...
  - name: i-0
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Instance
      spec:
        forProvider:
          region: us-east-1
          ami: ami-0a3c3a20c09d6f377
          instanceType: t3.small
          subnetIdSelector:
            matchControllerRef: true
          associatePublicIpAddress: true
          keyName: foo
          vpcSecurityGroupIdSelector:
            matchControllerRef: true
          userData: |
            #cloud-config
            hello
---
apiVersion: compute.example.com/v1alpha1
kind: Cluster
metadata:
  name: my-test-cluster
spec: {}

Steps to Reproduce

Install crossplane via Helm chart 1.14.5
Install ec2 provider
Apply above manifests.

$ ssh -i ~/Downloads/foo.pem -o StrictHostKeyChecking=no ec2-user@$(kubectl get instances -o jsonpath='{.items[].status.atProvider.publicIp}') sudo cat /var/lib/cloud/instance/user-data.txt | xxd
00000000: 6f4e ba6f b7fb 69be f66f cd3d 7f7d b5e1  oN.o..i..o.=.}..
00000010: fe78 e37d 7be3 7e1f 7347 fc7d b7b8       .x.}{.~.sG.}..

$ ssh -i ~/Downloads/foo.pem -o StrictHostKeyChecking=no ec2-user@$(kubectl get instances -o jsonpath='{.items[].status.atProvider.publicIp}') curl -s http://169.254.169.254/latest/user-data

What happened?

I would expect to see #cloud-config\nhello as the output to each of those commands.

Relevant Error Output Snippet

No response

Crossplane Version

1.14.5

Provider Version

0.47.1

Kubernetes Version

1.29.0

Kubernetes Distribution

Talos

Additional Info

I was asked to open this ticket by @haarchri , for whom this is apparently a second report of the same bug. Please let me know if you need the missing portions of that Composition.

[turkenf]

Hi @dhaines-quera,

Thank you for raising this, it is hard to understand and reproduce this issue with the provided information, could you please give more/clear information?

• What did you expect? What happened?
• Example all MRs required to reproduce the issue.
• Is there anything specific to your environment?

[dhaines-quera]

Sure thing. The original report was here: https://crossplane.slack.com/archives/C05E0UE46S2/p1706134383962819

• I would expect to see the #cloud-config and hello in the output of the following two commands (assuming the instance is the only one in the region and an appropriately setup SSH key foo) instead of what actually occurs:

$ ssh -i ~/Downloads/foo.pem -o StrictHostKeyChecking=no ec2-user@$(kubectl get instances -o jsonpath='{.items[].status.atProvider.publicIp}') sudo cat /var/lib/cloud/instance/user-data.txt | xxd
00000000: 6f4e ba6f b7fb 69be f66f cd3d 7f7d b5e1  oN.o..i..o.=.}..
00000010: fe78 e37d 7be3 7e1f 7347 fc7d b7b8       .x.}{.~.sG.}..

$ ssh -i ~/Downloads/foo.pem -o StrictHostKeyChecking=no ec2-user@$(kubectl get instances -o jsonpath='{.items[].status.atProvider.publicIp}') curl -s http://169.254.169.254/latest/user-data

• The entire example:

---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: clusters.compute.example.com
spec:
  group: compute.example.com
  names:
    kind: Cluster
    plural: clusters
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties: {}
            required: []
    served: true
    referenceable: true
  claimNames:
    kind: ClusterClaim
    plural: clusterclaims
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: cluster
spec:
  compositeTypeRef:
    apiVersion: compute.example.com/v1alpha1
    kind: Cluster
  resources:
  - name: vpc
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: VPC
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.0.0/20
  - name: igw
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: InternetGateway
      spec:
        forProvider:
          region: us-east-1
          vpcIdSelector:
            matchControllerRef: true
  - name: rtb
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: DefaultRouteTable
      spec:
        forProvider:
          region: us-east-1
          route:
          - cidrBlock: 0.0.0.0/0
            gatewayIdSelector:
              matchControllerRef: true
          defaultRouteTableIdSelector:
            matchControllerRef: true
  - name: subnet-0
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Subnet
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.0.0/24
          availabilityZone: us-east-1a
          vpcIdSelector:
            matchControllerRef: true
  - name: subnet-1
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Subnet
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.1.0/24
          availabilityZone: us-east-1b
          vpcIdSelector:
            matchControllerRef: true
  - name: subnet-2
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Subnet
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.2.0/24
          availabilityZone: us-east-1c
          vpcIdSelector:
            matchControllerRef: true
  - name: subnet-3
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Subnet
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.3.0/24
          availabilityZone: us-east-1d
          vpcIdSelector:
            matchControllerRef: true
  # - name: subnet-4
  #   base:
  #     apiVersion: ec2.aws.upbound.io/v1beta1
  #     kind: Subnet
  #     spec:
  #       forProvider:
  #         region: us-east-1
  #         cidrBlock: 10.128.4.0/24
  #         availabilityZone: us-east-1e
  #         vpcIdSelector:
  #           matchControllerRef: true
  - name: subnet-5
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Subnet
      spec:
        forProvider:
          region: us-east-1
          cidrBlock: 10.128.5.0/24
          availabilityZone: us-east-1f
          vpcIdSelector:
            matchControllerRef: true
  - name: sg
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroup
      spec:
        forProvider:
          region: us-east-1
          vpcIdSelector:
            matchControllerRef: true
  - name: sgr-0
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroupIngressRule
      spec:
        forProvider:
          region: us-east-1
          ipProtocol: "-1"
          fromPort: 0
          toPort: 0
          securityGroupIdSelector:
            matchControllerRef: true
          referencedSecurityGroupIdSelector:
            matchControllerRef: true
  - name: sgr-1
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroupIngressRule
      spec:
        forProvider:
          region: us-east-1
          ipProtocol: "TCP"
          fromPort: 443
          toPort: 443
          cidrIpv4: 0.0.0.0/0
          securityGroupIdSelector:
            matchControllerRef: true
  - name: sgr-2
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroupIngressRule
      spec:
        forProvider:
          region: us-east-1
          ipProtocol: "TCP"
          fromPort: 60000
          toPort: 60001
          cidrIpv4: 0.0.0.0/0
          securityGroupIdSelector:
            matchControllerRef: true
  - name: sgr-3
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroupIngressRule
      spec:
        forProvider:
          region: us-east-1
          ipProtocol: "UDP"
          fromPort: 61000
          toPort: 61000
          cidrIpv4: 0.0.0.0/0
          securityGroupIdSelector:
            matchControllerRef: true
  - name: sgr-4
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: SecurityGroupIngressRule
      spec:
        forProvider:
          region: us-east-1
          ipProtocol: "TCP"
          fromPort: 22
          toPort: 22
          cidrIpv4: 0.0.0.0/0
          securityGroupIdSelector:
            matchControllerRef: true
  - name: lb
    base:
      apiVersion: elbv2.aws.upbound.io/v1beta1
      kind: LB
      spec:
        forProvider:
          region: us-east-1
          loadBalancerType: network
          subnetSelector:
            matchControllerRef: true
  - name: lbtg
    base:
      apiVersion: elbv2.aws.upbound.io/v1beta1
      kind: LBTargetGroup
      spec:
        forProvider:
          region: us-east-1
          name: foo
          protocol: TCP
          port: 6443
          targetType: ip
  - name: i-0
    base:
      apiVersion: ec2.aws.upbound.io/v1beta1
      kind: Instance
      spec:
        forProvider:
          region: us-east-1
          ami: ami-0a3c3a20c09d6f377
          instanceType: t3.small
          subnetIdSelector:
            matchControllerRef: true
          associatePublicIpAddress: true
          keyName: foo
          vpcSecurityGroupIdSelector:
            matchControllerRef: true
          userData: |
            #cloud-config
            hello
---
apiVersion: compute.example.com/v1alpha1
kind: Cluster
metadata:
  name: my-test-cluster
spec: {}

• I can think of nothing particularly interesting about our environment.

[ken-cogniac]

try to convert it to base64 in patch transform.

e.g:

- type: FromCompositeFieldPath
  fromFieldPath: status.abc
  toFieldPath: spec.forProvider.userData
  policy:
    fromFieldPath: Required
  transforms: 
    - type: string
      string:
        type: Convert
        convert: "ToBase64"

[dhaines-quera]

@ken-cogniac that didn't seem to do anything either way (and I made fromFieldPath a couple of values, both extant and not). Also, I don't understand what the purpose of having both userData and userDataBase64 fields would be if such a transform were necessary.

[sergenyalcin]

The upstream upjet fix for this issue: crossplane/upjet#358
PR where the corresponding fix is consumed: #1188

Fix was released on 0.47.3 and 1.1.1 versions of provider-aws.