InvalidParameterException: Override argument cannot be null, Creating Task in cluster null

[jlujan-invitae]

Jenkins and plugins versions report

Environment

Jenkins: 2.387.3
OS: Linux - 5.10.179-168.710.amzn2.x86_64
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.18.0
ace-editor:1.1
amazon-ecs:1.48
ant:487.vd79d090d4ea_e
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
artifact-manager-s3:822.vf129d4836c31
authentication-tokens:1.53.v1c90fd9191a_b_
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:108.v47b_fd43dfec6
aws-java-sdk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-cloudformation:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-codebuild:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ec2:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecr:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-efs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-elasticbeanstalk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-iam:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-kinesis:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-logs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-minimal:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sns:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sqs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ssm:1.12.481-392.v8b_291cfcda_09
aws-secrets-manager-credentials-provider:1.202.ve0ec0c17611c
blueocean:1.27.4
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.4
blueocean-commons:1.27.4
blueocean-config:1.27.4
blueocean-core-js:1.27.4
blueocean-dashboard:1.27.4
blueocean-display-url:2.4.2
blueocean-events:1.27.4
blueocean-git-pipeline:1.27.4
blueocean-github-pipeline:1.27.4
blueocean-i18n:1.27.4
blueocean-jwt:1.27.4
blueocean-personalization:1.27.4
blueocean-pipeline-api-impl:1.27.4
blueocean-pipeline-editor:1.27.4
blueocean-pipeline-scm-api:1.27.4
blueocean-rest:1.27.4
blueocean-rest-impl:1.27.4
blueocean-web:1.27.4
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.0-1
bouncycastle-api:2.28
branch-api:2.1109.vdf225489a_16d
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-bitbucket-branch-source:809.vc1d904b_30426
cloudbees-folder:6.815.v0dd5a_cb_40e0e
cobertura:1.17
code-coverage-api:4.7.0
command-launcher:100.v2f6722292ee8
commons-httpclient3-api:3.1-3
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
config-file-provider:938.ve2b_8a_591c596
configuration-as-code:1647.ve39ca_b_829b_42
credentials:1254.vb_96f366e7b_a_d
credentials-binding:604.vb_64480b_c56ca_
data-tables-api:1.13.4-2
display-url-api:2.3.7
docker-commons:419.v8e3cd84ef49c
docker-workflow:563.vd5d2e5c4007f
durable-task:507.v050055d0cb_dd
echarts-api:5.4.0-5
favorite:2.4.2
font-awesome-api:6.4.0-1
forensics-api:2.3.0
generic-webhook-trigger:1.86.3
git:5.1.0
git-client:4.4.0
git-server:99.va_0826a_b_cdfa_d
github:1.37.1
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1728.v859147241f49
github-checks:545.v79a_a_68b_ca_682
gradle:2.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
ivy:2.5
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:233.vdc1a_ec702cff
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:66.vd8fa_64ee91b_d
jenkins-design-language:1.27.4
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.84
jquery3-api:3.7.0-1
jsch:0.2.8-65.v052c39de79b_2
junit:1207.va_09d5100410f
labelled-steps:1.0
lockable-resources:1166.v7ef984f72fe2
mailer:457.v3f72cb_e015e5
matrix-auth:3.1.8
matrix-project:789.v57a_725b_63c79
maven-plugin:3.22
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
nexus-artifact-uploader:2.14
okhttp-api:4.11.0-145.vcb_8de402ef81
parameterized-scheduler:1.2
pipeline-aws:1.43
pipeline-build-step:496.v2449a_9a_221f2
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2141.v5402e818a_779
pipeline-model-definition:2.2141.v5402e818a_779
pipeline-model-extensions:2.2141.v5402e818a_779
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2141.v5402e818a_779
pipeline-stage-view:2.32
pipeline-utility-steps:2.15.4
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prism-api:1.29.0-7
pubsub-light:1.17
resource-disposer:0.22
role-strategy:633.v836e5b_3e80a_5
saml:4.418.vdfa_7489a_b_a_2d
scm-api:676.v886669a_199a_a_
script-security:1251.vfe552ed55f8d
slack:664.vc9a_90f8b_c24a_
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-credentials:305.v8f4381501156
sshd:3.303.vefc7119b_ec23
structs:324.va_f5d6774f3a_d
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:596.v8c21c963d92d
workflow-api:1215.v2b_ee3e1b_dd39
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3691.v28b_14c465a_b_b_
workflow-durable-task-step:1247.v7f9dfea_b_4fd0
workflow-job:1308.v58d48a_763b_31
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

Create new Cloud in Jenkins console
Configure Cloud for ECS cluster
Add Template to Cloud
Create Job with scripted pipeline that utilizes ECSTaskTemplate to inherit from the cloud template
Run the Job

Job

def dynamic_label = "${JOB_NAME}${currentBuild.number}".replaceAll('[^a-zA-Z0-9]', '').toLowerCase()

ecsTaskTemplate(
    cloud: 'on-prem-agent-cluster',
    label: dynamic_label,
    name: dynamic_label,
    inheritFrom: 'on-prem-agent-template',
    image: '<ECR Registry>/ruo-analysis-agent:latest',
    memoryReservation: 4096,
    cpu: "1024",
    overrides: []
)
{
  node(dynamic_label)
  ...
  }

Expected Results

Task is created in ECS.

Actual Results

Jun 17, 2023 9:48:12 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy

Started provisioning on-prem-agent-cluster-onpremtest68-htgt1 from on-prem-agent-cluster with 1 executors. Remaining excess workload: 0

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher

ECS: Launching agent


Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher

[on-prem-agent-cluster-onpremtest68-htgt1]: Creating Task in cluster null

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy

Received StrategyState{label=onpremtest68, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=1, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Selected Region: us-west-2

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Found dynamic agent task definition: arn:aws:ecs:us-west-2:<aws_account>:task-definition/on-prem-agent-cluster-onpremtest68-dz678:1

Jun 17, 2023 9:48:22 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask

[on-prem-agent-cluster-onpremtest68-htgt1]: Starting agent with task definition arn:aws:ecs:us-west-2:<aws_account>:task-definition/on-prem-agent-cluster-onpremtest68-dz678:1}

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Selected Region: us-west-2

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Found container definition with 1 container(s). Assuming first container is the Jenkins agent: on-prem-agent-cluster-onpremtest68-dz678

Jun 17, 2023 9:48:22 PM WARNING com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch

[on-prem-agent-cluster-onpremtest68-htgt1]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[on-prem-agent-cluster-onpremtest68-htgt1]
com.amazonaws.services.ecs.model.InvalidParameterException: Override argument cannot be null. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: cc490add-edd4-49fa-ae71-5ee09bc50121; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.ecs.AmazonECSClient.doInvoke(AmazonECSClient.java:5067)
	at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:5034)
	at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:5023)
	at com.amazonaws.services.ecs.AmazonECSClient.executeRunTask(AmazonECSClient.java:3546)
	at com.amazonaws.services.ecs.AmazonECSClient.runTask(AmazonECSClient.java:3515)
	at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:510)
	at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:213)
	at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launchECSTask(ECSLauncher.java:149)
	at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:110)
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

Experiencing strange behavior in the UI when opening the Cloud Configurations, All clouds show as "Edited". Using JCasC to configure an existing cloud and manually adding the cloud that is not functioning. Inspecting the cloud through the script console shows the correct clusterArn. Manually running the created task definition in the ECS cluster through AWS Console/CLI also functions as expected.

[jlujan-invitae]

Creating Task in cluster null is coming from the use of agent.getClusterArn() on ECSLauncher.java:146. This will not work as the clusterArn is not set on the agent until after task creation at

amazon-ecs-plugin/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSLauncher.java

Line 227 in 4fe6005

agent.setClusterArn(cloud.getCluster());

It looks like the ECSSlave is tied to the cloud by constructor so getClusterArn() should simply return cloud.getCluster()
in

amazon-ecs-plugin/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSSlave.java

Line 102 in 4fe6005

return clusterArn;

[jlujan-invitae]

The root cause was the instance Jenkins URL not being set correctly from JCasC upon restart. Opening the global configuration page and saving, then opening the Cloud Configuration and saving appears to have corrected the issue.

This should still be considered a bug in ecs-plugin as getDockerRunCommand appends a null to the command list when the Jenkins URL is null. This results in the InvalidParameterException: Override argument cannot be null and a lot of wasted time.