Use jenkinsci/ec2-fleet-plugin instead of awslabs/ec2-spot-jenkins-plugin
The EC2 Fleet Plugin launches EC2 Spot or On Demand instances as worker nodes for Jenkins CI server, automatically scaling the capacity with the load.
This plugin uses EC2 Spot Fleet or Auto Scaling Group to launch instances instead of directly launching them by itself. Both maintain your fleet's target capacity as Spot prices change to maintain the fleet within the specified price range. For more information, see How Spot Fleet Works.
- Support EC2 Spot Fleet or Auto Scaling Group as Jenkins Workers
- Supports all features provided by EC2 Spot Fleet or Auto Scaling Groups
- Auto resubmit failed jobs caused by Spot interruptions
- No delay scale up strategy: enable
No Delay Provision Strategyin configuration - Add tags to EC2 instances used by plugin, for easy search, tag format
ec2-fleet-plugin:cloud-name=<MyCloud> - Allow custom EC2 API endpoint
- Auto Fleet creation based on Job label (details)
EC2-Plugin is a similar Jenkins plugin that will request EC2 instances when excess jobs are detected. The main difference between the two plugins is that EC2-Fleet-Plugin uses EC2 Spot Fleet and ASG to request and manage instances instead of doing it manually with EC2 RunInstances. This gives EC2-Fleet-Plugin all the benefits of Spot Fleet and ASG: allocation strategies, automatic availability zone rebalancing (ASG only), access to launch configurations and launch templates , instance weighting, etc.
| EC2-Fleet-Plugin | EC2-Plugin |
|---|---|
| Supports On-Demand & Spot Instances | Supports On-Demand & Spot Instances |
| Scales with EC2 Spot Fleet or ASG | Scales with RunInstances |
| EC2 Spot Fleet and ASG Allocation Strategies | No Allocation Strategies |
| Use launch config/template to set instance settings | Manually set instances settings within plugin |
| Custom instance weighting | No custom instance weighting |
This plugin is using SemVersion which means that each plugin version looks like
<major>.<minor>.<bugfix>
major = increase only if non back compatible changes
minor = increase when new features
bugfix = increase when bug fixes
As a result, you can safely update the plugin to any version until the first number is different than what you have.
Releases: https://github.com/jenkinsci/ec2-fleet-plugin/releases
Go to AWS account and follow instructions.
Specify programmatic access during creation and record the credentials. These will
be used by Jenkins EC2 Fleet Plugin to connect to your Spot Fleet.
Alternatively, you may use AWS EC2 instance roles
Add an inline policy to the IAM user or EC2 instance role to allow it to use EC2 Spot Fleet and Auto Scaling Group. AWS documentation about this
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeSpotFleetInstances",
"ec2:ModifySpotFleetRequest",
"ec2:CreateTags",
"ec2:DescribeRegions",
"ec2:DescribeInstances",
"ec2:TerminateInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeSpotFleetRequests"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:UpdateAutoScalingGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:ListInstanceProfiles",
"iam:ListRoles"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"*"
],
"Condition": {
"StringEquals": {
"iam:PassedToService": [
"ec2.amazonaws.com",
"ec2.amazonaws.com.cn"
]
}
}
}
]
}Create EC2 Spot Fleet
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-requests.html#create-spot-fleet
Make sure that you:
- Check
Maintain target capacitywhy - Specify an SSH key that will be used later by Jenkins.
Alternatively, create an Auto Scaling Group
https://docs.aws.amazon.com/autoscaling/ec2/userguide/GettingStartedTutorial.html
Once the Spot Fleet or ASG is ready, you can use it by adding a new EC2 Fleet cloud in the Jenkins configuration.
- Goto
Manage Jenkins > Plugin Manager - Install
EC2 Fleet Jenkins Plugin - Goto
Manage Jenkins > Configure System - Click
Add a new cloudand selectAmazon EC2 Fleet - Configure AWS credentials, or leave empty to use the EC2 instance role
- Specify EC2 Spot Fleet or Auto Scaling Group which you want to use
More information on the configuration options can be found here.
You can specify the scaling limits in your cloud settings. By default, Jenkins will try to scale the fleet up if there are enough tasks waiting in the build queue and scale down idle nodes after a specified idleness period.
You can use the History tab in the AWS console to view the scaling history.
Below is a Groovy script to setup EC2 Spot Fleet Plugin for Jenkins and configure it. You can run the script with Jenkins Script Console.
import com.amazonaws.services.ec2.model.InstanceType
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.DirectEntryPrivateKeySource
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
import com.cloudbees.jenkins.plugins.awscredentials.AWSCredentialsImpl
import hudson.plugins.sshslaves.SSHConnector
import hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy
import com.cloudbees.plugins.credentials.*
import com.cloudbees.plugins.credentials.domains.Domain
import hudson.model.*
import com.amazon.jenkins.ec2fleet.EC2FleetCloud
import jenkins.model.Jenkins
// just modify this config other code just logic
config = [
region: "us-east-1",
// EC2 Spot Fleet ID
// or Auto Scaling Group Name
fleetId: "...",
idleMinutes: 10,
minSize: 0,
maxSize: 10,
numExecutors: 1,
awsKeyId: "...",
secretKey: "...",
ec2PrivateKey: '''-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----'''
]
// https://github.com/jenkinsci/aws-credentials-plugin/blob/aws-credentials-1.23/src/main/java/com/cloudbees/jenkins/plugins/awscredentials/AWSCredentialsImpl.java
AWSCredentialsImpl awsCredentials = new AWSCredentialsImpl(
CredentialsScope.GLOBAL,
"aws-credentials",
config.awsKeyId,
config.secretKey,
"my aws credentials"
)
BasicSSHUserPrivateKey instanceCredentials = new BasicSSHUserPrivateKey(
CredentialsScope.GLOBAL,
"instance-ssh-key",
"ec2-user",
new DirectEntryPrivateKeySource(config.ec2PrivateKey),
"",
"my private key to ssh ec2 for jenkins"
)
// find detailed information about parameters on plugin config page or
// https://github.com/jenkinsci/ec2-fleet-plugin/blob/master/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetCloud.java
EC2FleetCloud ec2FleetCloud = new EC2FleetCloud(
"", // fleetCloudName
awsCredentials.id,
"",
config.region,
config.fleetId,
"ec2-fleet", // labels
"", // fs root
new SSHConnector(22,
instanceCredentials.id, "", "", "", "", null, 0, 0,
// consult doc for line below, this one say no host verification, but you can use more strict mode
// https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/src/main/java/hudson/plugins/sshslaves/verifiers/NonVerifyingKeyVerificationStrategy.java
new NonVerifyingKeyVerificationStrategy()),
false, // if need to use privateIpUsed
false, // if need alwaysReconnect
config.idleMinutes, // if need to allow downscale set > 0 in min
config.minSize, // minSize
config.maxSize, // maxSize
config.numExecutors, // numExecutors
false, // addNodeOnlyIfRunning
false, // restrictUsage allow execute only jobs with proper label
)
// get Jenkins instance
Jenkins jenkins = Jenkins.get()
// get credentials domain
def domain = Domain.global()
// get credentials store
def store = jenkins.getExtensionList('com.cloudbees.plugins.credentials.SystemCredentialsProvider')[0].getStore()
// add credential to store
store.addCredentials(domain, awsCredentials)
store.addCredentials(domain, instanceCredentials)
// add cloud configuration to Jenkins
jenkins.clouds.add(ec2FleetCloud)
// save current Jenkins state to disk
jenkins.save()Sometimes you need to prepare an agent (an EC2 instance) before Jenkins can use it. For example, you need to install some software which is required by your builds like Maven, etc.
For those cases you have a few options, described below:
AMI allows you to create custom images for your EC2 instances. For example, you can create an image with Linux plus Java, Maven etc. Then, when EC2 Fleet launches new EC2 instances with this AMI they will automatically get all the required software. Nice =)
- Create a custom AMI as described here
- Create EC2 Spot Fleet with this AMI
EC2 instances allow you to specify a User Data script that is executed when an instance first launches. This allows you to customize the setup for a particular instance.
EC2 instances don't provide any information about the User Data script execution status, so Jenkins could start a task on a new instance while the script is still in progress. Most of the time Jenkins will repeatedly try to connect to the instance during this time and print out errors until the script completes and Jenkins can connect.
To avoid those errors, you can use the Jenkins SSH Launcher Prefix Start Agent Command setting
to specify a command which should fail if User Data is not finished. In that way Jenkins will
not be able to connect to the instance until the User Data script is done. More information on configuring the SSH
launcher can be found here.
- Open Jenkins
- Go to
Manage Jenkins > Configure System - Find proper fleet configuration and click
Advanced...for SSH Launcher - Add checking command into field
Prefix Start Agent Command- example
java -version &&
- example
- To apply for existing instances, restart Jenkins or Delete Nodes from Jenkins so they will be reconnected
Check out the FAQ & Gotchas page here.
Plugin usage statistics per Jenkins version can be found here
https://jenkins.io/doc/developer/publishing/releasing/
mvn release:prepare release:performhttps://issues.jenkins-ci.org/browse/JENKINS-53954
Regular script:
sudo yum install java-1.8.0 -y
sudo yum remove java-1.7.0-openjdk -y
java -version User Data Script:
Note sudo is not required, -y suppresses confirmation.
Don't forget to encode with Base64
#!/bin/bash
yum install java-1.8.0 -y && yum remove java-1.7.0-openjdk -y && java -versionContributions are welcome! Please read our guidelines and our Code of Conduct.