Skip to content

Latest commit

 

History

History
409 lines (263 loc) · 19.7 KB

install-jenkins.md

File metadata and controls

409 lines (263 loc) · 19.7 KB

Jenkins installation guide

In these guide, we explain how to setup Jenkins to automatically package and pre-compile an Unreal Engine project.

⚠️ These guide use the JDK 11 as it's the only version available for P4V plugin, but the JDK will be deprecated in September 2024. Still it works, but it won't have the last security update.

Jenkins home page Jenkins dashboard page

Installation

Install Jenkins Main docker

  1. Copy the docker-compose.yaml file in your server :

    version: '3'

services:
    jenkins-main:
        image: wesleypetit/jenkins:lts-jdk11
        container_name: jenkins-main
        restart: unless-stopped
        ports:
            - "8080:8080"
            - "50000:50000"
        volumes:
            - jenkins_home:/var/jenkins_home
        # networks:
        #     - nginx-proxy

# networks:
#     nginx-proxy:
#         external: true

volumes:
    jenkins_home:

    It will automatically install several plugins : - P4 : Perforce Client plugin for the Jenkins SCM provider. - Matrix Authorization Strategy : Offers matrix-based security authorization strategies (global and per-project). - PostBuildScript : A plugin for the Jenkins CI to run several configurable actions after a build, depending on the build result. - Pre SCM BuildStep : This plugin allows build steps to be performed before the SCM step performs an action. - SCM Skip : Plugin adds functionality of preventing a Job to be built when a specific pattern ([ci skip]) in SCM commit message is detected. - Environment Injector : This plugin makes it possible to set an environment for the builds. - Discord Notifier : Discord Notifier provides a bridge between Jenkins and Discord through the built-in webhook functionality. - SAML : A SAML 2.0 Plugin for the Jenkins Continuous Integration server.

  2. (Optional) If you want to use an anonymous account, pull wesleypetit/jenkins:lts-jdk11 from portainer or run :

    sudo docker pull wesleypetit/jenkins:lts-jdk11

  3. (Optional) If you install Nginx Proxy Manager, add a new proxy host to forward the port 8080 and add a new stream to forward the port 50000.

  4. Deploy your container :

    sudo docker-compose up -d

  5. Finally, set up jenkins using this documentation.

  6. Follow this documentation to disable building on the built-in node.

Install a Windows Agent

  1. Install Windows on your build server.

  2. In the windows agent, install several programs :

    ℹ️ I write a small script to install Unreal Engine prerequisites (Visual Studio, .NET core, JDK11 and Jenkins). Only the Epic Games launcher is not automatically installed, as it does not provide an official package.

  3. Now, go Manage Jenkins > Nodes in your Jenkins Main Dashboard.

  4. Click on add New Node, enter a node name (e.g windows-unreal-engine) and pick Permanent Agent.

  5. Specify the Remote root directory dedicated to Jenkins an click on Save.

    Windows Node configuration

  6. Now, click on the node and execute the given command in the windows agent.

Configuration

SSO with Auth0

We use Auth0, because it proposes a free tier without credit card that is limited to 7500 active user.

  1. Create an account on Auth0, it will unify authentication for Helix and other services. Complete all steps depending on your location.

  2. In Auth0, create a Regular Web Application without specifying the technology. It will allow Jenkins to use Auth0 for Authentication.

  3. Now, go to Applications > <YOUR_NEWLY_APPLICATION_CREATED> > Settings and keep aside your Client ID and Domain, as it will be needed for Jenkins settings.

  4. In Allowed Callback URLs, add <YOUR_JENKINS_URI>/securityRealm/finishLogin

  5. In Allowed Logout URLs, add <YOUR_JENKINS_URI>.

  6. Scroll and open Advanced Settings section and click on Endpoints.

  7. Copy the SAML Metadata URL field and open it in your browser. It's your IdP Metadata file needed for Jenkins settings.

  8. Click on Save Changes.

  9. Now, go back to your Jenkins dashboard.

  10. Go in Manage Jenkins > Security and change Security Realm to SAML 2.0.

  11. Copy and paste your IdP Metadata file in the appropriate field.

  12. Set the Username Attribute to nameid and Email Attribute to email.

  13. Add your IdP logout url (e.g https://<YOUR_AUTH0_DOMAIN>/v2/logout?client_id=<YOUR_AUTH0_CLIENT_ID>&returnTo=<YOUR_JENKINS_URI>).

  14. Click apply and test on another browser. The idea is to modify settings without losing access.

Add Perforce Credentials

  1. Go to Manage Jenkins > Credentials and click on (global).

  2. Click on Add Credentials.

  3. Enter your P4PORT (without the ssl prefix).

  4. If your helix core server use ssl connection, enable SSL connection checkbox and add your server fingerprint in the Trust field. Here is a command to get your server fingerprint :

    p4 -u <YOUR_USER_NAME> -p <YOUR_P4_PORT> trust -l

  5. Enter your jenkins username and generate a long-lived login ticket :

    p4 -u <YOUR_USER_NAME> -p <YOUR_P4_PORT> login -ap

  6. Click on Test Connection to debug your credentials then click on Save.

Automatic builds for Unreal Engine

In these guide, we explain how to setup a Jenkins pipeline to automatically package an Unreal Engine project and (optionally) send a discord notification containing a download link to the package. These guide is heavily inspire by Patrice Vignola guide, I updated the instructions with Unreal Engine 5.

Jenkins package pipeline

Setup your pipeline

  1. Go to Dashboard > New Item to create a new project.

  2. Enter a name and pick Freestyle project.

  3. Select Perforce Software in the Source Code Management section.

  4. Pick a Perforce Credentials, change the Workspace behavior to Streams and define on which stream you want to package in Stream Codeline.

  5. (optional) Setup build periodically for build triggers. In our case, we package the game every working day at 8AM (e.g H 8 * * 1-5).

  6. Enable Inject environment variables to the build process and add these fields in the Properties Content field :

    PROJECT_NAME=<YOUR_PROJECT_NAME>
BUILD_CONFIG=BuildCookRun
PLATFORM=Win64
CONFIG=Shipping
ARCHIVE_DIRECTORY=%WORKSPACE%\Build
ARTIFACT_NAME=%PROJECT_NAME%_Win_CL-%P4_CHANGELIST%_ID-%BUILD_ID%
UE_REBUILD_BAT=C:\Program Files\Epic Games\UE_5.3\Engine\Build\BatchFiles\RunUAT.bat

    Here is a full explanation of each fields : - PROJECT_NAME : It corresponds to your .uproject filename, we expect to find it under the workspace root folder. - BUILD_CONFIG : Unreal automation commands (BuildCookRun, BuildGame...). You can find more using the command <YOUR_ENGINE_FOLDER>\Build\BatchFiles\RunUAT.bat -List. - PLATFORM : Platforms to build, join multiple platforms using + (e.g Win64+PS4+XboxOne). - CONFIG : Configurations to build, join multiple configurations using + (e.g Development+Test). - ARCHIVE_DIRECTORY : Directory to archive the builds to. - ARTIFACT_NAME : Final build name. In our case, we add the changelist and the jenkins build id in the name (e.g AwesomeProject_Win_CL-100_ID-0). - UE_REBUILD_BAT : Path to RunUAT.bat file corresponding to your Unreal Engine version.

Setup build steps

  1. In the Builds Steps section, click on Add Build Step and pick Execute Windows batch command.

  2. Add the following command to package your game :

    CALL "%UE_REBUILD_BAT%" %BUILD_CONFIG% -project=%WORKSPACE%\%PROJECT_NAME%.uproject -Platform=%PLATFORM% -configuration=%CONFIG% -archive -archivedirectory="%ARCHIVE_DIRECTORY%" -clean -cook -skipstage -build -pak -package -makebinaryconfig -distribution -SkipCookingEditorContent -ForceMonolithic -NoP4 -NoSubmit

IF %ERRORLEVEL% NEQ 0 (EXIT %ERRORLEVEL%)

    ℹ️ For package arguments, the official Unreal Automation Tool documentation is an good overview. If you need more information check out Botman repository.

  3. Add a new Windows batch command to rename the package folder :

    CALL rename Build\Windows "%ARTIFACT_NAME%"

  4. Add a new Windows batch command to zip the package folder :

    CALL "%UE_REBUILD_BAT%" ZipUtils -archive="%WORKSPACE%\%ARTIFACT_NAME%.zip" -add="%ARCHIVE_DIRECTORY%" -compression=9

IF %ERRORLEVEL% NEQ 0 (EXIT %ERRORLEVEL%)

Setup artifact

  1. In the Post-build actions section, click on Add post-build action and pick Archive the artifacts.

  2. To add the generated build as an artifact, change the files to archive field to :

    *.zip

  3. Click on Advanced and enable Archive artifacts only if build is successful.

(Optional) Setup discord notification

⚠️ Only Jenkins authenticated user can download artifacts.

  1. Follow the Introduction to Webhooks guide to generate a Webhook URL.

  2. Edit your jenkins pipeline.

  3. In the Post-build actions section, click on Add post-build action and pick Discord Notifier.

  4. Enter your Webhook URL.

  5. Click on Advanced and enable Enable URL linking to redirect the discord message to the jenkins build.

  6. Save, test and your are done !

Precompile binaries for Unreal Engine

In these guide, we explain how to setup a Jenkins pipeline to automatically precompile and submit the DLLs in Perforce. Artists/designers will receive the DLLs and no longer need to compile. Optionally, we will send a discord notification on failed build. These guide is heavily inspired by the Unreal Community Wiki, I updated the instructions with Unreal Engine 5.

Jenkins binaries compilation pipeline

Setup Perforce Stream

Here is the complete workflow :

  • Coders make code changes and submit them to the coder stream.
  • Jenkins monitors the mainline stream, syncs to the latest changes, builds the project, and submits the generated DLLs to the mainline stream.
  • Artists/designers sync the mainline stream to receive the updated DLLs.

On the technical aspect, you need two virtual stream : - coder : inherits from main but excludes all Binaries folders. - jenkins-binaries-compilation : inherits from main but excludes all Content folders since it won't need it.

Here is the stream hierarchy :

Stream hierarchy

  1. Create a virtual stream for coders :

    Coder virtual stream

  2. In the Advanced tab, add the following paths and adapt it to your project :

    share ...
exclude Binaries/...
exclude Plugins/AwesomePlugin/Binaries/...

    Coder virtual stream advanced settings

  3. Create a virtual stream for jenkins :

    Jenkins compilation virtual stream

  4. In the Advanced tab, add the following paths and adapt it to your project :

    isolate ...
exclude Content/...
exclude  Plugins/AwesomePlugin/Content/...
share Binaries/...
share Plugins/AwesomePlugin/Binaries/...

    Jenkins compilation virtual stream advanced settings

Setup your pipeline

  1. In Jenkins home page, go to Dashboard > New Item to create a new project.

  2. Enter a name and pick Freestyle project.

  3. Select Perforce Software in the Source Code Management section.

  4. Pick a Perforce Credentials, change the Workspace behavior to Streams and set the jenkins binaries stream in Stream Codeline.

  5. Under polling build filters, click on Add new build filter and select Exclude changes outside Java pattern.

  6. With the following pattern, a build we be triggered only when a changelist contains a code file (.h or .cpp) :

    //your_depot_path/main/.*\.cpp
//your_depot_path/main/.*\.h

  7. (optional) Enable Poll SCM for build triggers, Jenkins will scan the current workspace and compare with the server to trigger or not a build. In our case, we set the poll frequency to every 30 minutes (e.g H/30 * * * *).

  8. Enable Inject environment variables to the build process and add these fields in the Properties Content field :

    PROJECT_NAME=<YOUR_PROJECT_NAME>
BUILD_CONFIG=BuildEditor
PLATFORM=Win64
CONFIG=Development
UE_REBUILD_BAT=C:\Program Files\Epic Games\UE_5.3\Engine\Build\BatchFiles\RunUAT.bat

    Here is a full explanation of each fields : - PROJECT_NAME : It corresponds to your .uproject filename, we expect to find it under the workspace root folder. - BUILD_CONFIG : Unreal automation commands (BuildCookRun, BuildGame...). You can find more using the command <YOUR_ENGINE_FOLDER>\Build\BatchFiles\RunUAT.bat -List. - PLATFORM : Platforms to build, join multiple platforms using + (e.g Win64+PS4+XboxOne). - CONFIG : Configurations to build, join multiple configurations using + (e.g Development+Test). - UE_REBUILD_BAT : Path to RunUAT.bat file corresponding to your Unreal Engine version.

Setup pre-compile steps

  1. In the Builds Steps section, click on Add Build Step and pick Execute Windows batch command.

  2. Add the following command to compile your game :

    CALL "%UE_REBUILD_BAT%" %BUILD_CONFIG% -platform=%PLATFORM% -configuration=%CONFIG% -project="%WORKSPACE%\%PROJECT_NAME%.uproject" -WaitMutex -FromMsBuild -notools -ForceCompile

IF %ERRORLEVEL% NEQ 0 (EXIT %ERRORLEVEL%)

    ℹ️ For package arguments, the official Unreal Automation Tool documentation is an good overview. If you need more information check out Botman repository.

(Optional) Setup discord notification on failure

  1. Follow the Introduction to Webhooks guide to generate a Webhook URL.

  2. Edit your jenkins pipeline.

  3. In the Post-build actions section, click on Add post-build action and pick Discord Notifier.

  4. Enter your Webhook URL.

  5. Enable Send only failed.

  6. Click on Advanced.

  7. Enable Enable URL linking to redirect the discord message to the jenkins build and enable Send log file to discord.

  8. Disable Enable version info in footer and Enable artifact list.

Submit in Perforce

  1. In the Post-build actions section, click on Add post-build action and pick Perforce: Publish assets.

  2. Pick a Perforce Credentials, change the Workspace behavior to Streams and set the jenkins binaries stream in Stream Codeline.

  3. Change the Publish Options to Submit change.

  4. Enable Only publish on build success and Reopened files. These options makes sure you submit only correct binaries and that Jenkins checkout for the next build.

  5. Set a Purge Revisions (e.g 5) to only stores the last n revisions in Perforce.

  6. Click on Advanced and add the following paths :

    //your_depot_path/jenkins-binaries-compilation/Binaries/...
//your_depot_path/jenkins-binaries-compilation/AwesomePlugin/Binaries/...

    ⚠️ Submit with publish paths ignore p4ignore files.

  7. Click on Save.

References