Unreal: support in Deadline #1

iLLiCiTiT · 2024-07-08T16:46:19Z

Changelog Description

This PR should add possibility to render Unreal project on Deadline with Perforce integration (and possibly without P4).

Additional info

It requires ynput/ayon-version-control#2, working Perforce server with user credentials and P4 workspace created following some P4 depot and stream

Testing notes:

set info about P4

2,provide P4 user credentials (in Project Settings > Site Settings)
Deadline must have some environment variables set

either via OS environments
OR via env vars set on Unreal in Applications addon

Expected env var: (applicable values for DL WORKER, not artist machine)

   "UnrealExecutable": {"windows": "C:/Program Files/Epic Games/UE_5.3/Engine/Binaries/Win64/UnrealEditor-Cmd.exe"},
   "UnrealExecutable_5_3": {"windows": "C:/Program Files/Epic Games/UE_5.3/Engine/Binaries/Win64/UnrealEditor-Cmd.exe"},
   "P4PORT":"localhost:1666",
   "P4USER":"admin",
   "P4PASSWD":"pass"

Copy ayon_core/modules/deadline/repository/custom/plugins/UnrealEngine5 to DEADLINE_REPO/custom folder
Render for DL uses existing creator and follow existing workflow, only added Render target
Provide path for render config and render queue ayon+settings://unreal/render_config_path and make sure that it exists in Unreal project

(expected value /Game/Ayon/DefaultMovieRenderQueueConfig.DefaultMovieRenderQueueConfig)
(expected value for Render Queue Path - /Game/Ayon/renderQueue
There must be at least 2 instances to be published, changelist_metadata creates instance with information about change list

(and render instance set to Farm publishing. Presence of changelist_metadata triggers Perforce sync before rendering on Deadline workers, implementation works even Perforce though.)

Additional info

PR moved from ynput/ayon-core#670
Its cleaned up version of ynput/ayon-core#183 .
Unreal part in ynput/ayon-unreal#8
Version control part in ynput/ayon-version-control#2

Official Unreal5 Deadline plugin is there as it contains additional 2 scripts for Perforce which needs to run before Unreal jobs. Its a question how to easily deploy those to DL repo, as it is not possible to just add these 2 to custom/Unreal5 to be picked up in addition to main Unreal5 plugin.
To add them to GlobalJobPreload seems a bit weird. But open for recommendations how to add them for easy deployment

AY-990

Whole Unreal5 plugin copied here as it is not possible to add to custom folder only `JobPreLoad.py` and `UnrealSyncUtil.py` which is handling Perforce. Might need to be revisited as this would create dependency on official Unreal5 plugin. `JobPreLoad.py` and `UnrealSyncUtil.py` handles Perforce syncing, must be triggered before Unreal rendering job. It would better to have here only these two files here, but deployment wouldn't be straightforward copy as for other plugins.

…adline_in_unreal

Subfolders were added to limit imports in different DCCs

m-u-r-p-h-y

was requested to test, but I'm lost and without clear test description can not help . . .

It should be relative path from workspace dir until folder with project

'env' is controlled by AYON, os.getenv might not Stream is required

antirotor

Just few notes, otherwise it works.

Just for the record, we are including UnrealEngine5 Deadline plugin with small bugfixes as is. Major part is unused by AYON at all, so in the next steps we need to decide what to do with it - as maintaining is probably nothing for us.

client/ayon_deadline/plugins/publish/unreal/submit_unreal_deadline.py

client/ayon_deadline/repository/custom/plugins/UnrealEngine5/JobPreLoad.py

Co-authored-by: Ondřej Samohel <[email protected]>

antirotor

Some question about Deadline plugin configuration....

antirotor · 2024-08-06T17:17:43Z

client/ayon_deadline/repository/custom/plugins/UnrealEngine5/JobPreLoad.py

+        version = deadlinePlugin.GetPluginInfoEntry("EngineVersion")
+        deadlinePlugin.LogInfo('Version defined: %s' % version )
+        version_string = str(version).replace(".", "_")
+        executable_key = f"UnrealEditorExecutable_{version_string}"


So with removing of hardcoded Unreal executable path by b57f535 this needs basically UnrealEditorExecutable_5_4 to be preset in the param (possibly UnrealEngine5.param) file, but currently there's nothing. Shouldn't we at least provide entries there for the latest 3 versions (5.2, 5.3 and 5.4)? Without it, it crashes:

Error: Job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\PHOBOS\plugins\66b25541b95e063dbc2eee1c\JobPreLoad.py': Unreal Engine 5.4 entry not found in .param file (FranticX.Scripting.PythonNetException) at Deadline.Plugins.DeadlinePlugin.FailRender(String message)

Or even better - set UnrealExecutable environment variable using AYON environments?

I am setting UnrealExecutable in env vars, see in Description

oh right, I am doing this too, but still getting thit error 🤔
There is even this entry in Job log:

2024-08-07 10:11:43: 0: INFO: Setting Process Environment Variable UnrealExecutable to C:\Program Files\Epic Games\UE_5.4\Engine\Binaries\Win64\UnrealEditor-Cmd.exe

so it should take both UnrealExecutable or UnrealEditorExecutable_5_4 (notice that extra Editor). So the problem is, that these variables will not make it to the UnrealEngine5 plugin proces (os.getenv() in JobPreLoad.py doesn't have them)

antirotor

it works with this

client/ayon_deadline/repository/custom/plugins/UnrealEngine5/JobPreLoad.py

Co-authored-by: Ondřej Samohel <[email protected]>

antirotor

works! thanks

kalisp added 2 commits July 8, 2024 18:43

Added unreal to deadline plugins

de75b45

iLLiCiTiT mentioned this pull request Jul 8, 2024

Unreal: support in Deadline ynput/ayon-core#670

Closed

iLLiCiTiT and others added 2 commits July 8, 2024 18:48

Merge branch 'develop' into feature/support_deadline_in_unreal

42be0b7

Merge remote-tracking branch 'origin/develop' into feature/support_de…

f747671

…adline_in_unreal

tatiana-ynput self-assigned this Jul 23, 2024

Moved unreal deadline plugin to subfolder

03a1675

Subfolders were added to limit imports in different DCCs

iLLiCiTiT assigned m-u-r-p-h-y Jul 23, 2024

iLLiCiTiT requested a review from antirotor July 23, 2024 12:53

iLLiCiTiT assigned antirotor Jul 23, 2024

m-u-r-p-h-y reviewed Jul 23, 2024

View reviewed changes

iLLiCiTiT assigned kalisp Jul 23, 2024

kalisp added 2 commits July 24, 2024 12:15

Fix resolution of PerforceGamePath

6d2fa50

It should be relative path from workspace dir until folder with project

Fix querying P4PORT from env

0fc1dcd

'env' is controlled by AYON, os.getenv might not Stream is required

antirotor requested changes Jul 24, 2024

View reviewed changes

antirotor mentioned this pull request Jul 24, 2024

Moves Sync dialog outside of Unreal ynput/ayon-version-control#2

Merged

antirotor added the type: feature Adding something new and exciting to the product label Jul 24, 2024

kalisp and others added 3 commits July 31, 2024 11:14

Update imports

c6211be

Co-authored-by: Ondřej Samohel <[email protected]>

Update logging

d9a3f6d

Co-authored-by: Ondřej Samohel <[email protected]>

Removed hardcoded unreal engine path

b57f535

kalisp requested a review from antirotor July 31, 2024 09:20

kalisp added 2 commits July 31, 2024 11:31

Renamed family to changelist_metadata

bed1687

Merge branch 'develop' into feature/support_deadline_in_unreal

672bc31

antirotor requested changes Aug 6, 2024

View reviewed changes

antirotor requested changes Aug 7, 2024

View reviewed changes

client/ayon_deadline/repository/custom/plugins/UnrealEngine5/JobPreLoad.py Outdated Show resolved Hide resolved

antirotor and others added 2 commits August 7, 2024 10:40

🐶 fix ruff warnings

da8f96e

Change to process environment variables

2feff1d

Co-authored-by: Ondřej Samohel <[email protected]>

antirotor approved these changes Aug 7, 2024

View reviewed changes

antirotor merged commit c975526 into develop Aug 7, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unreal: support in Deadline #1

Unreal: support in Deadline #1

iLLiCiTiT commented Jul 8, 2024 •

edited by kalisp

Loading

m-u-r-p-h-y left a comment

antirotor left a comment

antirotor left a comment

antirotor Aug 6, 2024

antirotor Aug 6, 2024

kalisp Aug 7, 2024

antirotor Aug 7, 2024

antirotor Aug 7, 2024

antirotor left a comment

antirotor left a comment

Unreal: support in Deadline #1

Unreal: support in Deadline #1

Conversation

iLLiCiTiT commented Jul 8, 2024 • edited by kalisp Loading

Changelog Description

Additional info

Testing notes:

Additional info

m-u-r-p-h-y left a comment

Choose a reason for hiding this comment

antirotor left a comment

Choose a reason for hiding this comment

antirotor left a comment

Choose a reason for hiding this comment

antirotor Aug 6, 2024

Choose a reason for hiding this comment

antirotor Aug 6, 2024

Choose a reason for hiding this comment

kalisp Aug 7, 2024

Choose a reason for hiding this comment

antirotor Aug 7, 2024

Choose a reason for hiding this comment

antirotor Aug 7, 2024

Choose a reason for hiding this comment

antirotor left a comment

Choose a reason for hiding this comment

antirotor left a comment

Choose a reason for hiding this comment

iLLiCiTiT commented Jul 8, 2024 •

edited by kalisp

Loading