Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug Report: Random Jenkins Build Failures at asset_profile step. #672

Open
galibzon opened this issue Feb 13, 2024 · 1 comment
Open

Bug Report: Random Jenkins Build Failures at asset_profile step. #672

galibzon opened this issue Feb 13, 2024 · 1 comment
Assignees
@gadams3
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/build Categorizes an issue or PR as relevant to SIG Build. sig/graphics-audio Categorizes an issue or PR as relevant to SIG graphics-audio.

Comments

@galibzon
Copy link
Contributor

Describe the bug
The o3de-extras jenkins jobs fail randomly during the asset_profile step. The failure is always the same, but it occurs randomly:

[2024-02-13T16:39:20.258Z] AssetProcessor: -----------------------------------------
[2024-02-13T16:39:20.258Z] AssetProcessor: Asset Processor Batch Processing complete
[2024-02-13T16:39:20.258Z] AssetProcessor: ---------------FAILED ASSETS-------------
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_0_depthpasstransparentmin.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_1_shadowmappass_customz.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_2_depthpass.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_3_shadowmappass.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_4_depthpasstransparentmax.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: D:/workspace/o3de/AutomatedTesting/Cache/Intermediate Assets/materialcanvas/graphdata/nodes/materialoutputs/standardpbr/materialgraphname_common_5_depthpass_customz.shader
[2024-02-13T16:39:20.258Z] AssetProcessor: -----------------------------------------
[2024-02-13T16:39:20.258Z] AssetProcessor: Number of Assets Successfully Processed: 2102.
[2024-02-13T16:39:20.258Z] AssetProcessor: Number of Assets Failed to Process: 6.

Some examples of jenkins jobs that have failed like that:
https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/development/111/pipeline
https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/PR-664/8/pipeline

Assets required
N/A

Steps to reproduce
Run a jenkins a job under the o3de-extras repo.

Expected behavior
asset_profile should pass

Actual behavior
asset_profile fails randomly, sometimes requiring to re-run the job at least 3 times for around 9hrs+ of waste compute time.

Screenshots/Video
N/A

Found in Branch
latest development of o3de-extras + o3de

Desktop/Device (please complete the following information):

  • Occurs on Jenkins

Additional context
Add any other context about the problem here.
@galibzon galibzon added kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 13, 2024
@galibzon galibzon added sig/build Categorizes an issue or PR as relevant to SIG Build. sig/graphics-audio Categorizes an issue or PR as relevant to SIG graphics-audio. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 13, 2024
@amzn-changml amzn-changml mentioned this issue Apr 17, 2024
Merged
@amzn-changml
Copy link
Contributor

Note that this appears to pass if using a clean build: https://jenkins.build.o3de.org/blue/organizations/jenkins/o3de-extras/detail/development/123/pipeline, however the subsequent incremental runs fails again.

We may want to consider having Jenkins set this error as a non-blocking, rather than failing the AR run completely. Here's an example of how that's implemented in Linux: o3de/o3de#8832

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gadams3 gadams3

Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/build Categorizes an issue or PR as relevant to SIG Build. sig/graphics-audio Categorizes an issue or PR as relevant to SIG graphics-audio.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@amzn-changml @galibzon @gadams3