Houdini: Better support for single frame outputs

[BigRoy]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior:

A lot of extractor seems to rely on instance.data["frames"] to be a list of files:

• ayon-core/client/ayon_core/hosts/houdini/plugins/publish/extract_bgeo.py

  Lines 37 to 42 in ee6e42e

  	output = instance.data["frames"]
  	_, ext = lib.splitext(
  	output[0], allowed_multidot_extensions=[
  	".ass.gz", ".bgeo.sc", ".bgeo.gz",
  	".bgeo.lzma", ".bgeo.bz2"])

• ayon-core/client/ayon_core/hosts/houdini/plugins/publish/extract_composite.py

  Lines 32 to 34 in ee6e42e

  	output = instance.data["frames"]
  	_, ext = splitext(output[0], [])
  	ext = ext.lstrip(".")

• ayon-core/client/ayon_core/hosts/houdini/plugins/publish/extract_ass.py

  Lines 37 to 49 in ee6e42e

  	# Unfortunately user interrupting the extraction does not raise an
  	# error and thus still continues to the integrator. To capture that
  	# we make sure all files exist
  	files = instance.data["frames"]
  	missing = []
  	for file_name in files:
  	full_path = os.path.normpath(os.path.join(staging_dir, file_name))
  	if not os.path.exists(full_path):
  	missing.append(full_path)
  	if missing:
  	raise RuntimeError("Failed to complete Arnold ass extraction. "
  	"Missing output files: {}".format(missing))

• and more.

However it's a single file if the instance has no frame range data.

If it has frame range data but a it's a single frame then I believe these extractors end up setting a single file list like: representation["files"] = ["single_file.1001.exr"] - which the integrator does not tend to like.

For example, a single frame COP2 image publish errors with the following report:

publish-report-240325-17-47.json

The integrator wants single files to be a single filename instead of a list.

Expected Behavior:

There should be simple consistency across the API, so I suppose instance.data["frames"] should at best always be a list. We should then however just restructure the representation["files"] to be a single filename instead of the list of files.

Version

1.0.0

What platform you are running on?

Windows

Steps To Reproduce:

1. Publish a single frame of a sequenced product type, e.g. image sequence (COP2), bgeo, vdb

Are there any labels you wish to add?

• I have added the relevant labels to the bug report.

Relevant log output:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Ynput\AYON\dependency_packages\ayon_2403061937_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process
    runner(*args)
  File "E:\dev\ayon-core\client\ayon_core\plugins\publish\integrate.py", line 232, in process
    six.reraise(*sys.exc_info())
  File "C:\Program Files\Side Effects Software\Houdini 20.0.590\python310\lib\site-packages\six.py", line 719, in reraise
    raise value
  File "E:\dev\ayon-core\client\ayon_core\plugins\publish\integrate.py", line 219, in process
    self.register(instance, file_transactions, filtered_repres)
  File "E:\dev\ayon-core\client\ayon_core\plugins\publish\integrate.py", line 412, in register
    op_session.commit()
  File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\operations.py", line 701, in commit
    self._con.send_batch_operations(
  File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\server_api.py", line 6575, in send_batch_operations
    raise FailedOperations((
ayon_api.exceptions.FailedOperations: Operation "6ec98b42-d9db-4f78-8830-ff13b2f0af7a" failed with data:
{
    "id": "6ec98b42-d9db-4f78-8830-ff13b2f0af7a",
    "type": "create",
    "entityType": "representation",
    "entityId": "7372cb33eac711eeab71a3d64c91bc2b",
    "data": {
        "id": "7372cb33eac711eeab71a3d64c91bc2b",
        "versionId": "7366107deac711ee9003a3d64c91bc2b",
        "files": [
            {
                "id": "737e65b5eac711ee9c30a3d64c91bc2b",
                "name": "ynts_char_hero_imagesequenceMain_v003.",
                "path": "{root[work]}/ayontest/asset/char_hero/publish/imagesequence/imagesequenceMain/v003/ynts_char_hero_imagesequenceMain_v003.",
                "size": 66396,
                "hash": "ynts_char_hero_imagesequenceMain_v003,|1711385284,74326|66396",
                "hash_type": "op3"
            }
        ],
        "name": "",
        "data": {
            "context": {
                "root": {
                    "work": "C:/projects"
                },
                "project": {
                    "name": "ayontest",
                    "code": "ynts"
                },
                "hierarchy": "asset",
                "folder": {
                    "name": "char_hero"
                },
                "product": {
                    "name": "imagesequenceMain",
                    "type": "imagesequence"
                },
                "version": 3,
                "ext": "",
                "asset": "char_hero",
                "task": {
                    "name": "modeling",
                    "type": "Modeling",
                    "short": "mdl"
                },
                "subset": "imagesequenceMain",
                "family": "imagesequence",
                "representation": "",
                "username": "admin",
                "user": "admin"
            }
        },
        "attrib": {
            "path": "C:\\projects\\ayontest\\asset\\char_hero\\publish\\imagesequence\\imagesequenceMain\\v003\\ynts_char_hero_imagesequenceMain_v003.",
            "template": "{root[work]}/{project[name]}/{hierarchy}/{folder[name]}/publish/{product[type]}/{product[name]}/v{version:0>3}/{project[code]}_{folder[name]}_{product[name]}_v{version:0>3}<_{output}><.{frame:0>4}><_{udim}>.{ext}"
        }
    }
}
Detail: 1 validation error for RepresentationPostModel
name
  string does not match regex "^[a-zA-Z0-9_]([a-zA-Z0-9_\.\-]*[a-zA-Z0-9_])?$" (type=value_error.str.regex; pattern=^[a-zA-Z0-9_]([a-zA-Z0-9_\.\-]*[a-zA-Z0-9_])?$).

This is an example where e.g. the extractor fails to retrieve the extension because it's expecting a list of files, not a string.

Additional context:

No response

[cuID:AY-4795]

[MustafaJafar]

The integrator wants single files to be a single filename instead of a list.

I do remember something like that. I faced that in #9
tbh, I accepted it as a fact in my PR.

ayon-core/client/ayon_core/hosts/houdini/plugins/publish/collect_files_for_cleaning_up.py

Lines 60 to 90 in d5101b7

	files = []
	# Non Render Products with frames
	frames = instance.data.get("frames", [])
	staging_dir, _ = os.path.split(filepath)
	if isinstance(frames, str):
	files = [os.path.join(staging_dir, frames)]
	else:
	files = [os.path.join(staging_dir, f) for f in frames]
	# Render Products
	expectedFiles = instance.data.get("expectedFiles", [])
	for aovs in expectedFiles:
	# aovs.values() is a list of lists
	files.extend(sum(aovs.values(), []))
	# Intermediate exported render files.
	# TODO : For products like Karma,
	# Karma has more intermediate files
	# e.g. USD and checkpoint
	ifdFile = instance.data.get("ifdFile")
	if self.intermediate_exported_render and ifdFile:
	start_frame = instance.data.get("frameStartHandle", None)
	end_frame = instance.data.get("frameEndHandle", None)
	ifd_files = self._get_ifd_file_list(ifdFile,
	start_frame, end_frame)
	files.extend(ifd_files)
	# Non Render Products with no frames
	if not files:
	files.append(filepath)

[MustafaJafar]

Also, I'd like to thank you because I finally found the issue with image sequence in Houdini!
It only works if frame range is more than 1!

[antirotor]

This should be partly solved by #105 - but it will need some additional work on Houdini support side

[BigRoy]

The actual bugs that existed here should've been solved by #676 because it moved the extraction logic into a single plug-in which meant less duplicated code involving this. So we could close this - unless we want to keep it open to track a larger redesign of how we're dealing with files, etc. But I feel #105 already represents that maybe?

Should we close this one?

[MustafaJafar]

IMO, this issue was made originally to track the issue that most extractors expected instance.data["frames"] to be a list.
which was solved by as you have mentioned #676 .
if we want to suggest new design for ayon we can create another issue.