Houdini: Publish any ROP node (Generic Creator)

[BigRoy]

Implement Generic ROP publish from ynput/ayon-core#542

---

Changelog Description

This PR implements an idea for "lower level publishing" in Houdini. This implement Generic ROP publishing. Just create any Houdini ROP node (or custom Rop node HDA) and publish your product types from it!

This PR adds a Generic Creator:
Currently, It injects the ayon meta data onto ROP nodes to behave as though they were produced by various creators from Ayon. This ensures that validators and collectors are retained without loss.
e.g. using Generic creator while setting product type to pointcache should yield similar/same results as using pointcache creator
Here is an example using labs karma node and make it behave as though it was made by karma creator.
ynput/ayon-core#542 (comment)

Additional info

As part of the Ynput Houdini Workgroup sessions I developed this quick prototype to expose a way to batch publish and ingest files. Consider it more of an exploration of what's possible then a "drop it in production now" ready-to-go solution.

Explainer

Yes, this requires some explanation. Here you go.

2024-05-22.20-48-11.mov

What I forgot to add is that it currently still relies on detecting what the output files are for a ROP node based on a "file" parm that is often unique in Houdini per node. If anyone knows a way to just query the expected output files for a ROP node (similar to what PDG/TOPs seems to do I'd love to know!) but otherwise we'll just need to expand that list.

However, I also played with the idea of having "custom file list" attributes on the Node that when enabled could override the "Collector" logic and would instead use that list of files as the publish instance's files. So that e.g. one instance could also publish multiple representations. For that, @MustafaJafar did this lovely non-functional 'prototype' but it does get the idea across:

Demo scene file

The demo scene file:

ynts_char_hero_pdg_v012.zip

TODO 📔 ✏️

• Use $OS as default variant name.
• Add "OnCreated" callback to allow a studio to configure which nodes to automatically imprint as 'publishable'. See Fabia's first video here at 08:10 and code here
  • This is currently a first iteration - should still be exposed to project settings and allow user configuration.
• The product types being an enum with predefined standards per node type.
• The comment field being available on the ROP nodes
  • This does actually already work if you'd add the product type to ayon+settings://core/publish/collect_comment_per_instance ✅ - I guess this is fine for now.
• Add "Publish" button directly on the node similar to "self publish" or whatever we had for others
• Allow exposing to settings which node types may need to be "automatically imprinted" on create for a studio to customize, with also settings for allowed product types (the default type, default variant name, etc.)
• Implement good support for multiple representations, e.g. like this and Fabia's second video near the end.
• Make sure this API approach remains a goal of the PR.
• Keep in mind Fabia's usage where their AX Publish Submitter can submit just the publish of existing files to the farm and create e.g. a nuke dependency job that generates custom previews of the intermediate media, etc.
• The product name is currently an 'exposed field' but should actually be locked off to the user for manual editing and should always adhere to the get_product_name logic of the Creator and the product types so that they adhere to project templates correctly.
  • We need to make sure this isn't an expression that is heavy and re-evaluates the whole time. May need to be based on callback on "product type" and "variant" changes.

Known issues / Blockers 📔 🔥

• The publisher, creator and create context can't differentiate attributes per instance, as such it can't e.g. differentiate whether "split export and render job" makes sense or not for a particular instance's attribute definitions. See comment
• It will only detect the single file defined by the ROP node's output path and does not recognize whether e.g. a USD ROP or GLTF rop may generate other sidecar files (which often also if moved along should also remap paths, etc. so it might be non-trivial to support those out of the box)
  • This basically makes it useless for GLTF exports, and in many cases useless for USD (if it were to write multiple files which also need remapping) and rendering (which may generate export files, need dedicated Deadline plug-ins to maybe render certain outputs (e.g. Husk) or may generate multiple sequences of files (like AOVs to separate files).
• We do not know or detect the colorspace of the output files (there is no 'generic' way for ANY ROP to return what colorspace it may result in.
• If certain ROPs may override the output FPS other than the workfile, we do not know it.
• Due to Creators enforcing a single product type for which the attribute definitions are populated/created based on the Creator's product type we run into issues with a generic Creator that basically switches product type on the instances dynamically. See comments here.

Testing notes:

1. Check out the branch
2. Check out the explainer video
3. Test the demo scene file
4. Comment with great ideas on how to improve

Resolves #16

[krishnaavril]

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

[BigRoy]

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

Interesting request - when would you like the 'clear' this color or sticker? I can imagine it otherwise always having the sticker after the first publish you've done from the workfile.

[krishnaavril]

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

Interesting request - when would you like the 'clear' this color or sticker? I can imagine it otherwise always having the sticker after the first publish you've done from the workfile.

Yes, when its done fitst publish sounds good.

Another instance I can think is It helps a lot when the filecache node publishes, coz in sops its easy to mess around to find the publish node.

[MustafaJafar]

I've a question about collect batch families type.

[MustafaJafar]

Is it possible to add some condition here to add more items based on the instance ?

I tried but I found out that it works per creator not per instance.

[BigRoy]

No - unfortunately not. The current Creator system design and the Publisher UI do not allow it unfortunately. @antirotor I believe is aware of it and it's part of the "Extended Publisher" epic.

[BigRoy]

@MustafaJafar it may be that this PR ynput/ayon-core#860 by @iLLiCiTiT may solve this. The question then becomes WHAT would define which of the targets would be available?

[fabiaserra]

is use existing frames (farm) not supported yet?

[BigRoy]

is use existing frames (farm) not supported yet?

I believe currently it is for some, not all, if I remember correctly. Not sure about it exactly.

[MustafaJafar]

is use existing frames (farm) not supported yet?

Render targets feature is not yet implemented for all product types.

[MustafaJafar]

@BigRoy I gave it a test and it actually works although it was hard for me to learn how to use it. I can't really tell if there are any alternative solutions but, we can discuss that on discord.
ynput/ayon-core#860 (review)

[krishnaavril]

Adding use exiting frames using farm is essential IMHO, there is a dicussion happened in couple of days ago, about this feature.
Local publish is tkaing huge amount of time and leaving houdini freezed until the publish get done. This a time saving for artists.

[BigRoy]

Adding screenshot of what it looks like now:

[krishnaavril]

Hey!

Just testing this branch, we cannot publish anything from houdini(workfiles/caches etc), somehow its getting conflict, I've attatched the error when we tried to publish workfiles with this branch

Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.4-dev.1\ayon_core\pipeline\create\context.py", line 2484, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 393, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 641, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 624, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.9libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

[BigRoy]

Hey!

Just testing this branch, we cannot publish anything from houdini(workfiles/caches etc), somehow its getting conflict, I've attatched the error when we tried to publish workfiles with this branch

Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.4-dev.1\ayon_core\pipeline\create\context.py", line 2484, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 393, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 641, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 624, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.9libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Thanks @krishnaavril - should be solved now. It will now log a warning on creation.
It's because it tries to create it twice somehow? you have the creation of self publish button enabled in settings.

But that would only have been the case if you triggered the generic creation on a node that was already a publishable instance. I've changed the code, but would still be lovely if you could share a screen capture of how you get it to the state that it breaks.

[krishnaavril]

Thanks, Yes I do have the self-publish button enabled in Ayon settings.

But that would only have been the case if you triggered the generic creation on a node that was already a publishable instance.
No, I tried this with our inhouse otl, adding the parameter and otl name hardcoded in the ayon code which allows publishing our own cache and render nodes.

Will test these pushed changes today, thanks

[krishnaavril]

I've tested the code with the labs karma node, I think the issue is still there

Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.5-dev.3\ayon_core\pipeline\create\context.py", line 1278, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 392, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 640, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 623, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.10libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

after create was done and when I clicked on publish this error popped up!

[MustafaJafar]

This should fix the bug reported in #2 (comment)

Suggested change

	publish_button_parm = hou.ButtonParmTemplate(
	"AYON_self_publish",
	"Publish",
	help="Directly publishes only this node and any inputs with a "
	"comment entered in a popup prompt. All other instances will "
	"be disabled in the publish.",
	script_callback="from ayon_houdini.api.lib import "
	"self_publish; self_publish()",
	script_callback_language=hou.scriptLanguage.Python,
	)
	ayon_folder.addParmTemplate(publish_button_parm)
	publish_button_parm = parm_group.find("AYON_self_publish")
	if not publish_button_parm:
	publish_button_parm = hou.ButtonParmTemplate(
	"AYON_self_publish",
	"Publish",
	help="Directly publishes only this node and any inputs with a "
	"comment entered in a popup prompt. All other instances will "
	"be disabled in the publish.",
	script_callback="from ayon_houdini.api.lib import "
	"self_publish; self_publish()",
	script_callback_language=hou.scriptLanguage.Python,
	)
	ayon_folder.addParmTemplate(publish_button_parm)

[MustafaJafar]

Also, the above logic always creates self publish button as it doesn't take the value of the following setting into consideration:

• ayon+settings://houdini/general/add_self_publish_button

[MustafaJafar]

we can also do this which leverage self.add_publish_button that is set in apply_settings.

Suggested change

	publish_button_parm = hou.ButtonParmTemplate(
	"AYON_self_publish",
	"Publish",
	help="Directly publishes only this node and any inputs with a "
	"comment entered in a popup prompt. All other instances will "
	"be disabled in the publish.",
	script_callback="from ayon_houdini.api.lib import "
	"self_publish; self_publish()",
	script_callback_language=hou.scriptLanguage.Python,
	)
	ayon_folder.addParmTemplate(publish_button_parm)
	publish_button_parm = parm_group.find("AYON_self_publish")
	if not publish_button_parm and self.add_publish_button:
	publish_button_parm = hou.ButtonParmTemplate(
	"AYON_self_publish",
	"Publish",
	help="Directly publishes only this node and any inputs with a "
	"comment entered in a popup prompt. All other instances will "
	"be disabled in the publish.",
	script_callback="from ayon_houdini.api.lib import "
	"self_publish; self_publish()",
	script_callback_language=hou.scriptLanguage.Python,
	)
	ayon_folder.addParmTemplate(publish_button_parm)

But, this has two downsides:

1. it won't add the button until we change any attribute (as it won't be marked as a change on its own)
2. the button will be added at the end of the list because we are using .addParmTemplate and folders doesn't support insertBefore.
   

[iLLiCiTiT]

Why it's not stored as json string? (talking about "publish_attributes" and "creator_attributes").

[BigRoy]

So that the attributes are actually accessible as options on the node for the user to click. We want them to be like native houdini parms so houdini users can do with them using houdini logic whatever they want - that's the whole point. ;)

[iLLiCiTiT]

So "handleStart" = 0 will become "handleStart": True (as an example).

BTW considering in progress feature on create context: "Value change based callbacks to update attribute definitions or values." will break this assumption that you can freely change create and publish attributes with no consequences.

[BigRoy]

Does exposing these publisher UI attributes on the node actually make sense?

BTW considering in progress feature on create context: "Value change based callbacks to update attribute definitions or values." will break this assumption that you can freely change create and publish attributes with no consequences.

Yes, makes sense. There's just a plethora of reasons why one WOULD want to actually exploit this type of full control in Houdini but I'm not confident whether those are production use cases that really make sense over time.

@fabiaserra @krishnaavril @MustafaJafar how much use would you say there is in actually having all these publisher UI options exposed as separate Houdini parms? Like, would you really ever have those parms reference some other attribute in a production scenario? Or is setting those via the publisher UI sufficient (it would allow us to create more dynamic UI over time without 'unexpected changes' coming back from Houdini).

See here for a screenshot with some of those exposed attributes.

In this case in particular the Creator Attributes and Publish Attributes

---

So "handleStart" = 0 will become "handleStart": True (as an example).

Luckily those are not exposed like that in Houdini 🙈 - but yes, that definitely points to a crucial bug that would need solving. However, 'setting' them om the attribute definitions in publisher UI themselves would then automatically turn that back into 0 and 1 so it may be 'ok' but it's definitely illogical code. Thanks for point it out!

[fabiaserra]

I'm not sure I understand the questions but let me try explain why:

how much use would you say there is in actually having all these publisher UI options exposed as separate Houdini parms?

What do you mean by "separate"? If you just mean if there's a point on having these options exposed on the Houdini nodes natively... yeah, there's a lot of use, it allows artists to use Houdini workflows and not be limited by another UI, making it very easy entry point for Houdini users (the advanced options are probably confusing and irrelevant for anyone though?)

Like, would you really ever have those parms reference some other attribute in a production scenario?

Potentially, that's the beauty of Houdini

Or is setting those via the publisher UI sufficient

Not as it is, but not because just of the UI setting parameters... the whole publish framework being coupled with the export of the nodes is very limited IMO as I have already explained a few times

[MustafaJafar]

Additionally, the node parameters and the publisher attributes are synced.
From the node to the publisher and from the publisher to the node.
That logic is implemented in collect_instances by creating the instance manually and adding it to the context and update_instances by the imprint method.

[krishnaavril]

Does exposing these publisher UI attributes on the node actually make sense?

BTW considering in progress feature on create context: "Value change based callbacks to update attribute definitions or values." will break this assumption that you can freely change create and publish attributes with no consequences.

Yes, makes sense. There's just a plethora of reasons why one WOULD want to actually exploit this type of full control in Houdini but I'm not confident whether those are production use cases that really make sense over time.

@fabiaserra @krishnaavril @MustafaJafar how much use would you say there is in actually having all these publisher UI options exposed as separate Houdini parms? Like, would you really ever have those parms reference some other attribute in a production scenario? Or is setting those via the publisher UI sufficient (it would allow us to create more dynamic UI over time without 'unexpected changes' coming back from Houdini).

See here for a screenshot with some of those exposed attributes.

In this case in particular the Creator Attributes and Publish Attributes

So "handleStart" = 0 will become "handleStart": True (as an example).

Luckily those are not exposed like that in Houdini 🙈 - but yes, that definitely points to a crucial bug that would need solving. However, 'setting' them om the attribute definitions in publisher UI themselves would then automatically turn that back into 0 and 1 so it may be 'ok' but it's definitely illogical code. Thanks for point it out!

Does exposing these publisher UI attributes on the node actually make sense?

BTW considering in progress feature on create context: "Value change based callbacks to update attribute definitions or values." will break this assumption that you can freely change create and publish attributes with no consequences.

Yes, makes sense. There's just a plethora of reasons why one WOULD want to actually exploit this type of full control in Houdini but I'm not confident whether those are production use cases that really make sense over time.

@fabiaserra @krishnaavril @MustafaJafar how much use would you say there is in actually having all these publisher UI options exposed as separate Houdini parms? Like, would you really ever have those parms reference some other attribute in a production scenario? Or is setting those via the publisher UI sufficient (it would allow us to create more dynamic UI over time without 'unexpected changes' coming back from Houdini).

See here for a screenshot with some of those exposed attributes.

In this case in particular the Creator Attributes and Publish Attributes

So "handleStart" = 0 will become "handleStart": True (as an example).

Luckily those are not exposed like that in Houdini 🙈 - but yes, that definitely points to a crucial bug that would need solving. However, 'setting' them om the attribute definitions in publisher UI themselves would then automatically turn that back into 0 and 1 so it may be 'ok' but it's definitely illogical code. Thanks for point it out!

Hello @BigRoy ! I think it was completly fine having properties in ayon publisher window, Honestly Me and my teams uses only visualise purpose, there is no change scenario seen in production from our end. (I was writing this coz we tested this rop publish when houdini is in core)

Can we write a logic where ayon filters the rop node inside any studio specific otl?
This will help to identify what kind of files will be writting(pointcache/renders) by node or by the location of node we adding(obj -> pointcache/usd, stage-> Usd/USD render rop), this will help on creating generic for any otl(without we hardcoded in the script), this should essentially open the possiblity of adding generic for any custom otl with no code changes(or a ayon frontend parameter will be good too, where we mention our node names and parameters). we also know studio uses there own otls for render, we can put in the documentation that to add duplicate there parameter and name it as a picture parameter, thus it refers the path and publishes.

Let me know if this make sense.