System for implementing Houdini Nodes in version controlled and IAC based system

[Lypsolon]

Is there an existing issue for this?

• I have searched the existing issues.

Please describe the feature you have in mind and explain what the current shortcomings are?

While BigRoy was working on a recent improvement on the USD front, it became necessary to implement an HDA to handle Asset Loading. In the implementation phase, it became apparent that .hda files do not conform to the idea behind the Plain Text workflow, making overall QA for PRs hard.
We decided not to take action against this in the PR because we belive there might be a value in taking some more time for it and having a more robust setup to handle this.

PR: #294

How would you imagine the implementation of the feature?

We Thought about 2 Options for implementing a system like this.

1 Implement a system that allows .hda creation from source code.

This would mean that you describe your nodes in, e.g. a Python file, and when you create a package, a .hda file will be built from this.

2 Implement a PythonClass system that allows us to describe Houdini nodes in a central and general way and cast away .hda completely.

In this case, we would bould some PyhtonClasses that allow you to describe nodes in an Opinionated way from inside ayon. they would need options like (depend, connect, ayon tools, etc.)
atop this setup, we would need to have a system that loads said classes as nodes into Houdini. for that, a member of the Houdini-WG mentioned a while back that it's possible to have a shelf button that is not bound to a shelf. This will then appear in the Tab menu and act like a Normal node. we would abuse this in order to load our PyNodes.

Are there any labels you wish to add?

• I have added the relevant labels to the enhancement request.

Describe alternatives you've considered:

No response

Additional context:

No response

[BigRoy]

Here @fabiaserra mentions there's a ready-to-go solution for this:

That's already a solved problem and it's called expanding HDAs and you can do so with the hotl binary that ships with Houdini: https://www.sidefx.com/docs/houdini/ref/utils/hotl.html

We expand all of our internal HDAs that we store in version control by running hotl -t directory hdafile, something like this:

hotl -t driver_ax.ax_publisher.1.0 driver_ax.ax_publisher.1.0.hda
rm driver_ax.ax_publisher.1.0.hda
mv driver_ax.ax_publisher.1.0 driver_ax.ax_publisher.1.0.hda

(You need to add the .hda suffix on the expanded result in order for Houdini to discover it on the HOUDINI_PATH)

Of course this can be easily automated through the build system of your repo (we did automate it in my prior company) but I haven't created that many HDAs yet to need to spend the time for doing that.

Moreover, in order to be able to reuse as much of the backend code of the nodes, similarly to you in the HDA python scripts module I simply import this class wrapper factory of the node so I can then just call the functions defined in my repository.

In my prior studio we had a more complex system that using monkey patching it would override the base hou.Node python class so our code looked just like an extension to the existing python code of Houdini, that way our functions looked like native in Houdini (i.e. hou.node("/out/geometry").my_custom_function() or ``hou.node("/out/geometry").render()`) so we could extend functionality of native houdini nodes and wrap the backend of any node by simply calling the class the same name as the node.

Probably best to continue the discussion here.

Thanks @fabiaserra - perfect. I'd have imagined there was a solution to this.

Does this mean we would ship the extended/expanded state in the repository, but somehow turn those into .hda just before houdini launch? Or when do we convert back? Do we even convert back? Or is Houdini capable of loading the expanded hda directly and did you end up doing so? I would assume there's a performance bottleneck there.

[fabiaserra]

Does this mean we would ship the extended state in the repository, but somehow turn those into .hda just before houdini launch? Or when do we convert back? Do we even convert back? Or is Houdini capable of loading the expanded hda directly and did you end up doing so? I would assume there's a performance bottleneck there.

Nah, you keep them always expanded, Houdini treats them the same way as a binary .hda. I think the bottleneck is pretty negligible, you can ask SESI or do some benchmarks but we had a lot of expanded HDAs and that wasn't a concern

[fabiaserra]

For parm interfaces that are common across nodes what you want to do is also abstract that away into code as described on these articles https://www.andynicholas.com/post/customising-houdini-nodes-part-1-hdas-or-dynamic-ui so it's easier to maintain

[fabiaserra]

If you check the result of expanding the HDA, it contains the PythonModule or the different event triggered files (i.e., OnInputChanged) so you could easily automate that on all the HDAs hosted in the repo so a user doesn't need to touch those files from within the HDA and it automatically wraps the HDA with your code

[MustafaJafar]

Hey guys,
Thanks for these wonderful discussion.

I was reading through this issue and comments (specifically customising-houdini-nodes and expanding HDAs comment ).
Let write a summary of it and please correct me if something is not wrong.

Expanded HDA

HDA	Expanded HDA

• The .hda file format is a binary archive format.
• The Expanded HDA directory is a readable format.

Changes are explicit in Expanded HDA. You can adjust the default parameters and monitor any modifications a user makes to the HDA through version control. Unlike binary format, which cannot be tracked or edited without launching Houdini.

Customize HDAs

Customizing a binary .hda file is no different from customizing an Expanded HDA.
This topic is discussed in further details in customising-houdini-nodes.
but in essence, we have static and dynamic customization.

Static Customization

There are couple of options:

• Save HDA as new with higher version.
• Create new HDA presets.

Dynamic Customization

1. Define OnCreate events in HOM location .It can be used to customize node parameters.
   
2. Override Tab Menu on Houdini launch which can done via Houdini api or Monkey Patching
3. Using a Shelf; create a new shelf script that create your node and apply your changes.
4. Extend OPMenu.xml which is done in Houdini: Extend op menu to create HDAs in a Houdinish way #504

Best Practices

Store the code separately from the HDA, for example, in a dedicated Python file. This method is not a substitute for other customization techniques of HDAs; rather, it complements them. Saving the code externally is the suggested approach for any substantial pipeline integration.

[fabiaserra]

The .hda file format is a binary archive format. The Expanded HDA directory is readable format. Expanded HDA directory make tracking changes in git more explicit.

Not more explicit, the binary .hda is not explicit at all

Keep in mind, we are already saving the code library in outside the HDA. So, tracking changes IMO is considered explicit.

Not true, having expanded HDAs is way more than just the internal code as you can tweak default parms and see any changes a user does to the HDA in version control, binary format is impossible to track and edit without launching Houdini

Customize HDAs

It's works with Expanded HDAs. But, it's more used with binary .hda files.

Not true, most big studios use expanded HDAs and there's no difference in customizing either.

2. Override Tab Menu on Houdini launch which can done via Houdini api or Monkey batching

It's called "monkey patching"

5. Save the code outside of the HDA. i.e. in a dedicated python file.

This is not an alternative to the other ways of customizing HDAs, all of those methods you can do it by saving the code outside of the HDA and that's the recommended approach for any significant pipe integration

[MustafaJafar]

@fabiaserra
Thanks a lot. let me update my comment.

[BigRoy]

For what it's worth - I've converted the Load Asset LOP into the expanded format using hotl binaries that comes with Houdini.
See #294 with commit 8aee5f1

[BigRoy]

This has been solved now - and implemented currently for the Load Asset LOP.

This is still a goldmine of information for those that don't know about this feature - but with the knowledge gained here the first PR using it has been merged recently and hence this problem is now 'solved'. Feel free to re-open on ayon-houdini if it becomes a problem in the future.