Folder batch publishing tool

[bradenjennings]

Changelog Description

Batch Publisher can be opened from AYON launcher tray.
It is a tool for batch publishing multiple files each as a separate product.

Testing notes:

1. git clone the OpenPype repo
2. checkout this branch "Feature/op 7692 folder batch publishing tool"
3. Setup dev environment on AYON server and set OpenPype directory for bundle to directory of OpenPype
4. Ask @bradenjennings for ayon internal testing data
5. Make sure your AYON server has a project available. For example make sure you setup AYON server on AY_CG_demo project .
6. Open Batch publisher for AYON launcher tray
7. Choose the browse button and pick the root directory of test data, here
   C:\errolDotting
8. The table view should show some fbx file paths
9. Choose your folder (asset) to make the line "defined"

[ynbot]

Task linked: OP-7692 Folder Batch publishing tool

[tokejepsen]

Could we get some testing notes and the hound comments resolved?

[bradenjennings]

please note you probably only want to review the widgets.py file (it will soon be broken up into multiple files for model view etc).

The rest of the code is for publishing using Deadline "OpenPype" plugin, and isn't working yet. The whole approach might be wrong anyway.

[tokejepsen]

@bradenjennings thanks for updating the testing notes. When you think the PR is a state to review, then mark it with the Ready for review button.

[BigRoy]

Batch Publisher can be opened from AYON launcher tray.
It is a tool for batch publishing multiple files each as a separate product.

I wonder, isn't that somewhat what the Tray Publisher was intended to be for?
If not, what in its design "gets in the way" and would that mean that maybe the issue is in the design of the tray publisher?

Could you share a screenshot of what it looks like?

I wonder if some work here might overlap with the work done here: #5795 since you describe that it shows the result of batch ingestion in a table, I wonder whether it could actually just generate a relevant CSV file and thus re-use the entry point from the CSV one. (or the other way around, have the CSV one use the 'batch ingest' entry point maybe being defined here)

Also Batch Publishing / Ingesting may have it overlap with this here: https://community.ynput.io/t/batch-publishing-ingest/840/18 - so just cross-referencing that as well @fabiaserra @mkolar

[fabiaserra]

Also Batch Publishing / Ingesting may have it overlap with this here: https://community.ynput.io/t/batch-publishing-ingest/840/18 - so just cross-referencing that as well @fabiaserra @mkolar

I mean most of the code on this PR is a subset of mine haha @bradenjennings why didn't you tag me?

Also I think this misses the point of the opportunity we have with this of abstracting the parts that can be reused for other modules and continues with the OP bad practice of just copy pasting code all over the place...

[bradenjennings]

The Batch Publisher model populates first by running each glob string search named GLOB_SEARCH_TO_PRODUCT_INFO_MAP, within the top level directory you pick.
For example a glob statement is
"glob": "*/fbx/*.fbx"
it then uses the product type and representation that is contained within the same level of the dictionary.
https://github.com/ynput/OpenPype/pull/6144/files#diff-1ccdf8c93903740c855ee943a69ae58fc5184d636ca92487f1187b77cff08ea2R16

Next I walk the top level directory again and try to add any more product types.
By checking the extension is in PRODUCT_TYPE_TO_EXT_MAP.
This allows a more general purpose way of adding many product types to batch publisher.
https://github.com/ynput/OpenPype/pull/6144/files#diff-1ccdf8c93903740c855ee943a69ae58fc5184d636ca92487f1187b77cff08ea2R49

representation name can come from GLOB_SEARCH_TO_PRODUCT_INFO_MAP
otherwise similar to @fabiaserra code representation can come from PRODUCT_TYPE_TO_EXT_MAP
https://github.com/ynput/OpenPype/pull/6144/files#diff-1ccdf8c93903740c855ee943a69ae58fc5184d636ca92487f1187b77cff08ea2R35

[moonyuet]

Did simple test on this ticket. It launches and publishes successfully in AYON, but it would be great if there is select/deselect all option for the files as It is quite painful for users to unclick each to publish when they want to publish some files from tons of versions. Not sure if the product type preference is preset or not. But it would be great to set product type accordingly with the chosen folder.

[bradenjennings]

Did simple test on this ticket. It launches and publishes successfully in AYON, but it would be great if there is select/deselect all option for the files as It is quite painful for users to unclick each to publish when they want to publish some files from tons of versions

There is already a right click / context menu option to toggle selected rows between enabled or not

[bradenjennings]

Not sure if the product type preference is preset or not. But it would be great to set product type accordingly with the chosen folder.

You can edit the product type per row, by double clicking the cell to activate the editor widget combo box.

The default product type is chosen by an internal dictionary mapping. This should be exposed to AYON settings before releasing it.

[antirotor]

this will crash in case you can split the file but it isn't sequence - or rather there the part supposed to be frame number actually isn't:

Traceback (most recent call last):
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\ui\window.py", line 129, in _on_dir_change
    self._table_view.set_current_directory(directory)
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\ui\batch_publisher_view.py", line 47, in set_current_directory
    self._model.set_current_directory(directory)
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\ui\batch_publisher_model.py", line 34, in set_current_directory
    self._populate_from_directory(directory)
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\ui\batch_publisher_model.py", line 198, in _populate_from_directory    self._product_items = self._controller.get_product_items(
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\controller.py", line 281, in get_product_items
    frames = self._get_frames_for_filepath(filepath)
  File "C:\Users\annat\Documents\Projects\Ayon\OpenPype\openpype\hosts\batchpublisher\controller.py", line 374, in _get_frames_for_filepath
    frame = int(filepath_parts[-2])
ValueError: invalid literal for int() with base 10: 'ma'

And I know, I ran it on totally random folder :)

[bradenjennings]

@antirotor

this will crash in case you can split the file but it isn't sequence - or rather there the part supposed to be frame number actually isn't:

I will work on a fix for this. But what was the file path to the problematic file anyway? It would be good to test on my end