A basic asset creation pipeline - batteries included.

• Keywords: Film, games, content creation, pipeline

• Objective:

  1. Provide an example of how one might go about building a pipeline around Pyblish.
  2. Demonstrate where publishing fits within a typical production pipeline
  3. Inspire further expansion upon basic ideas

• Mission: Demonstrate what is made possible with publishing.

• Motivation: I'm doing this for the same reason I created Pyblish. Because I see publishing as the single most important aspect of any production pipeline. It is on top of the advantages that it provides that the surrounding pipeline is made possible - e.g. browser and loader, builder and manager.

• Requirements: Reliably output correct data with minimal impact on artist productivity.

• Technology: Starter is built upon Pyblish, Python and bindings for Qt, and depends upon a Windows, Linux or MacOS operating system with Autodesk Maya.

• Audience: Technical directors interested in pipeline working in small- to mid-sized companies with a thirst for better ways of working.

• Discaimer: In the interest of simplicity, Starter is a very limited pipeline. If you have experience with an existing pipeline, or have thought about making your own, odds are this will look nothing like it. Take what you can from this project, and feel free to contribute your own ideas to make it simpler, or fork and expand upon it.

Before you start looking into pyblish-starter, it is recommended that you first familiarise yourself with Pyblish.

• learn.pyblish.com

To make the most out of pyblish-starter, some knowledge and experience is assumed.

Welcome to pyblish-starter, a basic asset creation pipeline - batteries included.

Table of contents

• Install
• Usage
• Description
• Batteries
  • Creator
  • Loader
• Looking Ahead
• API
  • Terminology
  • Filesystem API
    • Information Hierarchy
    • Workspace, Stage and Share
    • ls()
    • asset.json
    • version.json
    • representation.json
  • Starter API
  • Host API
• Contract
  • starter.model
  • starter.rig
  • starter.animation
• Homework
• Contributing
• Help

Starter takes the form of a Python package with embedded plug-ins.

$ pip install pyblish-starter

Starter is initialised by calling install() with an interface for your host.

>>> from pyblish_starter import install, maya
>>> install(maya)

From here, you model, rig and animate as per the contract below.

Build your own ASSET CREATION PIPELINE, starting with the basics.

Overview

Asset creation covers aspects related to producing content used in the production of film. A film is typically partitioned into sequences and shots, where each shot consists of one or more assets.

This project includes plug-ins and tools for 3 common types of ASSETS in a typical production pipeline.

• Modeling
• Rigging
• Animation

These illustrate how to (1) devise a contract - known as a family - for each kind of ASSET and (2) design their interface towards each other.

Batteries included

In addition to contracts, the project illustrates boundaries between publishing and setup. The definition of "setup" being anything that isn't strictly output, such as creating geometry or assigning attributes.

It includes a series of graphical user interfaces to aid the user in conforming to these contracts. These interfaces represent placeholders for your pipeline to fill in; you are invited to either continue from where they finish, or build your own from scratch.

In addition to providing a co-operative set of plug-ins, Starter also implements a minimal toochain for ASSET creation in a typical film pipeline.

Associate content with a family.

The family is what determins how the content is handled throughout your pipeline and tells Pyblish what it should look like when valid.

API

The creator respects families registered with Starter.

from pyblish_starter import api

api.register_family(
    name="my.family",
    help="My custom family"
)

For each family, a common set of data is automatically associated with the resulting instance.

{
    "id": "pyblish.starter.instance",
    "family": {chosen family}
    "name": {chosen name}
}

Additional common data can be added.

from pyblish_starter import api

api.register_data(
    key="myKey",
    value="My value",
    help="A special key"

Finally, data may be associated with a family.

from pyblish_starter import api

api.register_family(
    name="my.family",
    data=[
        {"key": "name", "value": "marcus", "help": "Your name"},
        {"key": "age", "value": 30, "help": "Your age"},
])

Visualise results from api.ls().

from pyblish_starter import api

for asset in api.ls():
    print(ASSET["name"])

API

The results from api.ls() depends on the currently registered root.

from pyblish_starter import api

api.register_root("/projects/gravity")

The chosen ASSET is passed to the load() function of the currently registered host.

from pyblish_starter import api, maya
api.register_host(maya)

A host is automatically registered on pyblish_starter.install().

Without publishing, in any shape or form, the following essential tools are but a dream.

Ideas

Here are a few things you might want to implement now that you have a solid framework upon which to build.

• Add a thumbnail to ASSETS by automatically extracting one per instance
• Enable search in Loader, and augment it by extracting additional metadata about an ASSET, such as author, use, size, origin or datatype.

Starter exposes a series of interrelated APIs to the end-user.

Starter reserves the following words for private and public use. Public members are exposed to the user, private ones are internal to the implementation.

	Term	Public	Description	Example
	PROJECT	X	Root of information	Gravity, Dr. Strange
	ASSET	X	Unit of data	Ryan, Bicycle, Flower pot
	VERSION	X	An ASSET iteration	v1, v034
	REPRESENTATION		A data format	Maya file, pointcache, thumbnail
	FORMAT		A file extension	.ma, .abc, .ico, .png
	WORKSPACE	X	Private data	Scenefile for v034 of Ryan
	STAGE		Transient data	Outgoing VERSION from scenefile
	SHARED	X	Public data	v034 of Ryan
	PRODUCER		Creator of data	You
	CONSUMER		User of data	Me

Data is organised into files and folders.

Some files and folders have special meaning in Starter.

The mental and physical model for files and folders look like this.

During the course of the creation of any ASSET, data moves between 2 of 3 states.

• Starter does not take into consideration the workspace and is therefore workspace-agnostic.
• The staging area is both implicit and transparent to the PRODUCER and CONSUMER, except for debugging purposes. This is where automatic/scheduled garbage collection may run to optimise for space constraints.
• The shared space is where ASSETS ultimately reside once published.

Workspace and Shared separation

A naive approach to content creation might be to refer to ASSETS straight from another artists workspace. Starter separates between data considered work-in-progress and data exposed to others.

Workspace data is highly mutable and typically private to an individual artist.

• Mutable implies transient data that is likely to change at any given moment.
• Private implies personal, highly irregular and likely invalid data.

Shared data on the other hand is immutable, correct and impersonal.

• Immutable implies that the data may be dependent upon by other data.
• Correct implies passing validation of the associated family.
• Impersonal implies following strict organisational conventions.

Each ASSET reside within the top-level root directory as follows.

Hierarchy	Example

Each ASSET contain 0 or more versions which in turn contain 0 or more representations.

Hierarchy	Example

Every extraction is made into the staging directory, regardless of whether it integrates successfully or not.

Hierarchy	Example

Each directory will contain everything that did extract successfully, along with its metadata, for manual inspection and debugging.

Communication with the filesystem is made through ls().

ls() returns available assets - relative the currently registered root directory - in the form of JSON-compatible dictionaries. Each dictionary is strictly formatted according to three distinct "schemas" (see below).

See below for a full list of members.

Example

import pyblish_starter

for asset in pyblish_starter.ls():
    assert "versions" in asset

    for version in asset["versions"]:
        assert "representations" in version

        for representation in version["representations"]:
            pass

Schemas

Available schemas are organised hierarchically, with the former containing the latter.

• asset.json
  • version.json
    • representation.json

A unit of data

An ASSET iteration

A data FORMAT.

pyblish-starter provides a stateful API. State is set and modified by calling any of the exposed registration functions, prefixed register_*, or automatically when calling pyblish_starter.install().

Public members of pyblish_starter

Public members of pyblish_starter.api

A host must implement the following members.

Some data within a host is special, and is identified via custom "tags".

Starter defines these families.

A generic representation of geometry.

Target Audience

• Texturing
• Rigging
• Final render

Requirements

• All DAG nodes must be parented to a single top-level transform
• Normals must be unlocked

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• geometry_SEL (geometry): Meshes suitable for rigging
• aux_SEL (any, optional): Auxilliary meshes for e.g. fast preview, collision geometry

The starter.rig contains the necessary implementation and interface for animators to animate.

Target Audience

• Animation

Requirements

• All DAG nodes must be parented to a single top-level transform
• Must contain an objectSet for controls and cachable geometry

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• cache_SEL (geometry): Meshes suitable for pointcaching from animation
• controls_SEL (transforms): All animatable controls
• resources_SEL (any, optional): Nodes that reference an external file

Point positions and normals represented as one Alembic file.

Target Audience

• Lighting
• FX
• Cloth
• Hair

Requirements

• None

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• None

Legend

	Title	Description
	Target Audience	Who is the end result of this family intended for?
	Requirements	What is expected of this ASSET before it passes the tests?
	Data	End-user configurable options
	Sets	Collection of specific items for publishing or use further down the pipeline.

With an understanding of this asset creation pipeline, here are some suggestions for what to explore next.

Difficulty	Task	Description
	Loader tooltip	Pick any of the available data from an asset, and add it to the tooltip of the item.
	Shots	Think of and document how the kind of workflow exemplified here - (1) creating, (2) publishing and (3) loading assets - can be applied to a shot- and task-based pipeline.
	A Creator Help Page	The creator interface is very limited. Try including a help-page and picture alongside each family illustrating what each are for.
	Loader thumbnails	The loader interface is even more limited, with room for expansion. Consider how you might go about extracting a thumbnail alongside each asset, and add it to the interface.

pyblish-starter, as Pyblish itself, is an open source effort and contributions are welcome.

For example, you could fork Starter, expand upon the graphical user interfaces and either make it your own or submit a pull-request to have it merge with the official project.

For more information on this, contact me and let's have a conversation!