Pyblish Starter

A basic asset creation pipeline - batteries included.

Keywords: Film, games, content creation, pipeline
Objective:
1. Provide an example of how one might go about building a pipeline around Pyblish.
2. Demonstrate where publishing fits within a typical production pipeline
3. Inspire further expansion upon basic ideas
Mission: Demonstrate what is made possible with publishing.
Motivation: I'm doing this for the same reason I created Pyblish. Because I see publishing as the single most important aspect of any production pipeline. It is on top of the advantages that it provides that the surrounding pipeline is made possible - e.g. browser and loader, builder and manager.
Requirements: Reliably output correct data with minimal impact on artist productivity.
Technology: Starter is built upon Pyblish, Python and bindings for Qt, and depends upon a Windows, Linux or MacOS operating system with Autodesk Maya.
Audience: Technical directors interested in pipeline working in small- to mid-sized companies with a thirst for better ways of working.
Discaimer: In the interest of simplicity, Starter is a very limited pipeline. If you have experience with an existing pipeline, or have thought about making your own, odds are this will look nothing like it. Take what you can from this project, and feel free to contribute your own ideas to make it simpler, or fork and expand upon it.

Prerequisities

Before you start looking into pyblish-starter, it is recommended that you first familiarise yourself with Pyblish.

learn.pyblish.com

Target Audience

To make the most out of pyblish-starter, some knowledge and experience is assumed.

	minimal	recommended
`personality`	curious	excited
`title`	technical director	pipeline technical director
`experience`	1 year in advertisements or games	5+ years in feature film
`software`	windows, linux or macos	maya

Pyblish Starter

Welcome to pyblish-starter, a basic asset creation pipeline - batteries included.

Table of contents

Install
Usage
Description
Batteries
- Creator
- Loader
Looking Ahead
API
Contract
Homework
Contributing
Help

Install

Starter takes the form of a Python package with embedded plug-ins.

$ pip install pyblish-starter

Usage

Starter is initialised by calling install() with an interface for your host.

>>> from pyblish_starter import install, maya
>>> install(maya)

From here, you model, rig and animate as per the contract below.

Description

Build your own ASSET CREATION PIPELINE, starting with the basics.

Overview

Asset creation covers aspects related to producing content used in the production of film. A film is typically partitioned into sequences and shots, where each shot consists of one or more assets.

This project includes plug-ins and tools for 3 common types of ASSETS in a typical production pipeline.

Modeling
Rigging
Animation

These illustrate how to (1) devise a contract - known as a family - for each kind of ASSET and (2) design their interface towards each other.

Batteries included

In addition to contracts, the project illustrates boundaries between publishing and setup. The definition of "setup" being anything that isn't strictly output, such as creating geometry or assigning attributes.

It includes a series of graphical user interfaces to aid the user in conforming to these contracts. These interfaces represent placeholders for your pipeline to fill in; you are invited to either continue from where they finish, or build your own from scratch.

Batteries

In addition to providing a co-operative set of plug-ins, Starter also implements a minimal toochain for ASSET creation in a typical film pipeline.

Creator

Associate content with a family.

The family is what determins how the content is handled throughout your pipeline and tells Pyblish what it should look like when valid.

API

The creator respects families registered with Starter.

from pyblish_starter import api

api.register_family(
    name="my.family",
    help="My custom family"
)

For each family, a common set of data is automatically associated with the resulting instance.

{
    "id": "pyblish.starter.instance",
    "family": {chosen family}
    "name": {chosen name}
}

Additional common data can be added.

from pyblish_starter import api

api.register_data(
    key="myKey",
    value="My value",
    help="A special key"

Finally, data may be associated with a family.

from pyblish_starter import api

api.register_family(
    name="my.family",
    data=[
        {"key": "name", "value": "marcus", "help": "Your name"},
        {"key": "age", "value": 30, "help": "Your age"},
])

Loader

Visualise results from api.ls().

from pyblish_starter import api

for asset in api.ls():
    print(ASSET["name"])

API

The results from api.ls() depends on the currently registered root.

from pyblish_starter import api

api.register_root("/projects/gravity")

The chosen ASSET is passed to the load() function of the currently registered host.

from pyblish_starter import api, maya
api.register_host(maya)

A host is automatically registered on pyblish_starter.install().

Looking Ahead

Without publishing, in any shape or form, the following essential tools are but a dream.

Name	Purpose	Description
browser	remove file-system dependence	Search and clear presentation of available data relative a given project or task.
loader	control what goes in	Keep tabs on where data comes from so as to enable tracking and builds.
builder	associate disparate ASSETS	Automatic composition of data that changes independently but need to somehow stay associated.
manager	stay up to date	Notification and visualisation of data in time.

Ideas

Here are a few things you might want to implement now that you have a solid framework upon which to build.

Add a thumbnail to ASSETS by automatically extracting one per instance
Enable search in Loader, and augment it by extracting additional metadata about an ASSET, such as author, use, size, origin or datatype.

API

Starter exposes a series of interrelated APIs to the end-user.

Name	Purpose
Filesystem API	Defines how the developer interact with data on disk
Starter API	Defines how the developer interacts with Starter
Host API	Defines how the host interacts with Starter

Terminology

Starter reserves the following words for private and public use. Public members are exposed to the user, private ones are internal to the implementation.

Term	Public	Description	Example
`PROJECT`	`X`	Root of information	Gravity, Dr. Strange
`ASSET`	`X`	Unit of data	Ryan, Bicycle, Flower pot
`VERSION`	`X`	An ASSET iteration	v1, v034
`REPRESENTATION`		A data format	Maya file, pointcache, thumbnail
`FORMAT`		A file extension	`.ma`, `.abc`, `.ico`, `.png`
`WORKSPACE`	`X`	Private data	Scenefile for v034 of Ryan
`STAGE`		Transient data	Outgoing VERSION from scenefile
`SHARED`	`X`	Public data	v034 of Ryan
`PRODUCER`		Creator of data	You
`CONSUMER`		User of data	Me

Filesystem API

Data is organised into files and folders.

Some files and folders have special meaning in Starter.

Information Hierarchy

The mental and physical model for files and folders look like this.

Workspace, Stage and Share

During the course of the creation of any ASSET, data moves between 2 of 3 states.

Starter does not take into consideration the workspace and is therefore workspace-agnostic.
The staging area is both implicit and transparent to the PRODUCER and CONSUMER, except for debugging purposes. This is where automatic/scheduled garbage collection may run to optimise for space constraints.
The shared space is where ASSETS ultimately reside once published.

Workspace and Shared separation

A naive approach to content creation might be to refer to ASSETS straight from another artists workspace. Starter separates between data considered work-in-progress and data exposed to others.

Workspace data is highly mutable and typically private to an individual artist.

Mutable implies transient data that is likely to change at any given moment.
Private implies personal, highly irregular and likely invalid data.

Shared data on the other hand is immutable, correct and impersonal.

Immutable implies that the data may be dependent upon by other data.
Correct implies passing validation of the associated family.
Impersonal implies following strict organisational conventions.

Each ASSET reside within the top-level root directory as follows.

Hierarchy	Example

Each ASSET contain 0 or more versions which in turn contain 0 or more representations.

Hierarchy	Example

Every extraction is made into the staging directory, regardless of whether it integrates successfully or not.

Hierarchy	Example

Each directory will contain everything that did extract successfully, along with its metadata, for manual inspection and debugging.

`ls()`

Communication with the filesystem is made through ls().

ls() returns available assets - relative the currently registered root directory - in the form of JSON-compatible dictionaries. Each dictionary is strictly formatted according to three distinct "schemas" (see below).

See below for a full list of members.

Example

import pyblish_starter

for asset in pyblish_starter.ls():
    assert "versions" in asset

    for version in asset["versions"]:
        assert "representations" in version

        for representation in version["representations"]:
            pass

Schemas

Available schemas are organised hierarchically, with the former containing the latter.

asset.json
- version.json
  - representation.json

`asset.json`

A unit of data

Key	Value	Description
`name`	`str`	Name of directory
`versions`	`list`	0 or more `version.json`

`version.json`

An ASSET iteration

Key	Value	Description
`version`	`int`	Number of this VERSION
`path`	`str`	Unformatted path
`time`	`str`	ISO formatted, file-system compatible time.
`author`	`str`	User logged on to the machine at time of publish.
`source`	`str`	Original file from which this VERSION was made.
`representations`	`list`	0 or more `representation.json`

`representation.json`

A data FORMAT.

Key	Value	Description
`format`	`str`	File extension
`path`	`str`	Unformatted path

Starter API

pyblish-starter provides a stateful API. State is set and modified by calling any of the exposed registration functions, prefixed register_*, or automatically when calling pyblish_starter.install().

Public members of pyblish_starter

Member	Returns	Description
`install(host)`	`str`	Install Starter into the current interpreter session
`uninstall()`	`str`	Revert installation

Public members of pyblish_starter.api

Member	Returns	Description
`ls()`	`dict`	List available assets, relative `root`
`root()`	`str`	Absolute path to current working directory
`format_staging_dir(root, name)`	`str`	Return absolute path or staging directory relative arguments
`format_shared_dir(root)`	`str`	Return absolute path of shared directory
`format_version(version)`	`str`	Return file-system compatible string of `version`
`find_latest_version(versions)`	`int`	Given a series of string-formatted versions, return the latest one
`parse_version(version)`	`str`	Given an arbitrarily formatted string, return version number
`register_root(root)`		Register currently active root
`register_host(host)`		Register currently active host
`register_plugins()`		Register plug-ins bundled with Pyblish Starter
`deregister_plugins()`
`registered_host()`	`module`	Return currently registered host

Host API

A host must implement the following members.

Member	Returns	Description
`create(name, family)`	`dict`	Build fixture for outgoing data (see instance)
`load(asset, version=-1)`	`str`	Import external data into container

Some data within a host is special, and is identified via custom "tags".

Name	Description	Example
`pyblish.starter.container`	Unit of incoming data	`...:model_GRP`, `...:rig_GRP`
`pyblish.starter.instance`	Unit of outgoing data	`Strange_model_default`

Contract

Starter defines these families.

Family	Definition	Link
`starter.model`	Geometry with deformable topology	Spec
`starter.rig`	An articulated `starter.model` for animators	Spec
`starter.animation`	Pointcached `starter.rig` for rendering	Spec

`starter.model`

A generic representation of geometry.

Target Audience

Texturing
Rigging
Final render

Requirements

All DAG nodes must be parented to a single top-level transform
Normals must be unlocked

Data

name (str, optional): Pretty printed name in graphical user interfaces

Sets

geometry_SEL (geometry): Meshes suitable for rigging
aux_SEL (any, optional): Auxilliary meshes for e.g. fast preview, collision geometry

`starter.rig`

The starter.rig contains the necessary implementation and interface for animators to animate.

Target Audience

Animation

Requirements

All DAG nodes must be parented to a single top-level transform
Must contain an objectSet for controls and cachable geometry

Data

name (str, optional): Pretty printed name in graphical user interfaces

Sets

cache_SEL (geometry): Meshes suitable for pointcaching from animation
controls_SEL (transforms): All animatable controls
resources_SEL (any, optional): Nodes that reference an external file

`starter.animation`

Point positions and normals represented as one Alembic file.

Target Audience

Lighting
FX
Cloth
Hair

Requirements

None

Data

name (str, optional): Pretty printed name in graphical user interfaces

Sets

None

Legend

	Title	Description
	Target Audience	Who is the end result of this family intended for?
	Requirements	What is expected of this ASSET before it passes the tests?
	Data	End-user configurable options
	Sets	Collection of specific items for publishing or use further down the pipeline.

Homework

With an understanding of this asset creation pipeline, here are some suggestions for what to explore next.

Difficulty	Task	Description
	Loader tooltip	Pick any of the available data from an asset, and add it to the tooltip of the item.
	Shots	Think of and document how the kind of workflow exemplified here - (1) creating, (2) publishing and (3) loading assets - can be applied to a shot- and task-based pipeline.
	A Creator Help Page	The creator interface is very limited. Try including a help-page and picture alongside each family illustrating what each are for.
	Loader thumbnails	The loader interface is even more limited, with room for expansion. Consider how you might go about extracting a thumbnail alongside each asset, and add it to the interface.

Contributing

pyblish-starter, as Pyblish itself, is an open source effort and contributions are welcome.

For example, you could fork Starter, expand upon the graphical user interfaces and either make it your own or submit a pull-request to have it merge with the official project.

For more information on this, contact me and let's have a conversation!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Pyblish Starter

Prerequisities

Target Audience

Pyblish Starter

Install

Usage

Description

Batteries

Creator

Loader

Looking Ahead

API

Terminology

Filesystem API

Information Hierarchy

Workspace, Stage and Share

`ls()`

`asset.json`

`version.json`

`representation.json`

Starter API

Host API

Contract

`starter.model`

`starter.rig`

`starter.animation`

Homework

Contributing

Files

README.md

Latest commit

History

README.md

File metadata and controls

Pyblish Starter

Prerequisities

Target Audience

Pyblish Starter

Install

Usage

Description

Batteries

Creator

Loader

Looking Ahead

API

Terminology

Filesystem API

Information Hierarchy

Workspace, Stage and Share

ls()

asset.json

version.json

representation.json

Starter API

Host API

Contract

starter.model

starter.rig

starter.animation

Homework

Contributing

`ls()`

`asset.json`

`version.json`

`representation.json`

`starter.model`

`starter.rig`

`starter.animation`