PipeGadget

Here is the main idea. Imagine your studio has a simple pipeline for the Hair stage:

This is a standard U-shape with a single edit step and a plugin export.

Even with this simple pipeline, there is a ton of metadata:

• filenames,

• export parameters,

• open source library dependencies on data or plugins,

• Perforce initialization and syncing data and defaults,

• bin directories for executables and plugins,

• source revision numbers, labels, git branches or pins

• same for asset,

• etc.

It sure would be nice to have a node graph of the pipeline that could serve as a resource for all this.

Enter PipeGadget. You create a node graph that describes your pipeline (using the patterns in this website/book) and store it on the PipeGadget Server on Amazon:

The Node Graph is a Map

You now have a map of your pipeline, available anywhere in the world,

• to artists via their browser, or

• to any running processes via http.

Pipeline as an Asset

PipeGadget tracks the node graphs via a git-based API, and provides a publishing / subscription mechanism so you can

• target different node graphs to different consumers,

• and thus roll out (and more importantly, roll back) new pipelines.

Building the Map

The pipeline node graphs are constructed (in python, programmatically) from nodes, patterns of nodes, or other pipelines.

The Hair Data Export Plugin needs a lot of metadata:

• number of keyhairs per subdivision patch

• number of CVs along each keyhair

• keyhair density

• references to hair deformation (curl, scraggle, etc.) shaders

In a naive pipeline, you'd hard code that in the plugin. In a more robust pipeline, you'd store it in an xml or json file. (But you'd have no external visibility in where that file is. You'd have to read the plugin source to see where it gets its settings.)

In PipeGadget you'd store the metadata in a dictionary on the node corresponding to the plugin, or better yet, on the node corresponding to the data file. By storing it on the file, then anything that writes or consumes the file can traverse the node graph to find the correct metadata contract.

Thus the Exporter Plugin would

• find itself in the graph — the red line,

• traverse to its output node,

• gather the metadata — the orange line,

• and apply the metadata during export.

Similarly, any downstream consumer of that file would also traverse to the correct node to determine what metadata to expect.

This has the highly desirable effect of:

• establishing a contract that the hair export file will honor

• storing that contract in one place — in the right place: the node representing the export file

• informing all upstream and downstream nodes what the contract is.

The Map Metaphor

Pipeline systems are fairly common these days. Most if not all attempt to provide a pipeline execution model; in other words, they are the pipeline.

In contrast to that, PipeGadget provides a visualization of the pipeline.

This is our overall metaphor:

• Your pipeline is like roads and bridges; your data is like cars and trucks.

• PipeGadget is the map. PipeGadget wants to know the roads and bridges, and how cars and trucks behave, but not the specific instances of cars and trucks.

• Your production management system is like the central office at a trucking company. They know that a truckload of widgets is due at the factory next Tuesday, and the truck is being driven by Fred.

  They do not know the turn-by-turn directions, namely: exit number 42B, left onto the service road, second right into the driveway. For that, they need a mapping system.

• Your asset management system is like the trucking company garage. It provides safe parking/storage for trucks, and supplies trucks as needed.

Thus PipeGadget diagrams have nodes for applications and programs. (These are the roads in our metaphor). In addition, PipeGadget diagrams have nodes for files and data, but only in template form. (These represent how cars behave but not the specific instances of cars).

See more on how the map metaphor influences the pipeline diagrams.

Maps are Useful

Maps are super useful for many things, including

• Survey — Where are the roads now? What do they connect to?

• Urban Planning — Where will we put the new airport? How does that impact the current roads?

• Route Mapping — How do I get from SF to Tahoe? I need a map. In pipeline terms, how do I get this asset from modeling through to rendering? I need a map.

• Traffic Routing — Turn this light red, that one green. Open or close these lanes. In pipeline terms, this means changing the filenames or metadata contracts.

• Traffic Monitoring — Where are the accidents, where is traffic flowing freely? In pipeline terms, this is Analytics. The node graph is the right shape to accumulate data — there is a node for each thing you care about.

• Mashups — Are there any coffee shops nearby? What are the hours? menu? wifi policy? In pipeline terms, this is Metadata. Again, the node graph is the right shape to store all the metadata contracts. (More on that below).