Website Pages:

Welcome to Pipeline Design Patterns
Thoughts on Vocabulary
A Vocabulary for Diagrams
A Theory of Pipeline
Pipeline Design Patterns
Case Studies

Sections in this Page:

A Vocabulary for Diagrams
Basic Diagramming Rules
Scope and Structure
Applications, Plugins, Source Files
References, Overides, and Indirection
Aggregation and Specialization
Special Cases
More on Synchronization
A Complex Example

A Vocabulary for Diagrams

No discussion about Pipeline is complete without a bunch of boxes and lines and arrows drawn on a whiteboard. We've all been drawing these for years. We scribble and scribble and kind of “know what we mean.” (Of course, anyone else who comes in the room will be lost.)

A few years ago I started giving presentations on pipeline, and quickly realized I needed a consistent visual grammar for diagrams.

After a lot of thought I arrived at this set of diagram rules and symbols, which I believe to be:

Simple — The diagrams are easy to draw, both manually and programmatically. Everything is whiteboard friendly. The diagrams use mostly boxes and lines, with a just a few rules. No specific colors are needed, but colored markers always help.

Robust — The techniques cover the most common pipeline concepts, like Reference and Override. This in turn gives enough power to represent things like Conform, Bake, Publish, and Subscribe.

Evocative of the key concepts — The rules align with our innate notions of data flow and structure. See more on the Map Metaphor.

Consistent and connectible — The diagrams snap together the way we wish our pipelines would!

Basic Diagramming Rules

Data and Processes

Data — such as Files, Assets, etc. — are drawn as simple rectangles. Data are located on the diagram based on representation and revision status.

Processes — (including Applications and Plugins) are drawn as slanted rectangles. Processes are located as needed, near the data they interact with. (In other words, locate by usage, not by where the binary file lives.)

Note: on a whiteboard, slanted rectangles are way less sloppy than circles, and after a few false starts, very easy to draw.

Click on diagram to Zoom/Unzoom.


Pipelines do two big things for us in digital production. First, they manage the conversion of data formats, and second, they manage the synchronization of data sets across infrastructure levels. Examples of the latter include Perforce syncing and cloud rendering.

Horizontal Axis — represents data conversions. In whiteboard sessions we find it most natural to draw conversions left to right, just as in the above diagram.

Vertical Axis — represents data synchonization. In whiteboard sessions we find it most natural to draw global datastores and cloud services at the top of the board, and local stuff lower on the board.


Click on diagram to Zoom/Unzoom.

Scope and Structure


Any distinct infrastucture level like Cloud, Server, Asset Server, Perforce Server, Local Workspace, etc. gets drawn in its own horizontal band.

This diagram shows the skeleton of a typical pipeline: store at server level, sync to local, process, sync to cloud, process in cloud, sync to server.

Click on diagram to Zoom/Unzoom.


Often files or processes need grouping.

Click on diagram to Zoom/Unzoom.

Applications, Plugins, Source Files

Applications and Source Files

Click on diagram to Zoom/Unzoom.

Instead of inputs on left, outputs on right—with a corresponding change in representation—we draw the application as loading and saving the same file.

Note the representation doesn't change during load and save, hence the vertical relationship. This also mirrors the way caches are handled, below.

I like to draw applications a little bigger. This gives them a visual weight which illustrates where the artists spend their time.

Application Import and Export

Click on diagram to Zoom/Unzoom.


Applications (and Processes in general) can have Plugins.

Click on diagram to Zoom/Unzoom.


Often it's important to represent the contents of a file, i.e. does it contain geometry or animation etc. It's most whiteboard friendly to indicate content with little colored badges, because you can add and remove badges without a lot of erasing.

This Asset has components for Model Structure, Geo, Rig, Shader Binding, Shading, and Texture.

Click on diagram to Zoom/Unzoom.

The badges help us stummarize complex structure.

Click on diagram to Zoom/Unzoom.

References, Overides, and Indirection

File References

Files often refer to other files, via include, import, load or other equivalent syntax. This is indicated as follows:

Click on diagram to Zoom/Unzoom.

Using lines to indicate references generates way too much visual clutter:

  • references can come from anywhere — often from far earlier in the pipeline;

  • also, as files propagate they often carry the references with them.


File references are often resolved via an override mechanism. This is indicated with a diamond, as an homage to the classic flow chart decision symbol.

By convention, the reference in the lowest scope wins. The following diagram shows a reference which resolves to Upstream File 2.

Click on diagram to Zoom/Unzoom.


Some define indirection to be the same as reference. A more common understanding of the term is reference to reference. That's what I'll use throughout this web/book.

At first glance this is similar to override. But a deeper look shows this to be very different.

  • Override — evaluate a list of options, and return the first one which works.

  • Indirection — provide a pointer into a list of equivalent options.

Click on diagram to Zoom/Unzoom.

The above diagram shows two references: blue and green.

  • The green (diamond) reference implements override. The override tries the local version of CoffeeCup, then the server version.

  • The blue/green combination reference on the CoffeeCup node implements indirection. If the CoffeeCup override falls through to Server, it is pinned (via indirection) to Revision 3 there.

References and Content

The Content section (above) has a diagram showing an Asset with badges for internal content.

That asset has a lot of internal references which were not originally drawn, but if we did draw them:

Click on diagram to Zoom/Unzoom.

i.e. the Model Structure refers to Geo, Rig, and Shader Bindings; and the Shader Bindings refer to a Shader; and the Shader refers to a Texture.

This is a common structure, but this drawing is hard to draw and read. We need a simpler diagram:

Click on diagram to Zoom/Unzoom.

This is a very useful diagramming shortcut:

  • left-pointing triangles represent internal references within a structure;

  • large triangles with stems represent references which are more widely separated.

Both are equivalent. The downstream (referring) file contains a pointer to the upstream file (the file referred to), via include, import, load, or other construct.

Note: the Shader Binding is an Indirection Layer to the Shader. and the Shader is an Indirection Layer to the Texture. (That is certainly correct, i.e. the diagramming rules are consistent.)

Aggregation and Specialization

Aggregation is where one file is simply a list of references to other files. For example, a Set is a list of Models.

# DinerSet
table_1 = Table(name='Table_1', position=(12,3,0))
cup_1_1 = CoffeeCup(name='CoffeeCup_1_1',
                    parent=table_1, position=(0,0,0))
cup_1_2 = CoffeeCup(name='CoffeeCup_1_2',
                    parent=table_1, position=(2,10,0),
cup_1_3 = CoffeeCup(name='CoffeeCup_1_3',
                    parent=table_1, position=(8,7,0),
# ... etc.

Special Cases

There are a few situations that need special handling.

File Drawn in Multiple Places

Imagine we are precalculating the animation poses for a shot. For an explanation, see the Bake Pattern. Prior to calculation, our shot looks like:

Click on diagram to Zoom/Unzoom.

And after caclulation:

Click on diagram to Zoom/Unzoom.

We need to draw the Shot structure as both input to and output from a procedure. The solution is a dashed line, no arrowheads:

The following three diagrams are equivalent:

Click on diagram to Zoom/Unzoom.
Click on diagram to Zoom/Unzoom.
Click on diagram to Zoom/Unzoom.

More on Synchronization


Synchronization typically runs both ways, which can lead to lots of arrows.

Click on diagram to Zoom/Unzoom.

To be specific, the synchronization/revision data flow is:

First vertical arrow (reading left to right) — Server data is checked out to the local workspace, which changes the local revision level to match the server.

Second vertical arrow — The data is loaded into the application, which syncs the internal state of the app to match the revision level.

Third vertical arrow — After being modified in the application (which bumps the revision level internally), the data is saved, which bumps the local revision level.

Note this new revision level has not been explicitly named or numbered. This will happen on checkin. For now, the source file is newer than what is on the server.

Fourth and last vertical arrow — Finally, the data is checked in or committed, which bumps the revision level on the server.

I prefer to elide some of these arrows:

Click on diagram to Zoom/Unzoom.

Note: In each case, the arrows terminate on the Source Control node, rather than the file itself. This represents the idea that check-in/out and sync operations engage with the source control system, not with the internal storage of the file itself.

Synchronization Processes

Typically there are are processes which manage the vertical syncing between scopes, (such as Perforce, Git, FTP, etc.) but I prefer to elide them for readability.

If we include them, the graph can get wide very quickly:

Click on diagram to Zoom/Unzoom.

A Complex Example

The following pipeline illustrates most of these concepts.

Click on diagram to Zoom/Unzoom.

Reading from left to right:

Geometry — The source file is stored at server level, synced down to an artist workspace, edited, and converted via plugin.

Handoff — The converted file is synced up to server so it can be synced down to a different artist, who works on...

UVs and Textures — The raw geo is imported after handoff. In addition, the UV and Texture source files are stored at server level and synced down to this workspace. After editing, the data is exported via plugin to a local asset structure. Once approved there, it is checked in

Asset — is rebuilt/installed at server level, where it is available for referencing via an indirection scheme.

Shot Creation — Again, the master shot manifest is stored at server level, synced down to a local workspace. Asset references are via override, so if the artist has his/her own local version, he/she will pick that up.