Website Pages:

Welcome to Pipeline Design Patterns
Vocabulary
A Theory of Pipeline
General Thoughts on Pipeline
The Map Metaphor
Patterns as Shapes
Pipeline Design Patterns
Case Studies

Sections in this Page:

General Thoughts on Pipeline
What is a Pipeline?
The Problem is Not Going Away
The Pipeline Analogy
Uses of the Term
Pipeline Design Patterns
A Theory of Pipeline
Why the Coffee Cup?

Definitions:

A Vocabulary for Diagrams
Glossary

General Thoughts on Pipeline

The concept of Pipeline is increasingly important in 3D graphics, especially in digital film and games production. It's a big topic, encompassing almost everything we do in digital production.

This increasing importance represents a shift in how we think about graphics, from small-scale to large scale. In the early days of computer graphics—thirty whole years ago!—the main problems were algorithmic and foundational: how to represent geometry, surface qualities, lights, and cameras. Fifteen years ago the main problems were how to embed these algorithms into powerful user interfaces, so that artists could take control of the process. Major advances in this area were made by applications such as Maya or Nuke.

But today the big problems are about scale, complexity, collaboration, and workflow. At SIGGRAPH we crowd the main festival room for presentations on the latest VFX blockbusters. We do not ask

  • How did you make that tree look real?

  • How did you make that building fracture into rubble?


Rather, we ask

  • How did you render the entire forest?

  • How did you destroy the entire city?

  • How did you do that in so many shots?

  • How many artists were involved?

  • How did they collaborate and avoid stepping all over each other?


Recent films and games have dramatically redefined our ideas of what's possible. Often these projects have hundreds of characters, dozens of sets, and thousands of individual models. The characters have numerous costume changes. The set models have many variations, and may break or shatter.

How do we create and manage all this? With a pipeline.

What is a Pipeline?

The fundamental problem is that we cannot make—not directly—the thing we need to deliver.

What we need to deliver is a film or a game, i.e. a large binary file or files that some system can play back on a projector or laptop or console or mobile device. We cannot write these files directly; they are too large, complex, and unforgiving. They can only be read by a computer program; they can only be written by a computer program.

The files we can write are at a very different level of abstraction. Our technical artists want to work as filmmakers, and be concerned with the staging of a scene, the timing of a smile, or the placement of a light or camera. So we create these abstractions out of the raw data of CG, manage them through the filmmaking process, and ultimately use them to create the dense deliverable files.


There are four fairly distinct levels of abstraction:

Basic CG objects like polygons, surface normals, texture maps, and so on. These are managed by the low level CG libraries.

Assets — filmmaking objects like characters, sets/props, cameras, and lights. These are built, by artists, from the basic CG objects, using a variety of applications.

Scenes, shots, sequences, and levels. These are built, by artists, from the assets. And they can become assets as well.

The deliverable film or game, built from the sequences and levels.


Each level of abstraction is built from the lower level. Typically this involves multiple stages with some ordering—hence the metaphor of pipeline—and for each stage, the authoring format is typically different from the delivery format.

At each stage, and overall, there is a lot of data, both in quantity and variety; the data is complicated; and it all needs to evolve, both technically (transformations between data representations) and artistically (make it bigger, redder, more saturated etc.). We will see that these aspects—evolution, representation, and variation—form a sort of basis for understanding the problem.

Also at each stage, and overall, authoring is a human enterprise. The data needs to be created by artists with skills specific to that stage. Often several artists must collaborate or synchronize their edits in some way. All the data needs to be reviewed and approved by other artists.

Finally, all the data needs to be managed, versioned, backed up, cached, etc.

This entire process, with all its complications, we call the Pipeline.

The Problem is Not Going Away

Some people think the Pipeline problem will go away because of (take your pick): better standards, better applications, more powerful computers, or even “the cloud.”

Some other people think there is one pipeline, or a best pipeline.

Both notions are false. The difference in authoring and delivery formats is not just a hard problem we haven't yet solved, it is intrinsic to what we are doing. We are managing and converting levels of abstraction. As long as artists continue to demand higher levels of abstraction, and hardware gurus continue to invent new displays with corresponding dense delivery formats, we will need new pipelines to manage the translation.

We cannot get rid of the pipeline problem.

The Pipeline Analogy

We use the term pipeline because it evokes a rich analogy with a physical pipeline—for example, an oil pipeline. It is worth exploring this analogy.

One-way Flow. In a pipeline, oil flows in one direction. Similarly, filmmaking data typically moves in one direction—from the modeling tool to the rendering tool but not back. Of course, sophisticated pipelines can have reverse data flows. For example, the rendering stage might send ambient occlusion or curvature back to the modeling stage to be baked into the geometry.

  • Note that this reverse flow in representation is accompanied by a forward flow in revision status. Time marches forward.


Stages. The oil well is not connected directly to the tank under the local gas station. Rather, the system has stages for pumping, refining, shipment, storage, and delivery. Similarly, filmmaking pipelines are arranged in stages: modeling, shading, lighting, rendering, and so forth.

Repetition. Once the pipeline is set up and the first barrel flows through it, all subsequent barrels follow on the same path. Similarly, once the filmmaking pipeline is set up for the first model or shot, subsequent models/shots go through the same set of operations.

Failure modes. Each stage is a potential point of failure, each in its own way: the well can have a blowout, the shipment can spill, the refinery can explode. Similarly, filmmaking pipeline stages have specific failure modes: the models can leak, the shaders can fail to compile, the lighting shadowmaps can be out of date, and the render commands can be pointing at wrong asset versions.

Throughput and Bottlenecks. The oil pipeline has a certain throughput capacity, related to bottlenecks at various stages, i.e. when refineries go down for maintenance, tankers line up at the ports. Similarly, filmmaking pipelines get blocked: lighting is waiting because the assets are hung up in shading.

Uses of the Term

We use the term Pipeline in all the following ways:

What happens to the data? When and how is data saved, translated, optimized, cached, baked, etc.?

Example: “Our pipeline saves the animation into a per-frame geometry cache, which is read by a rendering procedural.”

Process. The organizational stages of building, assembling, and approving any given part of the project, whether a prop, a set, a shot, a sequence, or the entire project. This usage is tied into the notion of inventory flow

Example: “In our pipeline, the modeling department feeds shading and sets, the sets department feeds layout, the layout department feeds animation, and the animation department feeds lighting.”

Toolchain. The specific tools and processes used in the production. Another common term for this is software stack.

Example: “Our pipeline/toolchain is based on Maya and RenderMan.”

Connections. The scripts, plugins, and policies that connect tools and applications. Other terms for this are glue and plumbing.

Example: “Last year I worked on lighting software, but this year I am working on pipeline/plumbing/glue.”

Big Picture. The production process as a whole, opposed to any specific detail within it.

Example: “You guys are worried about this one tool; I am worried about the pipeline.”

Pipeline Design Patterns

There is no single pipeline for all film and games.

In fact, this is true even with a given studio or for a given project. It is very common for a studio to create a new pipeline (or several new pipelines) for each project.

But most pipelines share some common ideas, or design patterns. In this blog I'll try to identify and study those patterns.

The design patterns approach is roughly as follows:

  • be descriptive rather than prescriptive,

  • identify things that we all seem to know and do,

  • give them precise names, and

  • search for common themes and connections.

Inspiration

The inspiration for this approach is a pair of books. Both are classics and hugely influential in their respective fields.

  • The first is the architecture book A Pattern Language: Towns, Buildings, Construction by Alexander, Ishikawa, and Silverstein, published in 1977.

  • The second is the software book Design Patterns: Elements of Reusable Object-Oriented Software by Gamma, Helm, Johnson, and Vlissides, published in 1994.


I assume that most readers are software savvy and have heard of the software book. But the architecture book is perhaps more influential, as it was the first to explore the idea of common patterns. In fact, the software book credits the architecture book as its chief inspiration.

These books have re-defined the vocabulary and key ideas for a generation of architects and software developers. The chief contribution of each book was to point out familiar things and give them names. For example, A Pattern Language analyzes patterns such as Shopping Street, Alcoves, or Closets Between Rooms. Similarly, Design Patterns analyzes patterns such as Abstract Factory, Visitor, and Iterator. Perhaps more important than the analysis is simply the idea that such analysis is possible and useful.

As Applied to Pipeline

There are a large number of common patterns in digital filmmaking pipelines that need this same analysis.

In this website/book I will try to

  • identify some of the most important ones,

  • describe them in general terms, propose some vocabulary,

  • analyze them in terms of other patterns,

  • discuss best practices, and

  • provide case studies to make the analysis concrete.

A Theory of Pipeline

I'm proposing a Theory of Pipeline which rests on two key observations.

First, What a Pipeline Does

The pipeline delivers artistic intent into a deliverable digital object. The artistic intent is expressed at a high level of abstraction, and the deliverable is expressed at a very low level of abstraction.

So the pipeline manages the abstraction level, in particular the translation of abstraction levels.

Second, How It Does It

A pipeline can be thought of as large and complex version management system. The interesting behaviors come about because the versions of things come in three distinct types:

Revision. Describes the evolution of things. First checkin, second checkin, etc.

Representation. Describes the different encodings of things, for a given artistic intent. For example: modeling source code vs optimized mesh, or shading node graph vs compiled shader.

Variation. Describes the artistic intent of objects: red vs blue, tall vs short, etc.

These essentially form a basis for understanding pipeline.

Why the Coffee Cup?

Forget the Teapot.

In this web/book you'll see a lot of references to my favorite asset, the CoffeeCup. I like to use it as the canonical example for all of computer graphics.

For example, if you propose a new rendering algorithm I will most likely say: “OK, sounds promising, but how that will make a picture of this CoffeeCup?”

I do this for for two reasons:

One, I like concrete examples and the cup serves very well. It has

  • interesting geometry (genus one surface),

  • interesting shading (subsurface scattering for porcelain, texture map logos, procedural coffee stains),

  • animation properties (OK, this is a stretch, but maybe the rim can deform or the handle can bend),

  • FX capabilities (as a collision volume for the coffee fluid sim, or as a shatterable object),

  • setdressing implications (lots of cups on the shelves, in lots of variations), and so forth.

And two, I like my concrete examples to be nearby, within reaching distance. And the object most often nearby (to me) is a CoffeeCup.

Let's get some coffee and get started.

Bill Polson