What is a Pipeline?
The fundamental problem is that we cannot make—not directly—the thing we need to deliver.
What we need to deliver is a film or a game, i.e. a large binary file or files that some system can play back on a projector or laptop or console or mobile device. We cannot write these files directly; they are too large, complex, and unforgiving. They can only be read by a computer program; they can only be written by a computer program.
The files we can write are at a very different level of abstraction. Our technical artists want to work as filmmakers, and be concerned with the staging of a scene, the timing of a smile, or the placement of a light or camera. So we create these abstractions out of the raw data of CG, manage them through the filmmaking process, and ultimately use them to create the dense deliverable files.
There are four fairly distinct levels of abstraction:
Basic CG objects like polygons, surface normals, texture maps, and so on. These are managed by the low level CG libraries.
Assets — filmmaking objects like characters, sets/props, cameras, and lights. These are built, by artists, from the basic CG objects, using a variety of applications.
Scenes, shots, sequences, and levels. These are built, by artists, from the assets. And they can become assets as well.
The deliverable film or game, built from the sequences and levels.
Each level of abstraction is built from the lower level. Typically this involves multiple stages with some ordering—hence the metaphor of pipeline—and for each stage, the authoring format is typically different from the delivery format.
At each stage, and overall, there is a lot of data, both in quantity and variety; the data is complicated; and it all needs to evolve, both technically (transformations between data representations) and artistically (make it bigger, redder, more saturated etc.). We will see that these aspects—evolution, representation, and variation—form a sort of basis for understanding the problem.
Also at each stage, and overall, authoring is a human enterprise. The data needs to be created by artists with skills specific to that stage. Often several artists must collaborate or synchronize their edits in some way. All the data needs to be reviewed and approved by other artists.
Finally, all the data needs to be managed, versioned, backed up, cached, etc.
This entire process, with all its complications, we call the Pipeline.