Building a GLSL Shader Compiler

This blog post is by Aeva Palecek. She leads development on the open source graphics engine M.GRL, and has an admirable amount of WebGL as well as embedded systems knowledge. Please follow her on Twitter at @ladyaeva and check out her work at https://github.com/Aeva/m.grl.

About a month ago, I wrote a GLSL to GLSL compiler for a project called M.GRL. The Midnight Graphics and Recreation Library - or M.GRL - is an open source[1] WebGL game engine that I have been hacking together over the last two years. M.GRL aims to provide a clean developer experience for authoring 3D games, while also being flexible enough to be desirable for more experienced graphics programmers to use.

The GLSL to GLSL compiler was created with a focus on adding features into GLSL itself to make programs written in the language extensible. Future versions of the compiler might also include strict syntax validation and standardized error output, linting, compile-time optimization, and support for more shader languages beyond GLSL.

Pluggable Functions

Consider the following non-standard GLSL fragment shader:

plugin vec4 red() {
return vec4(1.0, 0.0, 0.0, 1.0);
}
plugin vec4 green() {
return vec4(0.0, 1.0, 0.0, 1.0);
}
plugin vec4 blue() {
return vec4(0.0, 0.0, 1.0, 1.0);
}
swappable vec4 color() {
return vec4(0.5, 0.5, 0.5, 1.0);
}

void main() {
gl_FragColor = color();
}

The above example creates an implicit uniform variable called “color”, which we can now use to control which function generates the color of an object being rendered.

In the game’s code, we can use the “color” uniform variable to control the behavior of the “color” function in the shader. To use the default behavior of the function, one would do the following:

some_character.shader.color = null;

… and as a result, the object will appear gray.

To change the behavior of the “color” function so that it calls a different function, we assign the name of the desired override function to the “color” uniform variable:

some_character.shader.color = "red";

… and now the “red” function will be called, and the object will be appear red.

Different objects can have different values for uniform variables associated to them, which in the following example would result in a different function being called at render time for each character:

alice.shader.color = "blue";
erica.shader.color = "green";

How it Works

The compiler works by transforming a source file or collection of source files into an abstract syntax tree (AST), and then transforming that AST back into source code. A parser generates a token stream, and several transformation passes remove sections of the token stream and replace them with AST objects. The result is a list containing AST objects representing global scope variables (attributes, uniforms, varyings, and constants) and functions.

Producing the AST and printing out the new source code is broken out into different functions, and while M.GRL automates the entire process when it loads a new shader program, it is possible to call these functions manually, should one desire to tinker with or introspect a shader program’s AST.

The swappable functions feature introduces the “swappable” and “plugin” keywords to a function’s definition. When either keyword is detected, it is noted on the AST objects for the respective function definition. When printing the shader AST the following happens to make swappable functions work:

  1. All “plugin” functions with a matching type signature are enumerated.
  2. A uniform variable is added to control the behavior of the swappable function. The enumeration is noted in the metadata for this variable so that later a nice interface may be provided to the user.
  3. The body of the swappable function is re-written to be enclosed within a switch-case statement[2], which allows the added uniform variable to control what behavior should actually execute when the function is called.

To continue with the example shader in the first part of this post, the compiler produces the following output:

uniform int _mgrl_switch_color;
vec4 red() {
return vec4(1.0, 0.0, 0.0, 1.0);
}
vec4 green() {
return vec4(0.0, 1.0, 0.0, 1.0);
}
vec4 blue() {
return vec4(0.0, 0.0, 1.0, 1.0);
}
vec4 color() {
if (_mgrl_switch_color==1) {
return red();
}
else if (_mgrl_switch_color==2) {
return green();
}
else if (_mgrl_switch_color==3) {
return blue();
}
else {
return vec4(0.5, 0.5, 0.5, 1.0);
}
}
void main() {
gl_FragColor=color();
}

Rationale

M.GRL provides a collection of bundled shaders so that users may use them to create beautiful games with no knowledge of GLSL. The purpose of pluggable functions allows for these bundled shaders to optionally be extended by a developer’s own custom code, or by using the standard set of plugin functions provided.

An example of where this is useful is M.GRL’s dynamic lighting model, whereby a robust deferred rendering scheme is provided to the user. The use of this lighting model is optional, though pluggable functions allow it to be easily customized by advanced users.

A more experienced user of the game engine might want to add procedural textures to objects in their game, and they can do so by circumstantially overriding the “diffuse_color” function in a shader by way of pluggable functions. Another user might want to use generated texture coordinates for some objects instead of providing a UV map, and this system makes it trivial to do.

Future Plans

Having tight integration between a game engine and the shader compiler also stands to provide a lot more flexibility as to how the engine itself conceptualizes shader programs. In the near term, this will mostly result in simplifying M.GRL’s inner workings, and applying some obvious optimizations to eliminate current pain points.

It is also a goal for the compiler to compensate for inconsistencies between GL environments. For example, it should be possible for a shader to be written with multiple render targets in mind, and when only one render target is available, the compiler may work with the compositing graph to run such a shader in multiple passes automatically.

But why stop there? Here is a short list of features I hope to add into GLSL or the compiler itself:

  • type inferencing
  • Python-like namespacing
  • strict syntax validation and standardize error output
  • support for other shader languages (HLSL etc)
  • support for non-GLSL compilation targets (SPIR-V etc)

 

Footnotes

[1] M.GRL is released under the Lesser General Public License version 3. This means it does not dictate the terms in which your program is released, however it does require all modifications to M.GRL to also be released under the LPGLv3.

[2] A point of syntax: switch-case statements aren’t actually supported in the version of GLSL we are compiling to.

Programming with People

On the Software Engineering Interview