In this tutorial, we will make a simple toon shader. Sometimes, this is also called a cel shader. Also, sometimes people write it “cell shader”, but I’m [http://en.wikipedia.org/wiki/Cel pretty sure that “cel” is correct. No matter what it is called, though, they are the same, and they all have the same goal: to make a model (or entire game) look like it is drawn like a cartoon. It is an interesting effect, and it is actually pretty easy to do, with the stuff that we know already. There are, obviously, lots of different ways to accomplish this, and if this is something that you find interesting, you might want to look into some of the other ways that people do toon shading. We will do one of the simplest ways, but as you can see from the image below, that it is pretty easy to do.
There are basically two parts to toon shading. The first is that, rather than a continuous change of color, there is a sudden jump, and then no changes for a while, leaving larger regions that are all the same color. You can see this in the image above. Second, toon shaders usually have outlines around the objects. There are actually quite a few ways of doing this, including the Sobel operator , which is used for edge detection. We, however, will take a shortcut. It won’t give quite as nice of results, but it will do the job anyway.
This time, rather than trying to stick extra stuff into an existing shader from a previous tutorial, I am just going to give you the shader code (below), and then we will go through it piece by piece and discuss it. It actually isn’t too hard to add this stuff into any other shader, so feel free to do that if you want. Our shader is going to be pretty simple because I am going to just ignore ambient light (we’ll accomplish this another way) and specular light (this looks kind of weird because you have lots of. bands around the highlights, but it looks pretty decent). We will use texturing, although if your texture was extremely intricate, it might not look much like a cartoon. I would recommend sticking with simple textures for this toon shader.
Anyway, on to the shader!
//--------------------------- BASIC PROPERTIES ------------------------------
// The world transformation
float4x4 World;
// The view transformation
float4x4 View;
// The projection transformation
float4x4 Projection;
// The transpose of the inverse of the world transformation,
// used for transforming the vertex's normal
float4x4 WorldInverseTranspose;
//--------------------------- DIFFUSE LIGHT PROPERTIES ------------------------------
// The direction of the diffuse light
float3 DiffuseLightDirection = float3(1, 0, 0);
// The color of the diffuse light
float4 DiffuseColor = float4(1, 1, 1, 1);
// The intensity of the diffuse light
float DiffuseIntensity = 1.0;
//--------------------------- TOON SHADER PROPERTIES ------------------------------
// The color to draw the lines in. Black is a good default.
float4 LineColor = float4(0, 0, 0, 1);
// The thickness of the lines. This may need to change, depending on the scale of
// the objects you are drawing.
float LineThickness = .03;
//--------------------------- TEXTURE PROPERTIES ------------------------------
// The texture being used for the object
texture Texture;
// The texture sampler, which will get the texture color
sampler2D textureSampler = sampler_state
{
Texture = (Texture);
MinFilter = Linear;
MagFilter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};
//--------------------------- DATA STRUCTURES ------------------------------
// The structure used to store information between the application and the
// vertex shader
struct AppToVertex
{
float4 Position : POSITION0; // The position of the vertex
float3 Normal : NORMAL0; // The vertex's normal
float2 TextureCoordinate : TEXCOORD0; // The texture coordinate of the vertex
};
// The structure used to store information between the vertex shader and the
// pixel shader
struct VertexToPixel
{
float4 Position : POSITION0;
float2 TextureCoordinate : TEXCOORD0;
float3 Normal : TEXCOORD1;
};
//--------------------------- SHADERS ------------------------------
// The vertex shader that does cel shading.
// It really only does the basic transformation of the vertex location,
// and normal, and copies the texture coordinate over.
VertexToPixel CelVertexShader(AppToVertex input)
{
VertexToPixel output;
// Transform the position
float4 worldPosition = mul(input.Position, World);
float4 viewPosition = mul(worldPosition, View);
output.Position = mul(viewPosition, Projection);
// Transform the normal
output.Normal = normalize(mul(input.Normal, WorldInverseTranspose));
// Copy over the texture coordinate
output.TextureCoordinate = input.TextureCoordinate;
return output;
}
// The pixel shader that does cel shading. Basically, it calculates
// the color like is should, and then it discretizes the color into
// one of four colors.
float4 CelPixelShader(VertexToPixel input) : COLOR0
{
// Calculate diffuse light amount
float intensity = dot(normalize(DiffuseLightDirection), input.Normal);
if(intensity < 0)
intensity = 0;
// Calculate what would normally be the final color, including texturing and diffuse lighting
float4 color = tex2D(textureSampler, input.TextureCoordinate) * DiffuseColor * DiffuseIntensity;
color.a = 1;
// Discretize the intensity, based on a few cutoff points
if (intensity > 0.95)
color = float4(1.0,1,1,1.0) * color;
else if (intensity > 0.5)
color = float4(0.7,0.7,0.7,1.0) * color;
else if (intensity > 0.05)
color = float4(0.35,0.35,0.35,1.0) * color;
else
color = float4(0.1,0.1,0.1,1.0) * color;
return color;
}
// The vertex shader that does the outlines
VertexToPixel OutlineVertexShader(AppToVertex input)
{
VertexToPixel output = (VertexToPixel)0;
// Calculate where the vertex ought to be. This line is equivalent
// to the transformations in the CelVertexShader.
float4 original = mul(mul(mul(input.Position, World), View), Projection);
// Calculates the normal of the vertex like it ought to be.
float4 normal = mul(mul(mul(input.Normal, World), View), Projection);
// Take the correct "original" location and translate the vertex a little
// bit in the direction of the normal to draw a slightly expanded object.
// Later, we will draw over most of this with the right color, except the expanded
// part, which will leave the outline that we want.
output.Position = original + (mul(LineThickness, normal));
return output;
}
// The pixel shader for the outline. It is pretty simple: draw everything with the
// correct line color.
float4 OutlinePixelShader(VertexToPixel input) : COLOR0
{
return LineColor;
}
// The entire technique for doing toon shading
technique Toon
{
// The first pass will go through and draw the back-facing triangles with the outline shader,
// which will draw a slightly larger version of the model with the outline color. Later, the
// model will get drawn normally, and draw over the top of most of this, leaving only an outline.
pass Pass1
{
VertexShader = compile vs_2_0 OutlineVertexShader();
PixelShader = compile ps_2_0 OutlinePixelShader();
CullMode = CW;
}
// The second pass will draw the model like normal, but with the cel pixel shader, which will
// color the model with certain colors, giving us the cel/toon effect that we are looking for.
pass Pass2
{
VertexShader = compile vs_2_0 CelVertexShader();
PixelShader = compile ps_2_0 CelPixelShader();
CullMode = CCW;
}
}
Feel free to just copy this into a .fx file.
I’m going to skip over explaining the “Basic Properties” and the “Diffuse Light Properties”, because we’ve covered them in earlier tutorials, and so you probably know what to do with them.
We’ll jump on down to the “Toon Shader Properties”.
There are two variables here that you can play with.
Both involve how the outlines are drawn.
The first, LineColor, determines what color the lines will be drawn.
The second, LineThickness, will determine how much the vertices will be expanded to produce the outlines.
We’ll also skip over the texture stuff because we talked about that in another tutorial as well. The “Data Structures” section is probably pretty similar to what we have done in the past too. There shouldn’t be any surprises there.
The “Shaders” section is where things start to get interesting.
First of all, notice that we have four shaders, instead of two.
This is because our toon shader is going to take a couple of passes to render.
This is called multi-pass rendering, and we haven’t seen this before, in these tutorials.
Each pass will require its own vertex shader and its own pixel shader, so we have four shaders total.
The CelVertexShader is pretty simple.
Once again, you have probably seen all of this in the diffuse lighting HLSL tutorial, so I’m not going to say much about it.
The CelPixelShader is where we do our first interesting thing.
We first calculate the diffuse lighting intensity (and cut it off at 0). We then calculate the texture color like usual.
Then we break the color up into discrete chunks to get the cel-shaded effect that we want.
// Discretize the intensity, based on a few cutoff points
if (intensity > 0.95)
color = float4(1.0,1,1,1.0) * color;
else if (intensity > 0.5)
color = float4(0.7,0.7,0.7,1.0) * color;
else if (intensity > 0.05)
color = float4(0.35,0.35,0.35,1.0) * color;
else
color = float4(0.1,0.1,0.1,1.0) * color;
This code basically just forces the intensity to be one of four values, based on certain cutoff points. You can play around with these values if you want. Also, keep in mind that there are lots of ways to do this. I have seen quite a few toon shaders that do something like the following instead, which produces similar results:
color = round(intensity * 5) / 5 * color;
The OutlineVertexShader also does some interesting things that we will talk about.
This shader looks like this:
// The vertex shader that does the outlines
VertexToPixel OutlineVertexShader(AppToVertex input)
{
VertexToPixel output = (VertexToPixel)0;
// Calculate where the vertex ought to be. This line is equivalent
// to the transformations in the CelVertexShader.
float4 original = mul(mul(mul(input.Position, World), View), Projection);
// Calculates the normal of the vertex like it ought to be.
float4 normal = mul(mul(mul(input.Normal, World), View), Projection);
// Take the correct "original" location and translate the vertex a little
// bit in the direction of the normal to draw a slightly expanded object.
// Later, we will draw over most of this with the right color, except the expanded
// part, which will leave the outline that we want.
output.Position = original + (mul(LineThickness, normal));
return output;
}
The comments in this code explain it fairly well, I think, but the basic idea with this is that we will take the vertices and move them outwards just slightly from where they were originally at. “Outwards” is defined as “in the direction of the normal”, so that is what this shader does. The first line puts the vertex in the same location that it usually would be in. The second part transforms the normal as well. The last part is the interesting part because, in this line, we take the original vertex and move it just a little bit in the direction of the normal. It is pretty simple, but should basically expand our model ever so slightly.
The outline pixel shader is extremely simple. It just colors everything with the outline color:
// The pixel shader for the outline. It is pretty simple: draw everything with the
// correct line color.
float4 OutlinePixelShader(VertexToPixel input) : COLOR0
{
return LineColor;
}
The last part is also pretty interesting, so we will look at that too. In the last part, we define the technique with two passes. We haven’t done this before, so it is worth looking at.
// The entire technique for doing toon shading
technique Toon
{
// The first pass will go through and draw the back-facing triangles with the outline shader,
// which will draw a slightly larger version of the model with the outline color. Later, the
// model will get drawn normally, and draw over the top of most of this, leaving only an outline.
pass Pass1
{
VertexShader = compile vs_2_0 OutlineVertexShader();
PixelShader = compile ps_2_0 OutlinePixelShader();
CullMode = CW;
}
// The second pass will draw the model like normal, but with the cel pixel shader, which will
// color the model with certain colors, giving us the cel/toon effect that we are looking for.
pass Pass2
{
VertexShader = compile vs_2_0 CelVertexShader();
PixelShader = compile ps_2_0 CelPixelShader();
CullMode = CCW;
}
}
Notice that you can just put multiple pass definitions in a row, and they will be performed sequentially.
Pass1 does the outline, and Pass2 does the normal drawing with the toon-style coloring.
The other important thing we see here is the addition of the lines CullMode = CW; and CullMode = CCW;.
This goes back to a topic in graphics.
Usually, it is a lot of work to draw a single triangle, and if you are drawing lots of them, you’d be willing to get rid of as many as you can to speed things up.
So what they do is they assign a front and back face to the triangle and draw only the triangles that are facing toward the screen.
This theoretically means you only need to draw half as many triangles, which will speed things up a lot.
The front face and back face are determined by the “winding order” of the triangle.
Usually, the front face is the one where the vertices are given in counterclockwise order, and the back face is the one where the vertices appear in clockwise order.
The process of removing (and thus not drawing) backward triangles is called “backface culling”.
These two lines here indicate which triangles to cut out.
For our outline shader, we are actually telling it to ignore the front-facing triangles and only draw the backward ones.
Then we switch it around when we draw the front-facing ones with our regular shader.
Without this, the outline shader would draw the expanded model, and it would block out the rest of the model.
All we would see is a big blob of black.
But since we are ignoring triangles that are facing the screen during the outline pass, nothing will be drawn there, and we will be able to see the second pass.
As far as loading this into XNA, I think you probably know what to do by now, so I’m not going to give any specific directions. Just make sure that you are using the correct shader and that you set the matrices and texturing stuff up like you’ve done before. (The shader is the only real new thing here!)
ToonShader.zip