Source Filmmaker

Source Filmmaker

230 ratings
How-to Cel shade Guide
By Hoxdolum
Do you want real-time Cel shading for projects and video's?
Then this might be the guide for you!
6
3
2
   
Award
Favorite
Favorited
Unfavorite
1. Setting up
This'll seem confusing at first but once you do it, it'll be quick to apply to other models.
Requires no Post-processing or multiple renders to achieve this effect.



This was something i saw Anonymousgamer testing out awhile back.
Since then I wanted to get a fully working result that allowed Pesudo Pbr to work.

This requires A $PhongExponentTexture and $BumpMap to work

If you have Neither then we'll try to get around that.

( I HIGHLY RECOMMEND YOU MAKE AN ORIGINAL AND CEL FOLDER BEFORE PROCEEDING )

For this tutorial I'll be Using Mark2580 BL3 Lorelei Model
https://steamproxy.net/sharedfiles/filedetails/?id=1867822718

(Edited Version of Lorelei textures to compare with as you follow this tutorial)
https://drive.google.com/file/d/19i9diAs9Rk4hSOHofJZPPyxP8Ug3MtOu/edit

Now load up Crowbar or SFM and bring in the model


First step we'll first need to add this to our model's base VMT's

----------------------------------------
// Paste into VMT
$basemapalphaphongmask 1 // ( If you have something in BaseTextures Alpha, then disable this)
$COLOR2 0.25 0.25 0.25 // Change this for shadow tint Color adjustment
$phongalbedotint 1
$phongexponent 0.001
$phong 1
$phongboost 23.076
$phongfresnelranges 0.05 .225 1


----------------------------------------

Now your model will either Invert colours or become very dark.


We'll next need to edit some textures to get this working correctly.
2. Editing Textures
We'll need a photo editing program next to check RGBA channels.

First texture will be Basetexture, VTFedit and Export as PNG or TGA

Now check If there's is an Alpha layer:
+If nothing is occupying Alpha, Create Alpha channel and apply White
+( We can move onto Exponent section )

-If something is there ( etc Alpha for Hair/clothing) then leave BaseTexture as is.
-( Move onto Normal map section )

----------------------------------------------------------
Normal Map Section:

Export and open your Normal map texture:

*Check if there is a gloss Map in Normal Alpha channel , Invert it
-(We can also clear White but the model may glow more )

----------------------------------------------------------

Exponent Section:

+If the model has one , Wipe Green channel to White.
(This allows the models colours to come through)

-Your Model has none
We'll Cheese this and copy the green Channel from the BaseTexture into a new file.


Next is to duplicate layer. (This will be following Blueflytrap's PBR method for Exponent)

1st layer>Ajustments> Level > Move input middle to 0.28

2nd layer>Change Blend Mode to lighten.
Ajustments> Level > Move Output Right to 0.2 and Merge layers

Now go to Channel and wipe Green/Blue.
After that we should be left with a Cyan colour > Save as Exponent
+ (We could also use this for Normal map by Inverting>Pasting into Alpha)
! (Will require tweaking since we don't have the original Rough/Gloss map)


With All that completed, Export back to VTF format and test model.

Now we should have the model looking like this for comparison.
3. Finish
This is still experimental overall , Some values and parameters may conflict.

What each parameter does:

Control shadow/ colour:

$COLOR2 0.5 0.5 0.5 // Softer colour/contrast , Any higher will reduce The cell shade effect


$COLOR2 0.25 0.25 0.25 // very dark but seems to balance well


$COLOR2 0 0 0 // Completely back unless Radius is high

-----------------------------------------------------


$phongalbedotint 1 // Allow colour Tints on phong reflection

$phongexponent 0.001 // Tightens Shadow factor

$phongboost 23.076 // multiply's tint

$phongfresnelranges 0.05 .225 1 //The 1st value is very important , at 0 the Colours will not show
// Can start from 0.01 to adjust if end results are too bright.
-----------------------------------------------------
Extra info:

The values used for this tutorial adds a Rimlight as shown on right
If you wish to remove it, use the middle values



-----------------------------------------------------

Hopefully i've covered enough in this Tutorial to get you the desired results.
I'll try to respond to best of my knowledge if you get stuck.

Here's more examples of models I've tested with this method

22 Comments
JustSomeDog 14 Oct, 2023 @ 6:22pm 
actually, do you have discord? I'm having trouble again.
JustSomeDog 17 Aug, 2023 @ 10:42am 
Even COLOR2 is a "float" attribute? Awesome that's all I need to know!
Hoxdolum  [author] 17 Aug, 2023 @ 10:24am 
@Flick
No worries so the above are just float values

$basemapalphaphongmask 1// What this does is look at Basetexture Alpha layer for phong mask rather than $Bumpmap Alpha

Only reason to not use it is for area's like alpha hair/ alpha clothing that depends on it.

At 3 stage.Finish, it explains what each one does

$phongalbedotint 1 // Allow colour Tints on phong reflection

$phongexponent 0.001 // Tightens Shadow factor

$phongboost 23.076 // multiply's tint

$phongfresnelranges 0.05 .225 1 // The 1st value is very important , at 0 the Colours will not show
// Can start from 0.01 to adjust if end results are too bright.

You check the edited texture/ VMT's on the model in Crowbar , for SFM use:
mat_reloadallmaterials in the console to see how the values affect it.
JustSomeDog 16 Aug, 2023 @ 1:08pm 
I just wanna know what attributes you used to make all the list (The one that starts with $) $basemapalphaphongmask is it all "float" attributes? That's all I'm asking.
Hoxdolum  [author] 16 Aug, 2023 @ 9:02am 
@Flick Hi there
The $PhongExponentTexture is what controls the Phong Via Exponent texture.

It's hard to say when making an exponent texture from scratch using basetexture.
I choose green channel normally cus the character in this tutorial doesn't have any strong Green colours.
If you had a character who's mostly green, you'd want Red/Blue to avoid over exposure.


I could make a video if i find the time between projects.
In the end some models will require more/less work given complexity and if the necessary files are there to begin with.

Here's the Edited Loreli model to help with comparing along the tutorial.
https://drive.google.com/file/d/19i9diAs9Rk4hSOHofJZPPyxP8Ug3MtOu/edit
JustSomeDog 9 Aug, 2023 @ 8:52am 
So, PLEASE tell us which channel where suppose to pick.
JustSomeDog 9 Aug, 2023 @ 8:51am 
Mind making a video? Cause I don't know where to start. How do I get the $phongtexture or whatever is called. You didn't tell us which channel to use and type in that $texture.
Bonkie 13 Jul, 2023 @ 1:43pm 
Can you add me on steam for further discussion about this?
Tango112 28 Jun, 2023 @ 11:27am 
to bad I don't know what any of that means lol
KARMA The 『 BARMAN 』 21 Jun, 2023 @ 8:59am 
I presume this doesn't work with Paintable stuff, right?