A tad different from the usual type of questions around here. This is about realtime, NPR rendering: I've been playing the latest Legend of Zelda installment recently, and found myself mesmerized by this incredibly novel way of drawing toon outlines. To be clear, I'm talking about the light brightness transmitting through the edges of the object on all orientations, not the complete cel-shaded look. Reference images below. Not even able to classify how this effect is achieved. Almost tempted to call it NPR subsurface scattering, but it doesn't really look like SSS. Screenspace width rim shading? It appears most proeminently in the (substantially more cel-shaded) characters, but is also apparent in the terrain and object, which uses a relatively more traditional PBR look. Argh, this is frustratingly complicated.
And here is my (paltry half-assed) attempt at approximating the effect in Unity: (old video, using too much of the default PBR shader. Observe the outline, not the general shading) https://www.youtube.com/watch?v=HsFS8PbnfQQ It uses basically a combination of rim-shading with a cutoff, mixed with the dot of the camera and light directions, essentially making the effect stronger the more the camera is pointed at a light. The most obvious problem with this approach is that it looks horrible on almost-flat surfaces seen at a grazing angle; and also depends a lot on the relative curvature of the object. In the end it looks simply "wet". This is an old video, here's a screenshot of the latest updates, with annotations where it behaves correctly and where it behaves weirdly (green=good, red=bad): In the game, the outline width is mostly constant, varying globally per object, and camera distance. With the rim-based approach, the cutoff is constant, but the proportionality depends on the surface curvature. You can see it clearly on the last two screenshots, where a slight turn of the camera makes a whole erroneous section of the outline appear. Maybe the rim approach if we can scale it by some factor that negates its curvature dependency, but that doesn't make much sense since that's the whole point of a rim. I tried using Sobel, but it has way too many false-positives and isn't really suited for this task, since not all edges are actually highligted, mostly those bordering the model or overlapping other parts of the model. Initially I won't post code since it's not a debugging question, but how to do it conceptually.