Tutorial – Creating outline effect around objects

Outline effect as post process – tutorial

Introduction

In this tutorial I will present a way of creating outline effect for meshes inside UE4. There are two popular methods of creating outline/stroke effect around objects in the game:

  • Render mesh twice. Draw first mesh with a standard material and then draw second mesh with slightly higher scale and emissive material.
  • Add outline to objects in post-process using edge detection

I’ll focus on the second method, as it requires very little effort to integrate it into existing projects. This tutorial assumes, you are familiar with basics of Unreal Engine 4 and it’s systems (material editor in particular). If you wish to fully understand how to implement edge detection algorithm as UE4 material, some basic image processing knowledge will be useful.

Basic steps

In order to implement outline effect as post process you need to follow these steps:

  • Create a post process material that will add stroke effect for objects that have Custom Depth rendering enabled.
  • Add your post process material into blendables list in your scenes’ post process volume.
  • For each skeletal mesh or static mesh that you wish to be highlighted, enable parameter „Render Custom Depth”.

I’ll explain each of those points later on. If you are here just for the material, you can grab it in download section at bottom of the page.

Custom depth map

One of UE4’s cool features is ability to render individual meshes into separate depth texture and use it in materials. A depth texture keeps information about distance between each individual pixel in world space and a camera. A typical depth map will look like the image below.

Our custom depth map looks similar, expect there are visible only objects that have „Render Custom Depth” option selected in their properties.

This makes it super convenient to implement effects such us Outline, in screen space.

Creating a material

This is the hardest part. In our material we will implement sobel operator and convolve it with the custom depth map. In other words we will apply edge detection filter on our custom depth map to find out where we should draw stroke.

We will start with implementing our convolution algorithm. It’s rather sophisticated math behind this, but everything really boils down to following these steps:

  1.       for each pixel P
  2.            for each pixel PN adjacent to P
  3.            Multiply PN’s value with weight found in kernel’s cell relative to pixel’s origin
  4.            add result to output value
  5. return output value

Sampling our depth map is simple: Drop Scene Texture node and connect it to the Emissive output. In node’s properties set Texture ID to CustomDepth

Now, we need to sample adjacent pixels. We can use the UV parameter of Scene Texture node for this. Problem is, UV parameters work in texture space, which means they take values from Top-Left corner(0,0) to Bottom-Right corner(1,1) of the texture. To do this we need to take inverse of our texture width and height, multiply by our offset and add it to UV coordinates of current pixel.

In material editor it will look like this:

Next, we need to duplicate those expressions for remaining seven adjacent pixels. The final setup for calculating pixel offsets:

We have a set of UVs for all adjacent pixels, so we can now sample actual values from the CustomDepthMap. I’ve created a simple material function for this. It accepts UV coordinates as input and returns value of the sampled pixel.

If we use this function to sample adjacent values, we will have all the data to do the actual edge detection. Let’s create another material function that will perform the convolution:

On the left side there are two sets of input parameters. First 9 input vectors are the values of pixels that we will be working on. The other three, provide values of our kernel. It’s really just a 3×3 matrix, but since there is no matrix type in UE4 material editor, I divided mine into three Vector3 parameters. Up to this point, we have all pieces of algorithm. All we have to do is put everything together inside the final material.

The sobel operator actually involves two convolution operations: vertical and horizontal. As you can probably notice, the only difference between those two is the input kernel.

Now we need to combine our two values:

CombineSobel

In material editor we can just create vector from calculated values and return it’s length. The output will be black image with white outline around meshes that have custom depth rendering enabled. The last thing to do is blending our outline into final image. In this case I used „IF” expression to get solid stroke around objects, but it should be fairly easy to adjust the effect for your needs.

REMEMBER: Make sure that your material needs to be in Postprocess Domain, and not in Surface. You can change that in your material’s properties.

Scene setup

Our awesome material is finished, and now we need to plug it into our scene’s post process to see actual results. In your level select Post Process Volume and search for „Blendables”. Add new item to blendables list and pick your material from the list. If you don’t have any post process volumes in your leve, you should create one. Also double check that your volume has „Unbound” property set to true. Otherwise you will see the outline around objects only when you are inside that volume.

blendables

You can test results by dropping any mesh into scene and checking it’s „Render custom depth” property.

Blueprint example

In our sample project, static meshes get highlighted, when player aims at them. I’ve used FPS template for starter. In MyCharacter blueprint I’ve added a new function called Trace. It is called by timer every 0.1 seconds. It checks if player is looking at a static mesh. If yes then it sets RenderCustomDepth to true. When player stops aiming at that mesh, the RenderCustomDepth variable is set to false. You can check it out in action by downloading sample project provided below.

Conclusion

In this tutorial I’ve presented a fairly simple and easy to integrate method of rendering outline around objects. In fact this material is capable of doing much more than this particular effect. By convolving image with different kernels, we can achieve a variety of effects like blur, emboss or sharpen(see http://en.wikipedia.org/wiki/Kernel_(image_processing) for more information).

Demo video


 

Downloads

Outline effect – Material

Material used in this tutorial.

Outline effect – Sample Project

40 Comments

  1. Hi. Excellent tutorial! Thank you very much – it helped me a loooot! However I’m currently struggling with another thing. I’m trying to get similar effect but only on occluded parts of object. For example – only render outline on the part of cube hidden behind the wall. I’m fighting with this for couple of days now, sadly – without success. I’ve tried many operations between custom depth, pixeldepth and scenedepth, but nothing I’m nowhere closer to solving this. Can you please give me at least some tip on how to create the mask only for occluded parts of object? Will be very very grateful 🙂

    Reply
    • Hi!

      Apologies for late response. Hope you found solution to your problem.
      If not, here’s a little hint that may help you.

      You can compute difference of CustomDepth and SceneDepth. That way, you will get values = 0 where object is not occluded, and >0 where it’s occluded. You can try playing with that. Unfortunately, UE4 will not render objects that are fully occluded, so you won’t see the outline(unless these are translucent objects). You can fix that by modifying engine’s rendering code but that’s rather non trivial thing to do.

      Reply
  2. This was very helpful man!

    Just one Issue, I had to redo part of a material, as some material functions where not ticked to be visible in library, so the material didn’t compile.

    Thanks for the tutorial!

    Reply
  3. Awesome stuff. It works great for me. Thanks for not only putting the material up for download but also for explaining some of how this process works for drawing the outline.

    Reply
  4. Hello, great tutorial.
    Can you help me, I want to make this effect for 3d person player.
    How to make it? The problem is in camera.

    Reply
  5. Hello,

    first of all I would like to thank you for the awesome tutorial. I am actually having some issues getting it to work on my project. Once I reach the final step (set Blendables), my screen imminently switch into the colour that my material blueprint (either the main or instance material) is set to. Please advice.

    Regards,
    Jacquelyn.

    Reply
    • Hello Jacquelyn,

      Glad you like it. It appears that somehow my material doesn’t properly detect edges in your scenes. Are you experiencing this in all your scenes or just specific ones? Does this happen if you create new blank project? Hard to tell what may be causing this. If you could isolate the issue into test scene and send it over or just provide more info how to reproduce this issue, then maybe I could help.

      Cheers!

      Reply
  6. Thanks for your awesome tutorial.!! I have a problem setting the mat_function MF_EXTRACT DEPTH. It gives the following error :-
    Error [SM5] SceneTexture expressions cannot be used in opaque materials

    Reply
  7. Hi, thanks for this very useful tutorial.

    I was just wondering if there was a way to choose which objects are allowed to be „traced”. So far, I noticed that only objects with Collision get traced. I would like to be able to choose objects that can be traced not depending on the fact that they have a collision or not. Maybe create a parameter „can be traced” that we can activate per object? Or a list of objects?

    Any idea is appreciated 🙂

    Thanks !

    Reply
  8. Is there any way to use this with the scene depth so that all objects, not just custom depth ones, can have the outline?

    Reply
  9. Hi there,

    Great tutorial! Could you explain exactly how to change the thickness of the outline? So that the effect is quite distinct, thanks 😀

    Reply
  10. Hi Guys,
    Is there anyway I can just highlight certain object? Because right now all static meshes is my scene is highlighted wherever my cursor goes

    Reply
  11. Hi!
    Thanks for tutorial!
    I have question similar to one above.
    I have custom meshes imported, when I enable render custom depth, they lit up and when I play scene, exact opposite happens.. When I look at them they lit off, when I turn away they lit up. For basic geometry like cubes etc. it works as excepted and I even don’t have to turn on „Render Custom Depth”.
    What I am doing wrong? UE 4.8.3
    Thanks!

    Reply
  12. Hey PawelM, thanks for the tutorial. This has been a really good journey on my learning of post-processing and material functions.

    I had just 1 problem thou, my edges flicker. I’ve used this calculation on the SceneDepth and realized that the edges are basically all flickering, would you know what could cause that?

    Reply
  13. nice tutorial! i’ve been looking this for a long time! i’m trying to read it…..uhh you know i’m a chinese…still, thanks for your sharing of the project!

    Reply
  14. If you want this to only occur on specific objects all you need to do is create a blueprint interface. Within that blueprint interface create a function called LookAtTrace, than go inside your character blueprint and your object you wish to interact with and add that BP interface. In your character blueprint add an event using your LookAtTrace function that executes the line trace, and finally go into your interact object and add a interface message from your LookAtTrace that enables the custom depth.

    Reply
  15. Is there a way of making this not occlude through walls? I am using this as an edge shader for characters and it displays the lines through the walls. I tried messing with scenedepth instead of custom depth but it just added the effect to the whole world.

    Reply
  16. You can make only a material with contour?
    I was looking in this process, but it is a complex medium, do not know how i could do this on my project runner infinity example of the epic – youtube…

    Reply
  17. Hi.
    I’m using Unreal Engine 4.12.3 and I have tested it your material on it but I can’t use it.
    The material function MF_EXTRACT_DEPTH throws the error: Error [SM5] SceneTexture expressions cannot be used in opaque materials.

    Do you know how can I fix this error?

    Thanks.

    Reply
  18. Hey! Thanks a lot, your material is really helpful.

    I’m having some trouble, though. In your „Trace” function you have a variable named „current target”. If I create that variable and try to connect it with „Set Render custom depth” the same way as you do Unreal won’t let me.

    It says „actor reference is not compatible with Primitive Component Reference”.

    Reply
  19. This looks really great! Though, I don’t see any mention of a license for the actual assets you show off at the end. Can I assume you intend to provide them free of charge for commercial use since you’ve provided a tutorial on how to make them in the first place? If so, would you require that I credit you on anything I publish that uses this?

    Thanks! Again, really love this effect, so I appreciate you taking the time to teach folks. 😉

    Reply
  20. I’m having some trouble unfortunately. I get an error when compiling the PP material saying:
    „[SM5] Function NeighbourUVs: Missing function input „Texture UV””
    I’ve triple checked both my PP material and the material function for for acquiring UVs of adjacent pixels.

    Any help would be thoroughly appreciated!

    Reply
  21. I am using UE 4.13.1
    I downloaded Matreal.
    PP_OutLiner_M don’t link. So I haved to link again.
    I tried a little but succeeded nicely.
    Thank you so much for sharing a good idea.

    Reply

Leave a Comment.