Tag Archives: debugging

Integrating RenderDoc

A few days ago a new version of RenderDoc was released, and while reading the changelist I discovered that Temaran has made a really cool plugin which integrate RenderDoc directly in Unreal Engine 4.

This is extremely useful. More than once I launched debugging from Visual Studio, found a weird bug, and had to launch it again from RenderDoc, trying to reproduce the bug.

So I looked at the source code uploaded on Github by Temaran, removed the UE4 related code and only kept a single class to be able to load RenderDoc and trigger a capture directly from my engine.

You can find the code here: https://github.com/oks2024/RenderDoc-Manager

In the end it’s just two header and one cpp file. You just have to provide some paths, like where you want to store the captures and your RenderDoc folder. In my case I use the portable version and stored it in Perforce. Keep in mind that your build target must match the RenderDoc version, you can’t mix x86 and 64bits.

You can either bind a key to RenderDoc that will trigger a capture or use the StartFrameCapture()/EndFrameCapture() functions. I use the latter because it allows me to skip the update part of my engine, capturing only the rendering functions.

It’s working great in my small engine, I’m using it for a couple of days and hadn’t noticed any issue. I know that it can slow down the resources creation so for a bigger engine it’s not be something you want to have always enabled.

As you can see it’s very a basic code skeleton, and most of the code is from Temaran’s plugin, but I found it really usefull and thought it worth sharing.

I think I will add functions (at least to set capture options without having to recompile) as I need them, and I will try to keep the Github repository up to date. And if you have any suggestion, it’s on github, so feel free to leave a comment or add your modifications :).

Apart from debugging, it may also be usefull for creating automated tests. With the appropriate script you can load a level, move the camera through several positions, and take captures. And after that it should be possible to get the images (and maybe even timings) from the captures, and compare them to make sure your last submit did not break the rendering.

Baldur Karlsson is doing an amazing work on RenderDoc, with regular updates and new features. I was already one of my favorite tool, and it’s not going to change !

Visual Studio has triggered a breakpoint

Yesterday I wasted a lot of time tracking a bug which turned out to be pretty instructive.

I am working on a tiled deferred renderer, and after adding a bunch of features, and before starting new ones, I spent some time on cleaning the development mess.

The DirectX debug device was complaining about some objects I forgot to release, so I made sure that my destructors were all doing their job. But then each time I closed the program, this message showed up:

 Visual Studio has triggered a Breakpoint

Well, it’s not really self-explanatory. I had set no breakpoint, visual studio has triggered the breakpoint itself. Clicking break leads to a SAFE_RELEASE() in the destructor of one of my singletons. When I tried “continue” the program terminated without any other errors/messages.

I first tried to comment the supposed faulty line. No more errors, but some DirectX objects are still alive. I thought that maybe the device had to be released last. I tried that, but the message came back, and breaking stop to SAFE_RELEASE(m_pd3dDevice). In fact I understand that it will always break on the last D3D released object, but if I let an object alive the message will not pop up (but an object will not be destroyed, that’s not a solution).

Obviously the bug was memory related, so I tried some analysis tools. Each one pointed me a different direction, far from the real solution. So I begin a less subtle debugging method: comment everything!

Since it was a memory problem I let only creations and destructions of my objects, removing the calls to the update and renders functions. And surprisingly it worked, no error, everything was destroyed. I then called the update functions, ok, and the render functions, and the breakpoint is back.

I continued to isolate the problem inside the rendering functions, until I have only the following code left:

 

// Set render targets.

ID3D11RenderTargetView* RTViews[2] = { ppAlbedo, ppNormal};

engine->GetImmediateContext()->OMSetRenderTargets(2, RTViews, 0);

// Clear render targets

quadRenderer->SetPixelShader(m_pClearPixelShader);

quadRenderer->Render();

 

I started to think that I had messed up with my quad rendering or render targets, but in fact it turns out that the evil one was the apparently innocent SetPixelShader function. It’s a simple (yet horrible) setter:

void QuadRenderer::SetPixelShader(ID3D11PixelShader* ppPixelShader)
{
    m_pQuadPixelShader = ppPixelShader;
}

 

This is wrong, (I’ll come back to that later), but not harmful per se.  The true horror lies in the QuadRenderer destructor:

 

QuadRenderer::~QuadRenderer(void)
{
    SAFE_RELEASE(m_pQuadVertexLayout);
    SAFE_RELEASE(m_pQuadVertexShader);
    SAFE_RELEASE(m_pQuadVertexBuffer);
    SAFE_RELEASE(m_pQuadIndexBuffer);
    SAFE_RELEASE(m_pQuadPixelShader);
}

 

The pixel shader is released, but which pixel shader ? It wasn’t created by the QuadRenderer, his owner must have already released it. SAFE_RELEASE check that the pointer is not null, but in that case the pointer still points to something, something that had already been released, leading to the land of unknown behaviors, or worse.

I learned several lessons thanks to that bug.

-        Destructors are not the funniest part of the code, but it needs to be done carefully, you can’t just look at the member variables and delete them all, it can be trickier than that.

-        Despite of his name, the macro SAFE_RELEASE is not that safe (obvious, but it’s easy to forget).

-        Poor design can lead to annoying bugs. When I implemented the QuadRenderer class I was thinking: “Ok, I’ll need a vertex and pixel shader, but I want to be able set the pixel shader I want”. This is wrong. I don’t want my QuadRenderer to have a pixel shader, I want it to render with a particular pixel shader. There is no need to save it. This is something to be careful, so that your destructors can be trivial.

Well, now everything is clean and I’ve learned much more than I would have thought, I can start adding new features !