Note: this is an article dedicated to the windows platform. What am I going to show may or may not apply to other platforms.
Those pesky native crashes
If you are a long term unity developer, it must have happened to you: Unity suddenly closes itself and shows the dreaded unity bug report window. You found that 100% reproducible bug that would make Unity crash all the times. However you are a c# developer and you don’t have the tools to debug native crashes, so what to do? Luckily there are several ways to tackle the problem, as it’s useful to check if something you have done may have caused the problem.
Unity Bug Report
The first step is usually to check what the Unity Bug Report generates when a crash occurs. I am going to use some pictures I found on the internet here to show you visually the steps to take.
The window shows where unity stores the crash reports. Knowing where unit stores them is useful in case you need to ask someone to send you the report.
At this point, instead to send the bug report, you click Preview, which shows you the actual list of files generated that you can open for inspection:
it’s usual enough to open the error.log and check the stack trace to have already a good idea of what’s going on.
For example, a recent problem we were investigating was showing like:
Stack Trace of Crashed Thread 29040:
0x00007FFA3AE2CD5D (HavokNative) HP_UnlockPlugin
0x00007FFA3ACC0C2E (HavokNative) HP_UnlockPlugin
0x00007FFA3ACC03C8 (HavokNative) HP_UnlockPlugin
0x00007FFA3ACBBFFE (HavokNative) HP_UnlockPlugin
0x00007FFA3AE26345 (HavokNative) HP_UnlockPlugin
0x00007FFA3AE2AE9F (HavokNative) HP_UnlockPlugin
0x00007FFA3ABF2D98 (HavokNative) HP_SyncWorldIn
0x00007FFA1E7A95D2 (lib_burst_generated) Ordinal0
0x00007FFA32BD533C (UnityPlayer) UnityMain
0x00007FFA32BD59AF (UnityPlayer) UnityMain
0x00007FFA32BD391C (UnityPlayer) UnityMain
0x00007FFA32BD3D4A (UnityPlayer) UnityMain
0x00007FFA32BD4E70 (UnityPlayer) UnityMain
0x00007FFA32CBD1C8 (UnityPlayer) UnityMain
0x00007FFA75107BD4 (KERNEL32) BaseThreadInitThunk
0x00007FFA75F6CED1 (ntdll) RtlUserThreadStart
Stacks for Running Threads:
Call Stack for Thread 27156:
0x00007FFA75F9C144 (ntdll) NtWaitForSingleObject
0x00007FFA73108BC3 (KERNELBASE) WaitForSingleObjectEx
0x00007FFA3330E232 (UnityPlayer) UnityMain
0x00007FFA32BD4E1C (UnityPlayer) UnityMain
0x00007FFA32BD1E2C (UnityPlayer) UnityMain
0x00007FFA32DCB70B (UnityPlayer) UnityMain
0x000001E28736234A (UnityEngine.CoreModule) Unity.Jobs.JobHandle.ScheduleBatchedJobsAndComplete()
0x000001E28736228B (UnityEngine.CoreModule) Unity.Jobs.JobHandle.Complete()
0x000001E29225BDF3 (RobocraftX.StateSync) RobocraftX.StateSync.DeterministicStepTimingEngine.StepSimulation()
0x000001E292204D13 (RobocraftX.StateSync) <SimPhysicsAtCorrectFrameRate>d__12.MoveNext()
0x000001E27AC337D4 (Svelto.Tasks) Svelto.Tasks.Lean.SveltoTaskWrapper`2.MoveNext()
0x000001E27AC32E43 (Svelto.Tasks) Svelto.Tasks.Lean.LeanSveltoTask`1.MoveNext()
0x000001E27AC324DB (Svelto.Tasks) Process`1.MoveNext()
0x000001E27AC3229B (Svelto.Tasks) Process`1.Svelto.Tasks.Internal.IProcessSveltoTasks.MoveNext()
0x000001E27ABCA53C (Svelto.Tasks) Svelto.Tasks.Unity.Internal.RunnerBehaviourUpdate.ExecuteRoutines()
0x000001E27AC32203 (Svelto.Tasks) Svelto.Tasks.Unity.Internal.RunnerBehaviourUpdate.Update()
0x000001E27ABC8F40 (mscorlib) System.Object.runtime_invoke_void__this__()
0x00007FFA26D3CBB0 (mono-2.0-bdwgc) mono_get_runtime_build_info
0x00007FFA26CC2122 (mono-2.0-bdwgc) mono_perfcounters_init
0x00007FFA26CCB11F (mono-2.0-bdwgc) mono_runtime_invokewhich gave already enough clues to solve our problem.
Debug natively with visual studio or windbg
The rare times this is not enough, you can start some advanced debugging using the dump file generated after the crash. The dump file (.dmp) generated by Unity is quite barebone, so some times is not enough. If this the case, you will need to generate a full fledged dump file.
At the end of the following link, you can see how to ask Windows to create a full-size dump file every time any application crashes:
https://docs.unity3d.com/Manual/WindowsDebugging.html
Although I usually prefer using procdump, which is very simple to use:
https://docs.microsoft.com/en-gb/sysinternals/downloads/procdump
or even simpler debugdiag:
So generating a complete dump file is simple, debugging is not that much. You can debug dump files with visual studio or windbg. I prefer windbg over visual studio, but be sure to download the version from the windows app store, much more friendly than the original one! Debugging with windbg, while not too difficult, needs some learning and this is outside the scope of this article. We will use Visual Studio CE from now on.
Debugging a dump file is surely less powerful than debugging a live process. Attaching Visual Studio or WinDBG before unity crashes is simpler and provides more information.
- Dump files are to analyze, post mortem, crashes that happen rarely or happening on other people machines.
- Live debugging is to debug reproducible crashes
Debugging a live process or a dump is not useful only for analyzing crashing, but it’s useful also to analyze deadlocks! In the past I had several of those too, normally causes by third party plugins. Deadlocks freeze Unity and you know they are deadlocks because Unity is not freezing due to a simple infinite loop.
I am not going in details about how to debug deadlocks, but I will show you how Visual Studio looks like while debugging Unity.
Open Unity and Visual Studio. Start Visual Studio without code. Be sure you follow these instructions first:
Visual Studio Setup
Note: VS2010 and earlier do not function with http server symbol stores.
1. Go to Tools -> Options
2. Expand the Debugging Section, select Symbols
3. Specify a cache directory (if not already specified)
4. Add a “Symbol file (.pdb) location” of http://symbolserver.unity3d.com/
also be sure that Enable Just my code is disabled from the debugging options:
Unity Editor and Unity Standalone Clients ship with .PDB files, which helps the debugger naming the native code being executed. In this way, while you will still need to navigate through obscure assembly, you will see friendly names for functions called and stack trace.
Now that Visual studio is open, go to Debug and Attach Process
search for Unity and click on Attach
If you are debugging a crash, just wait for the crash to happen. The debugger will intercept it and show you exactly where the application crashes and the native exception that happened.
In my case I don’t have a crash, so I simply pause the execution and show you how the debugging would look like when I randomly stop on a random thread:
it’s actually quite easy to identify a thread and switch to the main thread if necessary. Just change the current thread as showed in this image:
If you want to have a complete view of what’s going on thread wise, Parallel Watches and Parallel Stacks are very powerful and intuitive tools:
nothing simpler than that really.
Showing C/C++ code instead of assembly
I don’t debug assembly. Who does that? I am not a hacker! So in the ultra rare case the stack is not enough, in order to have a good idea of what’s going on, having the source code may help to get more insight.
However Unity is closed source, so no luck there, except for few cases. For example Mono is open source and Unity distributes their mono hacked version on github.
https://github.com/Unity-Technologies/mono
so if a crash happens in mono, like it happened to me lately, having this code may help a lot!
When Visual Studio asks for the source code, just give the folder where you copied it and it will work out everything on its own.
Previously assembly will now show for example like:
Mixing Native Stack and c# stack
Unity mono doesn’t allow mixing Native stack with c# stack out of the box. This means that the only stack you will be able to see is the native one.
Some times though, it happens that native code is called by c# code. Think about all those marshalled c#/c++ functions! Unity DOTS introduces even more of those, but luckily Burst now provides a Native Debugging Mode.
For the times you don’t use Burst, enabling the Mixing Native Stack may be useful.
Now, how often it is going to be useful is a good question. It’s so rare (it happened to me 3 times in my 8 years), that Unity is not even advertising this option. It’s currently considered only an internal tool. However it works very well!
This is how Mixed Stack looks like:
There you go, now it’s possible even to understand from what c# code a possible native crash happens!
How to use this tool is explained here:
https://github.com/Unity-Technologies/UnityMixedCallstack
These are the relevant things to know to make it work:
- you don’t need to download any code from the github page
- you have to download a release (currently: 2.5) from: https://github.com/Unity-Technologies/UnityMixedCallstack/releases
- Install the Visual Studio Extension that will enable the magic
- Then you need to be sure that in your Environment Variables you have this enabled:
UNITY_MIXED_CALLSTACK must be set to 1. You will need to reboot after it’s set before to start debugging.
It’s time for the goodbyes
That’s all I had to say, but I need to add one last thing that you may be already aware of: You can attach a Native Debugger and a Managed Debugger at the same time to Unity. Both debuggers will work exactly as expected and debugging the same session could lead to more clues!