Debugging Enhancements

[mitchcapper]

dnSpy has had some great changes as of late to improve the debugging experience. A few things I was potentially going to try and implement were:

• Improved exception window rather than the fairly straight messagebox more in line with VS.

  • Adding the option to turn off the exception right there rather than searching and disabling it in the exceptions list.
  • Adding the option to add an except in / only in conditions that dnSpy already supports to this popup box
  • Add the new box to popup on any exception break rather than just the unhandled ones like VS.
  • Could add a similar InnerException toggling through like VS
  • Could add a copy option to copy direct to clipboard the data
  • In terms of the view details there are already spare watch windows, technically a watch window with just that in it floating is basically it but i see this as not very important

• Ability to save/load selected exceptions configurations. Sometimes end up sending a decent amount of time turning of a select set of exceptions only for one reckless failed set of filter to only enabled, disable all re-enable all but filter not properly on losing the config.

• Potentially add a "Disable all exceptions" for a specific assembly, could just manaully apply that setting to each of the exceptions in exception settings rather than storing new data. Would also need a Re-Enable exceptions for said module to make undo easier.

• Toggle to disable exceptions, technically there is a disable all shown/enable all shown but it can be error prone. if you don't want exceptions you have already disabled to show up you need to make sure you have filtered first. One slip and all reset. A global exceptions on/off would allow you to run a program to a point then start caring about exceptions.

• I don't know dnSpy internals or how break/tracepoints are matched up with dynamic assemblies. I am not sure if it is based on order they are loaded, hashing the memory module, mvid, or what have you. In the export I see an ID number assigned to dynamic modules, and maybe this is incremental based on load order and what i used. I have noticed that they can frequently no longer match up over multiple runs in some situations. This causes the module to be reloaded into the assembly side even if the bytes should be the exact same as it previously. Might look to see if hashing is not done if it is a practical option from a performance POV. Another in app option could be to give users the option to loosen BP restrictions to just name matches. The only way around this I have found is to breakpoint on load, then manually assign breakpoints again, then continue which can be time consuming.

• Ability to view linked deobfuscated assembly. There are some great deofuscaters out there but sometimes, despite their best efforts to not break things, they do (it could even just be anti-deobfuscation detection and thwarting). While using deobfuscated code for static analysis is nice, a basic function like go to MD Token in alternate assembly would be great for quickly looking at the obfuscated binary without copying the token, manually switching to the other assembly and going to that token. This likely would be best implemented as a 3rd party extension rather than as a core feature as im sure not useful to many. Maybe a second window that tries to stay in sync with the current tab would be interesting (as would somewhat work while debugging as well).

I have very little free time (like everyone I am sure) so even if some of these sound good it could be a bit before I can look at them. Still figured get some feedback so if I do get a chance to do so I can.

[mitchcapper]

I am not sure if this belongs in dnSpy or should just be left for de4dot and such.
If the original assembly had no debug directory entries then you can't use PDB's with it and debuggers like VS will not try to load any symbols. dnSpy is not effected but for modules saved out this could be 'fixed' if the option was enabled. The simple code I have been using for dnlib is:

var modCtx = ModuleDef.CreateModuleContext();
var module = ModuleDefMD.Load(dll, modCtx);
module.CreatePdbState(dnlib.DotNet.Pdb.PdbFileKind.PortablePDB);
var writerOptions = new NativeModuleWriterOptions(module, optimizeImageSize: true) {
	KeepWin32Resources=true,
	WritePdb=true,
};
writerOptions.MetadataOptions.Flags |= MetadataFlags.PreserveAll | MetadataFlags.KeepOldMaxStack;
module.NativeWrite(target, writerOptions);
File.Delete(target.Replace(".exe", "").Replace(".dll", "") + ".pdb");

I will do a patch for de4dot to have that, but it may fit in dnSpy as well. The actual PDB does not need to be generated as VS will be happy to do the real work there but this adds the needed entries on the file assembly itself.

[mitchcapper]

I have since modified my make things debuggable code to be more robust. If desired to add such a thing to dnSpy as an option for exports I can do a PR. Until #5 lands there are certainly times where debugging in VS is a better experience just due to the code being more understandable. VS is too much of a boy scout though, so this helps corrupt them.

/// <summary>
		/// pdb_assembly_name should be the original assembly name ie MyNamespace.MyAsm
		/// We don't write a real pdb out here use the ILSpy lib call to do so or manually use resharper dotPeek.  
		/// </summary>
		/// <param name="target_file"></param>
		/// <param name="pdb_assembly_name"></param>
		public void MakeDebuggable(String target_file, String pdb_assembly_name = null, bool OutputThePDB = false) {//we dont normally wee 
			InitStd();
			if (String.IsNullOrWhiteSpace(target_file) || target_file.Equals(source_file))
				throw new Exception("Target file must be separate and distinc from source file");

			RemoveCustomAttribute(module.Assembly, typeof(DebuggableAttribute));
			RemoveCustomAttribute(module.Assembly, typeof(SuppressIldasmAttribute));
			//AddCustomAttribute(module.Assembly, () => new DebuggableAttribute(DebuggableAttribute.DebuggingModes.Default | DebuggableAttribute.DebuggingModes.DisableOptimizations)); // | DebuggableAttribute.DebuggingModes.IgnoreSymbolStoreSequencePoints
			AddCustomAttribute(module.Assembly, () => new DebuggableAttribute(true,true)); 

			if (module.PdbState == null)
				module.CreatePdbState(dnlib.DotNet.Pdb.PdbFileKind.PortablePDB);

			module.PdbState.PdbFileKind = dnlib.DotNet.Pdb.PdbFileKind.PortablePDB;
			module.PdbState.RemoveAllDocuments();
			module.PdbState.UserEntryPoint = module.EntryPoint;
			
			var pdbOutFile = DNUtils.GetPDBFileFromExec(target_file);

			var finfo = new FileInfo(target_file);
			if (String.IsNullOrWhiteSpace(pdb_assembly_name))
				pdb_assembly_name = finfo.Name.Substring(0, finfo.Name.Length - finfo.Extension.Length);
			pdb_assembly_name += ".pdb";

			using var junkStream = new MemoryStream();
			var writerOptions = GetDefaultWriterOptions((x) => {
				x.PdbStream = OutputThePDB ? null : junkStream;
				x.PdbFileName = new FileInfo(pdbOutFile).FullName;
				x.PdbFileNameInDebugDirectory = pdb_assembly_name;
				x.WritePdb = true;//i promise if not true it WILL NOT write the debug directory entries in the main executable
			});

			module.NativeWrite(target_file, writerOptions);

		}

[ElektroKill]

Improved exception window rather than the fairly straight messagebox more in line with VS.

Yes, the exception dialog definitely needs to be reworked. Reading stuff like inner exceptions, perhaps showing them in some sort of treeview (since we can have multiple inner exceptions for stuff like AgregateException), and adding additional options are things to consider. Oh and of course, making it easy to copy and paste the exception information. It might even be possible to produce a nice syntax highlighted stack trace using ICorDebugExceptionObjectValue.EnumerateExceptionCallStack which gives us a stack trace in the form of a list of IL offsets, method tokens, and module references which would allow creating a string stack trace in a custom format or one akin to the regular ToString with stuff like jump to reference for the methods, syntax highlight and member name escaping for obfuscated files. There are tons of possibilities for this but implementation will not be easy due to how exceptions are currently communicated to the UI.

Ability to save/load selected exceptions configurations. Sometimes end up sending a decent amount of time turning of a select set of exceptions only for one reckless failed set of filter to only enabled, disable all re-enable all but filter not properly on losing the config.

This makes total sense and I'd definitely accept such a change.

Potentially add a "Disable all exceptions" for a specific assembly, could just manaully apply that setting to each of the exceptions in exception settings rather than storing new data. Would also need a Re-Enable exceptions for said module to make undo easier.

I do not really see a use case for this. Feel free to elaborate more on this, perhaps I'm just missing something.

Toggle to disable exceptions, technically there is a disable all shown/enable all shown but it can be error prone. if you don't want exceptions you have already disabled to show up you need to make sure you have filtered first. One slip and all reset. A global exceptions on/off would allow you to run a program to a point then start caring about exceptions.

A toggle to disable/enable breaking on handled exceptions is something that could be added to the Exceptions settings tool window.

I don't know dnSpy internals or how break/tracepoints are matched up with dynamic assemblies. I am not sure if it is based on order they are loaded, hashing the memory module, mvid, or what have you. In the export I see an ID number assigned to dynamic modules, and maybe this is incremental based on load order and what i used. I have noticed that they can frequently no longer match up over multiple runs in some situations. This causes the module to be reloaded into the assembly side even if the bytes should be the exact same as it previously. Might look to see if hashing is not done if it is a practical option from a performance POV. Another in app option could be to give users the option to loosen BP restrictions to just name matches. The only way around this I have found is to breakpoint on load, then manually assign breakpoints again, then continue which can be time consuming.

Currently, dynamic modules are assigned IDs depending on load order. This is because we have very little information about dynamic modules without resorting to reading their metadata which is an expensive operation. Names aren't reliable since some code that uses them will just use a random name every time making lookup by name not feasible.

Ability to view linked deobfuscated assembly. There are some great deofuscaters out there but sometimes, despite their best efforts to not break things, they do (it could even just be anti-deobfuscation detection and thwarting). While using deobfuscated code for static analysis is nice, a basic function like go to MD Token in alternate assembly would be great for quickly looking at the obfuscated binary without copying the token, manually switching to the other assembly and going to that token. This likely would be best implemented as a 3rd party extension rather than as a core feature as im sure not useful to many. Maybe a second window that tries to stay in sync with the current tab would be interesting (as would somewhat work while debugging as well).

This is pretty much near impossible to make work seamlessly since breakpoints are mapped by IL offset. deobfuscating things like control flow would completely change all the IL offsets making it impossible to remap a breakpoint from the deobfuscated code to the obfuscate code. The deobfuscation utilities would somehow have to produce some sort of mapping of original -> modified IL offsets for dnSpy to even try to get such a feature to work.

[mitchcapper]

produce a nice syntax highlighted stack trace using ICorDebugExceptionObjectValue.EnumerateExceptionCallStack

I don't know how you feel about adding another dependency but I love Ben.Demystifier and it isn't just some exception string prettier. I maintain another project and use it to massively cleanup stack traces but still have exact il offsets/md tokens/modules. https://github.com/mitchcapper/ProductionStackTrace/blob/enhancements/ProductionStackTrace/LogClasses.cs#L30-L60 . It is Apache2 so I believe is fine to use in GPL3 projects. I will need to look again at how the ST are sent to the UI, but the ProductionStackTrace package I maintain actually is designed to produce text based stacktraces that can be reparsed later to recover all the relevant information. I don't know its the best way to send the data through dnSpy as it is a bit hacky to stringify something only to immediately deserialize but it certainly works for ProductionStackTrace.

Potentially add a "Disable all exceptions"

I do not really see a use case for this.

Just as VS has "Except in this module" sometimes you care about an exception that is generic and may be thrown by several other libraries you don't want to be breaking on. I find this especially true with initial launches where you may have many try/catch exception testing.

Currently, dynamic modules are assigned IDs depending on load order.

I have certainly seen malware dynamically scrambling the names, but at the same time I would say it is more the exception than the rule. Maybe a user preference would be a good way to go? 3 items that come to mind for in memory modules would be module name, module size and module load order. Of course a hash of the module in memory might also be a good option if not truly generated on the fly (minus the perf of doing so too). Granted I assume the problem would be we don't just have a static list we are matching against (or knowing we have to match) so we don't know about collisions until it is too late (ie we need to know if the breakpoint is associated with this module at time of module load without knowing what is loaded later). To deal with collisions though we could simply store the dupe number by loader order as well. If my user preference is to track modules by name then when when a module is loaded it's ID would be its name followed by how many other modules currently loaded have that same name (so say System.Runtime#3). To determine at load time if it is the same module as before the same rules are simply applied. We don't know how many modules are to come but as we use load order first in gets spot #1 etc.

It is not foolproof but may provide better results in some situations. Given it is based on load order right now it isn't like we don't run into thinking it is the wrong module. It being a user preference would allow them to tweak as desired.

Ability to view linked deobfuscated assembly.

This is pretty much near impossible to make work seamlessly since breakpoints are mapped by IL offset

I think a starting point here (that would still be quite helpful) is simply a right click option to go to the first mdToken at or before the current cursor location in the reference assembly. It would require MD Tokens to be preserved but this would save currently of scrolling to the top of the method, holding alt, highlighting the token, copying the token, switching the active pane then going to that MD token. Again may be too niche for the core but I can see about trying a basic implementation as an extension.