ANN: Parnassus Parallel Debugger

[Dave Millington (personal) 42]

A new IDE plugin aimed at helped debugging multithreaded apps! https://blogs.embarcadero.com/new-ide-plugin-parnassus-parallel-debugger/

 

The plugin:

• Displays all thread call stacks next to each other
• Lets you step over, run, etc a thread by itself, not the whole process
• Shows CPU usage by thread
• Editor integration to see where threads are executing, so you can easily see what other threads are in the same area of code
• and more...

 

This has taken me five years to write! That said, most of that was downtime due to working too much time to spend time on this - this release is the result of working weekends since August. It's version 1, and I plan to add some more features over the next few months.

 

[Mike Torrettinni 198]

On 12/15/2020 at 5:15 AM, Dave Millington (personal) said:

A new IDE plugin aimed at helped debugging multithreaded apps! https://blogs.embarcadero.com/new-ide-plugin-parnassus-parallel-debugger/

 

The plugin:

• Displays all thread call stacks next to each other
• Lets you step over, run, etc a thread by itself, not the whole process
• Shows CPU usage by thread
• Editor integration to see where threads are executing, so you can easily see what other threads are in the same area of code
• and more...

 

This has taken me five years to write! That said, most of that was downtime due to working too much time to spend time on this - this release is the result of working weekends since August. It's version 1, and I plan to add some more features over the next few months.

 

 

Excellent! I don't use threads often, but can't wait to try it out! Great job @Dave Millington (personal)! 🙂

[Dave Millington (personal) 42]

4 hours ago, Mike Torrettinni said:

Excellent! I don't use threads often, but can't wait to try it out!

Thanks Mike! I hope the editor markup makes the editor more pleasant / modern to use, even if you don't use multithreading.

 

This GIF shows it: https://i1.wp.com/blogs.embarcadero.com/wp-content/uploads/2020/12/editor-before-and-after-7862533.gif?ssl=1

 

(Note for that to really shine, you need to edit your editor colour scheme to turn off solid backgrounds - just check the 'Use default' checkbox for each element, like a breakpoint, that changes the background colour.)

[Stéphane Wierzbicki 45]

Thanks, this looks really promising.

I just want to inform you that IDE is unstable since installing this plugin

 

Delphi is runniing on a Windows 2016 TSE server.

 

 

 

[Lars Fosdal 1793]

I had a failing assert in some CPP code - which I sort of assume is this plugin, since I've never seen it before.

It happened on a breakpoint in an exception handler, and it brought down the IDE.

Quote

Assertion failure: "index <= items.n()" in ..\win32src\DBKIMPL.CPP at line 2773 debugging

 

Asserts in production code is not a good practice.

 

[Lars Fosdal 1793]

Repeatable. And - it brings down the IDE.
Will be checking if it is possible to make a small example when I get time.

[Lars Fosdal 1793]

@David Millington
Unable to reproduce in small example.  But, uninstalled the Parallel Debugger - and the Assert fail no longer happened.

Created a QP
https://quality.embarcadero.com/browse/RSP-31944

[Lars Fosdal 1793]

Just want to add that I loved the way it worked up until that error which hopefully can be quickly remedied.

[Dave Millington (personal) 42]

I suppose bug reports had to start arriving at some point...!

 

2 hours ago, Lars Fosdal said:

It happened on a breakpoint in an exception handler, and it brought down the IDE.

Quote

Assertion failure: "index <= items.n()" in ..\win32src\DBKIMPL.CPP at line 2773 debugging

 

Asserts in production code is not a good practice.

 

This, unfortunately, is in the debugger itself. I agree re asserts - I brought this up with the debugger team once, I will need to revisit.

 

In terms of fixing it, since it's a debugger bug, the main way is to work around it in the plugin. I have code already to do this for some areas. Could you create a QP report with the full call stack at the time of the assert, please?

 

One reason you might not have seen it before, even it being a debugger bug, is that the plugin uses the debugger a lot more than the IDE does by default - the IDE looks at a single thread at a time; this plugin scales debugger usage by the number of threads every single time you pause, step, etc. So a rare debugger bug may become more visible. Luckily we can work around many.

3 hours ago, Stéphane Wierzbicki said:

I just want to inform you that IDE is unstable since installing this plugin

Can you enter a QP with that call stack please?

 

As it is, it doesn't look like it's in the plugin itself. The comment I wrote above to Lars about seeing debugger issues may apply here.

[Lars Fosdal 1793]

6 minutes ago, Dave Millington (personal) said:

Could you create a QP report with the full call stack at the time of the assert, please?

David, The full callstack at the time of the assert is in the first attached QPInfo file of https://quality.embarcadero.com/browse/RSP-31944
Added the extract in a comment.

[Anders Melander 1815]

3 hours ago, Lars Fosdal said:

Asserts in production code is not a good practice.

Do you prefer Access Violations or other random errors?

[Lars Fosdal 1793]

Anders,

With Assert
If In_The_Shit then Panic

Without Assert
If In_The_Shit then DoOurBestToCope

 else BizAsUsual;

[Dave Millington (personal) 42]

4 hours ago, Lars Fosdal said:

David, The full callstack at the time of the assert is in the first attached QPInfo

Thanks Lars. That sentence was meant for @Stéphane Wierzbicki, to ask for a QP entry with the same info as in the comment.

 

4 hours ago, Lars Fosdal said:

Without Assert
If In_The_Shit then DoOurBestToCope

 else BizAsUsual;

This is what I usually try to do.

 

Years ago, there was the "crash fast, crash early" concept. I worked on software built like that and the idea was, if anything was wrong, bail and it would be fixed as a bugfix. Lovely concept, but it made the software seem really unstable to users. In practice, many things can be caught and either worked around, or at least shown as an error message or raised some other way, and slowly over time we changed the software to work like that. It functioned just as well and customers were happier.

 

Personally I think the debugger here, as a service invoked by the ToolsAPI, should never show UI but should exit safely and return an error that the invoker can handle as it sees fit.

[Darian Miller 366]

4 hours ago, Dave Millington (personal) said:

One reason you might not have seen it before, even it being a debugger bug, is that the plugin uses the debugger a lot more than the IDE does by default - the IDE looks at a single thread at a time; this plugin scales debugger usage by the number of threads every single time you pause, step, etc. So a rare debugger bug may become more visible. Luckily we can work around many.

 

 

Does that mean that more debugger issues will be found and fixed, or that your IDE Plugin will simply ignore more as time goes on?  😉

 

[Anders Melander 1815]

4 hours ago, Lars Fosdal said:

Without Assert
If In_The_Shit then DoOurBestToCope

 else BizAsUsual;

That's not what Assert is for.

An Assert is a verification that the assumption under which the code is written holds. If the assumption proves to be wrong, i.e. the Assert fails, then all bets are off. There is no "best effort" that can save it.

 

There's nothing to prevent the application from presenting an assertion error the same way as all other exceptions, but the fact is that it's an internal error that the user can do nothing about, and the application shouldn't just pretend that everything is fine.

Since I write a lot of framework and library code I use assertions a lot but at one of my former employers, a company in the medical device sector, I was told to remove all assertions because it "made their software look bad". Pointing out that that if there was assertion errors then looking bad was the least of their problems didn't go down well. Apparently they preferred random catastrophic  errors of unknown origin to immediate errors that could be traced back to their software.

[Dave Millington (personal) 42]

30 minutes ago, Darian Miller said:

Does that mean that more debugger issues will be found and fixed, or that your IDE Plugin will simply ignore more as time goes on?

More issues fixed, I would hope! We've actually found and already fixed or plan to fix several already, which were found by this plugin.

 

[mvanrijnen 123]

Looks good so far, sometimes a liitle bit slow with repainting the windows, but very usefull!

 

That there are some issues matches the host application 🙂

[Lars Fosdal 1793]

@Anders Melander - I prefer the method that David describes.  Return a documented error that can be handled.  The criteria of the assert means you actually know the situation and should be able to do so.

For me, asserts are contract validators that I use in debug mode.  If the asserts hits me, I love to see it straight away in the debugger.  I am not a fan of asserts in production code.

 

Silly car analogy:
Assert means you will drive off the road when you hit an unexpected turn.

Asserts off and condition validation and error handling means you hit the brakes instead.

[mvanrijnen 123]

17 hours ago, Anders Melander said:

That's not what Assert is for.

An Assert is a verification that the assumption under which the code is written holds. If the assumption proves to be wrong, i.e. the Assert fails, then all bets are off. There is no "best effort" that can save it.

 

There's nothing to prevent the application from presenting an assertion error the same way as all other exceptions, but the fact is that it's an internal error that the user can do nothing about, and the application shouldn't just pretend that everything is fine.

Since I write a lot of framework and library code I use assertions a lot but at one of my former employers, a company in the medical device sector, I was told to remove all assertions because it "made their software look bad". Pointing out that that if there was assertion errors then looking bad was the least of their problems didn't go down well. Apparently they preferred random catastrophic  errors of unknown origin to immediate errors that could be traced back to their software.

I've had an employer like that also. And then they go mad because tackling problems with the software while in production takes ages for consultants and after them the developers, because they get no errors 🙂 

 

 

[Tommi Prami 131]

Not most likely possible to implement for this plugin, but "Run until project code is reached" would be super cool.

Multithreaded app is munching a way, maybe looping for ever, but don't have clue where. One could pause app, and Run until project code is reached  (Or Sun until code open in IDE is reached (Named/in any thread).

Don't even know would this be possible for debugger to do, but it would be super cool. It is very hard to track down where the app is spending it's time, when it does it in a place you did not expect it to be.

 

Just throwing the idea out there, maybe it is possible or even easy to implement, dunno...

 

-Tee-

[Guest]

17 minutes ago, Tommi Prami said:

Just throwing the idea out there, maybe it is possible or even easy to implement, dunno...

It is near impossible to implement right, and when it will be right it will be useless and waste of time, why ?

There is two things to know 

1) Debugger does work with threads not code ( any code, machine or not), so while it is possible to make it recognize ranges of the project code in memory location it will make little sense when the project is build with runtime packages, in this case it will not be able to differentiate DLL or BPL, wasting time on such functionality doesn't worth it.

2) Threads, not all are initialized form your project, any assumption that every thread is application started is wrong, OS will add and control few of its own, you can see them in an empty VCL application, i am looking now and see that the VCL app started with 3 extra threads, and in less than one minute the OS will add extra 2 then end the first 3, so monitoring these is waste of time, in fact deep down Delphi RTL responsible for some of them but all.

 

Now to your question on how to find where the code responsible for specific loop execution, it is easy, just pause the application using the debugger and switch between these threads and examine the stack, this is the way to track threads and code, the stack is your friend and you should walk it back, in some cases the stack will show you code outside your project, the stack will have address belongs to OS DLL or unknown, these most likely are not yours, yet it might be, eg there is many OS callback functionality that start a thread to call your application like with many MM operation, to track these you should use an API monitor and track CreateThread itself, by doing so you will have the stack of the caller code which will lead you to your project code.

 

ps: real debuggers like x64dbg or OllyDbg have extra functionality, like you can look at the memory map in full then add breakpoint on access/read/write/execution, these breakpoint will stop the debugger watching full memory block.

[Tommi Prami 131]

23 minutes ago, Kas Ob. said:

Now to your question on how to find where the code responsible for specific loop execution, it is easy, just pause the application using the debugger and switch between these threads

Not true, in my experience, most of the time it'll pause in RTL code etc or outside of the scope (windows maybe dunno), and while stepping for ages you are sill somewhere at "unknown areas". 

I've never been so lucky with pausing that it'll pause in easy place to step through. Maybe this plugin will help a ton because you see them in parallel. Have not tested threading part of it yet.  

[Guest]

6 minutes ago, Tommi Prami said:

Not true, in my experience, most of the time it'll pause in RTL code etc or outside of the scope (windows maybe dunno), and while stepping for ages you are sill somewhere at "unknown areas". 

I don't want to handle "Not true" in wrong way, but may questions will be more productive

 

I ran an empty VCL project showing an empty form, then clicked pause on the debugger, hit CTRL+ALT+T calling the thread window here is it

Yes everything looks like unknown and there is no stack, double clicking on them some have this

that is a thread belongs to the OS, but this is the one you need this is clearly from the stack is the main thread

 

Can you see where it did started ?

Can you find its calling origin in your project code or RTL ?

 

ps: there is one with no calling stack at all, this belongs to the debugger you should discard it.

[Dave Millington (personal) 42]

14 hours ago, Tommi Prami said:

most of the time it'll pause in RTL code etc or outside of the scope (windows maybe dunno)

With this plugin, it will still pause in the same place (it won't wait for execution to move) but if there is a call stack entry which is in your source, which calls the code that is in, eg, Windows, it will show you that line. Ie it tries to show you the bit of your app you have control over when you pause.

 

14 hours ago, Kas Ob. said:

but this is the one you need this is clearly from the stack is the main thread 

This plugin handles this too - unless there is a reason to pause and make a non-main-thread thread the current thread (eg an exception was thrown in a thread) then when you pause, it will show you the main thread by default.

 

You can control this: it's called the 'thread of interest', which is the thread that will become the current thread when the process is paused. It just defaults to the main thread if it hasn't been set another way. Each thread's call stack has its own pause button, and clicking that makes that thread the thread of interest when pausing - ie, the function is "pause in this thread".

15 hours ago, Tommi Prami said:

"Run until project code is reached" would be super cool.

Super, super cool.

 

I agree, difficult to implement. I can think of a couple of ways - run until return in a loop, until the top call stack entry is in your source; or examining the stack and placing a breakpoint where it returns (hard, because you have to figure out where it will return to, ie the next instruction.)

 

It is worth thought. Thanks for the suggestion.

[Guest]

@Dave Millington (personal) Judging from frequent questions in this forum, i think there is a gap in public knowledge on how to utilize the debugger the right way, specially with multithreading, so i want to suggest that you make tutorials with videos to cover this in great detail, as there is so many developers using Delphi for many years, yet they wasted huge time on small bugs due to lack of understanding on how to use debugger, how read the stack, how thread naming should be used....

 

The plugin looks beautiful and useful but such help and educational material should cover debugging with or without it.