We use DUnitX and it discovers all our silly mistakes before release

[Lars Fosdal 1793]

Fresh code: https://github.com/VSoftTechnologies/DUnitX

Typical code that we test with DUnitX:
- value to string and string to value
- lookup functions
- generic classes

- formatters

as well as database integration tests (CRUD)

 

How do you guys use it?

[FredS 138]

6 hours ago, Lars Fosdal said:

use it

Essentially all system wide methods are run through DUnitX now.
This is how I know that RSP-15040 is still in Rio 10.3.1, I use that when anyone asks why I built my own string helper.

 

Depending on complexity I use TestInsight while coding.

[Der schöne Günther 316]

This might come across as ignorant, but I never really got behind what DUnitX does better than DUnit. For sure, it does things differently. But better? What are the advantages? Why should I consider migrating all DUnit test projects to DUnitX?

[Stefan Glienke 2018]

Among a few minor things it allows for a more declarative approach of writing tests by using attributes to provide the test data.

In a classic DUnit test if you have to test an algorithm with 20 different value combinations you need to write 20 tests (either as different methods or by putting them into one test).

 

However DUnit can be pimped to allow the same so you don't need to migrate (*) - since DUnit is kinda abandoned development wise Vincent decided to roll his own library instead of modifying and possibly cleaning up the code from DUnit which dates way back. 

 

(*) I did that way back even before DUnitX existed: https://stackoverflow.com/a/9006662/587106 and later also put that into a unit of Spring4D.  As you can see in the screenshot you have multiple tests shown although there is only one method declared the extension takes care of producing them so you can run them individually (if for example certain data produces a failure) while you are fixing it.

 

Another thing (which personally never bothered me much) is that with DUnitX you can write testcase classes without inheriting from a base class (this can also be done with an extension for DUnit - did it but never put it anywhere because it was not useful for me).

 

The last thing that I remember is the fact that DUnitX can do is have fixture setup/teardown - they only run once even if there are multiple tests in the class - classic DUnit runs Setup/TearDown methods before and after every single test - guess what? You can also plug that onto DUnit.

 

My personal opinion: do automated unit tests but it does not matter what framework you are using. I use DUnit with the before mentioned extensions and we do so at work because its powerful and compact. And regardless which one you are using - use them with TestInsight and make them part of your CI. 😉

Edited February 18, 2019 by Stefan Glienke

[David Heffernan 2353]

I know I go on about this but it's really important, in my view, to stress that the testing framework doesn't discover the bugs. It's the developer that writes good tests that discover the bugs. 

 

You need a testing framework to help manage the test code. But the strength of the test is always the quality of the test code rather than the framework. There are plenty of libraries that ostensibly have test suites but are full of bugs because the test suites are incomplete.

[Stefan Glienke 2018]

32 minutes ago, David Heffernan said:

There are plenty of libraries that ostensibly have test suites but are full of bugs because the test suites are incomplete.

If we just had some easy to use and nicely integrated into the IDE way to measure code coverage ...

[Vincent Parrett 762]

6 hours ago, Stefan Glienke said:

If we just had some easy to use and nicely integrated into the IDE way to measure code coverage ...

 

@Stefan Glienke weren't you working on something like that? That is something I'd pay for! 

[Tommi Prami 131]

12 hours ago, FredS said:

Essentially all system wide methods are run through DUnitX now.
This is how I know that RSP-15040 is still in Rio 10.3.1, I use that when anyone asks why I built my own string helper.

 

Depending on complexity I use TestInsight while coding.

That issue is closed, is there duplicate, or why is that? Any idea? Issue states fixed in 10.2 if I understand...

[Uwe Raabe 2063]

 

11 hours ago, Stefan Glienke said:

If we just had some easy to use and nicely integrated into the IDE way to measure code coverage ...

5 hours ago, Vincent Parrett said:

 

@Stefan Glienke weren't you working on something like that? That is something I'd pay for! 

I showed this during EKON last year. Unfortunately I couldn't find the time working on it again. It heavily lacks documentation, but perhaps it can serve as a start.

https://bitbucket.org/uweraabe/codecoveragemagician

Edited February 19, 2019 by Uwe Raabe

[Daniel 417]

I have moved the discussion about the TStringHelper into a separate topic:

https://en.delphipraxis.net/topic/738-mis-behaviour-of-tstringhelper/

 

[luebbe 26]

I have used https://github.com/DelphiCodeCoverage/DelphiCodeCoverage for a while.

The results are ok. You can run your unit test with it on a build server and import the output file. The disadvantage is that it multiplies the run-time for the unit test by a factor of at least ten.

Our tests ran for an hour instead of five minutes. So we turned it off again. Nobody misses it. It could be ok though to run a separate job on the build server once per night just for the code coverage.

 

[Stefan Glienke 2018]

That it what I looked into some years ago and started refactoring out some for me unnecessary code and cruft.

 

You have to know that a coverage profiler is basically a debugger that puts a breakpoint into every single executable line (or at least those where you specified you are interested in).

Now finding out what lines that are is the interesting part - the other is presenting the data nicely inside the IDE when executed under TestInsight (at least for me).

[Wagner Landgraf 43]

Just bumping this topic to ask: regardless of all the issues, are you guys still using some code coverage tool (if yes, which one), or just do not use it at all? I also miss a good tool for that.

[Fr0sT.Brutal 900]

I tried https://github.com/DelphiCodeCoverage/DelphiCodeCoverage and it's pretty good

[Der schöne Günther 316]

Excessively writing unit tests for new parts, but not using code coverage tools for the whole project. More than 50% of our code is 10-20 years old and simply cannot be tested.

 

Yes, I have read Working Effectively with Legacy Code by Michael C. Feathers (goodreads.com), how to get out of this. But ain't nobody got time for that.

[Stefan Glienke 2018]

On 8/27/2021 at 4:47 PM, Fr0sT.Brutal said:

I tried https://github.com/DelphiCodeCoverage/DelphiCodeCoverage and it's pretty good

Working on some better integration and result format that can be consumed by state of the art tooling such as https://github.com/danielpalme/ReportGenerator or https://github.com/marketplace/codecov but thats a side project that I don't spend much time on.

Also still fighting some spurious crashes with DelphiCodeCoverage that result from it putting its breakpoints where it really should not (I suspect some incorrect debug information from the map file).

Edited August 31, 2021 by Stefan Glienke

[Fr0sT.Brutal 900]

2 hours ago, Stefan Glienke said:

Also still fighting some spurious crashes with DelphiCodeCoverage that result from it putting its breakpoints where it really should not (I suspect some incorrect debug information from the map file).

I also experienced crashes on a code that runs CreateProcess but didn't investigate that yet. Alas, the coverage doesn't check multiple branches sharing the same line so its results aren't 100% reliable. Moreover, Delphi compiler strips out unused functions even in full debug mode so untested lines seem identical to ignored ones.

[Guest]

On 2/18/2019 at 8:30 PM, David Heffernan said:

I know I go on about this but it's really important, in my view, to stress that the testing framework doesn't discover the bugs. It's the developer that writes good tests that discover the bugs. 

Yes! TDD would be completely detrimental to my development processes and my personal creativity. Kept my hands off that. Will keep doing that.

[Guest]

Code coverage can only yield furtive information in real use-cases, no?

So for that to be productive it must be implemented like madExcept/EurekaLog/Clientside logging? No?

That would bog down the end-users devices (if you are not running it server-side, there sampling could yield something, but faces with a real problem that would not be the first thing to try).

So, my question, or rather my theory that i would want you-all to falsify: code coverage in a test/lab environ will never yield anything useful, apart from the rubber-ducking you already gain from the code-compile-test cycle.

It is a bit like telemetry, in order to make it useful, you have to get into your users pants. No? Abusing privacy and GDPR and a sleuth of other humane practices?

I prefer to have an intimate relation with my client's bosses, middle-bosses and end-users. If they trust me, they will call when appropriate. And i can do my systemic  problem-analysis devoid and free of idiotic speculations (a forum, bug-reporting-system, esp using e-mails, is usually pissed down with end-user speculations that i have to put time and effort into answering because of bad office-politics and the usual self-centred middle-layer-bosses fuckups.

I do use code coverage analysis and tools but for optimisation of critical part of what i deliver. Very seldom, though, usually common sense is cheaper and works better.

Humbly. Me.

[Uwe Raabe 2063]

4 minutes ago, Dany Marmur said:

Code coverage can only yield furtive information in real use-cases, no?

Well, Code Coverage can also tell you which parts of a tested method was left out and thus needs more tests. OK, not that much helpful when you don't do tests.

 

It is not meant to be executed in the customers production use, there you are right. That is more what collecting the areas actually used are for.

[Guest]

@Uwe Raabe, aha! I missed that it was discussed in the context of TDD. Now that makes sense. Thanks!

[Vincent Parrett 762]

6 hours ago, Dany Marmur said:

Code coverage can only yield furtive information in real use-cases, no?

So for that to be productive it must be implemented like madExcept/EurekaLog/Clientside logging? No?

If only this were actually possible.. but your customers would not be happy with your application's performance - codecoverage tools are literally debuggers that put a breakpoint on ever line of code and record when the breakpoint is hit - it slows things down dramatically. So only really practical to use it against unit tests. Our unit tests for FinalBuiilder only take a few seconds to execute, but when running with code coverage it takes about 20 minutes - I tend to turn it on only when making major changes or writing new tests.  

[Laurens 1]

On 8/31/2021 at 11:22 AM, Stefan Glienke said:

Working on some better integration and result format that can be consumed by state of the art tooling such as https://github.com/danielpalme/ReportGenerator or https://github.com/marketplace/codecov but thats a side project that I don't spend much time on.

Also still fighting some spurious crashes with DelphiCodeCoverage that result from it putting its breakpoints where it really should not (I suspect some incorrect debug information from the map file).

Are you using a recent release? In march this year I have fixed an issue with spurious crashes in multi-threaded code where the breakpoint was not correctly cleared (and could cause the CPU to execute a NOP where a real instruction should be).

On 8/31/2021 at 1:36 PM, Fr0sT.Brutal said:

I also experienced crashes on a code that runs CreateProcess but didn't investigate that yet. Alas, the coverage doesn't check multiple branches sharing the same line so its results aren't 100% reliable. Moreover, Delphi compiler strips out unused functions even in full debug mode so untested lines seem identical to ignored ones.

I have seen the same result once, but several years ago, do you a minimal workable example? If so, I would appreciate if you create an issue so we can investigate and fix it.

 

The DelphiCodeCoverage tool is a good tool to start with. I prefer to integrate it to more common tools like SonarQube. We use this for analyzing our Delphi, C# and TypeScript code in one tool.

[Fr0sT.Brutal 900]

3 minutes ago, Laurens said:

I have seen the same result once, but several years ago, do you a minimal workable example? If so, I would appreciate if you create an issue so we can investigate and fix it.

The test involved my wrapper for createprocess with threads, IO pipes etc, it's somewhat complex to extract MRE. I have it in my backlog but hardly will put my hands on it in near future...

[Stefan Glienke 2018]

7 minutes ago, Laurens said:

Are you using a recent release?

No, I took the state from SourceForge years ago and removed all the cruft that I did not need and refactored some things. Also due to the fragmentation of the project it was kinda hard to see what is actually the most recent one. I just learned that probably your fork is.