GUI automation tool for Firemonkey apps ?

[salvadordf 32]

I've been asked to find a GUI automation tool for a Firemonkey desktop application for Windows.

There're several tools that work for VCL applications but I can't find anything for FMX apps.

Many well known tools can't find the FMX controls and they can only simulate mouse and keyboard events, compare screenshots, OCR, etc.

All I could find is this StackOverflow question :
https://stackoverflow.com/questions/30197921/how-can-i-automate-delphi-firemonkey-ui-testing-without-access-to-source-code

That question has 2 suggestions :
- Use the "FireMonkey Accessibility Package".
- Build a "Test harness".

The "FireMonkey Accessibility Package" for Delphi 10.2 is here :
https://cc.embarcadero.com/Item/30780
It seems to work with Delphi 10.3 if we apply this fix :
https://quality.embarcadero.com/browse/RSP-25532

My question to all of you is this :
What GUI automation tool do you use for Firemonkey applications?

Are we limited to building "Test harnesses" or using the "FireMonkey Accessibility Package"?

Thanks!

[Dave Novo 51]

related to this, one key to the testing harness that we wrote for VCL is the SendInput windows API   Does anyone know if there is an analog for FMX that works across platforms? We really only care about OSX for mac. But knowing android and iOS would be helpful for others.

[Andrea Raimondi 13]

Hi! 

What, exactly, are you looking to test? 

A lot can be accomplished, for example, by unit testing even for the UI using mock objects. 

 

A

[Dave Novo 51]

I am not sure what @salvadodrf wants the automation for, but for me it is for testing. The debate over the utility of GUI unit testing has been beaten to death all over the internet. The advantages/disadvantages over mocking your entire system can be found all over. Without getting into a religious debate over whether or not GUI testing is good, I am just wondering if there already exists FMX tools to allow you to drive the GUI for an FMX app like the SendInput app does for Windows. Of course, SendInput alone cannot do everything, you have to combine with VCL methods to find coordinates of controls etc, but SendInput is really the key and I have not found an analog under FMX.

[salvadordf 32]

@Andrea Raimondi It would be used to test the whole application but they would like to be able to check every control property. I also asked this question in other places and it seems that all we have is mouse and keyboard events automation. Some tools like "Sikulix" also have OCR and image finding functions.

 

[TurboMagic 92]

One issue with this is, that there didn't ask enough people about this yet. That's at least what SmartBear the company behind TestComplete told me.

Why don't all users liking to have such a thing ask all these vendors about it to show there's demand?

I'd start with TestComplete and with Ranorex, which even belongs to Idera but can only do VCL so far, as far as I know.

[Guest]

Darn, i did not even now there was GUI driven test-suites for VCL!

Anyone used such a setup with DevExpress controls?

On 6/16/2020 at 6:27 AM, Dave Novo said:

Without getting into a religious debate over whether or not GUI testing is good

Please do! And esp. in a Delphi 10+ VCL/FMX context.

 

[My actual need are in the webbrowser-javascript domain - that did not work well because of too much trottling and DOM maninpulation, i use knockout.js]. But having  favourite in the VCL domain would not hurt.

[Dave Novo 51]

@Dany Marmur - the crux of the debate is that GUI testing is hard to maintain. If you do it badly, every time you make a slight change to your GUI, tons of test cases start failing for no reason. Also, GUI testing has all sorts of issues to do with knowing when the action is completed. i.e. if you click on a button, then check for a result, how long do you wait to check for the result. Often test code doing the GUI automation is in another thread, and its hard for that thread to know when the application is done what its supposed to do and then test. If you check some state too early, then the test will fail because the app is done. But then you try on your own computer, and the test passes perfectly, because the app was slightly faster on your computer compared to the testing farm. If you do GUI testing well, all these are less of an issue.  Some people like the very localized testing of very specific issues that you can get from a very specific test case.  With a GUI test, things can fail for all sorts of reasons, and its hard to track down.  But that is also the benefit, you get failures in things you did not think of.

[Anders Melander 1782]

1 hour ago, Dave Novo said:

the crux of the debate is that GUI testing is hard to maintain.

Amen to that.

I've been involved in attempts to implement GUI testing using TestComplete three times in three different companies. Each time the project (the GUI test part) was abandoned after using massive amounts of resources writing and maintaining the test scripts (and by massive I mean several man years). How are one ever going to keep up if it takes a test engineer hours to update the tests for every small UI, workflow or timing change a developer makes. It's much more feasible to write test instructions and have the QA department do the tests manually. They can use TestComplete and the like for simple keyboard/mouse record/playback but the result verification is done manually.

 

1 hour ago, Dave Novo said:

you get failures in things you did not think of.

I remember a place where we outsourced the testing to an Indian company. I was quite impressed by the amount of bugs they initially found with no domain knowledge whatsoever and very little knowledge about how the application was supposed to work. From what I could see from the bug reports they had simply tried pressing every possible key combination or just smashed their hand into the keyboard in every single dialog. They found "features" we didn't even know we had. It made me think of the infinite monkey theorem.

 

I have yet to see automated GUI test implemented successfully anywhere.

[Dave Novo 51]

Well, if you come to Pasadena, you can visit and see it running beautifully 😀 We have a bank of 5 servers. Each one is 24 cores and 200GB of ram. They each run about 75 headless windows machines each in a citrix environment. We have 4000 visual tests that are run multiple times per day and it takes about 1.5 hours to complete a full run on that bank.

 

The key is that we wrote our own GUI automation over the years. All we do is send keystrokes and mouseclicks to the UI. However, there are several caveats that we have developed over the years to make it work.

1. the test case NEVER contains screen coordinates. We have methods like ClickOnMenu(aMenu:TMenu,'File|Open'). That  method queries the TMenu and finds coordinates of the 'file' menu item, clicks on it and finds coordinates of the open menu item, then clicks on it. Similar is ClickOnControl(form:TForm;control:TControl). The testing system finds the open form, finds the control. finds the coordindates and then sends a mouse click in the middle of the control. There are other variants if you dont exactly want to click in the middle. So in the test is says ClickOnControl(form1, form1.button1). That way, if the developer changes the form, i.e. moves button1 around, then the test is fine. If they delete button1, the test stops compiling. That being said, we wrote tons of these types of methods that are specific for the controls we use, to accommodate their oddities, like virtual string tree etc. It did not happen overnight.

 

2. We have a small delay after every input except certain cases. This small delay is generally enough. We also have hooked into the idle processing of the program so after sending a mouseclick / keyboard press it waits for the onIdle so the program is not processing more messages. There are a few more tricks as well. However, the upshot is that we rarely have a problem with timing any longer. For really variable processes, we have our GUI automation thread wait on a TEvent in the program and the program triggers the TEvent when the process is done so that the test knows to continue. But that is rare. Not so similar from an end user, that just sits there waiting for the screen to refresh.

 

So, the upshot is that it was a lot of work, and likely only worth it if you are going to maintain the same program for a long time. But it can be done and today, new developers in our company have no idea the hard work that went into it and they just write scripts in a fairly high level way and the infrastructure does the rest.

 

The test cases find the problem they are designed to test, about 50% of the time. The rest of the failures are because of bugs that are involved in things the test was doing to get to the point where it was trying to test.

 

The other upshot is that we very, very rarely have regressions. The vast, vast majority of bugs that make it to production are in new features that have not (in hindsight) been sufficiently testing.

 

 

[David Schwartz 426]

Tangential story here: I started working at this new company earlier this year. My first programming task, as it were, was to move this app from a WinXP VM to a Win10 VM. That's really all it was supposed to be. 

 

It seemed to work fine in XP but kept randomly failing in Win10. I guess several people had "researched" it for a while and had concluded there were certain problems, and they gave me a bunch of directions and things to test. Regardless, they thought it should only take a couple of days but may need a little tweaking in the app that was controlling things. So I started to dig in.

 

First discovery: they were using a Windows app -- Version 1.0 -- that ran OCR on a big batch of forms that were scanned in and delivered to us as PDF files. This was a batch-mode process, so they had a control program that stuffed keystrokes and mouse events into the Windows message queue to control the app. I guess they had a lot of variance in the timing because everything had a 5-sec delay. Except the OCR function itself, which had a 45-second window. It frequently timed-out.

 

Second discovery: all of that "research" people did was useless. It turns out, they were making observations on a folder thinking it contained files that were being scanned and then imported into the system. Turns out the stuff in this folder was more of a "discard" pile of slag left over as part of the OCR process. The fact that the app kept crashing at the same apparent point made it look like it was choking on one particular thing, which appeared to be some failure to rename a file. Later on, I figured out the failure was because it failed to rename MOST of the files, whereas the last one they thought was the problem was actually done correctly.

 

An early thought we had was maybe this old app had problems running in Win10, so we contacted the company to buy an upgrade. The company has a page with a "Buy Now" button, but it goes to a form with a bunch of questions that are sent to a marketing guy who was trying to determine what the optimum solution might be. We just wanted the latest version of the same app, but they refused to sell it to us. We didn't say exactly how we were using it, and I suspect our use violated their license anyway, which some others at the company also suspected. Instead, the marketing guy was trying to sell us a hosted version that we estimated would cost us about $10k/mo. Never mind that it wasn't HIPAA or PCI compliant, which we required. Their self-hosted solution was around $75k from what I could tell. The only thing I could see for sure was that the old app wasn't 100% reliable running in the Win10 environment; it def. had problems.

 

So I started looking around for another OCR solution. I found one from a British chap that cost under $100 and was a Delphi library (not a DLL) that had no license restrictions that might affect our use.

 

I hooked it up, did some testing, and it ran amazingly fast. When I cropped part of the page to restrict the scanning to the part with data we wanted, it went even faster.

 

The original solution took 6-8 hours to process most batches of data we got, while this one completes the same input data in 20-40 minutes.

 

The other thing I discovered about the original code was that the error handling was worthless. (I've found that in a lot of our code now.) I got it working and discovered a TON of errors were happening AFTER the OCR process ran, and that implied about 25% of the data it scanned was not valid or did not correlate with known data as it should have. I guess it was getting a lot of false hits because it was scanning the entire page, which often had weird artifacts that matched things we were looking for, only they were in the wrong place on the page, so the neighboring data it captured was wrong or missing.

 

I rewrote the whole damn thing in a way that allowed us to switch between the two OCR engines. The new library worked a whole lot better. The older one still worked; it was far more robust and didn't fail as often, but it still took hours vs. minutes.

 

Management wasn't happy because it took me way longer than they expected. I was surprised they weren't concerned about the opaqueness of errors, poor scanning quality, or the overall execution time. Or even the fact that all of their initial assessments were 100% off-base.

 

Oddly, someone who heard about how the original code worked expressed interest in maybe adapting it to do GUI-based testing. The mere thought of using the method they used here to drive this OCR app, but for more generalized testing, just made me shudder.

 

Most of this software does not trap run-time exceptions, so it just terminates with useless messages like: "error reading item(0); attempt to read from address 00000026", or "error in numeric conversion: '' is not a valid number". This is after consuming tens of thousands of input records. Their solution: just step through it with the debugger. In 99% of the cases, it's bad input data.

 

 

 

 

Edited October 12, 2020 by David Schwartz