Creating FMX controls in a background thread

[havrlisan 24]

I'm aware that the creation and manipulation of FireMonkey controls are not thread-safe, but what does it take for Embarcadero to fix/reimplement to enable this behavior? I've gathered that the main issues most certainly arise from using the TList class without locks, such as TFmxObject.FChildren and TComponent.FComponents. Besides the properties, there's also the TMessageManager.Default which returns an instance of TFixedMessageManager, which is also not thread-safe (because it also uses TList without any locks).

Besides these, what else could be an issue?

[Dalija Prasnikar 1396]

44 minutes ago, havrlisan said:

Besides these, what else could be an issue?

Plenty.

 

User interfaces in general (not just Delphi) are not thread-safe. Thread safety comes with hefty price. You would have to lock just about everything and it would significantly slow down the UI. Also, UI controls are complex and have plenty of interactions. Due to that complexity, making UI thread-safe would be complicated and error-prone. Not only that, but UI needs to interact with OS layer where UI controls are also not thread-safe as such where you cannot access the same control from different threads. That means if you would construct one control in background thread to be used in main thread afterward, not only you would have to make sure that interaction with Delphi UI side of controls have to be thread-safe, but control would have to avoid triggering any code that interacts with OS part which is almost impossible.

 

UI is not thread-safe by design and that is something that will not change as it is not practical nor beneficial. 

 

There are some parts around handling images that could be fixed to fully support processing images in background threads and then passing them to the UI for presentation.

[havrlisan 24]

26 minutes ago, Dalija Prasnikar said:

Plenty.

 

User interfaces in general (not just Delphi) are not thread-safe. Thread safety comes with hefty price. You would have to lock just about everything and it would significantly slow down the UI. Also, UI controls are complex and have plenty of interactions. Due to that complexity, making UI thread-safe would be complicated and error-prone. Not only that, but UI needs to interact with OS layer where UI controls are also not thread-safe as such where you cannot access the same control from different threads. That means if you would construct one control in background thread to be used in main thread afterward, not only you would have to make sure that interaction with Delphi UI side of controls have to be thread-safe, but control would have to avoid triggering any code that interacts with OS part which is almost impossible.

 

UI is not thread-safe by design and that is something that will not change as it is not practical nor beneficial. 

 

There are some parts around handling images that could be fixed to fully support processing images in background threads and then passing them to the UI for presentation.

Thanks for the detailed explanation. So if I understood correctly, FMX is designed in such a way that it is nearly impossible to separate the rendering part (which is done in the main thread) from the rest of the code? If, for example, one would rewrite the framework from scratch, would it be possible to separate the whole control creation and manipulation from the actual canvas drawing? To my understanding, the only part that must be run in the main thread is the drawing on canvas (which directly uses graphics API).

[Lars Fosdal 1792]

The challenge is that to do the drawing, you need a long series of calls to retrieve device info as well as setting up values and UI elements.

When accessing the UI in it self is not threadsafe in the underlying OS - it becomes pretty much impossible.

Windows UI APIs - not threadsafe

Linux Gnome GDK/GTK+ - not threadsafe

Android UI toolkit - not threadsafe

Apple iOS UIKit - not threadsafe

 

and creating the control structure is such a miniscule task in itself.

Of course you can build data structures, models, etc in a thread, but the real work happens towards the UI - which cannot be made treadsafe.

[Dalija Prasnikar 1396]

1 hour ago, havrlisan said:

Thanks for the detailed explanation. So if I understood correctly, FMX is designed in such a way that it is nearly impossible to separate the rendering part (which is done in the main thread) from the rest of the code? If, for example, one would rewrite the framework from scratch, would it be possible to separate the whole control creation and manipulation from the actual canvas drawing? To my understanding, the only part that must be run in the main thread is the drawing on canvas (which directly uses graphics API).

In theory when writing and GUI framework from scratch, it could be possible to make that framework thread-safe.

 

But drawing is not the only part that needs to run in the main thread. Any interaction with the OS - creating OS handles (like windows, graphic objects...), mouse interaction, keyboard interaction, OS notifications, needs to run in the main thread. too. Also access to all resources would have to be protected by locks. That means every single field and property inside control. So while background threads would be running doing some work on those controls, UI thread would not be able to use them. Now it may look that this would still be solution if you want to just create and destroy controls in background threads, but again to do that safely ALL access ALL the time have to run through locks, That means even if you never use background threads, accessing and working with UI controls would be much slower ALL the time. Multithreading is not cost free. 

 

I don't know what is your intent behind the question. Again, the fact that FMX is not thread-safe is not a flaw in its design, or anything similar. It is because thread-safe UI frameworks are basically non existent (I am saying basically, because like I said in theory they are possible, but I don't know about any). All major OS-es have thread-unsafe UI frameworks: Windows, Linux, Android, iOS, macOS - simply because making those thread-safe does not make sense. 

 

Thread safety is not just about throwing locks around some lists, but also defining what interactions need to logically happen without any interruptions from other threads. That would mean locking would have to extend to much broader code besides just accessing lists, and the more complex the scenarios (and UI are extremely complex frameworks with complex code paths) the more chances are that there will be bugs and concurrency issues that cannot be easily resolved.

 

Patching FMX to add thread-safety is impossible as it is not just adding few locks here and there, it would have to be thorough rewrite. If you want to write your own framework, you are always free to do so, but I would have to ask "You and what army?" 

[havrlisan 24]

2 hours ago, Dalija Prasnikar said:

In theory when writing and GUI framework from scratch, it could be possible to make that framework thread-safe.

 

But drawing is not the only part that needs to run in the main thread. Any interaction with the OS - creating OS handles (like windows, graphic objects...), mouse interaction, keyboard interaction, OS notifications, needs to run in the main thread. too. Also access to all resources would have to be protected by locks. That means every single field and property inside control. So while background threads would be running doing some work on those controls, UI thread would not be able to use them. Now it may look that this would still be solution if you want to just create and destroy controls in background threads, but again to do that safely ALL access ALL the time have to run through locks, That means even if you never use background threads, accessing and working with UI controls would be much slower ALL the time. Multithreading is not cost free. 

 

I don't know what is your intent behind the question. Again, the fact that FMX is not thread-safe is not a flaw in its design, or anything similar. It is because thread-safe UI frameworks are basically non existent (I am saying basically, because like I said in theory they are possible, but I don't know about any). All major OS-es have thread-unsafe UI frameworks: Windows, Linux, Android, iOS, macOS - simply because making those thread-safe does not make sense. 

 

Thread safety is not just about throwing locks around some lists, but also defining what interactions need to logically happen without any interruptions from other threads. That would mean locking would have to extend to much broader code besides just accessing lists, and the more complex the scenarios (and UI are extremely complex frameworks with complex code paths) the more chances are that there will be bugs and concurrency issues that cannot be easily resolved.

 

Patching FMX to add thread-safety is impossible as it is not just adding few locks here and there, it would have to be thorough rewrite. If you want to write your own framework, you are always free to do so, but I would have to ask "You and what army?" 

This is exactly what I was asking for, thank you very much for the explanation. I got some things mixed up in my head and started concluding that such implementation may be beneficial, hence why I asked about it. My initial thought was that TBitmap should be creatable and drawable in a background thread, but that doesn't seem possible because of global Canvas lock, and IIRC it also uses the TMessageManager (and that's where I expanded my opinion to all other components). That seems like a good idea, no?

Edited February 22, 2023 by havrlisan

[Lars Fosdal 1792]

TBitmap can be created and manipulated in a thread, but there are caveats.

https://www.google.com/search?q=delphi+tbitmap+threadsafe

[havrlisan 24]

6 minutes ago, Lars Fosdal said:

TBitmap can be created and manipulated in a thread, but there are caveats.

https://www.google.com/search?q=delphi+tbitmap+threadsafe

If by caveats you're referring to forced canvas locking or unpredictable behavior, then "can be created" doesn't really mean anything.

Creating TBitmap creates TBitmapImage, which calls CanvasClass.InitializeBitmap() method that initializes a bitmap handle that is specific to the graphics engine. In Windows case, the class is TD2DBitmapHandle, and wouldn't you know, this is in its constructor:

FContextLostId := TMessageManager.DefaultManager.SubscribeToMessage(TContextLostMessage, ContextLostHandler);

and it all goes down the drain.

[Lars Fosdal 1792]

Looks like you are right, and according to this thread, 

Quote

The alternative is to not use TBitmap at all, just use the Win32 API directly for all GDI operations in a worker thread.

 

[aehimself 396]

What if the form and all the controls are created runtime by the thread? Then once the thread is completed the main thread can show it.

 

I recall a post about a background thread being able to build and even show a form in VCL; it's just two threads should not access it the same time.

Even if so, maybe that is not even applicable under FMX...

 

Anyway, I thought it worth to ask.

[Dalija Prasnikar 1396]

2 hours ago, havrlisan said:

My initial thought was that TBitmap should be creatable and drawable in a background thread, but that doesn't seem possible because of global Canvas lock, and IIRC it also uses the TMessageManager (and that's where I expanded my opinion to all other components). That seems like a good idea, no?

From OS perspective bitmap operations can be performed in background threads and then bitmap could be transferred to UI for painting. This is where FMX has some issues that could and should be fixed. Same goes for working with 3D graphics primitives.

[Dalija Prasnikar 1396]

53 minutes ago, aehimself said:

What if the form and all the controls are created runtime by the thread? Then once the thread is completed the main thread can show it.

This is not possible because constructing forms and controls also interacts with the OS in thread-unsafe way. For instance creating window has thread affinity and window belongs to a thread that creates it. You cannot construct window in one thread and pass it to another. If you remove initialization code that interacts with OS and other thread-unsafe parts, controls would need to have separate initialization that would need to be synchronized with the main thread. This would be rather messy as different controls have different requirements and construction of controls would get additional layer of complexity which would also only slow things down. Forms streaming also uses global lock mechanism to protect global namespace (data modules can be constructed and streamed in background depending on used components), so adding synchronization during loading forms would also open potential for deadlocking, and preventing the deadlocks would also require additional calls and checks which would again slow everything down.

53 minutes ago, aehimself said:

I recall a post about a background thread being able to build and even show a form in VCL; it's just two threads should not access it the same time.

Even if so, maybe that is not even applicable under FMX...

This never worked in VCL and it does not work in FMX. Threading issues can be hard to catch. Just because you can create some components in background thread without immediately bumping into issues does not mean such code is bug free.

[Brandon Staggs 277]

5 hours ago, havrlisan said:

This is exactly what I was asking for, thank you very much for the explanation. I got some things mixed up in my head and started concluding that such implementation may be beneficial, hence why I asked about it. My initial thought was that TBitmap should be creatable and drawable in a background thread, but that doesn't seem possible because of global Canvas lock, and IIRC it also uses the TMessageManager (and that's where I expanded my opinion to all other components). That seems like a good idea, no?

This is a design flaw in FMX.

[Fr0sT.Brutal 900]

On 2/22/2023 at 11:20 AM, Lars Fosdal said:

Windows UI APIs - not threadsafe

Why not? It is built on thread-safe message queue. Talking about VCL, if a component is hanging in the air (without a parent), not using globals like Application and Screen and not touched concurrently there's a chance it could be handled in bg thread.

[Lars Fosdal 1792]

@Fr0sT.Brutall Why has been answered in the thread. 

[Fr0sT.Brutal 900]

16 minutes ago, Lars Fosdal said:

Why has been answered in the thread. 

No it wasn't. I consider WinAPI UI pretty thread-safe.

[Lajos Juhász 293]

16 minutes ago, Fr0sT.Brutal said:

I consider WinAPI UI pretty thread-safe.

You can have a strong belief that the Earth is flat but that will not change the fact that isn't.

 

Yes, I know that the Flat Earth Society has members from all around the globe.

[Fr0sT.Brutal 900]

7 minutes ago, Lajos Juhász said:

You can have a strong belief that the Earth is flat but that will not change the fact that isn't.

Similarily, you can have a strong belief that banana is fruit but that will not change the fact that isn't.

How about proofs? My position: as long as communication to WinAPI control is done via Post/SendMessage (which is the 99% of cases), it is thread-safe. Your move.

Edited March 16, 2023 by Fr0sT.Brutal

[Dalija Prasnikar 1396]

19 minutes ago, Fr0sT.Brutal said:

My position: as long as communication to WinAPI control is done via Post/SendMessage (which is the 99% of cases), it is thread-safe.

There is way more in UI thread safety than thread safe communication between controls.

 

In Windows API windows can communicate across thread-boundaries, but each window still belongs and is bound to a thread where it is created and that thread's message queue. This basically prevents you to create window control in one thread (background) and use it in another (main).

[A.M. Hoornweg 144]

1 hour ago, Fr0sT.Brutal said:

No it wasn't. I consider WinAPI UI pretty thread-safe.

No it isn't. Because it isn't stateless.  For example, if you draw a shape by a sequence of lines:

MoveToEx (hdc,100,100,NIL);

LineTo (Hdc, 200,200);

LineTo(Hdc, 10,100);

 

the commands draw contiguous line segments because each segment starts at the position where the previous command ended.  If multiple threads draw in parallel on the same Hdc, that is not the case anymore.

 

 

 

[Fr0sT.Brutal 900]

1 hour ago, Dalija Prasnikar said:

This basically prevents you to create window control in one thread (background) and use it in another (main). 

Why prevents?

I underscore - we're talking about a control fully isolated inside a non-main thread - this means it has no parent window belonging to another thread of the app. With pure WinAPI window and bg thread running message loop, what issues could arise when talking to that control from main thread only by messages?

47 minutes ago, A.M. Hoornweg said:

No it isn't. Because it isn't stateless.  For example, if you draw a shape by a sequence of lines: 

Fair point. However, if this procedure is executed only as reaction to WM_PAINT that is only launched by WndProc, seems it couldn't be interfered because WndProc calls seem to be serialized by OS. Though MS docs don't clearly say so (they only state SendMessage directly calls WndProc) my test shows that while WndProc is running, SendMessage's from another threads just wait for it to finish.

 

I could be wrong here but I want to know what exactly makes WinAPI thread-unsafe.

[programmerdelphi2k 237]

A question about a doubt that torments me sometimes:

if visual controls are inherited from the operating system (the parent classes), at its core, and used by Delphi in its pertinent classes, like TEdit(... TWinControl), after all:

Which thread actually creates the visual control (i.e. the object used by Delphi), the MSWindows API or the pseudo class in Delphi?

 

And, if the final object (if it comes from AP Windows) is created by the OS, and then perfected by Delphi (in its pertinent classes), there would not be a transfer of "ownership" between the threads of the operating system and the thread Delphi application main?

 

Another thing: if basically everything comes via message from the operating system, and intercepted by the application (thread main) Delphi, a very tight semaphore could really degrade future actions in the interface, no?

[Dalija Prasnikar 1396]

2 hours ago, Fr0sT.Brutal said:

Why prevents?

I underscore - we're talking about a control fully isolated inside a non-main thread - this means it has no parent window belonging to another thread of the app. With pure WinAPI window and bg thread running message loop, what issues could arise when talking to that control from main thread only by messages?

Yes, you can have control that completely runs in the different thread that runs its own message loop. That is different scenario from the one I said - constructing the controls in background thread and then using them in main thread. Using (inside main thread) and merely communicating (sending messages are different things. There are boundaries between controls that belong to different threads. In other words there are different levels of thread-safety. Since Windows API has thread affinity, it is not fully thread-safe. 

 

In practical terms having such application means that you need to deal with more complexities in places where you otherwise wouldn't, Using UI in same thread means you can use SendMessage to perform operations quickly because in the context of same thread SendMessage will avoid the message queue and will simply run the windows proc. However, when you SendMessage to window that belongs to different thread, SendMessage will go through message queue. But, SendMessage is blocking call and in complex to-way communication between two threads with SendMessage you can easily cause deadlock. OK, solution to that is that you don't use SendMessage, but only non blocking communication, which is slower. and that would probably work if you handle all those windows manually directly through Windows API and you exactly know to which thread they belong to. Assing any abstraction layer on top your be a nightmare (this is where we start stepping into Delphi RTL and visual frameworks).

 

3 hours ago, Fr0sT.Brutal said:

I could be wrong here but I want to know what exactly makes WinAPI thread-unsafe.

Again, thread affinity is what makes it thread-unsafe. Now, you may have different viewpoint on what thread-safe is in this context, but not being completely thread-unsafe is still far from being completely thread-safe.

 

Some additional reading:

 

https://devblogs.microsoft.com/oldnewthing/20051010-09/?p=33843

 

https://devblogs.microsoft.com/oldnewthing/20051011-10/?p=33823

 

https://devblogs.microsoft.com/oldnewthing/20051012-00/?p=33803

 

https://devblogs.microsoft.com/oldnewthing/20051013-11/?p=33783

 

[Dalija Prasnikar 1396]

5 minutes ago, programmerdelphi2k said:

Which thread actually creates the visual control (i.e. the object used by Delphi), the MSWindows API or the pseudo class in Delphi?

The thread from which you call Windows API. When you call API like CreateWindow or you create object in Delphi there are no other threads involved. (Unless specific API or object is specifically meant to handle thread functionality or is part of some asynchronous API)

[programmerdelphi2k 237]

39 minutes ago, Dalija Prasnikar said:

The thread from which you call Windows API.

thanks @Dalija Prasnikar

Ok, but if we use "Create" from Delphi, as it is the default and widely used for this task, and... Delphi calls (uses) the class of this control defined in MSWindows, for example, TEdit uses the definitions from the MSWindows Edit control, so aren't we actually calling a task defined in MSWindows and transferring its owner to Delphi?
... that is, an OS API thread (performs the actions of a lower level) transfers to a thread (main) in Delphi (of a higher level)... isn't that what happens?

I don't know if I managed to explain my train of thought...