Multithread and thread-safe TClientDataSet?

[snodev 0]

I new using TClientDataSet and would like some help.

 

My application fetch heavy stream data from an external web-socket like API (from multiple channels), proccess it asap in worker threads and then show to the user when requested... nowdays I do it manually with lists, arrays and iterate to populate the components (manually).

 

I want to improve and standartize the data in the application memory, the approach I thought is to go with TClientDataSet. The problem is that the application works with a heavy multithread system (which works pretty well, its not the problem), and I know that the TClientDataSet is not thread-safe, and could have alot of problems to deal with, so I would like to ask who already faced this problem before, if I on the right way.

 

My approach:

    •    The worker threads (which will receive the data to proccess) will write in a own TClientDataSet, only the worker thread may write data in its dataset;

    •    Operations (insert/update/delete) in worker thread TClientDataSet wouldn't be synchronized;

    •    Operations and cloning will be protected by a sync object (TCriticalSection like);

    •    The UI that must shown data may clone the worker thread TClientDataSet to have a read-only copy, which can be linked in VCL controls (like TDBGrid) and also iterated, filtered, etc.

 

Anyone that experienced this scenario (or similar) can say if I on the right way to avoid thread synchronization and deadlock problems dealing with TClientDataSet in a multi-thread system?

Edited February 12 by snodev

[Anders Melander 1894]

If by "clone" you mean a cursor clone, then the master and the clone shares the same data and modification from one thread will alter data seen by other threads (with no automatic synchronization AFAIR). So: Not thread safe.

If by "clone" you mean a distinct private copy of the data, then yes it's safe.

 

Apart from that, do you have a specific reason to use TClientDataSet instead of TFDMemTable?

 

TFDMemTable also has clonable cursors (with pretty much the same restrictions as TClientDataSet I would think). With TFDMemTable you at least have the source and can see for yourself what goes on.

 

[snodev 0]

Thanks for the reply @Anders Melander!

 

By "clone" I meant a cursor clone. Good to know that even cloning the cursor won't make it thread safe. I didn't know that.

 

What would be a "private distinct copy" that you said? You mean like TClientDataSet.CopyDataSet/TClientDataSet.CopyRecords?

 

No specific reason to not use TFDMemTable, I think they do almost the same thing with few differences. I'll take a look into that, but I must solve the thread safety before.

[Anders Melander 1894]

1 minute ago, snodev said:

You mean like TClientDataSet.CopyDataSet/TClientDataSet.CopyRecords?

Yes.

 

1 minute ago, snodev said:

I must solve the thread safety before.

I think you should drop the idea of the workers writing to a dataset. Write to a lock free queue instead and then read from the queue from a separate thread.

[snodev 0]

19 minutes ago, Anders Melander said:

I think you should drop the idea of the workers writing to a dataset. Write to a lock free queue instead and then read from the queue from a separate thread.

I do this currently. But instead of a dataset I use lists and dicts to hold the data, so my thread notify the mainthread (UI) to retrieve the data when necessary.

 

My idea is precisely to change this usage, using a single TClientDataSet (by worker thread) that can synchronize with the UI transparently... which also let me work with more flexible UI components (like TDBGrid)…

[aehimself 402]

The only benefit I see here is the usage of data aware controls.

A Dataset descendant is never going to be as efficient in storage or manipulation as arrays are. Also keep in mind that you are doubling the resources used every time you are cloning a Dataset, which - by your own words - is already pretty large.

 

To simplify/standardize I would create my custom data storage class with thread safe access and then look into DataBinding or writing my own helpers to display the data in this storage. If speed is not an issue you also can use an indexed TFileStream to cut back on memory requirements.