Dalija Prasnikar

44 minutes ago, havrlisan said:

Besides these, what else could be an issue?

Plenty.

User interfaces in general (not just Delphi) are not thread-safe. Thread safety comes with hefty price. You would have to lock just about everything and it would significantly slow down the UI. Also, UI controls are complex and have plenty of interactions. Due to that complexity, making UI thread-safe would be complicated and error-prone. Not only that, but UI needs to interact with OS layer where UI controls are also not thread-safe as such where you cannot access the same control from different threads. That means if you would construct one control in background thread to be used in main thread afterward, not only you would have to make sure that interaction with Delphi UI side of controls have to be thread-safe, but control would have to avoid triggering any code that interacts with OS part which is almost impossible.

UI is not thread-safe by design and that is something that will not change as it is not practical nor beneficial. 

There are some parts around handling images that could be fixed to fully support processing images in background threads and then passing them to the UI for presentation.

11 hours ago, David Schwartz said:

This kind of stuff is B-O-R-I-N-G and SHOULD BE AUTOMATED, IMHO, but I don't know of anything that does it.

It takes 10 minutes to create such generator. I have few that generate boring code based on my coding patterns. I just need to write field declaration, copy-paste that to generator and I get all property declarations and assign method out of it. Yes, I need to adjust some of them afterwards, depending on their type, but that is only because my generator is extremely dumb parser and I never bothered to improve it as I don't need it that often.

I mean, if you find AI useful for some tasks, then by all means use it. But it is not a magical tool, and it has serious limitations. If you are aware of those, then you can get the most of it, but don't expect any intelligent solutions out of it.

The problem happens when outer For loop consumes all available threads and hits maximum where no new threads are being allocated for newly requested task. And then if inner tasks cannot get free thread to run, they cannot complete their job.

I have test case that will break more easily as number of needed threads is higher, and sleep also increases the time inner thread runs and gives more time for outer for loop to consume all available threads. Using separate pool for inner for loop solves the problem. This can still be an issue if you have code that indirectly uses nested parallel loops.

begin
	var counter := 0;
  var pool := TThreadPool.Create;

	const COUNT = TThreadPool.Default.MaxWorkerThreads * 4;

	TParallel.For(
		0,
		Pred(count),
		procedure(i: Int64)
		begin
			TParallel.For(
				0,
				Pred(count),
				procedure(i: Int64)
				begin
					TInterlocked.Increment(counter);
          Sleep(10);
				end
			, pool);
		end
	);

  Writeln(counter);
  Readln;
end.

Probably the best solution for parallel for loops that can spawn too many tasks would be to create dedicated thread pool for each loop. The cost of creating a thread pool is not big when dealing with longer running tasks. And for fast running tasks, like in this example, parallel loop is overkill anyway as it can run slower than simple for loop in single task.

11 minutes ago, Rollo62 said:

Thanks, my goal was to solve all special, unforseeable OS hardware exception cases by double-nesting, not necessarily the inline issue.

Inline behaves as if you have written the code directly within the try..except method. Just like double nesting does not help with the inlined procedure it will not help with any other code that is written directly in try..except block. Only wrapping the code that can cause exception in additional function, procedure or method is the solution.

28 minutes ago, Rollo62 said:

Not sure if that is flawless for any kind of exceptions, any ideas ?

If the G! is inlined then exception will not be caught by second level try..except block. It can only be caunght by exception handler outside the method where exception happens.

var
  P: ^Integer = nil;

procedure G1; inline;
begin
  P^ := 42;
end;

procedure TForm1.G2;
begin
  try
     try
       G1;
     except
       Memo1.Lines.Add('Inner');
     end;
  except
    Memo1.Lines.Add('Outer');
  end;
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  try
    G2;
  except
    Memo1.Lines.Add('Proc');
  end;
end;

In other words, in above code exception will be caught by exception handler around G2 -> Proc

If we remove inline directive then exception will be caught by Inner exception handler.

I don't know whether LLVM can automatically inline functions that are not explicitly marked for inlining, but I would expect that we shouldn't worry about that as this would break exception handling not only for Delphi, but for other languages, too.

10 hours ago, Brandon Staggs said:

Just to be clear, the nil object method call to Clear was deliberately designed to cause an access violation.

Don't worry, it was evident that this was just an example to show the failure.  

Obviously in real code one would not set object to nil and then use if Assigned() to check whether object is nil before using it. But if you have scenario where some function call can return nil object as valid response - for instance finding object that satisfies condition in a collection, and returns nil if the appropriate object is not found, then you have to check for nil before using it. Once you have valid object, calling methods on it, even if they raise exceptions will be caught by surrounding exception handler. 

10 hours ago, Brandon Staggs said:

I am more concerned about the non-obvious things that happen outside of our control. In fact, my primary concern (not stated) is dealing with access violations inside of library calls; libraries we have to use but to not have control over. And in fact, I don't actually need to worry about that in particular, since calling a function in a dylib that leads to an AV will return to our function and execute the finally block anyway.

Again, in such cases you will be calling some library function, and unless you are calling something on nil object this will be fine. The potential issues may arise if the library itself (Delphi ones) is not written with LLVM based compilers in mind, where some exceptions will won't be caught by appropriate exception handlers within the library code. If that happens then the library code has to be fixed, you cannot solve the problem from your code.

2 hours ago, Brandon Staggs said:

So, what is the right way to handle this? Obviously we don't want to call methods on an uninitialized object reference, but what pattern can be followed on Mac to make sure a lock is released after running some code that could potentially fail outside of our control?

This issue happens on all LLVM backed compilers because LLVM is not capable of catching hardware exceptions unless exception happens within another function. 

You find more information on my blog post https://dalijap.blogspot.com/2018/10/catch-me-if-you-can.html I never got around to write the sequel, but the implications is that literally anything that all exception handling implicit or explicit is broken in such situations.

The solution is that you either wrap your code in separate function that will not have any exception handling within, and then the caller will be able to catch and handle raised exceptions. Another way of solving issues is to avoid code that can trigger such hardware exception and raise Delphi exception if code does not satisfy some requirement as explained in https://docwiki.embarcadero.com/RADStudio/en/Migrating_Delphi_Code_to_Mobile_from_Desktop#Use_a_Function_Call_in_a_try-except_Block_to_Prevent_Uncaught_Hardware_Exceptions

So in the context of your example, you should either check whether object is nil before trying to use it (this would actually be general advice, as accessing nil object, depending on the code, on Windows does not guarantee that you will get AV).

Note. I don't know what exactly following quote from documentation about macOS means: "structured exception handling (__try/__except) is not available". Linked page talks about hardware exceptions, but I am not sure whether there are some other implications here besides what I said in context of LLVM. At the moment I don't have my development environment set up in a way that would allow me to verify behavior on Mac. 

2 hours ago, Brandon Staggs said:

Also, I am concerned about the implications of this with TTask and the thread pool. Last I looked (and I do need to verify this anew), Delphi uses code in finally blocks to handle completion of tasks and passing off to new tasks, etc. If there is an external exception in one of those tasks, how can the worker threads not become corrupt?

While bugs are always possibility, RTL appropriately handles hardware exceptions in cross-platform code. 

2 hours ago, Brandon Staggs said:

Also, a rant: I feel like a trailblazing pioneer on the frontier of some unexplored land. Searching this issue out has led me to one and only one thread where someone brought this up on Stack Overflow, and that thread was incorrectly marked as a duplicate (neither of the linked answers are relevant to the question in the slightest).

I removed wrong duplicates and added appropriate one. 

1 hour ago, Die Holländer said:

Thanks Dalija ! Yes, It seems i'm just lucky and wonder why its working OK under Windows but indeed it solved the issue.

On Windows Delphi uses FASTMM memory manager and on Other platforms it uses POSIX memory manager. I am not that familiar with inner workings of a POSIX memory manager, but FASTMM allocates larger chunks of memory from the OS which is then used for sub-allocating object instances and other heap based data. When you release an object, data in its memory location can be still intact and accessing such data does not always crash. In such cases memory is held by FASTMM so there will be no crash in the OS side because for the OS it is valid memory allocated by the program.

If you use FASTMM in full debug mode during the development, then accessing such invalid object will be noticed by FASTMM and shown as error.

I cannot verify the issue at the moment, but since TJsonTextReader uses TStringReader it might be better to reverse the releasing order. It is likely that TJsonTextReader destructor is accessing the TStringReader instance during destruction process. Such code would also be wrong on Windows, but you are just lucky that it does not fail there.

LJsonTextReader.Free; 
LStringReader.Free;

1 hour ago, emileverh said:

A compiler hint would be nice if you forget to explicit mention the type in this case.

Until now most of my code did not add the type by using inline vars ( strings, integers, etc. ). I you forget to mention the type with interfaced objects then you have a memory leak

Compiler cannot give you appropriate hint at that place. The problem is that it doesn't know whether you need to use interface reference or not because some classes that have interfaces have reference counting disabled and are used through object references.

If compiler would give a hint that would result in many false positives and at the end such hints would be useless. This is similar situation as the one where developer declares regular variable, but uses wrong type. There are no hints for such scenarios either.

The inline variable in question will be of type TCar as its type will be inferred from type used in constructor: TCar.

In such code where object instances are reference counted, you will need to explicitly specify correct type: ICar because reference counted instances need to be stored in interface references for proper initialization of reference counting mechanism.

var car: ICar := TCar.Create; 

40 minutes ago, Rollo62 said:

Anyway, you have to check out for yourself.

I already checked. 

Domains where it might work better are in a creative domain where there are no right or wrong answers, but then it is just a parrot that repeats someone's thoughts.

40 minutes ago, Rollo62 said:

The AI developments and technology were sleeping ever since on a low level and only in the last few years some significant, exponential changes happened in that field.

I know the power of exponential growth and that it might show reasonable results soon.

So many people nowadays are getting hot on AI and working on that topic, don't you agree we will see a real chat or human-like AI soon, by fixing the last 5%-10% of issues ?

I do and I don't say that chatGPT is the last step of the evolution, its only the start.

When people talk about AI they like to focus on intelligence. But ChatGPT is just a language model, with some corrective mechanisms on top. 

What it means?

It means there is no intelligence involved. It is just fancy "text completion" model that uses probability to determine which word should follow next in some sentence. Even when the model grows, and its probability analysis improves, it is still just dumb "text completion" It will never be able to reason about what it writes. And again, it may get better in avoiding ridiculous mistakes, but you are still stuck with original issue: you cannot trust the information it gives you, because it may not be correct and you will not be in position to realize that.

13 minutes ago, Rollo62 said:

Maybe its a question of different expectations ?

1. You might expect answers like from a nobel price winning professor, which is fair enough, so that you can use the data 1:1 as is.

 

2. I would expect answers like from an half-educated sales assistant or a passenger on the street, giving only 25-50% of useful information but leading into the right direction,

    which leads to useful thoughts from where one can search deeper if he likes.

Main problem wit ChatGPT  is that if you are asking about something you don't know well, it can lead you in completely wrong direction and you will never know it because you lack the expertise. And it will lead you there with a confidence of a master. And it is not wrong only occasionally, it is wrong most of the time. If you like wasting your time on wrong leads and hints, then it is absolutely the right tool for any job.

17 hours ago, Wagner Landgraf said:

When developing, did you check https://github.com/spinettaro/delphi-event-bus and if yes, why it didn't serve your purpose?

I was already working on my event bus when I discovered DEB. I would still work on mine even if I knew about DEB sooner, because I also wrote about it in my book and I cannot reason about other people's thought process  

It is also published as part of book code examples, but it is a feature that deserves separate repository not tied to the book.

When it comes to why DEB would not fit my purpose (if I ignore the book part), is that it has more complicated code and more features that don't necessarily bring anything of value to me (for instance, using attributes for setting subscriptions), but on the other hand have big impact on performance. My even bus also supports all types as messages, so there is additional performance boost and avoiding unnecessary memory allocations when message fits into integer or small record type. And at last, my even bus has waitable subscription, so you can wait that all dispatched messages are being fully processed. This is critical feature for more complex cleanup scenarios - yes, it can also be achieved through other means, but it is easier when it is built in feature, and I use it a lot.

And the most personal reason is that my code is more compatible (similar setup and calls) with my old thread-unsafe observer framework, and I can more easily and gradually refactor old code to using new one.

9 minutes ago, Oberon82 said:

Delphi is dead end. Pay to see roadmap is ridiculous.

It is not pay to see the roadmap. There is no longer a roadmap as such. But those that participate in beta testing have a bit more information about what is coming next in immediate and for some parts the release after that sooner than it is publicly available.

29 minutes ago, Lars Fosdal said:

Why?  It is stupid and annoying.

Don't ask me. I don't know why.

2 hours ago, Lars Fosdal said:

Something else I find annoying.

EMBT insists on mailing me offers for products that I already subscribe to. 

 

Why? 

 

What are the odds that someone would add another license for Delphi at Black Friday, Week, Month,  Xmas time,  insert price reduction opportunity here?

Because their marketing database is not connected to customer database.

58 minutes ago, Brandon Staggs said:

Indeed, and when it is actually done, working properly, it will be great to have. Right now, I feel like I have walked backwards a decade or more when I use the IDE, which is almost every day... But downgrading is not really an option either, for numerous reasons!

Currently one of the greatest is that when it chokes on something it needs to be killed and restarted. This can be done more easily by creating menu item in IDE. See https://stackoverflow.com/q/74164165/4267244

1 minute ago, Brandon Staggs said:

I don't begrudge them trying to find ways to increase their revenue at all. I want them to make more money because I need the tools and want to keep using them for a long time.

 

However, quality is a serious problem with Delphi right now. The latest version is the most frustrating Delphi I have used in the last decade because of intellisense/code completion failing every five minutes, and Ctrl+Click almost never working (I have retrained my muscle memory to use Ctrl+G instead which seems to work more often, but still not 100%). It would seem to me that getting more people to test early would be helpful, but maybe I am wrong.

 

I could even see paying more to have some kind of active interaction with the people deciding what features to implement, etc, but just for beta testing a patch? I don't get it.

Trust me. Number of people testing is the least problem right now.

One of the issues with LSP is that it is a huge endeavor. Which on its own would be such an issue if they could have left the old one running beside it like in Sydney. However, old tech interfering with LSP integration and they had to remove it.

It seems like Premium subscription gives you "advanced" beta access. For Standard it says there is access to beta for each major version release.

So reading between the lines and knowing that current beta is just an update, this might mean that only Premium subscribers will have access to this beta.

https://www.embarcadero.com/support

5 minutes ago, limelect said:

@Dalija Prasnikar see in one of my answers I even

free all used components DB and link nothing helped.

Once the program reaches max memory nothing helps

But if I use 2 programs just for testing see above it works.

Even on the same DB

Then the problem is not in this particular code but all the other code that consumes all available memory. The general advice for reducing memory footprint I gave earlier still stand.

Yes, using separate applications will help because each will have its own memory limit if they are running on 64-bit OS.

30 minutes ago, limelect said:

"Don't use memory stream to load data to rich edit. Create temporary file and write the data into that file and then load the data from the filestream."

I tried all ready  weeks ago

The question is whether or not you have done that correctly. The moment you write the data to file, you need to release memory of all the temporary data that was used in that process before you try to load that into control.

  MyMemStream :=  ProjectsFDTable.CreateBlobStream(ProjectsFDTable.FieldByName('Description'),bmRead);
  try
    TMemoryStream(MyMemStream).SaveToFile('t.txt'); // using absolute path is preferred
  finally
    MyMemStream.Free;
  end;
  Memo1.BeginUpdate;
  try
    Memo1.LoadFromFile('t.txt');
  finally
    Memo1.EndUpdate;
  end;

On 12/9/2022 at 6:03 PM, limelect said:

MyMemStream:=  ProjectsFDTable.CreateBlobStream(ProjectsFDTable.FieldByName('Description'),bmRead);
 AdvRichEditor.BeginUpdate;
AdvRichEditor.LoadFromStream(MyMemStream);
AdvRichEditor.EndUpdate;

Don't use memory stream to load data to rich edit. Create temporary file and write the data into that file and then load the data from the filestream.

However, no matter what you do, at some point you will run into memory issues with 32-bit application if your application consumes a lot of memory. I suggest you switch to 64-bit. If you cannot switch to 64-bit, then the only other option is optimizing other parts of your application to reduce overall memory usage. 

First step for optimizing would be getting rid of all memory leaks - they can cause fragmentation which can be a huge problem.

Next, reducing total memory allocation, by releasing anything that is not used at the time.  Be careful with creating local components that have owner which you don't explicitly free. They will not show as memory leaks, but every time you call such method your memory consumption will increase.

If that does not help, then you can try reducing memory load by avoiding default string type which is Unicode since D2009 (you haven't said which Delphi version you use) and use UTF-8 encoded strings - UTF8String and convert to Unicode only when you need to display data to the user - this is really the last option as you may need to write a lot of supporting code to avoid accidental conversions to and from Unicode strings as they will slow your application down and use extra memory in the process.

What is main? 

You are most likely corrupting a memory by accessing same data from main thread and background thread. Also you shouldn't work with any visual control from the background thread without calling the TThread.Synchronize or TThread.Queue

Also you are working with pointers, so it is possible that you are accidentally overwriting something.

4 hours ago, alogrep said:

I found where the problem occurs (see apptached image). However the call stack doesn't go inside the procedure where the problem actually happes. This is the code

There is a thread involved. Most likely not all data that needs to be protected is protected and another thread is either releasing the object while it is still in use, or there is memory corruption because multiple threads are messing with the data.

But there is not enough code to pinpoint the problem.