Can ICS thread cause memory violation? (EOutOfResources error)

[PizzaProgram 9]

1 minute ago, Dalija Prasnikar said:

Wouldn't it be better to autodetect whether component is created within background thread and set that property automatically rather than manually?

A great idea!

But it depends, how much time (CPU cycles) would it take to get the Current Thread's ID.

(the main thread's could be stored when the program start, and a simply if ... <>  ... takes only one cycle.)

[Kas Ob. 91]

This 

 

AlphaSkin is innocent here, though these leaks is definitely GUI objects, they are not freed and most likely with not closed handles.

How this happen is easy to explain (i think), your introduced code most likely copied and pasted form ICS demos (assuming) but in all cases you are missing the part of converting and adjusting the read size for those strings, hence you are overflowing buffers and writing in places you should not, and because FastMM allocate memory once and reuse its parts will allow such overflow, also both AlphaSkin and ICs( the way you are doing it by not caching and reusing components) have very memory intensive operation, the code keeps overflowing and destroying GUI handles and may be other things, and that making AlphaSkin after each failed operation to respawn/recreate these handles and allocate memory on them. 

 

ps: many of the OpenSSL objects have zeros in its contexts or will be zeroed for security reason, so the overflow is writing 0's which make AlphaSkin think it is not allocated objects and not throw an exception.

 

[Angus Robertson 526]

Quote

SSL header always changes by goverment request, so SSL has to be re-re-re-initialized anyway. 

 

I was talking about loading the OpenSSL DLLs and initialising the environment, not making a request.   The SslContext should be initialised when the thread starts, once, I thought I made that clear before.

 

Your application is making no attempt to check you are actually communicating with the government servers, no certificate chain checking. 

 

SSL servers accept hundreds of requests a minute without needing to re-initialise anything. 

 

Angus

 

[Dalija Prasnikar 1341]

11 minutes ago, PizzaProgram said:

But it depends, how much time (CPU cycles) would it take to get the Current Thread's ID.

Those few cycles mean nothing comparing to constructing the whole component over and over again. So this kind of safeguard can be easily implemented without having any negative side effects. The property itself could be left as-is, but initial value can be set in constructor depending on the thread ID value:

 

FMultiThreaded := TThread.CurrentThread.ThreadID <> MainThreadID;

 

[Fr0sT.Brutal 897]

7 minutes ago, Kas Ob. said:

but in all cases you are missing the part of converting and adjusting the read size for those strings

I looked twice and still don't see anything wrong.

[Kas Ob. 91]

On 8/19/2023 at 1:47 PM, PizzaProgram said:

written in Delphi7,

 

9 minutes ago, Fr0sT.Brutal said:

I looked twice and still don't see anything wrong.

 

[Fr0sT.Brutal 897]

9 minutes ago, Kas Ob. said:

written in Delphi7,

Still nothing.

[Angus Robertson 526]

ICS components could set the MultiThreaded property automatically, but the developer would still need to call IcsWndControl.ProcessMessages somewhere in the thread or the code would just stall, except in the simplest of applications.

 

Angus

[Dalija Prasnikar 1341]

14 minutes ago, Angus Robertson said:

ICS components could set the MultiThreaded property automatically, but the developer would still need to call IcsWndControl.ProcessMessages somewhere in the thread or the code would just stall, except in the simplest of applications.

Yes, you are right.  I am trying to solve one problem, while forgetting the broader picture.  Although it might be easier to figure out what is wrong if the thread would just stall, comparing to having random issues somewhere else in the application.

Edited August 21, 2023 by Dalija Prasnikar

[PizzaProgram 9]

1 hour ago, Kas Ob. said:

you are missing the part of converting and adjusting the read size for those strings, hence you are overflowing buffers and writing in places you should not

Please tell us the right answer!

How to get the right size under Delphi 7 ?

What exactly did I wrong?

lg       := Length(JsonUTF8);
SslHttpCli.SendStream.Write(JsonUTF8[1], lg);

Maybe lg-1 ?

[PizzaProgram 9]

4 minutes ago, PizzaProgram said:

Maybe lg-1 ?

Well, I've just tried that, and the server responded:

... Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])\n at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 951]"}]}

 

So I think my code was good how it was before.

(Except the ".MultiThreaded := True" part of course)

[Kas Ob. 91]

13 minutes ago, PizzaProgram said:

Maybe lg-1 ?

No !

 

The thing is : the bug might be not in this code you pasted and could be in other places even may be in ICS , Delphi 7 is not supported and things might be missed, also your suggestion of this "-1" explain that you need to research the difference between string (default) with WideString and AnsiString and UTF8String ..

 

Here if UTF8 is 2 bytes per char then it should be 

Quote

lg       := Length(JsonUTF8);
SslHttpCli.SendStream.Write(JsonUTF8[1], lg * 2);

or to make it safer when you are in doubt then, hard to me to advice this but 

Quote

//lg       := Length(JsonUTF8);
SslHttpCli.SendStream.Write(JsonUTF8[1], Length(JsonUTF8) * SizeOf(JsonUTF8[1]));

 

but as i said the problem might be not only here (i mean these 2 lines), you have to research and understand these strings and their size then adjust your code accordingly.

 

While i can't be sure 100% it is the problem but with high confidence it is the culprit but overflowing memory write.

[Dalija Prasnikar 1341]

10 minutes ago, Kas Ob. said:

Here if UTF8 is 2 bytes per char then it should be

UTF8String is always one byte per char, regardless of a Delphi version.

[Kas Ob. 91]

9 minutes ago, Dalija Prasnikar said:

UTF8String is always one byte per char, regardless of a Delphi version.

May be, !

 

I might be wrong here but there is bug with string copying somewhere.

 

Edited August 21, 2023 by Kas Ob.

[Kas Ob. 91]

Also was there a UTF8String in Delphi 7 ? and how it being handled in , what did Length() return, length in bytes or in chars ?

 

I have no idea.

[PizzaProgram 9]

1 hour ago, Angus Robertson said:

ICS components could set the MultiThreaded property automatically, but the developer would still need to call IcsWndControl.ProcessMessages somewhere in the thread or the code would just stall, except in the simplest of applications.

Sorry, but I don't fully understand that part yet.

Does it mean, that now, that I've enabled MultiThreaded property, I need to call this process too?

 

Also, if this is not the "right way" to automate thing:

FMultiThreaded := TThread.CurrentThread.ThreadID <> MainThreadID;

How about the opposite, at least stop the execution and warn us, programmers that we almost did a huge error:

// before post or get:
if (FMultiThreaded = False) and (TThread.CurrentThread.ThreadID <> MainThreadID) then begin
  raise Exception.Create('Warning! It is not safe to call this method from a background thread, if .MultiThreaded property is False! Exiting...');
  Exit;
end;

?

(and save us 200+ wasted hours for searching the error...)

 

2 minutes ago, Kas Ob. said:

I have no idea.

Thank you for your concern! I'm really happy you have looked into the code too.

You are right about "this COULD be a problem" at first site. I think too it's very risky to work with memory streams.

Luckily this part has been well tested. There are 5+ other programmers using the same code without problems.

 (We have all shared our knowledge with each other to finish until deadline and worked 3 month long to find out how to communicate with the gov. server.)

 

I was the only one using inside a Thread, and none of us knew about this "hidden" property. That's why the other guys had no problems.

[Angus Robertson 526]

17 minutes ago, Kas Ob. said:

was there a UTF8String in Delphi 7

ICS defines Utf8String and Unicode string for old compilers.  We still support Delphi 2007 since I still support my own commercial applications using it, but I stopped using Delphi 7 15 years ago. 

 

The original poster is making life more complicated for himself by using old ICS components. 

 

ICS v9 has a new sample Snippets, with several examples of making REST requests, one of which is: 

 

SslHttpRest := TSslHttpRest.Create (self) ;
    try
        try
            SslHttpRest.RestParams.AddItem('username', myusername);  
            SslHttpRest.RestParams.PContent := PContUrlEncoded;      
            StatCode := SslHttpRest.RestRequest(httpGET, myurl, False, '');  // sync request, no extra parameters
            AddLogText ('HTTP Rest Request Response: ' + IntToStr(StatCode)) ;
            if StatCode = 200 then begin
                AddLogText (SslHttpRest.ResponseRaw);                  
                AddLogText ('Address: ' + SslHttpRest.ResponseJson.AsArray[0].S['address']);    
            end;
        except
            AddLogText ('HTTP Error - ' + IcsGetExceptMess (ExceptObject)) ;
        end ;
    finally
        FreeAndNil (SslHttpRest) ;
    end ;

 

You completely ignore SslContext, encoding, Json, input and output steams, in most circumstances.

 

There is a POST snippet, but it uploads a file.

 

Angus

 

 

 

[uso 0]

6 hours ago, Dalija Prasnikar said:

UTF8String is always one byte per char, regardless of a Delphi version.

Off-topic: UTF-8 is 1 to 4 byte(s) per char depending on the respective code point.

 

@PizzaProgram
It is not a trival task to get the exact byte size of an UTF-8 coded string.

You need to put into consideration every code point. If you know your input languages, you may be able to approximate it - most of the time...

Edited August 21, 2023 by uso

[Dalija Prasnikar 1341]

45 minutes ago, Kas Ob. said:

Also was there a UTF8String in Delphi 7 ? and how it being handled in , what did Length() return, length in bytes or in chars ?

Yes, there is a UTF8String in Delphi 7. 

 

7 minutes ago, uso said:

Off-topic: UTF-8 is 1 to 4 byte(s) per char depending on the respective code point.

That is not what character means. It is how much code units are required for encoding Unicode code point in specific encoding.

 

Unicode character is called a code point and depending on encoding each code point can occupy from 1 to 4 bytes https://stackoverflow.com/questions/2241348/what-are-unicode-utf-8-and-utf-16

 

Character in Delphi can have different meanings:

 

One is Char type which can have size of one or two bytes depending on Delphi version (pre 2009 or 2009+). There are also other character types which can have different byte sizes.

 

Another one is used in terms of character type used in various string types: character size for AnsiString and UTF8String in all Delphi versions is one byte, for default string type in pre Unicode versions size of character is one byte, for Unicode versions size is two bytes.

 

Length function always returns the size in characters for particular string type which means the actual number of bytes will depend on the character size for that string type. for UTF8String length will always equal the number of bytes because size of character in UTF8String is one byte, regardless whether some Unicode characters (code points) require more that one byte when encoded.

 

19 minutes ago, uso said:

It is not a trival task to get the exact byte size of an UTF-8 coded string.

Yes it is. Just call the Length function on it.

 

20 minutes ago, uso said:

You need to put into consideration every code point. If you know your input languages, you may be able to approximate it - most of the time..

You only need to consider how many bytes are in codepoint if you are parsing such data byte by byte, where accessing UTF8String by index will not necessarily give you complete information about Unicode code point stored as you will get only one byte of it.

[Fr0sT.Brutal 897]

3 hours ago, Kas Ob. said:

you need to research the difference between string (default) with WideString and AnsiString and UTF8String ..

You'd be right if it was WideString / string after D2009 but Utf8String is just an AnsiString having char size = 1 byte with special codepage. Not talking about code points (or symbols, letters, etc) here because they're not relevant. This exact code gets "chars" which are in fact bytes (char - in Delphi terms, an item of a utf8 string variable, could have no sense as it could be just a part of encoded Unicode "character" aka code point).

Edited August 21, 2023 by Fr0sT.Brutal

[uso2sc 1]

3 hours ago, Dalija Prasnikar said:

Yes, there is a UTF8String in Delphi 7. 

 

That is not what character means. It is how much code units are required for encoding Unicode code point in specific encoding.

 

Unicode character is called a code point and depending on encoding each code point can occupy from 1 to 4 bytes https://stackoverflow.com/questions/2241348/what-are-unicode-utf-8-and-utf-16

 

Character in Delphi can have different meanings:

 

One is Char type which can have size of one or two bytes depending on Delphi version (pre 2009 or 2009+). There are also other character types which can have different byte sizes.

 

Another one is used in terms of character type used in various string types: character size for AnsiString and UTF8String in all Delphi versions is one byte, for default string type in pre Unicode versions size of character is one byte, for Unicode versions size is two bytes.

 

Length function always returns the size in characters for particular string type which means the actual number of bytes will depend on the character size for that string type. for UTF8String length will always equal the number of bytes because size of character in UTF8String is one byte, regardless whether some Unicode characters (code points) require more that one byte when encoded.

 

Yes it is. Just call the Length function on it.

 

You only need to consider how many bytes are in codepoint if you are parsing such data byte by byte, where accessing UTF8String by index will not necessarily give you complete information about Unicode code point stored as you will get only one byte of it.

I apologize for being that dumb about my coding environment.

Never the less, I think "UTF-8 is always one Byte per Char" is confusing if it comes to Unicode.

But you clarified, so I go in shame and keep quite. 🖖

Edited August 21, 2023 by uso2sc

[uso2sc 1]

Back to On-Topic:

@PizzaProgram

If I were you, I would check the creation and destruction of ressources.

It's mentioned already -- heading over the stream of messages led me to the fact it may got lost.

Concentrate on the image/ressource creation and destruction - not on strings and ICS in the first place. GDI has been a problem in my company since MFC got popular (I know that MFC is not Delphi, but they share some problems - mostly hard to find bugs).

Reuse your resources, resize if memory is a concern, do not recreate them.

If this helps, you know how to keep the business running and you won time to solve the real problem.

GDI and non-UI Threads are "special" friends...

If I'm completely wrong, slap me.

[PizzaProgram 9]

I'd like to summarize last week's work and experiences:

- I was very happy to read that Setting .MultiThreaded := True; could solve all my problems, but it did NOT. 😞
  (I've checked the code and saw running step-by-step the "thread safe processMessage".)

- GDI Memory Errors are the same, if ICS is running in background thread. No problems, if ICS is not running.

 

I've also reverted my code back to old AlphaSkin 2019 version, where I know it was stable. Same problem occurs.

So the only thing left to try to load OpenSSL once only:

On 8/21/2023 at 11:31 AM, Angus Robertson said:

2 - More seriously, the ICS components are being created and perhaps destroyed for each HTTPS request made, which is probably the cause of the memory leak, and is also highly inefficient. 

 

Specifically, OpenSSL is being loaded automatically by the components when the SslContext is automatically iniitialised by the request starting, and perhaps being unloaded when the request ends.  The SslContext is designed as something to be shared by components, initialised once and then reused.  Or OpenSSL can be loaded once when the program starts, to allow use with multiple SslContexts, in servers for instance that use multiple certificates.  Many of the ICS samples show how to load OpenSSL early.

 

So my question is:

 - How do I ensure ICS 8.70 is never unloading OpenSSL ?

 - while using it in both the main and multiple background threads separately?

 - and how do I "reset" safely TSslHttpCli to be re-used? (both buffers, headers, and everything else...)

 

1. Is it enough if I create a fake TSslHttpCli + TSslContext component pair in the main thread that is destroyed only when the program ends? (and may never get used...)
2. Does ICS know that it should not unload OpenSSL in the background thread? (has it something like reference-counting?)
3. Should I create only ONE sslContexts, or 1-1 for each thread?

 

Thank you very much for any further help!

Edited August 27, 2023 by PizzaProgram

[Angus Robertson 526]

I never said MultiThreaded would solve your problem, I said it was a mis-use of ICS for threads not to use it, it might work in simple cases, but not in most applications. 

 

I've already answered most of your other questions with previous comments. 

 

ICS how no knowledge of threads.  It does reference count loading OpenSSL, but that only works if you free components correctly, so OpenSSL also gets unloaded correctly.

 

Clue: what happens with errors when there is a large amount of code in a try/finally/end, and when you don't close connections first. 

 

Angus

 

[Fr0sT.Brutal 897]

14 hours ago, PizzaProgram said:

Does ICS know that it should not unload OpenSSL in the background thread? (has it something like reference-counting?)

Windows automatically does it. So to avoid unloading you can create a life-time socket or load DLLs manually