Can ICS thread cause memory violation? (EOutOfResources error)

[Angus Robertson 574]

The usual way to ensure OpenSSL is only loaded once is to drop an SslContext on the form, or create it once when the program starts. 

 

Ideally you initialise it once as well, since that is when the DLLs are loaded, and check to see if the DLLs are actually available to report errors before requests start. 

 

The high level components and servers have multiple SslContexts so in that case you call OverbyteIcsWSocket.LoadSsl when the form is created, and OverbyteIcsWSocket.UnLoadSsl;  when it's destroyed, as illustrated in numerous samples. 

 

You normally set several global variables before calling LoadSsl depending on whether you want old or new OpenSSL versions to be loaded, or from a specific directory, whether you need the legacy DLL, or checking the code signing signature for malware, again in all those samples.

 

Angus

 

[Kas Ob. 121]

15 hours ago, PizzaProgram said:

So my question is:

 - How do I ensure ICS 8.70 is never unloading OpenSSL ?

 - while using it in both the main and multiple background threads separately?

 - and how do I "reset" safely TSslHttpCli to be re-used? (both buffers, headers, and everything else...)

 

1. Is it enough if I create a fake TSslHttpCli + TSslContext component pair in the main thread that is destroyed only when the program ends? (and may never get used...)
2. Does ICS know that it should not unload OpenSSL in the background thread? (has it something like reference-counting?)
3. Should I create only ONE sslContexts, or 1-1 for each thread?

I think you are following the wrong thing here, even if this problem caused by loading and unloading OpenSSL which i doubt, but yet you can just log the loading and unloading operations with their result to a file and be sure.

 

If my hunch is right and it is caused by the intense allocating and reallocating operation done by both AlphaSkin and ICS with OpenSSL, then minimizing these operation by caching objects and reusing will only minimize the effect and delay the inevitable, this bug is just like ticking bomb and you were unlucky enough that is not crashing sooner.

 

I don't have My Delphi 7 at hand so checked an old projects and looked at the code without even running it, and here saw this gem 

 

In Utf8Encode there is this Length * 3, which assuming the size of UTF8 char would be 3 bytes at max, among this pasta with a call to WStrToPChar, which is mindboggling in the middle of converting a string to convert it first then convert it again, it seems Delphi 7 may be wrong in handling UTF8, it is +20 after all, and the fact UTF8String is used as RawByteString, raise even more questions.

 

Now assuming ICS (with dropped Delphi 7 support) is right and not missing with such conversion, i suggest you try the following

1) Use UnicodeToUtf8 and Utf8ToUnicode directly instead of Utf8Encode and Utf8Decode.

2) Use more aggressive memory manager to detect corruption, really you should try EurekaLog, it is the best ability to all sort of memory malpractices along with leaks resources like GDI handles.

3) Log all the strings in raw format to a file, write them to a a memorysteam separated with some chars then flush that stream to the disk, se for yourself the result.

 

Hope that help.

 

[Fr0sT.Brutal 900]

56 minutes ago, Kas Ob. said:

In Utf8Encode there is this Length * 3, which assuming the size of UTF8 char would be 3 bytes at max

Nothing wrong here. Hint: Widestring also doesn't have exactly one WideChar for every Unicode code point

 

@OP: try to run the app without using SSL (have ICS connect to plain socket server). This way you can either locate the issue or exclude whole SSL stuff

Edited August 28, 2023 by Fr0sT.Brutal

[Kas Ob. 121]

1 hour ago, Fr0sT.Brutal said:

Nothing wrong here. Hint: Widestring also doesn't have exactly one WideChar for every Unicode code point

How is this not wrong if this function assumes a UTF8 char can be only 3 bytes max instead of 4 ?

What else is wrong with that conversion function, have you seen these two calls in one line of code, are we juggling strings left and right through many types from WideString, AnsiString then UTF8String with wrong max length and expect it to not be wrong, because it does work sometimes ?

 

 

[Dalija Prasnikar 1396]

1 hour ago, Kas Ob. said:

How is this not wrong if this function assumes a UTF8 char can be only 3 bytes max instead of 4 ?

What else is wrong with that conversion function, have you seen these two calls in one line of code, are we juggling strings left and right through many types from WideString, AnsiString then UTF8String with wrong max length and expect it to not be wrong, because it does work sometimes ?

The function is not wrong. If Unicode character would require 4 bytes in UTF8 encoding, then it would also require two wide characters (UTF16 encoding) which means reserved number of bytes for such character would be 2 * 3 = 6. Also if the supplied buffer is not enough then the called conversion function would fail (without causing buffer overrun) and would return 0 and such scenario is fully covered by UTF8Encode function.

[Kas Ob. 121]

OK, i don't want this to as debate, but few things here should be pointed,

1 hour ago, Dalija Prasnikar said:

If Unicode character would require 4 bytes in UTF8 encoding, then it would also require two wide characters (UTF16 encoding) which means reserved number of bytes for such character would be 2 * 3 = 6.

True, but, UTF16 is either one or 2 of 16bit units ( it is called unit in Wikipedia), so it is either 2 bytes or 4 bytes, why this wasn't ever a problem with all versions of Delphi ?,

The is : simply because all the input and output where either internally generated data or by the OS ( mostly Windows), Windows API always refer to them as Wide, anyway this shouldn't a problem as long as the input and output is from Delphi application or the Windows, but in this very case the data are coming from remote place over the wire who knows the origin, also being handled by OpenSSL, so assuming it is will be 2 bytes UTF8 or even 2 bytes UTF16 is at least doubtful.

1 hour ago, Dalija Prasnikar said:

Also if the supplied buffer is not enough then the called conversion function would fail (without causing buffer overrun) and would return 0 and such scenario is fully covered by UTF8Encode function.

I really looked and looked then looked again, i just don't see failing point in both functions Utf8Encode and UnicodeToUtf8, this UnicodeToUtf8 though does trim but never fail, will return 0 if the input is an empty string other than that it will trim the result according to supplied buffer if it is shorter than needed, but will not fail.

It would be very helpful if someone paste the code of these functions here, of course if that not violating some license.

[PizzaProgram 9]

I've finished rewriting my program to allocate :

- 1x global cti + context

- 1x local for the thread

 

So I'll publish this version now and wait a few days again for customer experience and madExcept logs...

 

Maybe this whole thing isn't even caused by ICS directly, but OpenSSL?
I've red an article about a bad vulnerability fix that overwrote memory segments "for safety" it should have not. (Maybe don't remember well and can not find it any more.)

Looking at SSL3.0.7+ fixes I see a few buffer overruns...

 

4 hours ago, Kas Ob. said:

How is this not wrong if this function assumes a UTF8 char can be only 3 bytes max instead of 4 ?

You are right that it would be safer to assume the "maximum", but in my language it is 100% that even *2 would be enough, since most of the JSON text are English chars, maybe 1% is special. If I remember well, 4byte UTF8 is for some ancient Egypt and Klingon symbols.

Edited August 28, 2023 by PizzaProgram

[Kas Ob. 121]

@PizzaProgram Good luck !

 

11 minutes ago, PizzaProgram said:

If I remember well, 4byte UTF8 is for some ancient Egypt and Klingon

Not only for that 😎, also for a slice of pizza 🍕 

[Dalija Prasnikar 1396]

16 hours ago, Kas Ob. said:

17 hours ago, Dalija Prasnikar said:

If Unicode character would require 4 bytes in UTF8 encoding, then it would also require two wide characters (UTF16 encoding) which means reserved number of bytes for such character would be 2 * 3 = 6.

True, but, UTF16 is either one or 2 of 16bit units ( it is called unit in Wikipedia), so it is either 2 bytes or 4 bytes, why this wasn't ever a problem with all versions of Delphi ?,

What problem?

 

In context of wide string we can talk about wide characters where each such character stores one 16-bit code unit. This is important for determining the length of the wide string which is measured in number of wide characters - 16-bit code units.

So full Unicode character can be stored either in single or two 16-bit wide characters, which means that for characters that span two wide characters reserved space will be 2 * 3 which is more than enough to store that whole Unicode character in UTF8 representation which can be maximum 4 bytes. There is no Unicode character that fully fits into single 16-bit code unit that requires more than 3 bytes when encoded in UTF8.

 

16 hours ago, Kas Ob. said:

The is : simply because all the input and output where either internally generated data or by the OS ( mostly Windows), Windows API always refer to them as Wide, anyway this shouldn't a problem as long as the input and output is from Delphi application or the Windows, but in this very case the data are coming from remote place over the wire who knows the origin, also being handled by OpenSSL, so assuming it is will be 2 bytes UTF8 or even 2 bytes UTF16 is at least doubtful.

 

I have no idea what you want to say here. 

 

16 hours ago, Kas Ob. said:

17 hours ago, Dalija Prasnikar said:

Also if the supplied buffer is not enough then the called conversion function would fail (without causing buffer overrun) and would return 0 and such scenario is fully covered by UTF8Encode function.

I really looked and looked then looked again, i just don't see failing point in both functions Utf8Encode and UnicodeToUtf8, this UnicodeToUtf8 though does trim but never fail, will return 0 if the input is an empty string other than that it will trim the result according to supplied buffer if it is shorter than needed, but will not fail.

I was in a hurry and I took wrong turn while looking at code. Yes, you are right, the conversion function does not fail. The main point still stands and it does not cause buffer overrun. And again coming from UTF8Encode, conversion buffer allocates enough space for successful conversion of WideString content.

 

So the bug in the code we are discussing here does not come from handling strings and their buffers, at least not in the code that was presented here.

[Kas Ob. 121]

33 minutes ago, Dalija Prasnikar said:

Quote

The is : simply because all the input and output where either internally generated data or by the OS ( mostly Windows), Windows API always refer to them as Wide, anyway this shouldn't a problem as long as the input and output is from Delphi application or the Windows, but in this very case the data are coming from remote place over the wire who knows the origin, also being handled by OpenSSL, so assuming it is will be 2 bytes UTF8 or even 2 bytes UTF16 is at least doubtful.

 

I have no idea what you want to say here. 

Sorry i missed a word "the answer is :", i was answering my own question the one you answered above, and yes this is wasn't problem.

Yes it is exactly what you said, as long that char fits in 2 bytes UTF16 or a widechar with 2 bytes, then it will fits in 3 bytes UTF8, no debate there.

This is not a problem as long the input from Delphi or from a system like Windows OS which traditionally couldn't and didn't in the past support these UCS-4 aka UTF32, but now windows does support them like Android, and this makes a problem.

Take an example this pizza slice emoji 🍕, in the past there wasn't away to input from keyboard but now windows does support it, this pizza slice is 3 bytes in Unicode, but it is 4 bytes in UTF8 and UTF16, and Delphi with its String or WideString will not be able to handle it, but again since long time ago Delphi RTL switched to delegate all these operation to WideCharToMultiByte and MultiByteToWideChar. 

This is not the case with Delphi 7, and away from that emoji char, the OP did say he didn't have exotic chars in the traffic, can be the data received and handled by OpenSSL contain 4 bytes char, just because he is using HTTP request does imply the probability.

I was saying exactly what you did say, and expanded on the fact when using traffic from internet we can't expect it to fit in Delphi 2 bytes per char strings.

 

 

Anyway, i sounded my opinion about such GDI leak from AlphaSkin based on my point of view, and still see the cause by overflowing buffers or overwriting memory, to me the logical and easiest way to explain such behavior based on what OP said all happen with introducing ICS with OpenSSL and threading, and the fact if this overflowing was having anything else than zero's AlphaSkin code should generated hell of exceptions, threading problem can't be so consistent with zeroing memory, this leads me to deduct the only place that could make this happen with Delphi 7 and it outdated way to handle strings with different encoding coming from OpenSSL input and output, that is it.

[Dalija Prasnikar 1396]

37 minutes ago, Kas Ob. said:

This is not a problem as long the input from Delphi or from a system like Windows OS which traditionally couldn't and didn't in the past support these UCS-4 aka UTF32, but now windows does support them like Android, and this makes a problem.

Take an example this pizza slice emoji 🍕, in the past there wasn't away to input from keyboard but now windows does support it, this pizza slice is 3 bytes in Unicode, but it is 4 bytes in UTF8 and UTF16, and Delphi with its String or WideString will not be able to handle it, but again since long time ago

Delphi can perfectly handle pizza slice emoji with either UTF8String or WideString. How do you think it was handling Chinese characters that were spanning two UTF-16 code units?

 

Unicode encoding is a standard no matter what OS you are using. The most that can happen is that you cannot properly display the character because you are lacking appropriate font.

 

Now I cannot claim that Delphi 7 encoding/decoding functions are capable of handling all those correctly as I haven't tried, but even in Delphi 7 those functions could be replaced with the ones that handle them properly.

[Dalija Prasnikar 1396]

52 minutes ago, Kas Ob. said:

Anyway, i sounded my opinion about such GDI leak from AlphaSkin based on my point of view, and still see the cause by overflowing buffers or overwriting memory, to me the logical and easiest way to explain such behavior based on what OP said all happen with introducing ICS with OpenSSL and threading, and the fact if this overflowing was having anything else than zero's AlphaSkin code should generated hell of exceptions, threading problem can't be so consistent with zeroing memory, this leads me to deduct the only place that could make this happen with Delphi 7 and it outdated way to handle strings with different encoding coming from OpenSSL input and output, that is it.

There is still a lot of code running in that thread that we haven't seen and don't know how it interacts with other code and might cause memory corruption for various reasons. But the string encoding/decoding code we have seen is not it, no matter how outdated or not.

[Kas Ob. 121]

7 minutes ago, Dalija Prasnikar said:

Delphi can perfectly handle pizza slice emoji with either UTF8String or WideString. How do you think it was handling Chinese characters that were spanning two UTF-16 code units?

It handle a translated version without colors, while browsers handle this more elegantly and translate it back to the 4 bytes version, to my knowledge Windows started to support the emoji Unicode in full with WIndows 10 version 2H(something!!) and Windows 11 does it too, but older versions all does some translate to an equivalent Unicode versions, browsers does the opposite and translate to the the newer, also all languages are supported from decades ago, but a newer versions also exist.

 

Take this PizzaSlice on this browser it does appear like this 🍕 , colorful as in this image , in Delphi EditText you can try and input from Windows directly by opening the emoji keyboards with WinKey+. 

Now in Delphi it will appear like this 

Delphi can handle this as two bytes WideChar perfectly and Windows will take the short one just fine, and any browser on paste will translate it back to the color one,

Lets try something different to capture the real bytes for this, open an empty text file then write that emoji, on my Windows 10 ver 1803 it does appear black, and lets save that file with UTF8 and with Unicode, here are the files

PizzaSlice_UTF8.txt

PizzaSlice_Unicode.txt 

 

And in hex their content ( bytes)

PizzaSlice_UTF8 contains this EF BB BF F0 9F 8D 95 , EF BB BF is the UTF8 header  

 

PizzaSlice_Unicode contains this FF FE 3C D8 55 DF ,  FF FE is the Unicode header  

These bytes are the correct content for this emoji without any translation and can be confirmed by visiting this page and paste the pizza slice and look for the hex out put

https://onlinetools.com/unicode/convert-unicode-to-hex

 

 

 

PizzaSlice_UTF8.txt

[Angus Robertson 574]

Quote

've finished rewriting my program to allocate :

- 1x global cti + context

- 1x local for the thread

If I interpret your short hand correctly, you now have one global SslContext and a second one in the thread. 

 

Unless you actually initialise the global SslContext or attach it to a component and make an SSL request, it will not load the OpenSSL DLLs, so your wasteful problem of loading and unloading the DLLs several times a minute will remain. 

 

Please read my previous messages where I have explained how to do this properly, I'm not going to keep repeating myself.

 

Angus

[Dalija Prasnikar 1396]

17 minutes ago, Kas Ob. said:

It handle a translated version without colors, while browsers handle this more elegantly and translate it back to the 4 bytes version

This is not encoding, this is visual representation. Underlying encoding (numbers) is the same.

[PizzaProgram 9]

On 8/29/2023 at 11:41 AM, Angus Robertson said:

If I interpret your short hand correctly, you now have one global SslContext and a second one in the thread. 

YES. That is correct.

 

On 8/29/2023 at 11:41 AM, Angus Robertson said:

Unless you actually initialise the global SslContext or attach it to a component and make an SSL request, it will not load the OpenSSL DLLs, so your wasteful problem of loading and unloading the DLLs several times a minute will remain. 

Of course I am initializing the first one in the main thread, before creating the thread to use the 2th one, otherwise it would make no sense.

 

[OFF]

On 8/29/2023 at 11:41 AM, Angus Robertson said:

Please read my previous messages where I have explained how to do this properly, I'm not going to keep repeating myself.

No need to remind me again, since I'm usually reading all your posts 3-12x. Sometimes days later again.

Because:

 - That's the least I can do,

 - not to miss any hint or tip,

 - may see it from other angle next time, and

 - You are the one who knows much more about ICS, so i appreciate every answer You give!

 

I'm gladly taking every advice, Thank you very much !!!

Edited August 30, 2023 by PizzaProgram

[PizzaProgram 9]

Report:

One day has past, ca. 10 PCs are running so far the new version with "double ICS".

So far 0 GDI errors on those PCs!

 

But I'm not 100% satisfied so far, because:

UIB database opening still sometimes drops "Out of memory" errors!

 Once occurred, happens always, and never works again until app restart.

 

I'm using Unified InterBase component suite for 15 years with Firebird 2.5, without problems, but never actually in a thread. That's first to me too.

So I can not say it is related to ICS any more anyhow, before I know better.

The new version of my EXE I've released this night has more log capabilities that may bring me closer to the problem.

 (Yes, I'm creating ALL UIB components inside the thread, with separate connections.)

 

What I still wonder:

 - Can somehow Indy's (old 2019 version) SSL dll loadings interfere with ICS's SSL3 DLLs ?

(Indy REST calls are running in a different Thread, never had problems with them in the last 8 years.)

[PizzaProgram 9]

Bad bad news 😞 I was happy too soon!

 

The errors "disappeared" because of they were hidden by a try .. except inside AlphaSkin's code.

function CreateBmp32(const Width: integer = 0; const Height: integer = 0): TBitmap;
begin
  Result := TBitmap.Create;
  try
    Result.IgnorePalette := True;
    Result.PixelFormat := pf32bit;
    Result.HandleType  := bmDIB;
    if (Width < 0) or (Height < 0) then begin
      Result.Width := 0;
      Result.Height := 0;
    end
    else begin
      Result.Width := Width;
      Result.Height := Height;
    end;
  except
    Result.Width  := 0;
    Result.Height := 0;
  end
end;

It seems I do not see the memory consumption because it's happening at Virtual RAM!

A complete breakdown of the program is happening when Virtual RAM consumption is reaching ca. 1.87 GB. (plusz the normal RAM usage 180MB)

 

Neither madExcept, nor ProcessHacker does not show those "lost" memory blocks.

Also the UIB "Out of memory" was present on those PCs half day long continuously as I see from Log files.

... Process Hacker shows only that 118MB consumption. The list is not tooo long, and drops size radically The next page is down to 16KB.

 

So there are still 3 possibilites remains:

 1. ICS (or the OpenSSL3 dll call, or the dll itself is) is not thread safe, might overwrite memory blocks

 2. Indy10 <> ICS dll conflict

 3. UIB is not thread safe ( I'll have to create a sample program to test it somehow... )

 

The GDI errors are clearly only the symptoms of the real problem, because it's utilizing big RAM parts, so the problem occurs more often there.

I will activate "fullDebugMode" of FastMM4 too, but the problem is still that I' can not reproduce these at my PC, but can not ask the users.

I'm so tired of this 😞

 

Anyway, thanks in forward for any more hints I could try!

 

[FPiette 383]

1 hour ago, PizzaProgram said:

Neither madExcept, nor ProcessHacker does not show those "lost" memory blocks.

This probably means that there is no memory leak but memory fragmentation. You have 118MB memory split into thousands of block separated by small free blocks.

 

1 hour ago, PizzaProgram said:

might overwrite memory blocks

In madExcept, you can check for either memory overrun or under-runs. It is detected instantly and triggers an exception. Just check the checkbox in madExcept settings.

 

One of my first advice was you split the whole applications in small chunks, candidates are the threads. Run the code in another executable easier to debug and understand than the whole application. Divide to conquer!

[Cristian Peța 103]

On 9/1/2023 at 5:41 PM, FPiette said:

On 9/1/2023 at 4:33 PM, PizzaProgram said:

Neither madExcept, nor ProcessHacker does not show those "lost" memory blocks.

This probably means that there is no memory leak but memory fragmentation. You have 118MB memory split into thousands of block separated by small free blocks.

@PizzaProgram if fragmentation then this can be solved by running as 64 bit.

 

On 8/20/2023 at 5:02 PM, PizzaProgram said:

So logically in every 2-4 hours it can reach 32bit limit. (2GB)

Are you not using 4 GB?

{$SetPEFlags $0020} // Winapi.Windows.IMAGE_FILE_LARGE_ADDRESS_AWARE  { App can handle >2gb addresses }

Or better try 64 bit if possible.

[PizzaProgram 9]

On 9/4/2023 at 10:51 AM, Cristian Peța said:

Are you not using 4 GB?

{$SetPEFlags $0020} // Winapi.Windows.IMAGE_FILE_LARGE_ADDRESS_AWARE  { App can handle >2gb addresses }

 

1.) Thanks for the tip! I was always afraid to allow it, but it seems to work fine 🙂

 

On 9/1/2023 at 4:41 PM, FPiette said:

In madExcept, you can check for either memory overrun or under-runs. It is detected instantly and triggers an exception. Just check the checkbox in madExcept settings.

2.) It seemed a great idea, I've tested it for the last few days. "Luckily" never triggered any overrun or underrun exception.

Also realized: With madExcept's "active error search" >> [X] "report resource leaks" option turned on, the program consumes a total of:

800MB memory instead of 250.

 

3.) Also tried to activate FullDebugMode of FastMM4, but did not bring me any closer to find out anything about the leak.
madExcept's memory reports are much better.

 

4.) An other great app I've found is: VMMap. (See pic at 2. point.)

It shows me the whole fragmentation, total consumption, etc.

It helped me to see that actually the "Private WS" is growing, not the "Unusable".

(My only problem with it, I always have to press F5 to refresh, can not activate "realtime refresh from options menu". Also

 

Sometimes, when both Indy10 SSL and the ICS SSL is communicating in 1-1 background threads, I see some 5-7MB new mem-blocks appear, and not really disappear after.

So currently I do not think that memory fragmentation would be the real problem. I need more time to investigate because it's very time consuming.

Also I try to learn more about why I get so much "page faults".

 

Summarized:

 - Turning off madExcept's active error search (800MB > down to 250MB)

 - Turning off AlphaSkin before app terminate

 - Adding some try .. finally ReleaseObject(...) end; to AlphaSkin's code.

 - Allowing 3.2GB (4GB) max ... instead of just 2GB

 - fixing: ICS "multithreaded := True;"

 - fixing: ref counting OpenSSL (by creating +1 instance)

 - fixing: creating cti + SSL context only once per thread

 

The result:

 - Far from perfect, but pretty stable. Should now be able to run my app a full day or 2, without running out of memory, or showing error popups all the time.

 

Thank you very much for all the help so far to everyone! 🙂

 

[FPiette 383]

46 minutes ago, PizzaProgram said:

Also realized: With madExcept's "active error search" >> [X] "report resource leaks" option turned on, the program consumes a total of:

800MB memory instead of 250.

That's normal. Each memory block is guarded by additional data to detect overrun and under-run. madExcept is not intended to be delivered with the released product.

 

49 minutes ago, PizzaProgram said:

An other great app I've found is: VMMap. (See pic at 2. point.)

It shows me the whole fragmentation, total consumption, etc.

Pay attention that memory allocated with Delphi memory manager comes from virtual memory and is sub-allocated. In this sub-allocation, you get also fragmentation. Actual behavior is entirely dependent on the memory manager you use (You said FastMM4).

 

If you app consume memory but has no leak, then you have fragmentation.

[Lajos Juhász 293]

3 hours ago, FPiette said:

If you app consume memory but has no leak, then you have fragmentation.

 

Could be also objects with owners that are left in the memory. Those objects wouls use memory and be freed before the memory leak report is generated.

[FPiette 383]

9 minutes ago, Lajos Juhász said:

Could be also objects with owners that are left in the memory. Those objects wouls use memory and be freed before the memory leak report is generated.

Correct!

[PizzaProgram 9]

[Solved]

Just wanted to give a final report, that NO problems or memory leaks happened since I am:

 - setting SslHttpCli.MultiThreaded := True;

 - I am creating one global component in the main-thread at program start, and initializing OpenSSL3 DLLs from there
 - and destroying that global component only at program end. (This way OpenSSL can stay in memory, and initialized only once.)

 

(Tried to change the topic header to [Solved] ... but could not find a button to edit this late.)

 

Thank you very much for all the help You all have provided during this crisis ! 🙂

Edited October 3, 2023 by PizzaProgram