PizzaProgram

Sorry for disturbing You guys, just forget about my question, I'll stay with Indy, it works as expected, and Indy will keep downloading the DLLs for ICS, where TLS 1.3 is needed. Thanks for trying to help ! Bye.

They are already distributed everywhere all over the country. Some of them 3, 5, 10+ years ago !!!!!!!!!! Also there are "ready to use" windows-backup image files that contain all necessary setup and the windows itself, without those DLL we are missing now. It is already built-into my own exe. But my Exe needs to DOWNLOAD automatically the necessary "setup updater" ! Normally that would be a great idea, but it would increase the 7MB exe file to double size. Some regions have 20-50Kbit internet only and it's already taking 2+ minutes! to download that 7MB. I worked very very hard for years to keep it that small, while the "whole program" with all the pictures and DLLs and everything else is 150MB, which is normally "static data", so only the 7MB needs to be downloaded, sometimes daily.

Of course I did ! Also I have replied to it. I don't know how to explain it better: 1. I have and EXE on hundreds of PCs, that needs to communicate with both http and https endpoints. 2. ICS can not start, because some necessary SSL DLLs are missing. 3. ICS is trying to download those DLLs from a non-SSL server via http://... , but can not !!! 4. GOTO: #2.

No, it is impossible to do this manually on hundreds of PCs one by one. My EXE is auto-updating itself, no user action is necessary. If the new EXE needs some extra/updated DLLs, it must download them automatically and put them in the right subdirectory. Until now, Indy10 component did this completely automated. It has a simple way to check if http:// or https:// is necessary, and if no SSL is needed, it can work "alone" without any necessary extra DLLs. I was trying to get rid of Indy, (because it is 10+ years old and has some memory leaks,) and rewrite the whole auto-download system based on ICS. But it seems to be impossible. 😞

Yes, I NEED SSL, that's why I'm trying to download the necessary SSL DLLs first (from a non-SSL server). So it seems to me it's a 22's trap. Without those DLLs ICS is not working at all, but to download them, I need those DLLs first.

Sorry, need a bit more help, because it's not working as I've expected. (Still trying to solve it with ICS v8.70 yet.) The first step for this component to work, to download the necessary 3 files of SSL3 DLLs, so it can communicate with TLS1.3 servers via httpS . I've uploaded those files to my own http:// (no SSL!) server. But TIcsHttpMulti.Download() can not start, because it gives the error: Project ....exe raised exception class EIcsLibeayException with message 'Failed to load OpenSSL file libcrypto-1_1.dll'. Process stopped. Is it possible that ICS can not work without extra DLLs at all? Not even communicating with simple non-SSL http:// servers using the OS's built-in dll? (Only worked until now, because I've downloaded those DLLs with an Indy10 component, but now I've tried to update this part of my code to replace Indy everywhere.)

Yes, I know, thank You! (I just clicked the Post too late in this topic. That's why it appeared later than the other...) Here it is, so we have a link, if someone else is searching for the answer:

Yes, I understand, and it's clear for me now, but this could have been avoided, if there is a warning in the readme file + in the PAS file itself, stating that: "this component should NOT BE USED any more!" Use Instead: ... Maybe deleting the whole component + from examples completely would be the best? PS: My "lost 30 work-hours" ar still compensated by ICS itself. Thank you guys to developing + supporting it!

Just to confirm: - Is this new version recommended to use with Delphi7 - 32bit? (to replace v8.70 I'm using now) - or is it much more risky than using v8.70 ? - or it is impossible to install on Delphi7 anyway? I know, there is no more "official support" for D7, so I'm just asking for your advice/recommendation please. PS: it would be nice to include inside the readme files + on ICS 9.0 webpage + on the ICS main page of version listing: - what is the minimum Delphi IDE recommended - for which version? ... so the downloader can see at which version ended any of the support.

[! WARNING !] There should be a big fat warning at TMultipartHttpDownloader component, that nobody should use it ! version = 8.70 I've just spent 3 days to rewrite my Indy "downloader with progressbar" form to work based on this component, and the end result was a disaster: After each download the whole Delphi7 IDE froze, had to kill it by ProcessExplorer the "done" function was called not just once at the end, but like 5 times (while there were only 2 progress enabled) ... and several other problems, but worse is: The downloaded file was NOT THE SAME as the original! (a total scrambled version with mixed bytes) There are still many places where only DSL / 3G internet is available. (Having only 100/20Kbit down/up speed.) so I've hoped it helps but it made it worst.

[Solved] Just wanted to give a final report, that NO problems or memory leaks happened since I am: - setting SslHttpCli.MultiThreaded := True; - I am creating one global component in the main-thread at program start, and initializing OpenSSL3 DLLs from there - and destroying that global component only at program end. (This way OpenSSL can stay in memory, and initialized only once.) (Tried to change the topic header to [Solved] ... but could not find a button to edit this late.) Thank you very much for all the help You all have provided during this crisis ! 🙂

1.) Thanks for the tip! I was always afraid to allow it, but it seems to work fine 🙂 2.) It seemed a great idea, I've tested it for the last few days. "Luckily" never triggered any overrun or underrun exception. Also realized: With madExcept's "active error search" >> [X] "report resource leaks" option turned on, the program consumes a total of: 800MB memory instead of 250. 3.) Also tried to activate FullDebugMode of FastMM4, but did not bring me any closer to find out anything about the leak. madExcept's memory reports are much better. 4.) An other great app I've found is: VMMap. (See pic at 2. point.) It shows me the whole fragmentation, total consumption, etc. It helped me to see that actually the "Private WS" is growing, not the "Unusable". (My only problem with it, I always have to press F5 to refresh, can not activate "realtime refresh from options menu". Also Sometimes, when both Indy10 SSL and the ICS SSL is communicating in 1-1 background threads, I see some 5-7MB new mem-blocks appear, and not really disappear after. So currently I do not think that memory fragmentation would be the real problem. I need more time to investigate because it's very time consuming. Also I try to learn more about why I get so much "page faults". Summarized: - Turning off madExcept's active error search (800MB > down to 250MB) - Turning off AlphaSkin before app terminate - Adding some try .. finally ReleaseObject(...) end; to AlphaSkin's code. - Allowing 3.2GB (4GB) max ... instead of just 2GB - fixing: ICS "multithreaded := True;" - fixing: ref counting OpenSSL (by creating +1 instance) - fixing: creating cti + SSL context only once per thread The result: - Far from perfect, but pretty stable. Should now be able to run my app a full day or 2, without running out of memory, or showing error popups all the time. Thank you very much for all the help so far to everyone! 🙂

Bad bad news 😞 I was happy too soon! The errors "disappeared" because of they were hidden by a try .. except inside AlphaSkin's code. function CreateBmp32(const Width: integer = 0; const Height: integer = 0): TBitmap; begin Result := TBitmap.Create; try Result.IgnorePalette := True; Result.PixelFormat := pf32bit; Result.HandleType := bmDIB; if (Width < 0) or (Height < 0) then begin Result.Width := 0; Result.Height := 0; end else begin Result.Width := Width; Result.Height := Height; end; except Result.Width := 0; Result.Height := 0; end end; It seems I do not see the memory consumption because it's happening at Virtual RAM! A complete breakdown of the program is happening when Virtual RAM consumption is reaching ca. 1.87 GB. (plusz the normal RAM usage 180MB) Neither madExcept, nor ProcessHacker does not show those "lost" memory blocks. Also the UIB "Out of memory" was present on those PCs half day long continuously as I see from Log files. ... Process Hacker shows only that 118MB consumption. The list is not tooo long, and drops size radically The next page is down to 16KB. So there are still 3 possibilites remains: 1. ICS (or the OpenSSL3 dll call, or the dll itself is) is not thread safe, might overwrite memory blocks 2. Indy10 <> ICS dll conflict 3. UIB is not thread safe ( I'll have to create a sample program to test it somehow... ) The GDI errors are clearly only the symptoms of the real problem, because it's utilizing big RAM parts, so the problem occurs more often there. I will activate "fullDebugMode" of FastMM4 too, but the problem is still that I' can not reproduce these at my PC, but can not ask the users. I'm so tired of this 😞 Anyway, thanks in forward for any more hints I could try!

Report: One day has past, ca. 10 PCs are running so far the new version with "double ICS". So far 0 GDI errors on those PCs! But I'm not 100% satisfied so far, because: UIB database opening still sometimes drops "Out of memory" errors! Once occurred, happens always, and never works again until app restart. I'm using Unified InterBase component suite for 15 years with Firebird 2.5, without problems, but never actually in a thread. That's first to me too. So I can not say it is related to ICS any more anyhow, before I know better. The new version of my EXE I've released this night has more log capabilities that may bring me closer to the problem. (Yes, I'm creating ALL UIB components inside the thread, with separate connections.) What I still wonder: - Can somehow Indy's (old 2019 version) SSL dll loadings interfere with ICS's SSL3 DLLs ? (Indy REST calls are running in a different Thread, never had problems with them in the last 8 years.)

YES. That is correct. Of course I am initializing the first one in the main thread, before creating the thread to use the 2th one, otherwise it would make no sense. [OFF] No need to remind me again, since I'm usually reading all your posts 3-12x. Sometimes days later again. Because: - That's the least I can do, - not to miss any hint or tip, - may see it from other angle next time, and - You are the one who knows much more about ICS, so i appreciate every answer You give! I'm gladly taking every advice, Thank you very much !!!

I've finished rewriting my program to allocate : - 1x global cti + context - 1x local for the thread So I'll publish this version now and wait a few days again for customer experience and madExcept logs... Maybe this whole thing isn't even caused by ICS directly, but OpenSSL? I've red an article about a bad vulnerability fix that overwrote memory segments "for safety" it should have not. (Maybe don't remember well and can not find it any more.) Looking at SSL3.0.7+ fixes I see a few buffer overruns... You are right that it would be safer to assume the "maximum", but in my language it is 100% that even *2 would be enough, since most of the JSON text are English chars, maybe 1% is special. If I remember well, 4byte UTF8 is for some ancient Egypt and Klingon symbols.

I'd like to summarize last week's work and experiences: - I was very happy to read that Setting .MultiThreaded := True; could solve all my problems, but it did NOT. 😞 (I've checked the code and saw running step-by-step the "thread safe processMessage".) - GDI Memory Errors are the same, if ICS is running in background thread. No problems, if ICS is not running. I've also reverted my code back to old AlphaSkin 2019 version, where I know it was stable. Same problem occurs. So the only thing left to try to load OpenSSL once only: So my question is: - How do I ensure ICS 8.70 is never unloading OpenSSL ? - while using it in both the main and multiple background threads separately? - and how do I "reset" safely TSslHttpCli to be re-used? (both buffers, headers, and everything else...) Is it enough if I create a fake TSslHttpCli + TSslContext component pair in the main thread that is destroyed only when the program ends? (and may never get used...) Does ICS know that it should not unload OpenSSL in the background thread? (has it something like reference-counting?) Should I create only ONE sslContexts, or 1-1 for each thread? Thank you very much for any further help!

Sorry, but I don't fully understand that part yet. Does it mean, that now, that I've enabled MultiThreaded property, I need to call this process too? Also, if this is not the "right way" to automate thing: FMultiThreaded := TThread.CurrentThread.ThreadID <> MainThreadID; How about the opposite, at least stop the execution and warn us, programmers that we almost did a huge error: // before post or get: if (FMultiThreaded = False) and (TThread.CurrentThread.ThreadID <> MainThreadID) then begin raise Exception.Create('Warning! It is not safe to call this method from a background thread, if .MultiThreaded property is False! Exiting...'); Exit; end; ? (and save us 200+ wasted hours for searching the error...) Thank you for your concern! I'm really happy you have looked into the code too. You are right about "this COULD be a problem" at first site. I think too it's very risky to work with memory streams. Luckily this part has been well tested. There are 5+ other programmers using the same code without problems. (We have all shared our knowledge with each other to finish until deadline and worked 3 month long to find out how to communicate with the gov. server.) I was the only one using inside a Thread, and none of us knew about this "hidden" property. That's why the other guys had no problems.

Well, I've just tried that, and the server responded: ... Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])\n at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 951]"}]} So I think my code was good how it was before. (Except the ".MultiThreaded := True" part of course)

Please tell us the right answer! How to get the right size under Delphi 7 ? What exactly did I wrong? lg := Length(JsonUTF8); SslHttpCli.SendStream.Write(JsonUTF8[1], lg); Maybe lg-1 ?

A great idea! But it depends, how much time (CPU cycles) would it take to get the Current Thread's ID. (the main thread's could be stored when the program start, and a simply if ... <> ... takes only one cycle.)

Wow ! This could be it !!! Thank your VERY much for looking into the code! I've always thought this is safest method, instead of leaving a (most probably connection-error broken) component open for possible re-use. There are normally 1-2 request-sends pro 1 minute, and the SSL header always changes by goverment request, so SSL has to be re-re-re-initialized anyway. Some of my users have only 3G or DSL (max 40Kbit, instable) net connections, impossible to get better. And You also said: it won't be compatible with Delphi 7 any more. A new Delphi price is impossible high, and rewriting the whole codebase (1M+ lines) would be an impossible job. (For start: no IBX component available.) (Would be easier to drop delphi completely, what I don't really want to.)

I think I have to apologize: I've re-checked everything, and realised: - Yes, the "private bytes" used are minimal, - but after just 2 hours of app running the Virtual Memory consumption grows heavily ! (And I see continuously growing) Private bytes, 85,72 MB Peak private bytes, 116,15 MB Virtual size, 1,18 GB Page faults, 878 816 Hard faults, 2 114 Working set, 90,04 MB Peak working set, 119,59 MB Private WS, 73,84 MB Shareable WS, 16,2 MB Shared WS, 9,78 MB GDI < 900 Handles < 400 So logically in every 2-4 hours it can reach 32bit limit. (2GB) I'll investigate further. There are ca 100 "Private: Reserved" 7-8MB segments.

Dear Dalija, Thank you very much for the long and detailed explanation !! 🙂 But if that would be true, than a simple "process explorer" would show memory consumption of > 2GB, and not just 150MB. Am I wrong? (Actually I'm using a more advanced program for that: SystemInformer which shows every tiny byte of memory allocation.) Also MadExcept has a: ShowLeakReport(GetLeakReport() ); function, I have used before the whole APP is closing. There are no big leaks at my program! Exactly ! And it only happens, if the ICS background thread is RUNNING. Never if it's stopped, or not started at all. While if I check the GDI object number with a separate program, it is always less than 1000! (The default max is 10.000 in windows.) That's what I do not understand.

Good idea! ... 30 min later: Checked a few thousands of log entries, on 2 PCs, but nothing special. Only a few Schannel 40 and 70 errors, but found no evidence it would be my app related.