Leaderboard

Fixed the problem where some symbols (in particular generics) either resolved to the calling unit or to a wrong line number in the implementing unit. Thanks to @Stefan Glienke for pestering me about this until it got solved. Source committed and binary uploaded. And now... Pizza! 🍕

Growing impatient while building your FireMonkey app for macOS, iOS or Android? This post shows a way to (significantly) decrease the amount of time it takes for your app to build on these platforms. https://blog.grijjy.com/2021/04/05/build-speed/

I have got my first shot on Sunday. Monday morning I felt like 20 again, where I also had massive problems to get out of bed (albeit for other reasons). Took a nap after dinner (can't remember doing such thing before). Fortunately that fatigue went away in the evening. So I highly recommend a vaccination, too.

I have now implemented MSF intervals so the 16Mb pdb limit should be gone. I have however not been able to verify that it actually works beyond validating the files with cvdump and llvm-pdbutil because VTune hangs, burning CPU, when it tries to resolve symbols from the large pdb files I've thrown at it (the smaller ones still works). This could be a bug in VTune, it could be that I just need to wait longer than the 30 minutes I've tried or it could be that there's either something wrong with the pdb file or that I need to order the symbols in it differently. From examining the call stacks of the VTune threads it appears as if it's busy doing symbol lookup through msdia140.dll. Anyhow, source committed and new precompiled uploaded. Give it a go and let me know if you have better luck than me. P.S. It turned out to be just a tiny bit more complex than just some recursive stuff https://bitbucket.org/anders_melander/map2pdb/commits/95353cbf3ba5b284919e617ace970b02ca0bf0e2

While LLVM certainly is part of the situation it suffers from the same sickness that makes most C++ code compile terribly slow: templates - err, I should say generics. (precompiled headers anyone?) Especially FMX uses them like everywhere if it makes sense or not. Everything is a TList<something> and there are many generic types within FMX itself. So the compiler has to build all these generics for each and every unit. Yes, if you have 1000 units using TList<TComponent> then that code is being emitted into 1000 dcu files and in case of LLVM also .o files including all that RTTI and possibly gigantic debug information for that TList<T>. The linker later eliminates any duplicates so the final binary does contains the code and RTTI only once. I know this from refactoring spring4d where I redesigned the generic collections to produce as little code as possible and turned off RTTI for all the implementing classes which improvided compile time on some projects by a factor of 4. Unfortunately even with the new intrinsics introduced in XE7 that were for reducing binary size because it enables compiletime optimizations for the code (GetTypeKind and alike) there is some significant bloat happening as I reported some time ago. When you fancy and look into a map file for a typical FMX application you can see that a huge chunk alone is from system.generics.collections and most of that code is dead code because it's only in the binary because RTTI is turned on for those classes. Even if they are used as private fields in some classes and all that is ever called on them is Add and Delete the entire code is emitted by the compiler. So a significant part of the compiletime is being used producing garbage (emitting code into dcu/o files) that the linker later has to find and eliminate. Edit: To emphasize: generics alone are not bad - its the overuse of them and making them fatter than they need to be and having RTTI being turned on for them. Often in places where a class reference would have been enough instead of some fancy<T: TSomethingclass>

It took me a bit longer than expected to get here but I believe I've finally reached the goal. The following shows VTune profiling a Delphi application, with symbol, line number and source code resolution: Download Get the source here: https://bitbucket.org/anders_melander/map2pdb/ And a precompiled exe here: https://bitbucket.org/anders_melander/map2pdb/downloads/ The source has only been tested with Delphi 10.3 - uses inline vars so it will not compile with older versions. Usage map2pdb - Copyright (c) 2021 Anders Melander Version 2.0 Parses the map file produced by Delphi and writes a PDB file. Usage: map2pdb [options] <map-filename> Options: -v Verbose output -pdb[:<output-filename>] Writes a PDB (default) -yaml[:<output-filename>] Writes an YAML file that can be used with llvm-pdbutil -bind[:<exe-filename>] Patches a Delphi compiled exe file to include a reference to the pdb file -test Works on test data. Ignores the input file Example: Configure your project linker options to output a Detailed map file. Compile the project. Execute map2pdb <map-filename> -bind Profile the application with VTune (or whatever) Known issues The -bind switch must occur after the filename contrary to the usage instructions. PDB files larger than 16Mb are not valid. This is currently by design. 64-bit PE files are not yet supported by the -bind option. As should be evident I decided not to go the DWARF route after all. After using a few days to read the DWARF specification and examine the FPC source I decided that it would be easier to leverage the PDB knowledge I had already acquired. Not that this has been easy. Even though I've been able to use the LLVM PDB implementation and Microsoft's PDB source as a reference LLVM's implementation is incomplete and buggy and the LLVM source is "modern C++" which means that it's close to unreadable in places. Microsoft's source, while written in clean C and guaranteed to be correct, doesn't compile and is poorly commented. Luckily it was nothing a few all-nighters with a disassembler and a hex editor couldn't solve. Enjoy!

Update: I am no longer one-armed, but have about 1.65+ arms 😉 Mobility is decent and pain is limited, but there will be weeks of training to get back the strength and full use. To add insult to injury, I also caught the SARS-Cov-2 English mutation around March 16th, but I was lucky and got a very light progression, almost like a mild flu - but with an unusual amount of fatigue. Wife and stepson also got it, and they lost all sense of taste and smell, while I kept mine. Wife is currently getting her smell/taste back, but stepson still complains about food having no smell/taste or wrong taste, and that toothpaste tastes awful. We have no idea where we got it from, but I was visiting the hospital on that date, so it may even have been from there. The hardest part has been the fatigue. All my adult life, I have never been one to take a nap during the day, and typically have slept around 7 hours every night. The last weeks I've slept 7-8 hours during the night, and on several occasions 3-4 hours in the afternoon. I've been drained, both physically and mentally, and I still tire quickly, but it gets better every day. I highly recommend NOT contracting this shit! Keep your distance, wear that mask, wash those hands - and get vaccinated.

As the old saying goes, used regex to solve a problem? now you have two problems! 🤣 That said, I use regex extensively both in Delphi (I wrote the original System.RegularExpressions code) and .Net (keep it simple, and it's not for parsing html!) - but using it to find problems with my code? ah nope, for that I use Eyeballs 1.0 and static analysis tools like FixInsight and Pascal Analyzer (yes I know they have their limitations). Plus I guess I have learned a few things over the years that I apply in new code I write. As for the old code, well refactoring is a work in progress. Yesterday I rewrote a lexer/parser framework I initially wrote 10 years ago due to performance issues, and that issue was identified through profiling (see the thread on vtune). Switching to records took the memory manager from being a big % to insignificant (ie no longer shows up in the profiler). I thought I might get something like a 10% improvement, but was pleasantly surprised to see a 30% improvement! Those sort of gains don't come often in a mature code base.

ICS V8.66 has been released at: http://wiki.overbyte.eu/wiki/index.php/ICS_Download ICS is a free internet component library for Delphi 7, 2006 to 2010, XE to XE8, 10 Seattle, 10.1 Berlin, 10.2 Tokyo, 10.3 Rio and 10.4 Sydney, and C++ Builder 2006 to XE3, 10.2 Tokyo, 10.3 Rio and 10.4 Sydney. ICS supports VCL and FMX, Win32, Win64 and MacOS 32-bit targets. The distribution zip includes the latest OpenSSL 1.1.1k win32, with other versions of OpenSSL being available from the download page. Changes in ICS V8.66 include: 1 - Added a new TIcsInetAlive component to check for IPv4 and/or IPv6 internet connectivity, using Ping and/or HTTP, defaulting to www.msftconnecttest.com run by Microsoft for Windows 10 alive checking. The online and offline check intervals may be set, and event fires when online state changes. Sample OverbyteIcsHttpRestTst has a demo for new component. 2 - OpenSSL 1.0.2 and 1.1.0 ceased security fix support over 12 months ago so ICS now only supports 1.1.1, with 3.0 support due in the next few months. This removes a lot of legacy code and functions, and several old ICS functions needed for backward compatibility. Two components only used for 1.0.2 have been removed, TSslStaticLock and TSslDynamicLock which may give an error loading forms, until they are removed. SslContext SslOptions is now ignored, use SslOptions2 instead which has more modern options. Can not remove SslOptions because it's saved on too many DFMs. Most modern applications should not be using SslConext, but IcsHosts for servers for high level client components like TSslHttpRest. 3 - Undertook a major clean-up of OpenSSL functions that may require end user application changes if low level OpenSSL functions have been used, hopefully very rarely. All OpenSSL functions have been renamed to their original names removing ICS f_ prefix for commonality with other Delphi applications. Renamed literals BIO_CTRL_PENDING_, CRYPTO_NUM_LOCKS_, CRYPTO_LOCK_, OPENSSL_VERSION_ with underscore to avoid conflicts with functions of the same case sensitive names. Removed unit OverbyteIcsLibeayEx not used for a while. Most OpenSSL type pointers to Dummy array are now simple Pointer type. Changed most OpenSSL var parameters to pointers for commonality. New TSslWSocket method SslOK is TLS/SSL negotiated OK. LoadVerifyLocations no longer lets OpenSSL use default CA folder locations which is potentially a security risk. 4 - Bring server SSL/TLS security levels up to latest Mozilla recommendations. In 2021 Mozilla now recommends TLSv1.3 as modern ciphers and TLSv1.2/1.3 as Intermediate supporting all browsers from last five years, so IcsHosts now use sslCiphersMozillaSrvTLS12 as Intermediate level, also Mozilla recommends no cipher server preference so changed that. Updated SslSrvSecurity levels: sslSrvSecInter, sslSrvSecInterFS and sslSrvSecHigh now all the same TLSv1.2 or 1.3, sslSrvSecTls12Less now TLSv1.2 only, sslSrvSecSsl3 not supported, only sslSrvSecBack supports TLSv1 and 1.1, sslSrvSecTls13Only unchanged TLSv1.3 only. 5 - Added support for YuOpenSSL which provides OpenSSL in a pre-built DCU statically linked into applications, rather than using external OpenSSL DLLs. This make application distribution more reliable since it can not fail by users deleting the DLLs or copying incompatible versions into the directory. YuOpenSSL is a commercial product from https://www.yunqa.de/ and is supplied as separate compiled DCUs for Delphi 5 to 10.4. DEFINE YuOpenSSL in Include\OverbyteIcsDefs.inc determines whether the DCU is linked or the external DLLs. Note only one version of OpenSSL can be linked with YuOpenSSL, whereas different DLLs can be supported. Apart from setting the define and adding a path to YuOpenSSL.dcu, no other application code changes are needed unless you check or report the DLL directory, when GSSLStaticLinked can be checked to see if OpenSSL is statically linked. Also worth noting that YuOpenSSL provides access to the all OpenSSL APIs, while ICS only accesses those DLL APIs needed for functionality we support. 6 - Added a better way of configuring Socks proxy and HTTP Tunnel proxy settings in TWSocket, similarly to that added for THttpCli in V8.62. The new ProxyURL property sets SOCKS or HTTP Tunnel proxy settings using a single URL, ie proto://[user:password@]host:port where proto is socks5 or http. Also added new methods SetSocks and SetHTTPTunnel to set proxy properties in a single call. Socks proxy and HTTP tunnelling again support host names, broken in V8.56. The sample OverbyteIcsSocksTst has been updated to demonstrate use of the new ProxyURL property and also now supports HTTP Tunnel proxy. 7 - TIcsRestEmail has new OAAuthType property and event for OAuth2 browser URL perhaps via email for servers. TIcsRestEmail now sets OAuth2 errors, and clears old tokens. Increased default timeout for TIcsRestEmail AOuth2 login to 120 seconds since several web page warnings may be seen while attempting to login through the web page. 8 - TIcsHttpMulti has a new ParseLevels property that follow links on a parsed web page to lower level pages, which are also parsed, looking for files to download with a specific file extension, ie .avi. It ignores parent links without a file extension, and external links. This allows the component to download files indexed using HTTP from CCTV camera memory cards. 9 - In TSslHttpRest ensure SSL is initialised for non-REST HTTP requests like GET, POST and PUT. 10 - In TSslWSocketServer, added better error handling when IcsHosts SslCert is blank, so SslContext is still created for automatically created self signed certificates. 11 - In TSslWSocketServer, TSslHttpAppSrv and TSslHttpSrv, added new onBeforeCtxInit event which if set is called once before each IcsHosts SslContext is initialised to allow the context parameters to be adjusted for special ciphers or protocols. 12 - In THttpCli, AddrResolvedStr is now still available after the connection closes to report the IP address used for the connection. When using an HTTPS proxy with the CONNECT command don't send host and port which some servers are unable to handle, probably broken with V8.64. 13 - In TIcsMailQueue, added a new method CheckOAuthLogins that should be called before the queue starts when using OAuth2 email logins, which allows interactively checking with a browser of any OAuth2 login credentials, avoids asking for login first time email sent maybe days later. 14 - In TSslX509Certs, When starting the local web server for file challenges, check the server IP address still exists or change to 0.0.0.0, may not work if multiple IPs exist. 15 - Updated to OpenSSL 1.1.1k with two high security fixes, one introduced in the last version relating to a certificate chain validation that ICS does not use, the other relating to servers using TLSv1.2 and renegotiation, and that feature is removed in V8.66.

Just tested with a D2007 project and no problems at all.

On Windows, you can use the UniScribe API to get the glyphs from UTF-16 sequences. We did that for our https://github.com/synopse/mORMot/blob/master/SynPdf.pas library. But it is not cross-platform at all! ICU is huge, but the cross-platform way of using it. This is what we did in mORMot 2 - not for glyphs but for case folding, comparison and codepage conversions. Check also https://github.com/BeRo1985/pucu/tree/master/src which is a pure-pascal Unicode library... I recently found https://github.com/bellard/quickjs/blob/master/libunicode.h which is very well written, and eventually available in our mORMot2 QuickJS wrapper.

I assume "I get an OS error" means "CreateProcess(..) returns false and GetLastError() will then return ERROR_ELEVATION_REQUIRED". Is that correct? If so, it appears that CreateProcess(..) cannot be used to automatically handle the UAC prompt and everything beyond that for you. Use ShellExecute(..) instead. (Source: Using the ElevateCreateProcess Fix | Microsoft Docs)

Ouch.... I wish you full recovery.... I would also say speedy, but you don't recover fast from that thingy.... I also had mild symptoms (I had way worse flu infections), but fatigue is beyond extreme... I literally slept for a month. Absolutely. It is quite bad experience even with mild symptoms.

Hope you and your family recover fully @Lars Fosdal.

Why should programmers care about building times? https://xkcd.com/303/

Ok, here is the installable component. The dimming is not tied to visible anymore, but to a property Active, which can only be set at runtime. The control to be dimmed isn't tied to the parent at design time, instead there is a published property DimmedControl, which the parent will be set to at runtime (unless it's nil, then no dimming is possible). At design time the component can be dropped anywhere to design the child controls. DimmedControl can be changed at runtime. The attachment contains packages for runtime and design and a crude demo. I haven't changed aehimself's unit- and class names, which I should have done, but I'm too tired now. I'm using Delphi 10.3.3, it should work for 10.4.*, too. Instructions: Load BuildandDemo.groupproj Build DimPanel.bpl (runtime) for both Win32 and Win64 Build DimPanelD.bpl (design) Install DimPanelD.bpl Open and run DemoDim The component is on the "Additional" tab. If TDimPanel is grayed under Win64, try to close and reopen the IDE. Hope it roughly works 🙂 Edit: Bug in previous attachment The size was not set right initially, has been fixed by an override to Loaded DimPanel.zip

Not even to parse HTML?

I suggest that you read once again what @Remy Lebeau posted. You example is horrible and no amount of dancing around it will ever make it work properly. When you run it, and it works... that will only be an illusion caused by undefined behavior and a touch of luck.

It wasn't that, and I probably did forget the bind switch before I sent it to you, however with the new version.... we have lift off! Awesome work! E:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\FinalBuilder9.map -v -bind:I:\FBAT_HG9\Output\FB9\FinalBuilder9.exe map2pdb - Copyright (c) 2021 Anders Melander Version 2.0 Constructed a new PDB GUID: {CE1E6D08-67E6-4B5B-B42A-3FC719254F1E} Output filename not specified. Defaulting to FinalBuilder9.pdb Reading MAP file - Segments Warning: [ 8] Empty segment: .pdata [0006:00400000] - Modules - Symbols Warning: [20671] Failed to resolve symbol to module: [0005:00000608] SysInit.TlsLast Warning: [20673] Failed to resolve symbol to module: [0004:FF77A000] SysInit.__ImageBase - Line numbers Warning: [71563] Line number with zero offset ignored. Module:VSoft.IDE.Types, Segment:0001, Source:VSoft.IDE.Types.pas, Line:49 Warning: [71567] Line number with zero offset ignored. Module:VSoft.IDE.Interfaces, Segment:0001, Source:VSoft.IDE.Interfaces.pas, Line:328 Warning: [75481] Line number with zero offset ignored. Module:LMDRTLConst, Segment:0001, Source:LMDRTLConst.pas, Line:213 Warning: [75621] Line number with zero offset ignored. Module:intfLMDBase, Segment:0001, Source:intfLMDBase.pas, Line:44 Warning: [76980] Line number with zero offset ignored. Module:LMDDckCst, Segment:0001, Source:LMDDckCst.pas, Line:29 Warning: [79649] Line number with zero offset ignored. Module:VSoft.IDE.ActionListInterfaces, Segment:0001, Source:VSoft.IDE.ActionListInterfaces.pas, Line:136 Warning: [90056] Line number with zero offset ignored. Module:SynEditStrConst, Segment:0001, Source:SynEditStrConst.pas, Line:577 Warning: [98436] Line number with zero offset ignored. Module:VSoft.IDE.WizardManagerService, Segment:0001, Source:VSoft.IDE.WizardManagerService.pas, Line:61 Warning: [99799] Line number with zero offset ignored. Module:dwTaskbarList, Segment:0001, Source:dwTaskbarList.pas, Line:213 Warning: [103835] Line number with zero offset ignored. Module:RestException, Segment:0001, Source:RestException.pas, Line:26 Warning: [105461] Line number with zero offset ignored. Module:VSoft.Shared.UXLControlIntf, Segment:0001, Source:VSoft.Shared.UXLControlIntf.pas, Line:60 Warning: [107033] Line number with zero offset ignored. Module:VSoft.IDE.MS.ActionCollection.Interfaces, Segment:0001, Source:VSoft.IDE.MS.ActionCollection.Interfaces.pas, Line:1085 Constructing PDB file - Collecting source file names - Module streams - Strings stream - PDB Info stream - TPI stream - Symbols stream - DBI stream - IPI stream - Finalizing PDB file Patching PE file - PE32 image (32-bit) - PDB file name has been stored in debug data. - Header checksum has been cleared. - PE file has been updated. I was also able to create a pdb for the package E:\map2pdb>map2pdb.exe I:\FBAT_HG9\Output\FB9\VSoft.Core.map -bind:I:\FBAT_HG9\Output\FB9\VSoft.Core.bpl -v map2pdb - Copyright (c) 2021 Anders Melander Version 2.0 Constructed a new PDB GUID: {76323C0F-0787-44F8-BC72-2B6ACC9CBF82} Output filename not specified. Defaulting to VSoft.Core.pdb Reading MAP file - Segments Warning: [ 7] Empty segment: .tls [0005:00000000] Warning: [ 8] Empty segment: .pdata [0006:00400000] - Modules Warning: [ 1682] Module exceed segment bounds - ignored: OtlCommon.Utils [0005:00000000+256] - Symbols Warning: [ 6403] Failed to resolve symbol to module: [0005:00000000] OtlCommon.Utils.LastThreadName Warning: [ 8705] Failed to resolve symbol to module: [0005:00000100] SysInit.TlsLast Warning: [ 8709] Failed to resolve symbol to module: [0004:FFBF8000] SysInit.__ImageBase - Line numbers Warning: [64147] Line number offset out of range for module. Offset:003DAEEF, Module:.VSoft.Core [003DAEF0 - 003E5738] Constructing PDB file - Collecting source file names - Module streams - Strings stream - PDB Info stream - TPI stream - Symbols stream - DBI stream - IPI stream - Finalizing PDB file Patching PE file - PE32 image (32-bit) - PDB file name has been stored in debug data. - Header checksum has been cleared. - PE file has been updated. Now to hook it into every project in the group and see how it goes 😃 Thanks again for this awesome effort.. my only question now is, if you can do this in your spare time, why TF can't embarcadero do it 🙄

Hello Rolf, There is a file in the directory with the suffix, should be python35._pth that needs to be modified to import site. Here are the references I used to get past the issue: https://dev.to/fpim/setting-up-python-s-windows-embeddable-distribution-properly-1081 https://www.christhoung.com/2018/07/15/embedded-python-windows/

For those following this thread:

It appears that ICU is the way to go. ICU is now bunded in Windows, .NET 5 is based on ICU, also android Unicode and internationalization support | Android Developers and iOS ICU usage in Swift - Development / Standard Library - Swift Forums

No - when the already allocated block is large enough it doesn't do much. And when it has to actually allocate it just does a lock cmpxchg on a bool field of the block types - when its already locked by another thread it simply tries the next bigger one. Speaking of FastMM4 allocation was never the problem with many threads because of this mechanism - it could only lead to a bit memory overhead because some small blocks might be little overallocated because it had to go past the currently locked sub allocators. The issue was in deallocating because that needs of course to the allocator it the memory belongs to. This is what Pierre addressed in V5. FastMM having *a* global lock is a myth.

There's no supported way to do what you want and that's a good thing.

Recently upgrading to Android 10, I notice a few weird inconsistencies with the various App icons. It seems that the icon embedded in the App's APK is either ignored (replaced by an icon from an icon pack depending on whichever launcher is in use), shrunk down and fitting into some shape (circle, round-rect, etc) and even when shrunk into a shape, some icons are circle-cropped while other are just squares (with the embedded icon being either a square or a round-rect). Is there a manifest entry that lets the launchers know I just want my app's icon to display as-designed?