A.M. Hoornweg

Strings should not be used for byte manipulation (which does not contradict an earlier post of mine, explaining how to get legacy code libraries working if they do this).

Rather, write a new class or class helper that enhances tbytes and offers the functionality that makes strings so practical (insert, delete, append, concatenate, pos). Maybe even some new classes "tBytelist" and "tByteBuilder" as an analog to tStringlist and tStringbuilder.

On 9/22/2020 at 8:46 PM, c0d3r said:

Try deleting one component on an ancient form, causing all sorts of AVs on inherited forms. Turns out you have to open each inherited forms at design time after the deletion, to answer a dump question "if the reference is deleted or renamed"..., then save them.  What were these $&&*%$$ EMBR guys thinking?  The worse part was that some of inherited forms were OK no AVs and some of inherited forms weren't OK and AVs.

 

Imagine if you have hundreds inherited forms....

When I want to safely delete a component on an ancestor form, I do it outside of Delphi.

I use a search tool (Agent Ransack) to find out if the component appears in any descendant forms.   I simply search the folder for all *.dfm files which contain the text  "inherited mycomponentname".  Then I simply drag &drop the resulting list of file names onto an empty Notepad++ window and perform the deletion manually. 

When this is done, it is safe to delete the component on the ancestor form.

2 minutes ago, Anders Melander said:

That is not the purpose of the checksum. The purpose is to guarantee that the output is correct.

Which brings us back to the solution I proposed originally. Ensure that the compressed data is valid (with another checksum) before trying to feed it to the decompressor.

7 minutes ago, Anders Melander said:

I think the checksum is there to guard against a broken implementation. You can't validate the algorithm. Only the output it produces.

I'm not really sure what it is you're disputing.

If the data can only be verified after expansion and the expansion algorithm crashes, then we still don't know if the data or the implementation is broken.

1 hour ago, Anders Melander said:

Yes, you're right and it makes sense. The integrity of the data within a container is the responsibility of the container so the ZLib decompressor can assume that it's being given valid data and only uses its own checksum to verify that it is producing the correct output.

If that were true, a non-matching checksum would mean a broken algorithm and not necessarily broken data. It would tell us exactly nothing.

15 hours ago, Anders Melander said:

Didn't anyone Google "zlib checksum"?

According to the ZLib specification there is a checksum: https://tools.ietf.org/html/rfc1950

 

"contains a checksum value of the uncompressed data"

This isn't foolproof because the decompressor can only verify that checksum after decompression. What if the data is corrupted in such a way that it causes the decompressor to crash (I quote OP: "endless loop") during decompression?

One should also append a record containing the size and checksum of the compressed data at the end of the stream. That way one can check the integrity of the file before decompressing the data. 

9 hours ago, Lars Fosdal said:

"Let it crash" is not on my list of stability strategies. 

I had situations where an ADO connection to MS SQL Server would just "lock up" whilst executing a SQL statement and the call never returned. No exception, no error, just a lockup. I suspect it had something to do with our IT department's scheduled backup job saving the SQL Server VM because it always happened around that time of night.

Since lockups like that are out of the programmer's control, all I could do was design a workaround and it turned out that a watchdog thread worked brilliantly.

VMWare workstation will let you do this.

Just add two virtual serial ports to the VM, in the field "connection" specify "named pipe".  Both COM ports must use the same name for the named pipe (just call it "serialportpipe") and one port must be configured to be "server", the other must be "client".  That way, the two COM ports are crosswired.

Now you can start 2 applications, connect them up to the two COM ports and let them talk to each other.

1 hour ago, Steve Maughan said:

That was a question I had — why wasn't there a checksum error when it tried to decompress?

I don't know if Zlib writes a checksum or not, maybe you should append it yourself at the end of the compressed stream. Then first verify that checksum before trying to decompress it. 

4 hours ago, FPiette said:

I am using that solution with great success.

1) The main service launch a child process which does the work.

2) The main service is notified by Windows when his child exit (You can have a thread waiting for process termination).

3) The main service check child process health by sending a request using IPC. In my case the child service is accessed by clients so the main service use the same code to connect to the child process. If the connection fails, then the main service kill the child process.

4) When the child service gracefully terminate it set a flag so that the main service do not restart it but also close. The flag is actually a file created by the main service and deleted by the child process when terminating properly. If the main service detect the file is still there when the child process terminate, then the child process crashed.

 

I do it slightly differently:

- The main process launches a child process for the work.

- The child process creates a mutex.

The service application knows if the child process is still running by periodically checking the mutex.

- The child process has a separate thread called tWatchdogThread. 

- The main thread must call "tWatchdogThread.Notify" regularly to prove it's still alive.
- If that sign of life stays out for more than 10 seconds, the watchdog assumes the main thread has crashed and ends the process using TerminateProcess(). 

This mechanism has proved to be extremely reliable, I use it often.

11 hours ago, aehimself said:

Exactly my thoughts. When I was hit with the first encoding issue I switched to TBytes. Easy to handle, but really powerful; especially with the built-in TEncoding classes.

RawByteString, as it was mentioned earlier; but that also requires you to change the library itself

For my own source code I switched to tBytes too, long ago.  But I have several legacy third party libraries of companies that no longer exist and that are still very nice when used with Delphi 2007.  So when I came across Raymond Chen's blog I had the idea to salvage these libraries without mutilating the existing source code too much.  And I stated loudly and clearly that this is just a quick & dirty fix to get it done.

As Remy Lebeau stated, assigning RawByteString <-> String may still do a codepage conversion.  Ansistring(437) and Ansistring (28591) are safe.

26 minutes ago, Anders Melander said:

*cough* WideString *cough*

Available since D4 or thereabouts...

The VCL was strictly Ansi. I needed visual unicode compatibility in the early noughties and was forced to switch to a third-party VCL (first TNT, then LMD Elpack). 

7 minutes ago, Darian Miller said:

 

I dispute the concept that it was always considered bad practice.  Like many things, over time it was eventually agreed to be universally known as bad practice, but it's revisionist history to claim that it was always considered such.

How many Delphi Informant magazine articles, website articles and blog posts, along with commercial components and enterprise wide codebases were littered with this usage and there wasn't a peep about bad practice for years?

 

 

 

 

Fair point.  Moreover, the only tool in Delphi that could perform operations like copy/append/delete/insert/search on variable length data out of the box was "string". If all you have is a hammer, every problem looks like a nail.

I remember that Windows NT (from the mid-1990s) was already unicode-enabled but Delphi didn't support that until version 2009. So all that time, a char and a byte were basically synonymous in Delphi.

1 hour ago, David Heffernan said:

Kind of odd that you wouldn't just use a byte, TBytes. 

I do, all the time. But this is about salvaging older (possibly third party) ansi code without a rewrite. 

37 minutes ago, Fr0sT.Brutal said:

Well, while this example by R.Chen is good finding, it's a bit useless for Delphi. Why reinvent the wheel when Delphi already has codepage-neutral RawByteString that won't do any char conversion at all?

AFAIU only "uint8" <=> "uint16" conversion is performed when assigning Rawbytes to unicode and vice versa.

The documentation says that "Rawbytestring should only be used as a const or value type parameter or a return type from a function. It should never be passed by reference (passed by var), and should never be instantiated as a variable."   I read that as "you may not declare variables of type "RawByteString" and do stuff with them".

But if I define a Type Ansistring(437) and put binary data into it, the data is associated with a valid code page. A conversion to unicodestring will convert characters [#128..#255]  to some different unicode code points, which is uninteresting, because a back-conversion to ansi codepage 437 will produce the original binary data again.  So it is safe to use all normal unicode string functions as long as we remember to assign the final results back to an Ansistring(437).

Hello all,

when Delphi didn't know about unicode yet people would often stuff binary data into strings because strings were soooo practical and easy to manipulate.  Yes that is and was bad practice and highly frowned upon, but as we all know it was done anyway and people got away with it because Delphi didn't care about code pages at the time so it just worked. It was even done in professional libraries.  Code like that is very hard to port to newer Delphi versions which are unicode-enabled and codepage-aware and when you attempt a conversion, subtle errors may happen where you least expect it. 

If you still have precious old code libraries that do such things and which would be too costly or too complex to rewrite,  you may consider trying this workaround as a quick & dirty fix:

Type Binarystring= type Ansistring (437);

The nice thing about code page 437 is that all 256 possible ansichar values map to valid unicode code points so you can safely assign these strings to Unicodestrings and back again without data loss and Delphi's built-in string functions won't break the binary data contained in these strings. So you just may me able to salvage some legacy code by declaring this new type and then replacing all "string" declarations in the code with "binarystring" and all (p)Char with (p)Ansichar.  

And yes, it's still bad practice...

(The idea is originally from Raymond Chen:  https://devblogs.microsoft.com/oldnewthing/20200831-00/?p=104142 )

On 9/25/2020 at 8:25 PM, Remy Lebeau said:

https://quality.embarcadero.com/browse/RSP-31118

Voted!

6 hours ago, pyscripter said:

I suppose this is understandable.   The Windows world is on UTF-16 and the rest on UTF-8.  UCS4 (UTF-32) is very rarely used.

 

With the benefit of hindsight, UTF-8 would probably have been a better choice for Windows.  Now Microsoft is trying to bring UTF-8 back into Windows via the A routines and by offering to make UTF-8 the default character encoding.

Most programmers ignore that UTF-16 is a variable-length encoding and treat it like UCS-2, falsely assuming that one widechar corresponds to one codepoint.  While most of us probably don't handle exotic languages, this is the 21st century and sooner or later you'll stumble upon strings that contain Emoticons which just won't fit in one widechar ( https://en.wikipedia.org/wiki/Emoticons_(Unicode_block) .

Delphi could use some better support for that, like iterators etc. Something like this:

Function ReverseString (S:String):String;
VAR c:UCS4Char;
Begin
  Result:='';
  FOR c in s do 
   result:=c + result;
End;

And if you think this is far fetched, just look how elegantly Freepascal solves this, https://wiki.freepascal.org/for-in_loop#Traversing_UTF-8_strings . 

OK, I'll post a QC then.

Edit: https://quality.embarcadero.com/browse/RSP-31114

19 minutes ago, David Hoyle said:

The \\?\UNC is for unicode path up to 32K in length so you can work around the MAX_PATH limit. I wouldn't expect ExtractFileDrive to work with these as you would add the \\?\UNC prefix before calling CopyFileW() etc. There is similar syntax for drive letter. 

The word "drive" is a bit unlucky. What ExtractFileDrive() really does is to return the root of a volume, which can be either a drive letter or a file share.

ExtractFileDrive ('\\server\share\folder\filename.ext") returns "\\server\share", which is perfectly OK. I can use the result as the root path for a folder structure without problems.

ExtractFileDrive ('\\?\C:\folder\filename.ext") returns "\\?\C:" which is also perfectly OK. This result, too, can be used as the root path for a folder structure without problems.

The only corner case that isn't handled correctly is "\\?\UNC\" which is needed for long network paths. I think it wouldn't hurt anyone to support that syntax as well.

2 minutes ago, Anders Melander said:

One could argue that it's equally meaningless to use ExtractFileDrive on that path since whatever comes after the "\\?\" is handled directly by the file system and not by the file namespace parser.

I guess the correct thing to do would be to return an empty string as documented:

I do not agree with your suggestion that it should attempt to extract anything if the string starts with "\\?\" (or \\.\ for that matter). The result would be meaningless in the context.

I respectfully disagree.

The prefix "\\?\" is a well-known method to tell Windows that a program is able to parse file names longer than MAX_PATH. It seems that the prefix \\?\UNC\  extends this ability to network paths. Otherwise I wouldn't bother with them. 

But in this specific case I needed to test if two paths referred to the same volume so I used ExtractFileDrive, which failed on this syntax.

Hello all,

could it be that function UCS4StringToWideString misses the final character during the conversion?

VAR   s:UCS4String; W:String;
begin
    s:=[220,98,101,114,109,228,223,105,103];
    w:=UCS4StringToWideString(s);
    showmessage(w);
end;

I've just discovered that ExtractFileDrive misses a corner case on Windows and posted a QC about it. 

https://quality.embarcadero.com/browse/RSP-31109

I was experimenting with very long file and path names (see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) and stumbled upon a file name syntax which I hadn't seen before. It turned out that Delphi didn't know about it either.

The syntax is \\?\UNC\server\share which is just another way of writing \\server\share. 

Delphi's ExtractFileDrive returns "\\?\UNC" on such a path which is meaningless.

15 hours ago, Fr0sT.Brutal said:

Most of these checks could be performed with IF DECLARED clause. Thanks to it very few features must be defined manually, just some language features or unit relocations. I maintain a repo with defines for main compiler features here so use the provided info freely or make PR's/issues to add something that is missing.

This doesn't work for overloaded functions.  For example, function "strlen" (sysutils.pas) is ansi-only in Delphi 2007.  Then, in Delphi 2009 an overload was created for unicode so there were two versions.  In XE4 the ansi version was duplicated in unit Ansistrings so from then on there were three. And then in Delphi 10.4 Sydney the original ansi version in sysutils.pas got "deprecated". 

But I like your *.inc files, they look solid and they are a good reference as well. However I see that 10.4 Sydney hasn't been implemented yet?

14 hours ago, Fr0sT.Brutal said:

IDK what things you mean. Backward compatibility is pretty strong in current versions as main changes happened in D2009..XE2. So newer versions require very little changes, especially if you don't use newest features (and you have to avoid them if you care about older versions)

I maintain some libraries that are used in our company and that have to be compatible with a range of compilers (2007 .. current).  Some of our programs are used on embedded devices with older operating systems hence the need for D2007. We want our products to compile without hints or warnings.

Totally apart from the usual string problems (unicode/ansi and codepages), there's the issue of Embarcadero deciding to move RTL functions from one unit to another, of functions being declared "deprecated" and of functions suddenly being declared "inline".  So in order to suppress such warnings, I need to know when Embarcadero decided to pull off such changes.