Delphi and "Use only memory safe languages"

[David Heffernan 2345]

4 minutes ago, Dalija Prasnikar said:

I hate to be the one saying this... but ARC compiler was a step in that direction. You cannot have memory safety without automatic memory management.

I guess ARC per-se wasn't the big problem, it was more the attempt to support ARC compilers and non-ARC compilers with the same codebases??

[Der schöne Günther 316]

50 minutes ago, Cristian Peța said:

Linux and Windows sources are C, C++, Assembly....

and ... Rust which is often quoted in relation to memory-safety.

 

Linux:

It has been a (small) part of the Linux kernel since 6.1

Rust — The Linux Kernel documentation

 

Windows:

 

[Dalija Prasnikar 1396]

9 minutes ago, David Heffernan said:

I guess ARC per-se wasn't the big problem, it was more the attempt to support ARC compilers and non-ARC compilers with the same codebases??

Yes, that was the core problem. In other words, the ARC compiler alone on the existing codebases and Delphi component system which required DisposeOf would not be any more memory safe than code using non ARC compiler. Some loopholes that would be plugged by having automatic instance initialization and lifetime management, but DisposeOf opens other different loopholes.

 

So, if we would go that route, then all Delphi code (including RTL, VCL and FMX) would have to be written from scratch, which makes ARC compiler not really viable solution after all.

[Stefan Glienke 2004]

59 minutes ago, Dalija Prasnikar said:

ARC compiler was a step in that direction. You cannot have memory safety without automatic memory management.

With the exception that ARC is one of the worst ways of doing that - here are some reasons.

Edited March 6 by Stefan Glienke

[Brandon Staggs 278]

3 hours ago, Sherlock said:

Now there's something the folks over at Embarcadero can aspire to. Imagine built in memory safety in Delphi.

Did you read the abstract on the link? There is no way Embarcadero can do anything like that. They would have to literally transition away from Delphi.

 

Try to keep Delphi? You'd have to change the language too much. Their attempt to do that (ARC) failed for several reasons.

 

If you really need that kind of hand-holding, you need to move to a different language.

 

Expecting Embarcadero to do anything to help you transition away from Delphi (or C++) is not realistic. Expecting them to make it so you don't need to transition away (assuming, again, you want the hand-holding instead of having to manage memory yourself) from Delphi is also as unrealistic as expecting Google to somehow make C++ "memory safe."

[Brandon Staggs 278]

2 hours ago, David Heffernan said:

I guess ARC per-se wasn't the big problem, it was more the attempt to support ARC compilers and non-ARC compilers with the same codebases??

That's a big problem if you are trying to convince your existing customers that they can reuse their code on new platforms. Embarcadero obviously realized that was a losing strategy.

[dummzeuch 1505]

4 hours ago, Cristian Peța said:

Linux and Windows sources are C, C++, Assembly....

Some parts of the Linux kernel are being converted to Rust. Also, many parts of the Linux distributions are written in Python oder even shell scripts. Some Perl can still be found there too.

[David Schwartz 426]

I didn't dig very deeply into the nuts and bolts of this issue, and was quite surprised to see Delphi on the list. 

 

I read a lot of comments in the post I found, and it looked like the main thing people were objecting to vis a vis C and C++ is the highly idiomatic use of pointers to iterate through lists, which is so pervasive that it is found all over the STL and many common libraries. Such pointers aren't considered type-safe unless you tell the compiler to check them, so that could be the root of the problem.

 

While Pascal has Pointers, they have pretty much disappeared from use, except in highly-optimized code (which may be CPU-specific) and where Record types are employed. 

 

On the flip side, I can see why the White House might want to make an exception for Delphi since the VA may be the largest single organization in the world that's still heavily dependent on Delphi -- that and MUMPS. There are tons of vendors that are required to write Delphi apps that interface with their systems, and they're required to support them for ... decades (?).

 

I worked at a company 12 years ago that's still in business writing and maintaining code for the VA that's all written in Delphi and MUMPS. And every few months I see jobs posted on various job boards that are looking to hire contractors to work on projects with the VAs signature all over them. (They're looking for Delphi devs with some MUMPS exposure, familiarity with VistA, and a few other "code words" in the job reqs that give it away.) The VA's Delphi code is not going away any time soon, and continues to grow.

 

The point being, excluding Delphi would force Congress to invest a HUGE amount over the next decade to replace all of the Delphi code currently used in the VA. The Good News, however, is that by avoiding the use of pointers (which most of us do anyway) reduces the risk (that they seem to be focused upon) quite a bit.

 

That said, I found code examples in the code base I was working on that were written by a MUMPS programmer who learned Delphi and did a lot of coding on the app over time ... that led to the discovery of some very odd code expressions that are written as MUMPS expressions that actually compile in Delphi but don't work the same as they do in MUMPS!

 

When I found the first one, it looked like perfectly good Dephi code, but it made no sense. After staring at it for quite a while, it seemed similar to some C++ short-hand expressions, but the semantics don't match in Pascal. Then I remembered that the guy who I believe wrote this code was mainly a MUMPS programmer, and then I read it as MUMPS expressions and VIOLA! It made perfect sense! Oddly, it compiles in Delphi, but would throw intermittent errors depending on the data it was dealing with. It was a total fluke that it ran correctly over 99% of the time. (I guessed the error rate was between 1:1000 and 1:10k, but it was hard to nail down.)

 

I identified around 8 specific coding error patterns that occurred in over 100 places in the code. (The programmer used a lot of copy-and-paste techniques to save time, which made these snippets fairly easy to find once I knew what to look for.) A lot of them resulted in random data-dependent errors in code that had no error handling; they were also impossible to reproduce. I'd bet none of these have been fixed to this day.

 

If someone knew where this code was, it could probably be exploited. 

 

 

Edited March 7 by David Schwartz

[Sherlock 663]

16 hours ago, Brandon Staggs said:

If you really need that kind of hand-holding, you need to move to a different language.

There's an argument I can subscribe to: Spare us the syntactic sugar of the last 15 years and concentrate on the bug fixing.

Digit separators, generics, inline variable declaration, multiline string literals, string-like operations support for dynamic arrays? Can them all, and burn the heretics who asked for them

[Attila Kovacs 629]

15 hours ago, dummzeuch said:

Some parts of the Linux kernel are being converted to Rust

This rust appears to be yet another write-only language.

 

10 minutes ago, Sherlock said:

generics

Generics is fine. 🙂

 

[Kas Ob. 121]

I mentioned or to be exact dreamed about more evolved Pascal enhancement, mentioned it here at once as i recall, but really can't remember any comments or any discussions.

So will write it here as it is relevant for memory safety.

My suggestion is to introduce new modifier(s) to managed (and non managed) variables, this include classes/objects, the new main modifier is "Auto", example

procedure Test(aFileName: string);
var
  SList: TStringList; auto;           // the compiler will ensure calling the default constructor and destructor
begin
  //SList := TStringList.Create;      //  should stop the compiler with specific Error, you are not allowed to create an auto
  SList.LoadFromFile(aFileName);
  //SList.Free                        //  should stop the compiler with specific Error, you are not allowed to destreuct/free an auto
end

There is also similar modifier like "init/create" and "free" these will only their part and the compiler will ensure you can't abuse the memory layout or use after free.

This also can be applied to fields in TObject/Class, the compiler will put the default constructor and destructor in their right position and will stop compiling when the code is trying to call constructor and destructor.

 

But one could say using "auto" for the fields in classes will make initializing slower, no it will not as sooner or later it will initialized or used (except other usage, will follow down), will this cause degradation in performance?, no, not really, after all there is a price for pay.

 

Another modifier which i can argue will be very helpful, is "delayed" or "onuse".., this one is for places when you don't know if that class will be needed or not, so the compiler will not auto create it !, but will initialize it to 0 (nil), and will check against that nil on every first usage with in the current scope then call the constructor in-place, guaranteeing its readiness, while will enforce calling the destructor in every scope it mentioned (referenced) but again if it is nil then nothing to do.

With "delayed" modifier there is a performance hit for sure but it is merely "if assigned(x)" and as i mentioned above at the first usage in the current scope, so when in doubt i will leave to the compiler but when i am sure i will need that delayed to be ready for lets say a loop, then i can call specific function on it or just touch it in any code so the following code will not have to have the check for creation, of course if such delayed var is used in a loop then the compiler will put the constructor before entering the loop (at the loop limit preparing code)

 

Also to mention how clean the code will become not just eliminating user after free or use non initialized.

 

as for compiler: Delphi compiler is already equipped and has the ability to do that so minimal adjusting is needed, it does the same with all managed types (strings, arrays..) for the suggested modifiers above it will be close to this

Currently the compiler does the same with strings and arrays as seen here ( though this is my XE8 so not sure about the latest versions)

 

for combining the suggested "delayed" with the above the compiler could check for referencing one to another and adjust or rearrange the creation and destruction order.

 

Anyway, this i this another short by me to reach a discussion about this enhancement or call it evolution, also this is the right evolution toward more memory safety.

 

There is also can be a suggestion for different modifier "secure" "mute" (or something), with this one the compiler will allow any assigning to referencing (or double referencing), in other words and as example, X is a class, no code such Y := X is allowed, you need X (the object in X) you have to use it where is it which is X, This will ensure specific design and will limit pointer miss using, you want to load a bunch of the like of that X in an array then either declare them there with in the array and use them there, or, custom and precise steps should be done as  guarantee to the compiler that this is well thought of, otherwise we need to write better code and design the structure better, another example ; X is secure object and it is a field in the non secure Z, the compiler will not allow to use X out side the scope of this current scope and/or outside the Z, meaning, the compiler will not allow passing X as parameter to a procedure, but will allow Z, X and its value is not allowed to be used in any direct assigning ":=", still we have the ability to introduce assign as operator that will ship or duplicate X fields, but not X itself or its pointer, that pointer should not be reached or allowed to be accessed by the compiler, and when i say compiler i mean the new language constraints for these suggestions.

 

[JonRobertson 72]

6 hours ago, David Schwartz said:

While Pascal has Pointers, they have pretty much disappeared from use

Not on my system. A simple grep of 51,132 .pas files tells a different story. (Just a whole word search with no context).

pointer, 150,825 hits in 7,939 files

^,  110,934 hits in 6,326 files

 

Searching a few more popular (and current) open source libraries reveals significant use in current code. 8,625 hits was the highest I found in a single open source library on my system.

 

Delphi 12 source\rtl\sys - 2,837 hits. 1,871 hits in system.pas alone.

Delphi 12 source\rtl\common - 2,951 hits

Delphi 12 source\rtl\win - 6,442 hits. I was surprised to see 2,243 of those are in the winrt subfolder.

Delphi 12 source\rtl\osx - 4,751 hits

 

Then there are thousands of casts using TObject(), such as storing object references in the legacy TList or, ahem, storing references or pointer values in TComponent.Tag.

 

If I overlook RTL usage, VCL usage, and poor coding practices, every day Delphi coding uses pointers. At least I do.

 

// lstOfItems is a pointer. You may say it is a "typed pointer", but it is still a pointer. ;)
var lstOfItems := TStringList.Create;

 

[Brandon Staggs 278]

7 hours ago, David Schwartz said:

While Pascal has Pointers, they have pretty much disappeared from use, except in highly-optimized code (which may be CPU-specific) and where Record types are employed. 

 

That's a pretty bold statement. All of the non-trivial projects I work on make use of pointers. Saying they have "pretty much disappeared" is a bit of a stretch in my experience.

[Brandon Staggs 278]

6 hours ago, Sherlock said:

There's an argument I can subscribe to: Spare us the syntactic sugar of the last 15 years and concentrate on the bug fixing.

Digit separators, generics, inline variable declaration, multiline string literals, string-like operations support for dynamic arrays? Can them all, and burn the heretics who asked for them

I didn't mean it like that. I was speaking in practical terms. You are not going to get Delphi with Garbage Collection, and Delphi with ARC-everywhere (not just interfaces and dynamic arrays) has already failed. So if you need it, your only choice is to move away from Delphi. I didn't mean this in the "love it or leave it" sense, LOL!

[dummzeuch 1505]

1 hour ago, Brandon Staggs said:

You are not going to get Delphi with Garbage Collection

We already had that: It was called Delphi for dotNET. AFAIK the last Version was Delphi for dotNET 2007. Somehow it didn't catch on.

[David Schwartz 426]

16 hours ago, Brandon Staggs said:

That's a pretty bold statement. All of the non-trivial projects I work on make use of pointers. Saying they have "pretty much disappeared" is a bit of a stretch in my experience.

Sorry, y'all are obviously not reading my mind correctly. I was not referring to simple Pointer Aliases.

 

TL;DR -- Pointer aliases are inconsequential. Pointer math is extremely hazardous. In my experience, companies couldn't care less about making code run faster ... not a "little bit" (minutes saved) nor a "huge amount" (hours saved). Nothing justifies using pointer math in Delphi apps to make apps run faster due to the likelihood of errors that the compilar nor test suites can catch. Desktop apps are far different from real-time process control apps, or even the libraries used in them; NOBODY cares about making them run "a little faster" that a newer faster computer won't fix

 

I was referring to constructs like this (and I may not have even used them properly):

----------------------------------------------------------

var pXyz : pointer;  <----

var xyz: integer;

pXyz := ^xyz;  <----

var i : integer;

var myIntAry : array [0..maxVal] of Integer;

 

for i := Min(myIntAry) to Max(myIntAry) do

begin

  // this is an example of what I was referring to that's used frequently in C and C++

  ^pXyz := Random(1000);

  inc(pXyz);  

end;

----------------------------------------------------------

((sorry, the insert Code thing isn't working))

 

Yes, I realize Pointer aliases are used a lot in the VCL source code, but most of that was written in the 1990's and probably hasn't been updated much since then. ("If it ain't broken, don't fix it.") I also realize that Delphi class vars (eg, myClass : TMyClass) are basically pointers treated as references (like C++: TMyClass : &myClass = new(TMyClass); )

 

The pointers of concern (that use the ^ operator the same way C/C++ uses the * operator) are also the code used any time you have a Record that's allocated from the heap.

 

The thing is, some people still obsess over a few extra instructions that the compiler might generate that can be eliminated using pointers, but I suspect most of them are "old school" programmers from way back in the day when every CPU cycle and byte of memory that could be saved were significant. When programs were 50k, they could be copied to a 128k floppy for distribution, and nobody ever imagined a computer would need more than a few MB of RAM. I was taught that way, but shook off that habit in the 90's. 

 

Today, desktop computers ship with 8 GB or more of RAM. (My first HDD was a whopping 10 MB!) The CPUs are multi-core and have hyper-threading, and if you try to out-guess the compiler's optimizer, the code is more likely than not to run slower due to all of the NOPs injected into the instruction cache. All of that hand optimization is likely to work on only ONE CPU (or CPU family) and will need to be revised for every difference in number of cores, instruction cache size, and maybe even data cache size. 

 

I know there are always "exceptions". MOST people creating form-based desktop apps that interact with a back-end database using Delphi are NOT writing code like that! They _might_ declare a pointer alias here and there, but they certainly aren't writing code that anybody will notice if it runs 100ms faster.

 

At my last job, I was told to "port' something that was written for use by one specific client. It ran once a day and took a bunch of daily invoices scanned into PDF files at a dozen or so sales offices, with hundreds of pages saved into each PDF. It ran some OCR on each page and attempted to read an Invoice# from each one and then put the pages for each invoice into a separate PDF so the customer could view the pages for each invoice online if they wanted to, with all of the pages for that invoice in a single PDF file. It used a Version 1.0 DLL from an OCR app from 1997 that ran under WIndows 98 when it was released. This was 2018 and the code didn't run properly under Win10. For years, this thing ran in an hour or so. When I got hold of it, it took 8 hours to process a thousand or so invoices daily. I also discovered that it was unable to process about 30% of the invoices it was given and simply skipped over them. I completely rewrote it using a much newer OCR library written in Delphi, and my version ran in about 45 minutes for the same data that took the older one >6 hours. NOT ONE PERSON CARED THAT IT RAN SO MUCH FASTER! NOT EVEN THE CLIENT! In fact, my boss made a note on my annual review that this task took me 2 months to complete while they believed should have taken a week or two. (Never mind that their analysis was totally off-base.) The client rep contacted our support wondering why there were suddenly so many more invoices showing up every day in the process summary. I told our internal support person it was because the old code was not processing about 1/3 of them. When she told the client, they said, "Oh, ok. That's good to know." That was it. In use for over 15 years, they never noticed anything was wrong.

 

This is hardly an isolated experience I've had over the past 25 years writing and maintaining Delphi desktop apps for clients. Nobody really cares about even significant performance issues, and most don't notice small problems. Very few ever audit their data, either.

 

I worked at one place and discovered that medical invoices were being printed out and some separator pages were missing, and so customers were being mailed health records for other people accidentally stuffed into their envelopes, and nobody at our company that did the printing and mailing, nor the client company, ever knew! This should have been treated as a MAJOR HIPAA violation, but I was told to keep my mouth shut and just fix the problem.

 

You guys outside of America might be wondering how this happens, but from my experience, it's quite common. Every job and contract I've had, I've found at least one and sometimes dozens if not hundreds of things that had been operating improperly for years. Some were performance-related, but most were just data integrity problems. Management mostly did not want to hear about it. 

 

So I really don't want to hear about all of the exceptions and reasons why it's important to shave 50 ms off of the execution time of some loop you may be working on. The last place I worked printed bills and invoices, stuffed them into envelopes, and sent them out to the US Post Office -- 7 or 8 full 40' semi-trailers EVERY DAY, 7 days a week. They got pissed off if part of a line on an invoice was cut off on a print run of 50,000 invoices; not whether it took an extra 15 minutes to run a given print job, as long as the people who got the materials could read them and they looked "professional".

 

If customers didn't complain, that meant nothing was wrong. Nobody cared what we programmers might find.

 

One thing I've found over the past 15 years is that Management is far more scared of allowing programmers to refactor code than just about anything else. Because trying to shave a few seconds of processing time often leads to bugs that did not exist previously. And yes, that's because they don't have comprehensive test suites (which are notoriously hard to write for form-based Delphi apps).

 

 

Edited March 8 by David Schwartz

[Ian Branch 127]

On 3/8/2024 at 6:28 PM, David Schwartz said:

If customers didn't complain, that meant nothing was wrong. Nobody cared what we programmers might find.

Oh how true.....

[Rollo62 536]

This has bubbled up to an official statement now, I like these fast reactions 👍

https://blogs.embarcadero.com/is-delphi-a-memory-safe-language/?utm_source=Eloqua&utm_medium=email&utm_content=Article-240311-Multicontent

[David Heffernan 2345]

51 minutes ago, Rollo62 said:

This has bubbled up to an official statement now, I like these fast reactions 👍

https://blogs.embarcadero.com/is-delphi-a-memory-safe-language/?utm_source=Eloqua&utm_medium=email&utm_content=Article-240311-Multicontent

This is so stupid. Delphi and C++ are exactly the same in this regard.

[Attila Kovacs 629]

1 hour ago, Rollo62 said:

This has bubbled up to an official statement now, I like these fast reactions 👍 

https://blogs.embarcadero.com/is-delphi-a-memory-safe-language/?utm_source=Eloqua&utm_medium=email&utm_content=Article-240311-Multicontent

[Rollo62 536]

41 minutes ago, David Heffernan said:

This is so stupid. Delphi and C++ are exactly the same in this regard.

Who cares, it sounds good anyway. 

[JonRobertson 72]

1 hour ago, Rollo62 said:

Who cares, it sounds good anyway

I do, and I suspect many others do as well. What I was mostly disappointed about was the statement below.

Quote

the use of formal methods for static code analysis, in particular focused on security. We have been seeing a growing interest in this area also for Delphi, and we have been promoting third party tools that help in this space.

The company promoted in that link and the accompanying webinar was DerScanner. I watched the webinar last September and it intrigued me enough to ask approval to pursue having a small project analyzed and review the results.

 

Their home page had a short form to request a trial. Instead of providing a trial, I received an email that I will share, since it is typical marketing:

 

Quote

Our DerScanner product is a high-tech comprehensive application security solution that combines static (SAST), dynamic (DAST) and software composition (SCA) analysis methods in a single interface. DerScanner allows to identify vulnerabilities and undocumented features in the code of mobile and web applications. DerScanner is a unique tool that is able to analyze both source and binary code. It is the world leader in the number of supported programming languages (36 languages) that can also understand polyglot applications and significantly reduces false positives with its own Fuzzy Logic Engine patented technology Please let me know what exactly you are interested in so we may assist you in the best way.

 

I responded that I requested a trial from their home page. The reply was that my request for a trial was forwarded and included a whitepaper that "describes the key features of DerScanner". One of the key features was that DerScanner could analyze 36 programming languages, "even obsolete Delphi, COBOL and Visual Basic 6.0". I also noticed that the short form on their home page was changed from a trial request to "Get a Demo".

 

They did provide a trial account and I submitted the project along with a couple third-party libraries (with each vendor's permission). The resulting report contained thousands of incorrect analysis. One of the more prominent and obnoxious was “hardcoded encryption key”, for code such as "ctrlKey := GetKeyState(VK_Control) < 0", and every other line of code that contained these three characters in sequence: k e y, including static text.

 

The analysis that DerScanner provided was a complete joke and a total waste of time. The trial included two scans for free (I only used one), additional scans after the trial were $1,000 per scan. Sure, my employer can say "Yes, we are concerned about security. We had our application analyzed by a third-party that specializes in code analysis. They used a highly sophisticated tool named grep to analyze our code for vulnerabilities."

 

I am particularly disappointed in the recent statement because I had direct contact with Ian regarding my frustrating experience, and the laugh that I had after reviewing the generated report.

[JonRobertson 72]

5 minutes ago, JonRobertson said:

2 hours ago, Rollo62 said:

Who cares, it sounds good anyway

I do, and I suspect many others do as well

"Promoting tools to help in this area" sounds good. But "Sounds good" is marketing rubbish.

[FaFaFooey 6]

Can a memory manager for Delphi be modified or written to be as close to a garbage collector without being one? FastMM4/5 can detect leaks on shutdown but what about leaks during runtime..maybe it can monitor special types like TObjectCollect instead of TObject.

[JonRobertson 72]

11 minutes ago, FaFaFooey said:

Can a memory manager for Delphi be modified or written to be as close to a garbage collector without being one?

The memory manager would have to "know" when the memory was no longer needed, otherwise it would release allocated memory too soon.

 

Unless I don't understand the question, it sounds like Automatic Reference Counting, which was already attempted. And it was later removed for a few reasons.