Mike Torrettinni

This probably completely changes my process of export/import csv data. I have 100+ of arrays that I export, so I was testing if MOVE could improve the process of preparing data lines, a little bit. Of course once the data line is in string, the most time consuming is writing to file.

Aha, OK, I understand. I need csv file format, so it has to be non-binary text, I guess.

I forgot to add that I wanted to test this after this post suggesting using MOVE: Of course, in that case it was handling just strings, no integers or other types.

Are suggesting like this?: vFileOut.WriteLine(PrepareLineForExport(xData[i])); and vFileOut.WriteLine(PrepareLineForExport_MOVE(xData[i]));

I'm testing with David's Buffered streams: PrepareLineForExport: 6184 PrepareLineForExport_MOVE: 6561 uses System.Diagnostics, BufferedStreams; procedure TForm2.Button1Click(Sender: TObject); var vFileStrm: TWriteCachedFileStream; vFileOut: TStreamWriter; i: Integer; vLine: string; vSW: TStopWatch; begin FillTestData; TestData; vSW := TStopwatch.StartNew; vFileStrm := TWriteCachedFileStream.Create('c:\tmp\dataexport_'+FormatDateTime('hhnnsszzz',now())+'.txt'); vFileOut := TStreamWriter.Create(vFileStrm, TEncoding.UTF8, 512); for i := Low(xData) to High(xData) do begin vLine := PrepareLineForExport(xData[i]); vFileOut.WriteLine(vLine); end; vFileOut.Free; vFileStrm.Free; memo1.Lines.Add('PrepareLineForExport: ' + vSW.ElapsedMilliseconds.ToString); vSW := TStopwatch.StartNew; vFileStrm := TWriteCachedFileStream.Create('c:\tmp\dataexport_'+FormatDateTime('hhnnsszzz',now())+'.txt'); vFileOut := TStreamWriter.Create(vFileStrm, TEncoding.UTF8, 512); for i := Low(xData) to High(xData) do begin vLine := PrepareLineForExport_MOVE(xData[i]); vFileOut.WriteLine(vLine); end; vFileOut.Free; vFileStrm.Free; memo1.Lines.Add('PrepareLineForExport_MOVE: ' + vSW.ElapsedMilliseconds.ToString); end @Kas Ob.How can I export to text file your fastest Binary version?

True. Not perfect, but if you have only a few of these reported you can quickly manually check for issues and then disable the check. Probably that's why FixInsight doesn't have this check, because it's more of a hint then warning or error.

I see that PascalExpert has a check for this:

Attila showed similar case above, with const, that is also a cause of issues. Perhaps there should be a check when using same variable more than once as parameter in same method.

Interesting how such a simple const vs non-const usage can be the cause of issues. I guess when you have a lot of lines of code in the method, it's easier to make mistake like that. Or even in such a short method, like in example, when you are not careful. I guess, school is never out 😉

@Kas Ob. it's not that much different than other Sets, like Set of integer or Set of byte, same construct. If I may recommend read these resources, I'm sure you will find some value in them: http://docwiki.embarcadero.com/RADStudio/Sydney/en/Structured_Types_(Delphi) Also perhaps of interest: Marco Cantù's Essential Pascal Chapter 4 User-Defined Data Types: https://www.marcocantu.com/epascal/English/ch04udt.htm CharInSet is much slower than IN, should I fix W1050 warning hint?: https://stackoverflow.com/questions/35942270/charinset-is-much-slower-than-in-should-i-fix-w1050-warning-hint

@Tom F It's perfectly OK to complain, even better if you provide details when the issue is resolved or you made any progress with the Embarcadero. In last few years we see that just reporting bugs is not enough, they clear a lot of them but also introduce a lot of new ones. We all want Embarcadero to succeed in providing best Delphi versions, but if we keep our heads in the sand and not 'rock the boat' we are all doing ourselves a big disservice.

wrong post.

Are you thinking of making a commercial product? In that case I would first send your excel to some interested people and let them use it for a little bit. You will see how much customization is needed for their tracking expenses needs. Anybody knowing excel will probably even want to make some substantial changes. Then you can see if you can recreate such customization in Delphi. If you post it here, you could probably have some feedback really fast.

I was trying to run an example of wasteful, inefficient string manipulation to see what happens to memory consumption, without Fastmm. I expected the end result to be high memory consumption, de-fragmented, not leaving much free memory for project to use. But at the end of parsing the project uses almost the same system memory: Start: End: During the parsing the memory consumption is all over, the highest over 1GB: Here is example of string manipulation: uses StrUtils; const cLoop = 100; cMultiplyString = 100000; function MultiplyStr(aStr: string; aMultiplier: integer): string; var i: Integer; begin for i := 1 to aMultiplier do Result := Result + aStr; end; function Parse(aString: string): string; var s1, s2: string; i: Integer; begin s1 := MidStr(Copy(MidStr(aString, 1, Length(aString) - 2), 1, Length(aString) - 2), 10, Length(aString) div 2); for i := 1 to 100 do Delete(s1, 1, 1); s2 := s1 + s1; end; procedure TForm2.Button1Click(Sender: TObject); var s: string; i: integer; begin for i := 1 to cLoop do begin s := ''; s := Parse(MultiplyStr('STRING_TO_PARSE' + i.ToString, cMultiplyString)); end; end; Is Delphi's memory manager so good that even if the work is wasteful, inefficient string manipulation, when it ends it cleans up memory very well?

Thank you @Kas Ob. I think now I understand my question was really poorly formulated, I was focused on strings manipulation, everybody else thought I was stress testing MM. Makes sense that some of you wanted me to go back to school 🙂 I thought I was becoming better at asking questions, I guess I need more work. Even though it probably seemed like a waste of time for everybody else, I learned valuable things during preparing the example, thinking about it, reading about FastMM and 'discussing' in this topic: 1. I still need to work on improving question quality 2. I'm probably little rebellious when getting suggestions like 'google this and then we can talk' 😉 3. What I thought was very wasteful, bad example of string manipulation, was actually not! 😉 even Delphi 7 handled it really well. If I tested this example in D7 before posting, I would probably realize this is not worth even asking. But it led me to test D7 from references to FastMM - to test version without it. So, something that is probably very basic and very obvious fact for seasoned developers, it wasn't for me and wouldn't come to this conclusion without this topic,

Ok, thanks. I tested my example in D7, D2006 and D10.2.3 and end result is always quite clean memory consumption, even though D7 didn't have FastMM implemented, yet. So, pretty good outcome overall - of course we need to take into consideration Anders's observation that test cases probably could use more randomness.

I guess you are expecting something I can't deliver. My first post is best I can provide to explain my interest, I have explanation, example, I added a little more in comments, but not anything too much different than in first post. So, if you have a question what info you are looking, I might be able to add it, but if first post is so nonsense to you, I'm not sure what else I can do.

I guess I would say I'm not typical developer, but just a rookie developer that interest level into a topic is correlated to how quickly I can find out the most about a topic. I understand that could keep at rookie level forever, but I'm OK with it, I accepted my path - also probably too old to suddenly become what I should have been 20 years ago 🙂

We just have different approaches, I was expecting some short explanations, experience or cases where I can see a real problem. Your approach in this topic was to give me keywords to study, which I appreciate, but too broad for this topic.

I do that too, when topic is of interest to me. Not sure why you think this topic is nonsense, it did give me some valuable info. You are more than welcome to add your view on how string manipulation with Delphi is affecting OS memory and in which case it can cause any concern.

It's not that I don't want to, but google has 518M hits.... I'm not that interested reading books or hours of resources to get to a part that relates to my topic. Would love to read a page or two, but only what relates to this topic.

Ok, this makes sense. This was extremely exaggerated example from a few uses of MidStr, Delete and other string functions I have in my code. So I was thinking if any case of real data input is structured in a way that these functions get used in extreme way, what will be the effect. So, it was more like: can my code 'break' OS memory or should I spend days(weeks) trying to micro-optimize every use of MidStr, Delete and others, because I'm afraid how it will affect OS memory in extreme cases.

The feedback on this topic has been very clear and I like the conclusion that even non-perfect code will not break the system memory or cause it to run out of it. Like with memory leaks, most of the time when you exit the process, the memory gets released back to the system. I understand the OS is not perfect in memory handling, but if it handled such extreme example, it will also handle anything I throw at it in production code.

Is this Cached memory? If yes, I think Windows 10 handles this very well on its own, Here is my memory on start: and after 10x(!) running this wasteful example - i didn't measure exactly, but each run uses and releases 10GB+ of memory, so 10x runs used and released 100 GB+ memory: Cached went up by only 1GB (while my system running and not sure if other apps didn't contribute some cache also). I guess the conclusion still stands: Delphi is very good with managing and releasing memory, even if the code is absolutely bad in terms of string handling.

No, but doesn't windows handle the released memory quite efficiently? The memory consumption from just this test is 10GB+ and system has no less available memory after the test is done.