Example of wasteful, innefficient string manipulation

[David Heffernan 2345]

2 minutes ago, Mike Torrettinni said:

I like the conclusion that even non-perfect code will not break the system memory or cause it to run out of it.

I understand that you'd like that conclusion. It would be nice if it were true though. It isn't. 

[Anders Melander 1783]

On 12/24/2020 at 1:51 PM, Mike Torrettinni said:

I was trying to run an example of wasteful, inefficient string manipulation to see what happens to memory consumption, without Fastmm.

You test is a synthetic benchmark of metrics you don't understand yeilding results you don't know how to interpret. As usual this is completely pointless.

 

I see you've highlighted the peak working set size but do you even know and understand what a working set is?

 

If you really want to learn about how different memory management challenges can affect you then start by learning about what those problems are. Make an effort.

 

One of the reason why your benchmark isn't doing as bad as you'd expect is probable the lack of randomness in the test. You're repeating the same pattern 100 times. If your allocation were random this would fragment the heap badly but since you're allocating, freeing and then later reallocating the same block sizes again and again, there's good likelihood that the blocks can be reused.

[Mike Torrettinni 198]

5 minutes ago, Anders Melander said:

One of the reason why your benchmark isn't doing as bad as you'd expect is probable the lack of randomness in the test. You're repeating the same pattern 100 times. If your allocation were random this would fragment the heap badly but since you're allocating, freeing and then later reallocating the same block sizes again and again, there's good likelihood that the blocks can be reused.

Ok, this makes sense. This was extremely exaggerated example from a few uses of MidStr, Delete and other string functions I have in my code. So I was thinking if any case of real data input is structured in a way that these functions get used in extreme way, what will be the effect.

So, it was more like: can my code 'break' OS memory or should I spend days(weeks) trying to micro-optimize every use of MidStr, Delete and others, because I'm afraid how it will affect OS memory in extreme cases.

 

 

[David Heffernan 2345]

12 minutes ago, Mike Torrettinni said:

Ok, this makes sense. This was extremely exaggerated example from a few uses of MidStr, Delete and other string functions I have in my code. So I was thinking if any case of real data input is structured in a way that these functions get used in extreme way, what will be the effect.

So, it was more like: can my code 'break' OS memory or should I spend days(weeks) trying to micro-optimize every use of MidStr, Delete and others, because I'm afraid how it will affect OS memory in extreme cases.

 

 

Why aren't you curious to learn about how virtual memory works? Why aren't you curious to learn how sub-allocating memory managers work? Why aren't you curious to learn what address space fragmentation is? 

[Mike Torrettinni 198]

1 minute ago, David Heffernan said:

Why aren't you curious to learn about how virtual memory works? Why aren't you curious to learn how sub-allocating memory managers work? Why aren't you curious to learn what address space fragmentation is? 

It's not that I don't want to, but google has 518M hits.... I'm not that interested reading books or hours of resources to get to a part that relates to my topic. Would love to read a page or two, but only what relates to this topic.

[Attila Kovacs 629]

29 minutes ago, Mike Torrettinni said:

I'm afraid how it will affect OS memory in extreme cases.

do you want to talk about it? 😉

Edited December 25, 2020 by Attila Kovacs

[Attila Kovacs 629]

8 minutes ago, Mike Torrettinni said:

It's not that I don't want to, but google has 518M hits.... I'm not that interested reading books or hours of resources to get to a part that relates to my topic. Would love to read a page or two, but only what relates to this topic.

Sometimes, before I can make my decision in a particular matter (not programming) I do google/read/watch/test for days or for a week or once in a time even longer.

If you would use the time for the same instead of this kind of nonsense topics, you would have more of it. We too.

[Mike Torrettinni 198]

1 minute ago, Attila Kovacs said:

Sometimes, before I can make my decision in a particular matter (not programming) I do google/read/watch/test for days or for a week or once in a time even longer.

If you would use the time for the same instead of this kind of nonsense topics, you would have more of it. We too.

I do that too, when topic is of interest to me. Not sure why you think this topic is nonsense, it did give me some valuable info.

 

You are more than welcome to add your view on how string manipulation with Delphi is affecting OS memory and in which case it can cause any concern.

[David Heffernan 2345]

3 minutes ago, Mike Torrettinni said:

You are more than welcome to add your view on how string manipulation with Delphi is affecting OS memory and in which case it can cause any concern.

What's the point? You've already reached a false conclusion based on a flawed investigation. If we try to point this out you tell us that you aren't interested in learning the truth. 

[Mike Torrettinni 198]

Just now, David Heffernan said:

What's the point? You've already reached a false conclusion based on a flawed investigation. If we try to point this out you tell us that you aren't interested in learning the truth. 

We just have different approaches, I was expecting some short explanations, experience or cases where I can see a real problem. Your approach in this topic was to give me keywords to study, which I appreciate, but too broad for this topic.

[Attila Kovacs 629]

14 minutes ago, Mike Torrettinni said:

I do that too, when topic is of interest to me.

Why are you interested by something doesn't interest you?

[Mike Torrettinni 198]

4 minutes ago, Attila Kovacs said:

Why are you interested by something doesn't interest you?

I guess I would say I'm not typical developer, but just a rookie developer that interest level into a topic is correlated to how quickly I can find out the most about a topic. I understand that could keep at rookie level forever, but I'm OK with it, I accepted my path - also probably too old to suddenly become what I should have been 20 years ago 🙂

[Attila Kovacs 629]

Just now, Mike Torrettinni said:

to how quickly I can find out the most about a topic

I can't see any technical questions regarding to your fears. All I can see is some sort of blogging about some mind-blowing nonsense and that you are not even willing to read a word about it. I don't know how could we help. Or why.

[Mike Torrettinni 198]

Just now, Attila Kovacs said:

I can't see any technical questions regarding to your fears. All I can see is some sort of blogging about some mind-blowing nonsense and that you are not even willing to read a word about it. I don't know how could we help. Or why.

I guess you are expecting something I can't deliver. My first post is best I can provide to explain my interest, I have explanation, example, I added a little more in comments, but not anything too much different than in first post. So, if you have a question what info you are looking, I might be able to add it, but if first post is so nonsense to you, I'm not sure what else I can do.

[Attila Kovacs 629]

6 minutes ago, Mike Torrettinni said:

I guess you are expecting something I can't deliver.

And I guess, you are expecting something from us, we can't deliver.

 

Quote

Is Delphi's memory manager so good that even if the work is wasteful, inefficient string manipulation, when it ends it cleans up memory very well?

Yes. It's even better. In the case "very well" and "even better" are SI units.

[Mike Torrettinni 198]

2 minutes ago, Attila Kovacs said:

Yes. It's even better. In the case "very well" and "even better" are SI units.

Ok, thanks. I tested my example in D7, D2006 and D10.2.3 and end result is always quite clean memory consumption, even though D7 didn't have FastMM implemented, yet. So, pretty good outcome overall - of course we need to take into consideration Anders's observation that test cases probably could use more randomness.

[Guest]

About Memory Management: by MS

https://docs.microsoft.com/en-us/windows/win32/memory/about-memory-management

 

Heap Functions <--- relevant for your topic!!!

https://docs.microsoft.com/en-us/windows/win32/memory/heap-functions

https://docs.microsoft.com/en-us/windows/win32/memory/low-fragmentation-heap

 

by Jim Diroff II on StackOverFlow summary:

https://superuser.com/questions/1579851/what-is-standby-memory-in-ms-windows-os

 

the MSWindows (or others, I dont know), like this: "hey, user, maybe it's cool to have this in memory for later ... let's read, let's read ..."

Quote

... then ... we are like a diner full of food and almost throwing up from eating so much (reading).

 

[David Heffernan 2345]

On 12/24/2020 at 12:51 PM, Mike Torrettinni said:

I expected the end result to be high memory consumption, de-fragmented, not leaving much free memory for project to use

This was from the original post. I suspect you meant to say fragmented rather than defragmented.

 

When I asked how you assessed fragmentation you said that you hadn't, and that you didn't understand virtual memory.

 

I don't know what you think you have demonstrated, but I don't think you have demonstrated anything at all. 

[Guest]

@Mike Torrettinni It is hard to explain something when

 1) The point and the question to explain is not clear.

 2) We don't know what is your assumptions are, like do you have assumptions we consider as basic and simple information and hard to miss, this will throw the conversation in awkward way, we will skip explaining something so simple by our consideration and you simply didn't know it, you will feel the question is not answered, and we continue to scratch our head.

 

My suggestion is, please try to formulate the question in different way specially when you feel the point wasn't received clearly, and write what you think.

 

Now to the way you are assuming fragmentation work, it will not happen if you just allocated x item and freed them all, to get fragmented memory you should allocate x and free y from them (where y<x) while keep the rest there and repeat without freeing them all, Delphi MM will allocate big chunk then serve it as small chunks to your application, when application ask for more the MM will allocate another big allocation and so on, on freeing: when big allocation is completely unused, mean you application return all the small allocation to MM, here most likely MM will return the big allocation to the OS,

 

How to fragment this and stress the system, an example on how you can cause cause 1gb of memory to be reserved with only 1000 of very short string (1 char), 

Steps to do this : allocate many short string or arrays, let try explain this with arrays (easier), one 1 byte array like TBytes, the memory manager will allocate the smallest allocation for it it will need at least 1 byte+ 4 length = 5 bytes, the MM will allocate the smallest allocation and let assume it is 8, and here i will discard the look up table for it size (while there is at least 4), now to utilize 1gb we need 1gb/8b array, this size will require MM to allocate block with 1mb, so to keep 1mb indefinitely allocated you need just to leak one array per 1mb allocation ! try this on your own and allocate many strings or arrays and free all except 1 every X, find X on your own to cause 1000 unfreed string the use 1gb.

I hope this explanation give you a hint on how to stress the OS by miss using MM (or abusing the MM) for string or anything else, and by the way this is very similar on why the IDE is ever growing in memory usage, not a real leak but dangling pointers with may be very few wrongly managed interfaces, because each compile it will perform very similar behaviour with many small allocation and then keep few of them there.

 

 

ps:  Mean while i suggest to have a look at these great articles from Mark Russinovich https://techcommunity.microsoft.com/t5/windows-blog-archive/pushing-the-limits-of-windows-physical-memory/ba-p/723674

[Mike Torrettinni 198]

4 hours ago, Kas Ob. said:

@Mike Torrettinni It is hard to explain something when

 1) The point and the question to explain is not clear.

 2) We don't know what is your assumptions are, like do you have assumptions we consider as basic and simple information and hard to miss, this will throw the conversation in awkward way, we will skip explaining something so simple by our consideration and you simply didn't know it, you will feel the question is not answered, and we continue to scratch our head.

 

My suggestion is, please try to formulate the question in different way specially when you feel the point wasn't received clearly, and write what you think.

 

Now to the way you are assuming fragmentation work, it will not happen if you just allocated x item and freed them all, to get fragmented memory you should allocate x and free y from them (where y<x) while keep the rest there and repeat without freeing them all, Delphi MM will allocate big chunk then serve it as small chunks to your application, when application ask for more the MM will allocate another big allocation and so on, on freeing: when big allocation is completely unused, mean you application return all the small allocation to MM, here most likely MM will return the big allocation to the OS,

 

How to fragment this and stress the system, an example on how you can cause cause 1gb of memory to be reserved with only 1000 of very short string (1 char), 

Steps to do this : allocate many short string or arrays, let try explain this with arrays (easier), one 1 byte array like TBytes, the memory manager will allocate the smallest allocation for it it will need at least 1 byte+ 4 length = 5 bytes, the MM will allocate the smallest allocation and let assume it is 8, and here i will discard the look up table for it size (while there is at least 4), now to utilize 1gb we need 1gb/8b array, this size will require MM to allocate block with 1mb, so to keep 1mb indefinitely allocated you need just to leak one array per 1mb allocation ! try this on your own and allocate many strings or arrays and free all except 1 every X, find X on your own to cause 1000 unfreed string the use 1gb.

I hope this explanation give you a hint on how to stress the OS by miss using MM (or abusing the MM) for string or anything else, and by the way this is very similar on why the IDE is ever growing in memory usage, not a real leak but dangling pointers with may be very few wrongly managed interfaces, because each compile it will perform very similar behaviour with many small allocation and then keep few of them there.

 

 

ps:  Mean while i suggest to have a look at these great articles from Mark Russinovich https://techcommunity.microsoft.com/t5/windows-blog-archive/pushing-the-limits-of-windows-physical-memory/ba-p/723674

Thank you @Kas Ob. I think now I understand my question was really poorly formulated, I was focused on strings manipulation, everybody else thought I was stress testing MM. Makes sense that some of you wanted me to go back to school 🙂

I thought I was becoming better at asking questions, I guess I need more work.

 

Even though it probably seemed like a waste of time for everybody else, I learned valuable things during preparing the example, thinking about it, reading about FastMM and 'discussing' in this topic:

1. I still need to work on improving question quality

2. I'm probably little rebellious when getting suggestions like 'google this and then we can talk' 😉

3. What I thought was very wasteful, bad example of string manipulation, was actually not! 😉 even Delphi 7 handled it really well. If I tested this example in D7 before posting, I would probably realize this is not worth even asking.

But it led me to test D7 from references to FastMM - to test version without it.  So, something that is probably very basic and very obvious fact for seasoned developers, it wasn't for me and wouldn't come to this conclusion without this topic,

 

 

 

 

[Guest]

@Mike Torrettinni It is Ok and you are doing great.

 

2 hours ago, Mike Torrettinni said:

1. I still need to work on improving question quality

2. I'm probably little rebellious when getting suggestions like 'google this and then we can talk' 😉

Let me explain a thing here, and will give you an example, the other day i suggested a thing for Mahdi, he liked it and mentioned it will be tricky with DLL's, now i went on explaining it is easy ! and he went explain to me how it is not easy, see here we both were right, and both failed to explain the context of our point of view, for me i was strictly pointing to PPointer usage with DLL (per one module) for patching for this specific new feature, while he was thinking about expanding DDetours itself for runtime hooking, thus we failed to convoy the context as both we assumed the other is seeing and thinking the same way, now if you tried imagine how others read our awkward conversation, some will assume Mahdi was right and i am wrong, some will assume i was pointing to the right thing while Mahdi is overengineering it, and most will not give a jack and see us as wasted space on the internet, the point is no matter how good social skills you have we always will fail at some point with our limited focus and our own unsupervised bias.

 

And that why i suggested a book, a full book to read, and statistically speaking you will like it 😉 , and it will change you way of thinking, not only as programmer but as human being.

 

ps: it still very useful exercise if you tried my example above in real program, so if you got time try to make your leak 1gb with only 1000 string, the experience from doing is valuable, and you will get to know how to detect such leaks or dangling memory, the one that will not reported as a leak because it is not.