No one can help

[skyzoframe[hun] 4]

19 minutes ago, limelect said:

Right now from my point of view as a programmer, I catch the screen of a

program as a picture copy to clipboard saves to ADVrichedit so simple. 

https://download.tmssoftware.com/Download/Manuals/TMS TAdvRichEditor.pdf

 

 

Or you can store pictures on a local drive, and only the picture name has to be in the database. Or use a custom table for blob fields and call only when you have to show the picture.

Edited December 12, 2022 by skyzoframe[hun]

[Fr0sT.Brutal 897]

1 hour ago, limelect said:

As you can see the line is almost top of the screen

Any suggestion? Software wise

Okay I see you have all memory consumed.

Suggestion is to change tab to Processes and inspect YOUR APP. Stepping line by line in IDE.

[Attila Kovacs 617]

On 12/11/2022 at 9:07 AM, limelect said:

I never saw such behavior with hunters of projects behind me.

This is a warning to the Delphi community about what to expect.

I don't believe you a word. You can't even debug. You are just trolling us.

[programmerdelphi2k 235]

I said to him, that: first "The source code he sent me mixed "memory access" through "OpenProcess(), GetProcessMemoryInfo(), EnumProcess(), ... etc and lots of pointers...". Then said to eliminate any other types of tasks, and then, main, check if the "specific record" in the SQL DB is really OK, or there could be some kind of problem, like data corruption!

However, he told me that the record in question was "1GBytes" in size... which I thought was an exaggeration even for a large base database.... Naturally, the error is already found in the structuring of the database data, because surely something is not right in this configuration.

 

Of course, you could have a single record in your table, with a size of 1GB of data, however, this would certainly be something very peculiar and directed to a specific case of information storage.

I think the "base" is already contaminated so the results will be catastrophic when the weather warms up!!!

 

However, my opinion was not and will not be taken into account, even though he posted the request for help in a public way for anyone who wanted to help!

 

But as he himself already said in his previous posts: "that hardly anyone could help...." then, what is said is unsaid!

On 12/9/2022 at 12:08 PM, limelect said:

I doubt that anyone can help since it is not a database problem but a Delphi PROGRAM problem.

Once I use a database the program memory is large and unmanageable.

Not Delphi, my friend! but, Yours!

 

NOTE: I did (for him) a prototype using SQL DB read/write BLOB fields to show for him, how would be ? but I was ignored! just 22KB in 7zip + project + SQLite DB

 

Luck in the next Gigabytes in

VCL_SQLite_and_FireDAC_RichEdit_and_BLOB_field.7z

Edited December 12, 2022 by programmerdelphi2k

[limelect 47]

@programmerdelphi2k Plz stops attacking me it does not help.

And your demo is useless.

@Attila Kovacs Debug does not help since I finish my memory

and I know exactly when it happens when I load a large record.

It is a program too large to be in memory.

@Fr0sT.Brutal I know precisely what happens.

At first time, I thought it was an SQL problem now I know it is a memory problem.

@skyzoframe[hun] storing pictures into a file very fast I will

have thousand of them.

and this is exactly what I do

"Or use a custom table for blob fields and call only when you have to show the picture."

I insert it into a DB where I have no problem however bringing the record finishes the memory !!.

 

P.S an Idea. if I put the form and process into a DLL

load and unload each DLL (form) will it help for memory?

 

 

Edited December 13, 2022 by limelect

[Fr0sT.Brutal 897]

I give up. Just get more memory and chill.

[limelect 47]

@Fr0sT.Brutal You are right or design the program differently

Some programmers use many EXE in their program

Edited December 13, 2022 by limelect

[limelect 47]

Programming is fun this is why I chose this profession

0 physical memory while the program executes

[Lars Fosdal 1732]

30 minutes ago, limelect said:

I insert it into a DB where I have no problem however bringing the record finishes the memory !!.

Can you exclude the field(s) that holds the big chunk image data from the search view - and instead, when a single search item is focused - load just that field alone on demand?

[limelect 47]

@Lars Fosdal Unfornetly that is what I am doing.

I filter the database and left in the dbgride with 1 record. It does not matter how many records are left.

dblclick the record and read the field into the stream. No problem

   AdvRichEditor2.LoadFromStream(MyMemStream);<<< memory problem

 

before I read the blobe into a string memory error
It seems I have to redesign my program. because

at another time the record might even be larger.

I cannot know how many pictures and text richedit will hold.

 

P.S what amazed me that insert into db has no problems

 

Edited December 13, 2022 by limelect

[limelect 47]

@Lars Fosdal I just remembered one of my tests was to move

the record into DBmemory close and free all links and DB

and use the DBmemory. my thinking

was maybe it will reduce total memory it did not.

[Stano 140]

Kinda OT

As a lay amateur, I am puzzled by the fact that there are several files in one BLOB. Too much.
I would save each file separately from the very beginning. I was already taught that with modern DBs the number of records in the table is not a problem.
But I don't know your program, nor its philosophy and function.

[Lars Fosdal 1732]

2 minutes ago, limelect said:

It seems I have to redesign my program

I suspect it has to do with how the images are loaded into the TAdvRichEdit. Have you looked at TPictureContainer / THTMLPictureContainer that can be used to supply images to the editor? 

[limelect 47]

@Lars Fosdal I am writing you personally

Furthermore, even if I fix this one record the record mite

grow much bigger by the user than what?

Edited December 13, 2022 by limelect

[Lars Fosdal 1732]

I read / comment here whenever I have a few minutes to spare, but I can't commit to answer personal messages for support.
I prefer to do the communication here, as other may (or may not) chip in - since that is what a community is for.
 

[aehimself 388]

1 minute ago, Stano said:

Kinda OT

As a lay amateur, I am puzzled by the fact that there are several files in one BLOB. Too much.
I would save each file separately from the very beginning. I was already taught that with modern DBs the number of records in the table is not a problem.
But I don't know your program, nor its philosophy and function.

There can be multiple reasons

 

Wrong database design from the beginning

Either the person had no understanding how it should be done or thought that the small amount of data what it will hold won't cause further issues. Then it grew out of control.

To be honest (especially in newer applications) developers tend to completely ignore edge cases, lower hardware specs. "It works on my PC"

 

Legacy dragging on

This is exactly the case at us at work. Application was created ~20 years ago, when DB backends worked completely different and the architecture was functioning correctly. Now with thousands of tables and TBs of data stored it's not an easy task to start to split these so when a customer complaint comes in, we fix that one. At this speed the redesign won't be done in the next 5 years for sure and expose us to a risk that once done we'll have to start it again due to backend changes.

 

Can't reproduce

Not just the amount of data but the data itself can cause issues. Generally speaking, due to GDPR and confidentiality we, the developers aren't allowed to access and work with production data. When an export is needed, data is obfuscated before loaded in our test systems and this obfuscation itself is already changing the data. Once I spent days before figuring out that a rendering issue was caused by a BOM left in a BLOB field (customer just uploaded a file instead of using our editor), which was simply trimmed during obfuscation.

 

As quick as possible, no future in mind

This is especially true for commercial software unfortunately. Customers want more and more, quicker and quicker. In lots of occasions new features are added on a "quantity over quality"-basis. Just get it to work, get us paid, customer is happy - at least for now. This isn't necessarily a management issue as recognizing the need for, as well as optimization itself needs knowledge, time and manpower. Depending on the size, a company can lack many of this.

 

These are just my 2 cents. Nothing is unfixable, question is always: at what cost.

[limelect 47]

@aehimself The DB is NOT the problem. Since data (a record)

read OK. But the use of it. It finishes the memory.

@Stano Files are not the problem as they are data.

In my case, it is pictures and text as they are also data.

Can be asci or binary does not matter.

TAdvRichEdit uses this data with no problem

Edited December 13, 2022 by limelect

[Lars Fosdal 1732]

The format of the data and how they live in the application decides the memory consumption.

Think about web content.  Huge amounts of graphics and video in a 32-bit browser 32-bit OS, even with as little as 2Gb of memory.

How does it do that?  Load on demand and using temporary storage outside the browser such as file cache or direct streams.

 

If we load a block of data from a DB - and that data has to be loaded into a stream - and then fed into an editor - we are using that memory at least three times, maybe four - depending on how the control stores the data.

 

Divide and conquer.  Instead of one blob - use a separate table to hold individual images.  Explore if there are ways to load the image data on demand instead of embedding them in the control.

[Sherlock 636]

Just my two cents as a person who has been developing commercial software with Delphi for over 20 years now (where has all the time gone? sigh). Whenever there have been issues with the software, one thing turned out to be true: The DB was almost always the problem. That, or the way DB data was being handled. Rethinking table design and handling of DB-aware and not-DB-aware components have always done the trick.

 

But feel free to produce a minimal reproducible example for the folks at Embarcadero, if you are 100% sure that Delphi is at fault.

[Lars Fosdal 1732]

Also - a 32-bit application cannot use more than about 2 GB of memory, regardless of the amount of RAM in the machine.
Yes, there is a trick to extend that to about 3 GB - but that is only a temporary reprieve.

 

Another memory saving trick would be to scale images to the appropriate size and quality before storing them.  If you have a 12 Megapixel image that never will render bigger than 1024 x 768 - it should be scaled and compressed before storing it.

[limelect 47]

@sherlock I do not blame but myself.

I started with Delphi with #1 and before with visual and before 

with Pascal,c, and more. There were days I wrote with FORTRAN.

I never had such a problem not even with BDE.

I just finished making a large program that handles all company need

with MSSQL. many PC, a lot of electronics, and no memory problems.

The thing I blame is the components (not Delphi?) as they use a lot

of memory. Not my Pascal writing and I might be wrong.

@Lars Fosdal It will be too complicated but I will think about that option.

What you suggest is when reading to assemble all pictures that were

in records.

maybe.

The only problem I have programs already out there what do I do with them?

Edited December 13, 2022 by limelect

[limelect 47]

@Lars Fosdal It was so simple programming just  to copy to

richedit without any processing.

[Lars Fosdal 1732]

Even simple solutions can lead to complex problems.

[Fr0sT.Brutal 897]

49 minutes ago, limelect said:

I started with Delphi with #1 and before with visual and before 

with Pascal,c, and more. There were days I wrote with FORTRAN.

Then I really wonder why this topic lasts for 3 pages already and there's still no minimal reproducible sample here?

[Dalija Prasnikar 1341]

On 12/9/2022 at 6:03 PM, limelect said:

MyMemStream:=  ProjectsFDTable.CreateBlobStream(ProjectsFDTable.FieldByName('Description'),bmRead);
 AdvRichEditor.BeginUpdate;
AdvRichEditor.LoadFromStream(MyMemStream);
AdvRichEditor.EndUpdate;

Don't use memory stream to load data to rich edit. Create temporary file and write the data into that file and then load the data from the filestream.

 

However, no matter what you do, at some point you will run into memory issues with 32-bit application if your application consumes a lot of memory. I suggest you switch to 64-bit. If you cannot switch to 64-bit, then the only other option is optimizing other parts of your application to reduce overall memory usage. 

 

First step for optimizing would be getting rid of all memory leaks - they can cause fragmentation which can be a huge problem.

Next, reducing total memory allocation, by releasing anything that is not used at the time.  Be careful with creating local components that have owner which you don't explicitly free. They will not show as memory leaks, but every time you call such method your memory consumption will increase.

If that does not help, then you can try reducing memory load by avoiding default string type which is Unicode since D2009 (you haven't said which Delphi version you use) and use UTF-8 encoded strings - UTF8String and convert to Unicode only when you need to display data to the user - this is really the last option as you may need to write a lot of supporting code to avoid accidental conversions to and from Unicode strings as they will slow your application down and use extra memory in the process.