Zip Compression library

[Clément 148]

Hi,

In this project I need to ZIP huge text files ( over 1 GB ) for backup purposes. And I got disappointed.
I'm using Delphi library and third party components but I can't beat windows vanilla zip, neither in speed nor in size.
I am testing those libraries with 3 files: huge3.txt 1,926,179Kb, huge4.txt 797,946Kb and huge5.txt 650,767kb
(You really don't want to ask the sizes of huge1 and huge2 ).
Using windows (send compressed zip folder) in less then 2 minutes produces  a 255,940KB zip files.
Using delphi library (ZipForge, Abbrevia, VCL.Zip ) even at max compression level the best library produced a 316,690 KB. It took over 3 minutes.

Needless to say my customer will not accept such difference.
I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

 

Is there any hope?

Edited November 7, 2022 by Clément

[David Heffernan 2345]

Your program is 32 or 64 bit? There are fast third party components. You didn't say which ones you were using. 

[Clément 148]

17 minutes ago, David Heffernan said:

Your program is 32 or 64 bit? There are fast third party components. You didn't say which ones you were using. 

This test is 32 bit. Final version will have both available and the user will install the one he wants

[haentschman 92]

Hi...

https://componentace.com/zipforge_features.htm

I use it to make zip of a MSSQL database backup. All others stop at 2GB.
6GB -> 1GB

 

For Single Developer with Source Code   $149

 

[Rollo62 536]

6 hours ago, Clément said:

I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

7-Zip with using the *.7z format is able to produce compressed files larger than 2GB and can be integrated into Delphi app as well, by using the 7z.dll.

I think 7-Zip with using the *.zip format also stops at 2GB, but I haven't checked this with recent versions.

Edited November 7, 2022 by Rollo62

[shineworld 73]

Without to use 7z.dll there is a pure pascal implementation that I've used and works fine:

 

LZMA.442b.7z

[Rollo62 536]

@shineworld

Thanks, thats an interesting solution 👍

 

Unfortunately it seems that is not very well maintained, maybe from 2013/14, I can only find some references from here:

http://www.lab-z.com/lzma-压缩的例子/

http://www.birtles.org.uk/programming/

 

Do you know any more recent repository where the sources were maintained ?

 

[David Heffernan 2345]

6 hours ago, Clément said:

This test is 32 bit

I'd expect a 64 bit version to be faster. 

[David Heffernan 2345]

43 minutes ago, shineworld said:

Without to use 7z.dll there is a pure pascal implementation that I've used and works fine:

Given how poor the Delphi compiler is in terms of performance of generated code, I'd expect this to be slower. Is it? 

[dummzeuch 1505]

8 hours ago, Clément said:

Hi,

In this project I need to ZIP huge text files ( over 1 GB ) for backup purposes. And I got disappointed.
I'm using Delphi library and third party components but I can beat windows vanilla zip, neither in speed nor in size.
I am testing those libraries with 3 files: huge3.txt 1,926,179Kb, huge4.txt 797,946Kb and huge5.txt 650,767kb
(You really don't want to ask the sizes of huge1 and huge2 ).
Using windows (send compressed zip folder) in less then 2 minutes produces  a 255,940KB zip files.
Using delphi library (ZipForge, Abbrevia, VCL.Zip ) even at max compression level the best library produced a 316,690 KB. It took over 3 minutes.

Needless to say my customer will not accept such difference.
I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

 

Is there any hope?

Not that I have actually timed it: We use 7zip with the interface units from the jcl to produce .zip (not .7z) files.

[Attila Kovacs 629]

I did not believe those numbers so I did it myself.

 

 

source enwik9.zip http://mattmahoney.net/dc/textdata.html

 

compression routine

  zf := TZipFile.Create;
  zf.Open('MyFile.zip', zmWrite);
  zf.Add('enwik9', '', zcDeflate);
  zf.Close;

But, I don't know what TZipFile is using, own code or winapi.

 

[David Heffernan 2345]

There was an issue with some older versions of Delphi where the zlib .obj files supplied by Emba with the RTL were compiled without optimisation and the performance was poor.  Can't remember the version, but you'd be able to find the details by searching in Quality Portal to make sure you weren't suffering from that.

[Rollo62 536]

3 hours ago, David Heffernan said:

Given how poor the Delphi compiler is in terms of performance of generated code, I'd expect this to be slower. Is it? 

I also have not measured, but I'm well aware that 7-Zip with *.7z format, also with the original 7-Zip binary, is encoding much slower than using other ZIP encoder.

I'm not so sure about the decoding speed, probably its on par or even faster than ZIP there, but my critical path was the encoding side in an older project.

It's very likely that compiled with pure Delphi its even slower, that is true.

[shineworld 73]

3 hours ago, Rollo62 said:

@shineworld

Do you know any more recent repository where the sources were maintained ?

 

Sincerely I don't know where to find a more updated code.

I use it for years and for what I need works fine so never tried more updated versions.

Edited November 7, 2022 by shineworld

[Angus Robertson 574]

The USP of LZMA in 7Zip was higher compression size and better decompression speed, against a slower compression speed, which is usually done rather less frequently. 

 

ZLIB deflate as used by ZIP and HTTP compression is speed, not minimal size, with options for both.  The ZLIB library in Delphi is optimised C code which will be faster than the Pascal conversion of LZMA, I'm sure the DLL version will be faster, but then we are into DLL hell. 

 

Angus

[Fr0sT.Brutal 900]

Someone here has posted optimized/modernized Zlib OBJ files that gave speedup, probably this could be an option.

Edited November 7, 2022 by Fr0sT.Brutal

[Clément 148]

Some numbers.... I'm using zip forge clMax to try to get as close as possible to Windows10_X64 size.

64 bit zipForge (max): created at 10:18:17 - Last modified 10:20:56
32 bit zipForge (max): created at 10:14:29 - Last modified 10:17:51

64 bit TzipVCL (zcDeflate😞 created at 10:35:27 - Last Modified: 10:36:36

 

They are all at least 60MB greater.

 

 

 

 

 

Edited November 7, 2022 by Clément

[rvk 33]

7 minutes ago, Clément said:

I'm using zip forge clMax to try to get as close as possible to Windows10_X64 size.

That's with CompressionLevel to clMax. What was the CompressionMode and Zip64Mode setting?

 

If that is the maximum TZipForge can do I guess you can write that one off.

 

[Clément 148]

3 minutes ago, rvk said:

That's with CompressionLevel to clMax. What was the CompressionMode and Zip64Mode setting?

zipForge version 6.94.
CompressionLevel: clMax

CompressionMode: 9
EncryptionMethod caPkzipClassic,

Zip64Mode: zmDisabled
 

[Attila Kovacs 629]

downloaded and installed zipforge, pretty same as W10

 

[Clément 148]

I installed 7z.dll. Things are getting better (Last entry in the image )
Here is the code that beat windows zip! (everything 64bits)

 

   var lZip := TJclZipCompressArchive.Create('d:\temp 1\JCL_7ZDLL_x64.zip');
   try
    lZip.SetCompressionMethod(cmDeflate64);
    lZip.AddFile('d:\temp 1\huge3.txt', 'd:\temp 1\huge3.txt');
    lZip.AddFile('d:\temp 1\huge4.txt', 'd:\temp 1\huge4.txt');
    lZip.AddFile('d:\temp 1\huge5.txt', 'd:\temp 1\huge5.txt');
   finally
     lZip.Compress;
     lZip.Free;
   end;

I had to use cmDeflate64. cmDeflate was better than the other libraries but sill over 300MB.

Here I'm displaying the compression ratio.

But it took over 6 min

 

[rvk 33]

1 minute ago, Attila Kovacs said:

downloaded and installed zipforge, pretty same as W10

So CompressionMode 5 is better than 9 ?

 

[Clément 148]

2 minutes ago, Attila Kovacs said:

downloaded and installed zipforge, pretty same as W10

What zipForge's version are you using?

[Attila Kovacs 629]

Just now, Clément said:

What zipForge's version are you using?

the one I just downloaded.... must be the latest

[Clément 148]

4 minutes ago, Attila Kovacs said:

the one I just downloaded.... must be the latest 

I'm using a comercial version. I will retry with the personal one.