Jump to content
Clément

Zip Compression library

Recommended Posts

Hi,

In this project I need to ZIP huge text files ( over 1 GB ) for backup purposes. And I got disappointed.
I'm using Delphi library and third party components but I can't beat windows vanilla zip, neither in speed nor in size.
I am testing those libraries with 3 files: huge3.txt 1,926,179Kb, huge4.txt 797,946Kb and huge5.txt 650,767kb
(You really don't want to ask the sizes of huge1 and huge2 ).
Using windows (send compressed zip folder) in less then 2 minutes produces  a 255,940KB zip files.
Using delphi library (ZipForge, Abbrevia, VCL.Zip ) even at max compression level the best library produced a 316,690 KB. It took over 3 minutes.

Needless to say my customer will not accept such difference.
I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

 

Is there any hope?

Edited by Clément

Share this post


Link to post
17 minutes ago, David Heffernan said:

Your program is 32 or 64 bit? There are fast third party components. You didn't say which ones you were using. 

This test is 32 bit. Final version will have both available and the user will install the one he wants

Share this post


Link to post
6 hours ago, Clément said:

I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

7-Zip with using the *.7z format is able to produce compressed files larger than 2GB and can be integrated into Delphi app as well, by using the 7z.dll.

I think 7-Zip with using the *.zip format also stops at 2GB, but I haven't checked this with recent versions.

Edited by Rollo62
  • Like 1

Share this post


Link to post
43 minutes ago, shineworld said:

Without to use 7z.dll there is a pure pascal implementation that I've used and works fine:

Given how poor the Delphi compiler is in terms of performance of generated code, I'd expect this to be slower. Is it? 

Share this post


Link to post
8 hours ago, Clément said:

Hi,

In this project I need to ZIP huge text files ( over 1 GB ) for backup purposes. And I got disappointed.
I'm using Delphi library and third party components but I can beat windows vanilla zip, neither in speed nor in size.
I am testing those libraries with 3 files: huge3.txt 1,926,179Kb, huge4.txt 797,946Kb and huge5.txt 650,767kb
(You really don't want to ask the sizes of huge1 and huge2 ).
Using windows (send compressed zip folder) in less then 2 minutes produces  a 255,940KB zip files.
Using delphi library (ZipForge, Abbrevia, VCL.Zip ) even at max compression level the best library produced a 316,690 KB. It took over 3 minutes.

Needless to say my customer will not accept such difference.
I don't really need to produce ZIP files, I can use any compression I like, but the file must be opened by a popular compressor viewer.

 

Is there any hope?

Not that I have actually timed it: We use 7zip with the interface units from the jcl to produce .zip (not .7z) files.

Share this post


Link to post

I did not believe those numbers so I did it myself.

 

image.thumb.png.28a980a30a94b349dac17298cfc9052e.png

 

source enwik9.zip http://mattmahoney.net/dc/textdata.html

 

compression routine

  zf := TZipFile.Create;
  zf.Open('MyFile.zip', zmWrite);
  zf.Add('enwik9', '', zcDeflate);
  zf.Close;

But, I don't know what TZipFile is using, own code or winapi.

 

Share this post


Link to post

There was an issue with some older versions of Delphi where the zlib .obj files supplied by Emba with the RTL were compiled without optimisation and the performance was poor.  Can't remember the version, but you'd be able to find the details by searching in Quality Portal to make sure you weren't suffering from that.

Share this post


Link to post
3 hours ago, David Heffernan said:

Given how poor the Delphi compiler is in terms of performance of generated code, I'd expect this to be slower. Is it? 

I also have not measured, but I'm well aware that 7-Zip with *.7z format, also with the original 7-Zip binary, is encoding much slower than using other ZIP encoder.

I'm not so sure about the decoding speed, probably its on par or even faster than ZIP there, but my critical path was the encoding side in an older project.

It's very likely that compiled with pure Delphi its even slower, that is true.

Share this post


Link to post
3 hours ago, Rollo62 said:

@shineworld

Do you know any more recent repository where the sources were maintained ?

 

Sincerely I don't know where to find a more updated code.

I use it for years and for what I need works fine so never tried more updated versions.

Edited by shineworld

Share this post


Link to post

The USP of LZMA in 7Zip was higher compression size and better decompression speed, against a slower compression speed, which is usually done rather less frequently. 

 

ZLIB deflate as used by ZIP and HTTP compression is speed, not minimal size, with options for both.  The ZLIB library in Delphi is optimised C code which will be faster than the Pascal conversion of LZMA, I'm sure the DLL version will be faster, but then we are into DLL hell. 

 

Angus

  • Like 1

Share this post


Link to post

Someone here has posted optimized/modernized Zlib OBJ files that gave speedup, probably this could be an option.

Edited by Fr0sT.Brutal

Share this post


Link to post

Some numbers.... I'm using zip forge clMax to try to get as close as possible to Windows10_X64 size.

64 bit zipForge (max): created at 10:18:17 - Last modified 10:20:56
32 bit zipForge (max): created at 10:14:29 - Last modified 10:17:51

64 bit TzipVCL (zcDeflate😞 created at 10:35:27 - Last Modified: 10:36:36

 

They are all at least 60MB greater.

 

 

 

 

 

Results.png

zipFile_x64_prop.png

zipForge_64_clMax_Trans_prop.png

zipForge_32_clMax_Trans_prop.png

WIndows10_x64_prop.png

Edited by Clément

Share this post


Link to post
7 minutes ago, Clément said:

I'm using zip forge clMax to try to get as close as possible to Windows10_X64 size.

That's with CompressionLevel to clMax. What was the CompressionMode and Zip64Mode setting?

 

If that is the maximum TZipForge can do I guess you can write that one off.

 

Share this post


Link to post
3 minutes ago, rvk said:

That's with CompressionLevel to clMax. What was the CompressionMode and Zip64Mode setting?

zipForge version 6.94.
CompressionLevel: clMax

CompressionMode: 9
EncryptionMethod caPkzipClassic,

Zip64Mode: zmDisabled
 

Share this post


Link to post

I installed 7z.dll. Things are getting better (Last entry in the image )
Here is the code that beat windows zip! (everything 64bits)

 

   var lZip := TJclZipCompressArchive.Create('d:\temp 1\JCL_7ZDLL_x64.zip');
   try
    lZip.SetCompressionMethod(cmDeflate64);
    lZip.AddFile('d:\temp 1\huge3.txt', 'd:\temp 1\huge3.txt');
    lZip.AddFile('d:\temp 1\huge4.txt', 'd:\temp 1\huge4.txt');
    lZip.AddFile('d:\temp 1\huge5.txt', 'd:\temp 1\huge5.txt');
   finally
     lZip.Compress;
     lZip.Free;
   end;

I had to use cmDeflate64. cmDeflate was better than the other libraries but sill over 300MB.
image.thumb.png.6496ceacac3980c8ef086128475bcdc7.png

Here I'm displaying the compression ratio.

image.thumb.png.1dc03b103b4866907980c882d176a8be.png

But it took over 6 min

zipJCL_7ZDLL_x64_prop.thumb.png.184371b9f4f96968cbf08035389d56b0.png

 

Share this post


Link to post
1 minute ago, Attila Kovacs said:

downloaded and installed zipforge, pretty same as W10

So CompressionMode 5 is better than 9 ?

 

Share this post


Link to post
2 minutes ago, Attila Kovacs said:

downloaded and installed zipforge, pretty same as W10

What zipForge's version are you using?

Share this post


Link to post
Just now, Clément said:

What zipForge's version are you using?

the one I just downloaded.... must be the latest

Share this post


Link to post
4 minutes ago, Attila Kovacs said:

the one I just downloaded.... must be the latest 

I'm using a comercial version. I will retry with the personal one.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×