GZip compression in MARS

[Bjørn Larsen 3]

Hi, 

 

For those intereste;, I have implemented gzip compression in MARS.

https://github.com/larsenbjorn/MARS/commit/57d52646ee34f91e6d0b1791ea7c7ff28da803b2

 

I have not done any real benchmarks on this, but based on simple time measuring, the gain by enabling compression increases with slower connection and larger amounts of data (naturally) and in mobile apps.

The impact/gain is not noticeable when everything is running on the same computer. 

 

@Andrea Magni Do you see any problem with the implementation? If you find the solution usable I am happy to create a pull request.  

 

Best regards, 

Bjørn Larsen

  

 

[stijnsanders 37]

Does MARS do streaming? Or does MARS do transfer-encoding chunked? (If so, you're apparently supposed to gzip first and then do the chunking...)

[Andrea Magni 75]

Hi @Bjørn Larsen,

thanks for you contribution!

Some thoughts:

1. I would say compression should be done as last step before sending back the response to the client. I mean, we can take advantage of the AfterInvoke mechanism (attaching an handler or not according to Engine parameters). This should be more generally available to all MARS TMARSActivation implementation (only one in the official repository, I know). Another point is that some deployment methods (Apache/IIS) may provide compression natively and I would let this instead of our implementation.
   What about moving your implementation to the Server.Ignition unit (MARSTemplate demo)? See code below as draft... I guess one could even implement a symmetric BeforeInvoke to support compression also the other way around (from client to server), I don't know how this is popular/expected, however.
2. Using Response.SetHeader may lead to multiple Content-Encoding headers to be set at the same time (TPrimitiveTypesWriter, in MARS.Core.MessageBodyWriters.pas for instance sets the Content-Encoding to provide some information about text encoding. This may be a mine misunderstanding however... I need to check this sooner or later... if anyone has something to say about this, please do!). I switched to setting the Response.ContentEncoding property as I was having trouble with Postman (and Chrome) when multiple Content-Encoding headers were set.

 

    // Compression
    if FEngine.Parameters.ByName('Compression.Enabled').AsBoolean then
      TMARSActivation.RegisterAfterInvoke(
        procedure (const AActivation: IMARSActivation)
        var
          LOutputStream: TBytesStream;
        begin
          if ContainsText(AActivation.Request.GetHeaderParamValue('Accept-Encoding'), 'gzip')  then
          begin
            LOutputStream := TBytesStream.Create(nil);
            try
              ZipStream(AActivation.Response.ContentStream, LOutputStream, 15 + 16);
              AActivation.Response.ContentStream.Free;
              AActivation.Response.ContentStream := LOutputStream;
              AActivation.Response.ContentEncoding := 'gzip';
            except
              LOutputStream.Free;
              raise;
            end;
          end;
        end
      );

 

Let me know what you think about. I would be glad to accept your PR then...

 

Sincerely,

Andrea

 

Edited August 23, 2019 by Andrea Magni

[Andrea Magni 75]

8 hours ago, stijnsanders said:

Does MARS do streaming? Or does MARS do transfer-encoding chunked? (If so, you're apparently supposed to gzip first and then do the chunking...)

 

Frank Lauter suggested me to implement chunking in MARS but I haven't done it yet.

I guess one should implement some sort of caching in order to avoid continuously gzipping the same content just to serve a specific chunk to the client.

Would be very interesting to cover this as it may result in a great performance boost for large data transfers.

 

Sincerely,

Andrea

[stijnsanders 37]

Hmm, yes if it's static content, there would be sense in keeping a cache of the gzipped-data somewhere and serve that on subsequent requests. If it's dynamic content on the other hand, you might suspect this data to be one-off specifically for each request. So if there's a lot of it (and might take some time to generate, typically longer than it takes to transmit over to the client...) and you're having transfer-encoding chunked (which enables keeping the connection open when the response is sent over) then you should use TZCompressionStream's capability to zip a next chunk based on the existing stream...

[Bjørn Larsen 3]

On 8/23/2019 at 7:00 PM, Andrea Magni said:

Let me know what you think about. I would be glad to accept your PR then...

 

I totally agree with your input. In my case the api is going to be distributed to x number of local sites, where there will be no Apache/IIS. A simple setup is the key, and compression as part of executable is a big benefit. 

I have created a pull request for the changes to MARS.Core.Utils and the implementation in MARSTemplate demo.

 

[Andrea Magni 75]

Hi, sorry for the late (!). I accepted the pull request yesterday. Will be soon available also in the master branch (I am switching to trunk based development model).

 

Sincerely,

Andrea