Jump to content
Bjørn Larsen

GZip compression in MARS

Recommended Posts

Hi, 

 

For those intereste;, I have implemented gzip compression in MARS.

https://github.com/larsenbjorn/MARS/commit/57d52646ee34f91e6d0b1791ea7c7ff28da803b2

 

I have not done any real benchmarks on this, but based on simple time measuring, the gain by enabling compression increases with slower connection and larger amounts of data (naturally) and in mobile apps.

The impact/gain is not noticeable when everything is running on the same computer. 

 

@Andrea Magni Do you see any problem with the implementation? If you find the solution usable I am happy to create a pull request.  

 

Best regards, 

Bjørn Larsen

  

 

Share this post


Link to post

Hi @Bjørn Larsen,

thanks for you contribution!

Some thoughts:

  1. I would say compression should be done as last step before sending back the response to the client. I mean, we can take advantage of the AfterInvoke mechanism (attaching an handler or not according to Engine parameters). This should be more generally available to all MARS TMARSActivation implementation (only one in the official repository, I know). Another point is that some deployment methods (Apache/IIS) may provide compression natively and I would let this instead of our implementation.
    What about moving your implementation to the Server.Ignition unit (MARSTemplate demo)? See code below as draft... I guess one could even implement a symmetric BeforeInvoke to support compression also the other way around (from client to server), I don't know how this is popular/expected, however.
  2. Using Response.SetHeader may lead to multiple Content-Encoding headers to be set at the same time (TPrimitiveTypesWriter, in MARS.Core.MessageBodyWriters.pas for instance sets the Content-Encoding to provide some information about text encoding. This may be a mine misunderstanding however... I need to check this sooner or later... if anyone has something to say about this, please do!). I switched to setting the Response.ContentEncoding property as I was having trouble with Postman (and Chrome) when multiple Content-Encoding headers were set.

 

    // Compression
    if FEngine.Parameters.ByName('Compression.Enabled').AsBoolean then
      TMARSActivation.RegisterAfterInvoke(
        procedure (const AActivation: IMARSActivation)
        var
          LOutputStream: TBytesStream;
        begin
          if ContainsText(AActivation.Request.GetHeaderParamValue('Accept-Encoding'), 'gzip')  then
          begin
            LOutputStream := TBytesStream.Create(nil);
            try
              ZipStream(AActivation.Response.ContentStream, LOutputStream, 15 + 16);
              AActivation.Response.ContentStream.Free;
              AActivation.Response.ContentStream := LOutputStream;
              AActivation.Response.ContentEncoding := 'gzip';
            except
              LOutputStream.Free;
              raise;
            end;
          end;
        end
      );

 

Let me know what you think about. I would be glad to accept your PR then...

 

Sincerely,

Andrea

 

Edited by Andrea Magni

Share this post


Link to post
8 hours ago, stijnsanders said:

Does MARS do streaming? Or does MARS do transfer-encoding chunked? (If so, you're apparently supposed to gzip first and then do the chunking...)

 

Frank Lauter suggested me to implement chunking in MARS but I haven't done it yet.

I guess one should implement some sort of caching in order to avoid continuously gzipping the same content just to serve a specific chunk to the client.

Would be very interesting to cover this as it may result in a great performance boost for large data transfers.

 

Sincerely,

Andrea

Share this post


Link to post

Hmm, yes if it's static content, there would be sense in keeping a cache of the gzipped-data somewhere and serve that on subsequent requests. If it's dynamic content on the other hand, you might suspect this data to be one-off specifically for each request. So if there's a lot of it (and might take some time to generate, typically longer than it takes to transmit over to the client...) and you're having transfer-encoding chunked (which enables keeping the connection open when the response is sent over) then you should use TZCompressionStream's capability to zip a next chunk based on the existing stream...

Share this post


Link to post
On 8/23/2019 at 7:00 PM, Andrea Magni said:

Let me know what you think about. I would be glad to accept your PR then...

 

I totally agree with your input. In my case the api is going to be distributed to x number of local sites, where there will be no Apache/IIS. A simple setup is the key, and compression as part of executable is a big benefit. 

I have created a pull request for the changes to MARS.Core.Utils and the implementation in MARSTemplate demo.

 

Share this post


Link to post

Hi, sorry for the late (!). I accepted the pull request yesterday. Will be soon available also in the master branch (I am switching to trunk based development model).

 

Sincerely,

Andrea

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×