Jump to content

Andrea Magni

Members
  • Content Count

    110
  • Joined

  • Last visited

Posts posted by Andrea Magni


  1. 8 hours ago, stijnsanders said:

    Does MARS do streaming? Or does MARS do transfer-encoding chunked? (If so, you're apparently supposed to gzip first and then do the chunking...)

     

    Frank Lauter suggested me to implement chunking in MARS but I haven't done it yet.

    I guess one should implement some sort of caching in order to avoid continuously gzipping the same content just to serve a specific chunk to the client.

    Would be very interesting to cover this as it may result in a great performance boost for large data transfers.

     

    Sincerely,

    Andrea


  2. Hi @Bjørn Larsen,

    thanks for you contribution!

    Some thoughts:

    1. I would say compression should be done as last step before sending back the response to the client. I mean, we can take advantage of the AfterInvoke mechanism (attaching an handler or not according to Engine parameters). This should be more generally available to all MARS TMARSActivation implementation (only one in the official repository, I know). Another point is that some deployment methods (Apache/IIS) may provide compression natively and I would let this instead of our implementation.
      What about moving your implementation to the Server.Ignition unit (MARSTemplate demo)? See code below as draft... I guess one could even implement a symmetric BeforeInvoke to support compression also the other way around (from client to server), I don't know how this is popular/expected, however.
    2. Using Response.SetHeader may lead to multiple Content-Encoding headers to be set at the same time (TPrimitiveTypesWriter, in MARS.Core.MessageBodyWriters.pas for instance sets the Content-Encoding to provide some information about text encoding. This may be a mine misunderstanding however... I need to check this sooner or later... if anyone has something to say about this, please do!). I switched to setting the Response.ContentEncoding property as I was having trouble with Postman (and Chrome) when multiple Content-Encoding headers were set.

     

        // Compression
        if FEngine.Parameters.ByName('Compression.Enabled').AsBoolean then
          TMARSActivation.RegisterAfterInvoke(
            procedure (const AActivation: IMARSActivation)
            var
              LOutputStream: TBytesStream;
            begin
              if ContainsText(AActivation.Request.GetHeaderParamValue('Accept-Encoding'), 'gzip')  then
              begin
                LOutputStream := TBytesStream.Create(nil);
                try
                  ZipStream(AActivation.Response.ContentStream, LOutputStream, 15 + 16);
                  AActivation.Response.ContentStream.Free;
                  AActivation.Response.ContentStream := LOutputStream;
                  AActivation.Response.ContentEncoding := 'gzip';
                except
                  LOutputStream.Free;
                  raise;
                end;
              end;
            end
          );

     

    Let me know what you think about. I would be glad to accept your PR then...

     

    Sincerely,

    Andrea

     


  3. Hi @Yaron,

    I don't have actual numbers about the most reliable/best performance deployment method but my preferred deployment method for production is having an internal MARS REST server (Windows Service or Linux Deamon) and Apache acting as a reverse proxy (implementing SSL, caching, compression, eventually load balancing). This grants quite some granularity with respect to performance tuning and service availability (you can restart a single service or the whole thing).

    I also have deployed some MARS servers as Apache modules (on Linux).

     

    A couple of customers are using MARS with IIS: one is using IIRF (basically making IIS a reverse proxy and SSL implementor), keeping MARS servers standalone (not even Windows Services though this is on the roadmap). The other one is using ISAPI deployment.

    I have news from time to time from people using the library and there is not a clear trend among available deployment methods.

     

    Sincerely,

    Andrea

    • Like 1

  4. Hi @bioman,

    there are several approaches to the topic (CRUD operation against a DB). MARS doesn't dictate one specifically.

     

    Would you please specify if you are building a Delphi to Delphi solution (Delphi on the client and Delphi on the server)?

    You can have a dataset-oriented approach (client asks for a dataset, modifies it, changes are sent back to the server in form of delta updates) or a more general record-oriented approach (client asks for data, server sends a JSON array, client makes POST or PUT calls with modified/new data records).

    It also matters if you are looking for a data-centric approach (i.e. sending whole records from DB to client and back) or more interpolated approach (client commands REST server to perform operations that results in data manipulation).

     

    MARS is a lightweight library and I am trying not to dictate some aspects (more related to application than library IMO) so there are many ways to use it.

     

    Thanks for the kind words!

    Sincerely,

    Andrea

     

     


  5. On 7/30/2019 at 11:18 PM, NamoRamana said:

    I was wondering if MARS can ease this server migration?

    - in MARS, can I install multiple resources under the same server ? Idea here is that, each com server dll will act as own resource and its public methods will be the endpoints. For the better code management, is it

    possible to keep each resource code in its separate delphi project? how complex will be the installation of the REST server and these multiple resources, if we host them in IIS?

    - in current COM servers, Many of the method calls make heavy use of OleVarients as input and output paramas. Is there anything in MARS/REST that can keep those method signature same ?

    - in current COM env, we see some heavily used components slow down the end-user response a lot (sometimes, we have to reboot the host server). Do you think, MARS based REST service can handle a lots requests to a resource(I don't have request numbers)?

     

    Hi, welcome to this forum and to MARS.

     

    I've worked in the past to some migration process from Datasnap to MARS (and from RemObjects to MARS too). One thing you could take advantage is that TMARSActivation class (quite the core of request / REST method execution matching) can be replaced / tweaked.

    This means you may change the way an http request is matched against the Delphi code to be executed to produce response. Or you may want to have a different strategy/parsing technology to fill REST method arguments from the request.

    Obviously this is not the easiest task in the world but may have a lot of sense especially on large projects (I've migrated large datasnap applications to MARS with minimal effort and reducing to the minimum changes in the methods code).

     

    Coming to your questions:

    1) MARS can expose multiple engine (1 engine -> 1 port). Each engine can expose multiple applications. Each application can expose multiple resources. Each resource can expose multiple methods (http verbs and sub resources). You can organize resources as you like (one resource per unit, for example). I didn't get what you mean with "separate delphi project". You can deploy MARS server in several ways including ISAPI for IIS.

     

    2) MARS serialization/deserialization mechanisms are designed to be pluggable so you can implement your own readers/writers to handle OleVariants as you prefer.

     

    3) I have knowledge of MARS servers used in heavy load situations. Numbers are not significative because they actually depends on hardware/storage/purpose of REST methods. If you are looking for extreme performance, I would take a look to mORMot. But in 99% of my experience MARS is fast enough and REST architecture may help you to properly deal with scaling and performance boosts.

     

    Sincerely,

    Andrea


  6. On 7/25/2019 at 11:08 AM, Mike_Lob said:

    Hi,

    First of all: Thank you for providing us with this wonderuful implentation of your Rest library Andrea. I'm converting my own API servers to work with your library since it's far more enhanced than mine.

     

    Hi, thanks for the kind words and sorry for the delay.

     

    On 7/25/2019 at 11:08 AM, Mike_Lob said:

    1. How can I enforce Attributes provided with GET or POST calls?

    I have read somewhere in your GIThub that it should be possible but I have yet to figure out how to do this.

    I guess you are referring to QueryParam like "sort" in http://localhost:8080/rest/default/query/EMPLOYEE?sort=LAST_NAME

    You can use the Required attribute to let MARS check for the parameter's existence before calling the REST method.

     

        function GetDataset([QueryParam, Required] sort: string): TFDQuery;

     

    On 7/25/2019 at 11:08 AM, Mike_Lob said:

    2. How to dynamically change your database for a user.

     We have a Microsoft SQL server with a seperate database for every client.

     We also have one common database where we need to validate credentials and it will return the databasename from the sql server.

     For example: A user authenticates with an API key that we provied to the customer. That API key is linked to the users database and the SQL server returns that database name where we have to connect to - to perform specific user database queries.

    <SNIP>

    But I feel this should not be done like this. Also the .Server param doesnt exist in this context.

    Please forgive me if you see any 'newbie' questions. I start learning Delphi 4 months ago and I have yet to discover all Delphi concepts...

     

    I would add a specific value in the user's token, indicating the DB to use. You'll set this value in the token at authentication time (via API key or regular login).

    You can see this thread for a similar approach: 

     

    Keep in mind that your situation (if you store the connection name to be used inside the user's token) it's easier as you can use a connection name like 'TOKEN_CLAIM_DBCONNECTION':

     

    [Context, Connection('TOKEN_CLAIM_DBCONNECTION', True)] FD: TMARSFireDAC;

     

    And MARS will expand that connection name to the value of the DBCONNECTION name stored in the Claims of your (JWT) token.

     

    If you need some fallback mechanisms or more dynamic behavior, follow the instructions in the "FireDAC Connections" thread I linked here above.

     

    Feel free to ask whatever you need, sorry for the delay in this reply but I am abroad with family (vacation time).

     

    Sincerely,

    Andrea

     

     


  7. On 5/16/2019 at 6:24 PM, Andrea Magni said:

    I believe (I need to replicate this and investigate a bit) the Accept header sent from Angular matches also some MARS XML serializer.

    Will let you know here as soon as I figure out what's happening.

     

     

    It turned out it was the wildcard in the Accept header to cause this.

    I just pushed a fix that should also apply to your case and you may no longer need to instruct the Accept header all the times.

    Branch: develop, link to the commit: https://github.com/andrea-magni/MARS/commit/b8d5768b65072ff403de3c60e17688e8e95b1d9a

     

    Sincerely,

    Andrea

    • Like 1

  8. Hi @Stuart Clennett,

    I am happy you found a way to solve by yourself while I was in vacation 🙂

     

    Your solution works but has a couple of things to consider:

    1) that BeforeHandleRequest code gets executed for each incoming request (performance)

    2) manipulating Parameters is not suggested as App.Parameters is like a global object (IIRC).

     

    You have an other opportunity through (yet another) undocumented MARS feature:

     

        [Context, Connection('Dynamic_MyConnection', True)] FD: TMARSFireDAC;

     

    The Connection attribute specifies the name of the connection definition to be used with the TMARSFireDAC instance injected.

    The name supports macros (the second argument enables macro expansion).

     

    You can rely on built-in macros (i.e. using some part of the token, params, username or similar things. A list is available here: https://github.com/andrea-magni/MARS/blob/a8d323558bd591589ef667eac326324302d167a9/Source/MARS.Data.FireDAC.pas#L528 ). Last chance of the macro-expander is looking for custom providers.

    You can register your own in the initialization of one of your units or in the Server.Ignition.pas file:

     

      TMARSFireDAC.AddContextValueProvider(
        procedure (const AActivation: IMARSActivation;
          const AName: string; const ADesiredType: TFieldType; out AValue: TValue)
        begin
          if SameText(AName, 'Dynamic_MyConnection') then
          begin
            if Odd(SecondOf(Now))then
              AValue := 'OddConnection_' + AActivation.Request.HostName
            else
              AValue := 'EvenConnection_' + AActivation.Request.HostName;
          end;
        end
      );

     

    Beware: the connection name has to be this format: [PREFIX]_[SUFFIX](_[OTHER_SUFFIX])

    Using 'MyConnection' won't work for instance...

     

    Pros of this approach:

    1) code is executed only when needed (when the TMARSFireDAC instance gets created to be injected in your resources), all others REST endpoints won't be affected.

    2) it seems more isolated to me (but this can be a personal evaluation).

     

    If you need to fix some TFDConnection instances on your datamodules, try having a TMARSFireDAC instance injected in the datamodule itself and use the FD.ConnectionDefName (already expanded once injected) to fix the ConnectionDefName of your components (OnCreate event of the datamodule should be fine, or AfterConstruction method as well).

    If you encounter troubles doing this, write here again: there's another hidden feature for you to fix this (when registering the datamodule as resource you can specify an anonymous method to initialize it or to call a custom constructor instead of TDatamodule.Create(AOwner)).

     

    Sincerely,

    Andrea

     

     

     

     

    • Thanks 2

  9. Hi @Yaron,

    you can register an InvokeError procedure to deal with non-matching URLs, there's a template in Server.Ignition file, line 107 (https://github.com/andrea-magni/MARS/blob/daef64912b04036917cb336c4eee34438fa993a8/Demos/MARSTemplate/Server.Ignition.pas#L107).

     

    If you uncomment that registration, you can implement whatever policy using the AActivation argument to access Request and Response objects (you can set the StatusCode, Content and ContentType of the Response object as you like).

     

    Sincerely,

    Andrea

    • Like 1

  10. Unfortunately not...

    The TFrameStand functionalities should be the same, there are some changes due to refactoring (i.e. if you had event handlers for TFrameStand events, some argument changed type like TFrameInfo<TFrame> now being TSubjecTInfo, so you may need to adjust them on your side).

    TFormStand is the same concept but supporting TForm instead of TFrame.

     

    If you already have projects using TFrameStand, it would be helpful if you could confirm I broke nothing.

     

    Sincerely,

    Andrea


  11. 17 hours ago, Ruslan said:

    Would like to test new features and versions.

    Do use it in one of my products.

     

    Great, please check out the new version from https://github.com/andrea-magni/TFrameStand/tree/TSubjectStand (you can use Git or download a zip file if you are not confident with Git). Just beware to uninstall any previous version of TFrameStand before proceeding.

     

    Some changes in your code may be needed especially if you used TFrameStand events. For example where you had TFrameInfo<TFrame> as event handler argument, now you'll find TSubjectInfo (that is a new superclass of TFrameInfo<T>).

     

    If you encounter issues, just drop me a line.

    Thanks.

    Sincerely,

    Andrea


  12. 19 hours ago, dummzeuch said:

    Btw: Your github page says:

     

    ##Get started

    Unfortunately that link seems to be broken.

    http://www.andreamagni.eu/wp/tag/tframestand/

     

    It should probably be

    https://blog.andreamagni.eu/tag/tframestand/

    instead.

     

    Thanks, I relocated the blog some time ago and I set up a rewrite rule that for some reason is not working well.

    I changed the links and updated also the demo list. That document will change again soon however, I'll add section for TFormStand soon.

     

    Sincerely,

    Andrea


  13. I'm about to release TFormStand as a form-oriented twin of TFrameStand.

    To add support for TForm I had to refactor quite heavily TFrameStand code (now there is a common ancestor that is TSubjectStand and two inherited components TFrameStand and TFormStand).

     

    I am looking for:

    - people willing to test TFrameStand functionalities (regression tests) are still functioning;

    - people willing to test TFormStand

     

    Please reply to this message or contact me directly if you like to be involved.

     

    The new version is available in the TSubjectStand branch of TFrameStand GitHub repository: https://github.com/andrea-magni/TFrameStand/tree/TSubjectStand

    (Beware: uninstall any previous version of TFrameStand before installing the new one)

     

    Thanks

     

    Sincerely,

    Andrea

    • Like 1

  14. 12 hours ago, Yaron said:

    Is there a way to shorten the URL required to access the MARS server?

     

    What I did so far was to modify FEngine.AddApplication('DefaultApp', '/v', [ 'Server.Resources.*']); in "Server.Ignition".

     

    Ideally, I would like to eliminate the "/rest" and application name entirely, so the entry point would be whatever I define in [Path('')].

     

     

    Hi @Yaron,

    you can definitely get rid of the "/rest" prefix by setting the Engine BasePath property to empty string (by default the engine has 'DefaultEngine' as Name and '/rest/ as BasePath):

     

    class constructor TServerEngine.CreateEngine;
    begin
      FEngine := TMARSEngine.Create;
      try
        // Engine configuration
        FEngine.BasePath := ''; // <==============
        FEngine.Parameters.LoadFromIniFile;
    
        // Application configuration
        FEngine.AddApplication('DefaultApp', '/default', [ 'Server.Resources.*']);

    If you want to push this thing futher, you can even think about inheriting your own TMARSActivation class and shortcut request matching by overriding FindMethodToInvoke method.

     

    HTH

     

    Sincerely,

    Andrea

    • Like 1

  15. Sorry, it's my fault: MARS supports two http layer implementor on the client side: Indy and TNetHttpClient.

    Handling errors has some differences across the two, so it depends if you are using a TMARSClient (Indy) or TMARSNetClient (TNetHttpClient).

     

    Either ways, you can define an OnError handler for your TMARSApplication:

     

    uses FMX.Dialogs
    , MARS.Core.Utils, MARS.Client.Utils
    , IdHttp;
    
    var
      LException: EMARSClientHttpException;
      LIdException: EIdHTTPProtocolException;
    begin
      if AException is EMARSClientHttpException then
      begin
        LException := EMARSClientHttpException(AException);
        ShowMEssage('Error' + LException.Message + ' Details: ' + StreamToString(LException.Content));
        AHandled := True;
      end;
      if AException is EIdHttpProtocolException then
      begin
        LIdException := EIdHttpProtocolException(AException);
        ShowMEssage('Error' + LIdException.Message);
        AHandled := True;
      end;
    
    end;

     

    IIRC I was not able to find a way to collect error details (that are sent back from the server to the client using the body of the response) when using Indy.

    I am open to discuss how to improve the overall mechanism, please express your thoughts (everybody!).

    I'll try to update the old demos (such as HelloWorld) to include error handling at some point.

     

    Sincerely,

    Andrea

     


  16. Change TDataSet to TFDDataSet and you'll be able to produce both TMediaType.APPLICATION_JSON and TMediaType.APPLICATION_JSON_FireDAC (MARS will match what client prefer, looking for the Accept header's value):

      [POST]
      function GetData([BodyParam] AStatement: TStatement): TFDDataSet;
    function THelloWorldResource.GetData(AStatement: TStatement): TFDDataSet;
    begin
      Result := FD.SetName<TFDDataSet>(FD.Query(AStatement.SQL), AStatement.Name);
    end;

     

    Beware to remove the explicit Produces attribute you had.

    You may want to use a REST debugger (I like Postman) to make a call using Accept: application/json-firedac or Accept: application/json to see the two serializations.

     

    Now you can add a TMARSClientResourceJSON to your client application, note the SpecificAccept header where I am instructing to set Accept header to application/json-firedac media type:

    object MARSClientResourceJSON1: TMARSClientResourceJSON
      Application = MARSApplication
      SpecificAccept = 'application/json-firedac'
      SpecificContentType = 'application/json'
      Resource = 'helloworld'
    end

    Then use the following code (TMainDataModule is a datamodule containing the REST components, TMARSClient, TMARSApplication, TMARSClientResourceJSON):

     

    uses MARS.Core.JSON, MARS.Core.Utils, MARS.Data.FireDAC.Utils;
    
    procedure TMainDataModule.RetrieveData(const ASQL, AName: string;
      const AOnCompleted: TProc<TFDMemTable>; const AOnError: TProc<Exception> = nil);
    begin
      MARSClientResourceJSON1.POSTAsync(
        procedure (ABody: TMemoryStream)
        var
          LJSON: TJSONObject;
        begin
          LJSON := TJSONObject.Create;
          try
            LJSON.WriteStringValue('SQL', ASQL);
            LJSON.WriteStringValue('name', AName);
    
            JSONValueToStream(LJSON, ABody);
          finally
            LJSON.Free;
          end;
        end
      , procedure (ARes: TMARSClientCustomResource)
        var
          LData: TArray<TFDMemTable>;
        begin
          LData := TFDDataSets.FromJSON((ARes as TMARSClientResourceJSON).Response as TJSONObject);
          try
            if (Length(LData)>0) and Assigned(AOnCompleted) then
              AOnCompleted(LData[0]);
          finally
            TFDDataSets.FreeAll(LData);
          end;
        end
      , AOnError
      );
    end;

     

    Then you can have a call like:

     

      MainDataModule.RetrieveData('select * from EMPLOYEE', 'Employee'
      , procedure (ADataSet: TFDMemTable)
        begin
          FDMemTable1.Data := ADataSet.Data; // cloning dataset
        end
      , procedure (AError: Exception)
        begin
          ShowMessage(AError.ToString);
        end
      );

     

    FDMemTable1 is a TFDMemTable to store data on the client side.

     

    Please consider:

    1) code could have been more simple than this but I wanted to include the fact this call is actually made asynchronously (so if you need to transfer a large dataset or connection is not very fast, you are not going to hang your client application);

    2) this is a good example how to retrieve a dataset and then have a single (synchronized) point where the resulting dataset is then cloned to a local TFDMemTable that can be easily involved in databinding (LiveBindings) without being affected by the async call.

     

    Please let me know if this helps.

    Sincerely,

    Andrea

     

     


  17. TMARSFDResource is tightly coupled with a server side resource that has this interface:

     

        [GET, IsReference]
        function Retrieve: TArray<TFDDataSet>;
        [POST]
        function Update([BodyParam] const ADeltas: TArray<TFDMemTable>): TArray<TMARSFDApplyUpdatesRes>;

     

    One example of such resource is TMARSFDDataModuleResource (MARS.Data.FireDAC.DataModule.pas) that is the base class of the datamodule in FireDAC Basic demo.

    The logic of this resource does not allow the client to push statements to server (the Retrieve method is marked with GET verb so you don't have body available to send statements to be executed to sever side). Also, the Update method needs to know the structure (statements) of dataset in order to proprerly apply deltas to them.

     

    You can build a different resource on the same idea but adding statements as input to both methods. The Retrieve part we already covered in the previous message (using POST verb). You may want to have a PUT verb to apply updates.

    On the client side, you can use a TMARSClientResourceJSON and the TFDDataSets utility class (MARS.Data.FireDAC.Utils unit) to manage persistence of an array of datasets and/or an array of deltas (have a look at MARS.Client.FireDAC unit for some examples).

     

    As said, it is a bit unusual to send statements to be executed from the client to the server so there's no specific integration in MARS for this.

    It should not be hard to implement and all building blocks are already there. However if you find difficulties just write on this forum and I'll be happy to help.

     

    Sincerely,

    Andrea

     

×