Receiving multiples JSON on Rest API Horse

[Andre Capelli 0]

Hi guys!

I'm having a little problem, my rest api on horse (Backend, console application), it's working very nice, but, i'm about to receive about 1000 jsons, how can I control that? The response time 201 - Created is about 1 ~ 2 seconds for one json.

 

Thanks.

[Lars Fosdal 1792]

You need to give us a little more to work with.

How complex are the json structures?  Is it the backend or the client that will be written in Delphi?

Is it the backend or the client that will receive 1000 jsons ?

 

If the backend uses 1-2 seconds and it is not written in Delphi - that is not a Delphi problem.

 

[Andre Capelli 0]

11 minutes ago, Lars Fosdal said:

You need to give us a little more to work with.

How complex are the json structures?  Is it the backend or the client that will be written in Delphi?

Is it the backend or the client that will receive 1000 jsons ?

 

If the backend uses 1-2 seconds and it is not written in Delphi - that is not a Delphi problem.

 

Sorry, let me elaborate.

Yes, It's in Delphi, the Backend will receive the JSONs and Insert into the Database. The response time was just in case anyone needed.

 

But the problem is how I receive that much json and read and insert one per time, or all of them at the same time.

[Andre Capelli 0]

1 hour ago, Lars Fosdal said:

You need to give us a little more to work with.

How complex are the json structures?  Is it the backend or the client that will be written in Delphi?

Is it the backend or the client that will receive 1000 jsons ?

 

If the backend uses 1-2 seconds and it is not written in Delphi - that is not a Delphi problem.

 

The Backend works fine now, but it works receiving one json at time.

[Angus Robertson 574]

How are you receiving the JSON, with a web server or TCP service? 

 

I have a similar application that accepts data in various ways and writes to a SQL database, I use a FIFO queue (a stringlist) for the SQL stored procedures, so any that arrive faster than the SQL can accept them (about 20 per second) get queued.  If the volume is really high,  you can open multiple connections to the SQL server and do some in parallel.

 

Angus

 

[Andre Capelli 0]

3 minutes ago, Angus Robertson said:

How are you receiving the JSON, with a web server or TCP service? 

 

I have a similar application that accepts data in various ways and writes to a SQL database, I use a FIFO queue (a stringlist) for the SQL stored procedures, so any that arrive faster than the SQL can accept them (about 20 per second) get queued.  If the volume is really high,  you can open multiple connections to the SQL server and do some in parallel.

 

Angus

 

I have a Webhook pointing at the DDNS in the Cliente Server

[Lars Fosdal 1792]

Are you receiving multiple individual json packages (1), or is it one large package with an array of multiple json elements (2)?

1. {json}{json}{json} 

2. {"list":[{json},{json},{json}]}

 

How complex is each json element? Can you show us an example?

Is it the insert to the DB that is time consuming or the decoding of the json?

Are you opening and closing the DB connection for each json element?

[Andre Capelli 0]

4 hours ago, Lars Fosdal said:

Are you receiving multiple individual json packages (1), or is it one large package with an array of multiple json elements (2)?

1. {json}{json}{json} 

2. {"list":[{json},{json},{json}]}

 

How complex is each json element? Can you show us an example?

Is it the insert to the DB that is time consuming or the decoding of the json?

Are you opening and closing the DB connection for each json element?

Option 2, here it's an example of the json:

 

{
    "leads": [{
            "id": "1872278237",
            "email": "brodherjoao141186@gmail.com",
            "name": "Piu",
            "company": null,
            "job_title": null,
            "bio": null,
            "public_url": "http://app.rdstation.com.br/leads/public/55f0e6d6-418e-4d65-903b-663914ab9f22",
            "created_at": "2021-11-25T00:21:16.855-03:00",
            "opportunity": "true",
            "number_conversions": "1",
            "user": null,
            "first_conversion": {
                "content": {
                    "telefone": "71 99723-9477",
                    "nome": "Piu",
                    "traffic_source": "utm_source=Facebook%20Ads\u0026utm_medium=cpc\u0026utm_campaign=Vila%20Valqueire%20Telefone%20Branco",
                    "identificador": "Vila Valqueire Telefone Branco",
                    "email_lead": "brodherjoao141186@gmail.com",
                    "UF": null
                },
                "created_at": "2021-11-25T00:21:16.876-03:00",
                "cumulative_sum": "1",
                "source": "Vila Valqueire Telefone Branco",
                "conversion_origin": {
                    "source": "Facebook Ads",
                    "medium": "cpc",
                    "value": null,
                    "campaign": "Vila Valqueire Telefone Branco",
                    "channel": "Paid Search"
                }
            },
            "last_conversion": {
                "content": {
                    "telefone": "71 99723-9477",
                    "nome": "Piu",
                    "traffic_source": "utm_source=Facebook%20Ads\u0026utm_medium=cpc\u0026utm_campaign=Vila%20Valqueire%20Telefone%20Branco",
                    "identificador": "Vila Valqueire Telefone Branco",
                    "email_lead": "brodherjoao141186@gmail.com",
                    "UF": null
                },
                "created_at": "2021-11-25T00:21:16.876-03:00",
                "cumulative_sum": "1",
                "source": "Vila Valqueire Telefone Branco",
                "conversion_origin": {
                    "source": "Facebook Ads",
                    "medium": "cpc",
                    "value": null,
                    "campaign": "Vila Valqueire Telefone Branco",
                    "channel": "Paid Search"
                }
            },
            "custom_fields": {},
            "website": null,
            "personal_phone": "71 99723-9477",
            "mobile_phone": null,
            "city": null,
            "state": null,
            "tags": ["max filial vv", "vila valqueire - ads"],
            "lead_stage": "Lead",
            "last_marked_opportunity_date": "2021-11-25T00:29:39.508-03:00",
            "uuid": "55f0e6d6-418e-4d65-903b-663914ab9f22",
            "fit_score": "d",
            "interest": 0
        }
    ]
}

 

I'm treating only one json per time, that way, if the Backend receive another post, he stop decoding that json and start on the new one, I don't know how to control that.

And yes, i'm opening and closing the connection for each json, I only send the response 201 back when all is done.

[Lars Fosdal 1792]

Opening and closing the databaseconnection is expensive. 
You need a worker thread pool that where each worker thread keeps a connection to the database.

 

I am still not sure that I understand your architecture and dataflow.  

Your example shows a single element: {"leads": [{json}]}  - is that the typical REST request that you receive multiple times, or do you get {"leads": [{json1}], {json2}, ..., {json1000}]} ?

 

1 hour ago, Andre Capelli said:

I'm treating only one json per time, that way, if the Backend receive another post, he stop decoding that json and start on the new one, I don't know how to control that.

That doesn't sound right?  I don't know the Horse backend lib - so I don't know how to advise you on that.

[Andre Capelli 0]

1 hour ago, Lars Fosdal said:

Opening and closing the databaseconnection is expensive. 
You need a worker thread pool that where each worker thread keeps a connection to the database.

 

I am still not sure that I understand your architecture and dataflow.  

Your example shows a single element: {"leads": [{json}]}  - is that the typical REST request that you receive multiple times, or do you get {"leads": [{json1}], {json2}, ..., {json1000}]} ?

Yes, that's why I'm trying to rewrite that code, works only for one json.

 

The json is like this

 

{"leads":[{json}]} - one like this for each person 

{"leads":[{json}]} - another person

 

and that go on. Now, the Backend is working with a delay on the posts {"leads":[{json}]}, because it only process one at a time.

 

1 hour ago, Lars Fosdal said:

That doesn't sound right?  I don't know the Horse backend lib - so I don't know how to advise you on that.

Example:

the server send 3 posts with {"leads": [{json}]}, {"leads": [{json}]}, {"leads": [{json}]} in the same moment, how to control that at the same time?

[David Schwartz 426]

10 hours ago, Andre Capelli said:

the server send 3 posts with {"leads": [{json}]}, {"leads": [{json}]}, {"leads": [{json}]} in the same moment, how to control that at the same time?

I'm not all that familiar with JSON, but looking at this and what you're saying, I'm wondering if this shouldn't be structured as an array?

 

"a_lbl1": [  {"leads": [{json}]}, {"leads": [{json}]}, {"leads": [{json}]}  ]

[Lars Fosdal 1792]

@David Schwartz - I think that he means that the server sends individual REST posts.

 

@Andre Capelli - Just to be sure:

The Backend (which uses the Horse libs) is where your Delphi code is.

The Server is someone's server or service that place REST calls to your Backend.

How is the connection established? Do you make a request, and then your Backend is subscribing to individual, asynchronous updates from the Server?

 

IMO, if the Backend keeps receiving REST calls after connecting to the Server, a workable solution would be to queue the incoming JSON to a work queue, and then have a worker thread pool consuming from that queue. You do typically not want to do all the processing in the response handling, but hand it off to workers. It gets more complex if you must validate the results of handling the JSON to decide on the result code for the REST request, but if there is a Horse community, that would be the place to ask for how to deal with it.  It sounds odd that the current processing is interrupted when a new REST request arrives.

 

When you get multiple requests in Indy, each of them gets a thread of their own - and I use the above approach and have the response thread idle wait for a completion from the processing queue - or - after the specified time, return a timeout stat.

My solution is not typical REST, but JSONRPC - which sort of is synchronous RPC in JSON format, but the challenge is the same.  Worker threads save a lot of time when you need setting up the connections, lookup tables, and other forms of data that can be cached.

 

[Andre Capelli 0]

1 hour ago, Lars Fosdal said:

@David Schwartz - I think that he means that the server sends individual REST posts.

Exactly, I can't control.

 

1 hour ago, Lars Fosdal said:

 

@Andre Capelli - Just to be sure:

The Backend (which uses the Horse libs) is where your Delphi code is.

The Server is someone's server or service that place REST calls to your Backend.

How is the connection established? Do you make a request, and then your Backend is subscribing to individual, asynchronous updates from the Server?

Yes, the Backend it's where Delphi code is with Horse, the server is a client that sends posts. 

 

The server sends a post to a URL that redirects to the Backend, I just receive the rest calls and after the whole code I send back the response.

1 hour ago, Lars Fosdal said:

IMO, if the Backend keeps receiving REST calls after connecting to the Server, a workable solution would be to queue the incoming JSON to a work queue, and then have a worker thread pool consuming from that queue. You do typically not want to do all the processing in the response handling, but hand it off to workers. It gets more complex if you must validate the results of handling the JSON to decide on the result code for the REST request, but if there is a Horse community, that would be the place to ask for how to deal with it.  It sounds odd that the current processing is interrupted when a new REST request arrives.

 

I didn't understand too, after I created a log it's easy to see the processing stops and starting again with the new request

 

thanks for the help, I'll look up about creating a queue

[Angus Robertson 574]

You have still not clarified how exactly you are receiving the JSON, just a vague 'Cliente Server'. 

 

But if a new request is stopping an old request, it sounds like you have a single listener for receiving requests, whereas any proper server would accept multiple requests and handle each one separately, sometimes in a thread, but not necessarily with good program design and the ICS internet components. 

 

While a FIFO queue can help, there is a problem if you need to send a response for the SQL update status, particularly if you are continually opening and closing SQL connections. 

None of this design is really relevant to how many JSON records you receive, you just process them in one go, ideally with one SQL update.

 

Angus

 

 

[Lars Fosdal 1792]

2 hours ago, Andre Capelli said:

after the whole code I send back the response.

Does the response depend on the result of the processing?

I.e. Does the response need to contain data from the processing. or return a different status if the processing failed?

[Andre Capelli 0]

4 hours ago, Lars Fosdal said:

Does the response depend on the result of the processing?

I.e. Does the response need to contain data from the processing. or return a different status if the processing failed?

Yes, 201 for created and all information updated, and 400 if error on create or on insert any information ( the server will try again in 15 minutes, they control that)

 

I'm thinking in save every json on a txt file right on the post, and create a parallel thread to do all the working later, that can work right?

[Andre Capelli 0]

6 hours ago, Angus Robertson said:

You have still not clarified how exactly you are receiving the JSON, just a vague 'Cliente Server'. 

 

But if a new request is stopping an old request, it sounds like you have a single listener for receiving requests, whereas any proper server would accept multiple requests and handle each one separately, sometimes in a thread, but not necessarily with good program design and the ICS internet components. 

 

While a FIFO queue can help, there is a problem if you need to send a response for the SQL update status, particularly if you are continually opening and closing SQL connections. 

None of this design is really relevant to how many JSON records you receive, you just process them in one go, ideally with one SQL update.

 

Angus

 

 

Yes, I have to send the response back if occurs any error in any part of the SQL,

 

the server is a web site (RD Station)

[Angus Robertson 574]

If you have to send a response on SQL completion, you have to process all JSON in that request before sending a response, so a queue is not that useful, 

 

So this is down to your server design, if you are listening for connections that is a server, and you have still not explained how you are doing that, it is not a 'web site', it's a server.   Another web server may be sending your server requests, but that is not relevant.

 

Angus

 

[Lars Fosdal 1792]

I disagree. I'd still use work threads queues and a wait for a result queue to properly handle timeout situations.  Divide and Conquer.

[Andre Capelli 0]

On 12/4/2021 at 6:51 AM, Lars Fosdal said:

I disagree. I'd still use work threads queues and a wait for a result queue to properly handle timeout situations.  Divide and Conquer.

How would I do that? Can You give some examples? 

Edited December 7, 2021 by Andre Capelli

[ertank 27]

On 12/1/2021 at 8:05 PM, Andre Capelli said:

Sorry, let me elaborate.

Yes, It's in Delphi, the Backend will receive the JSONs and Insert into the Database. The response time was just in case anyone needed.

 

But the problem is how I receive that much json and read and insert one per time, or all of them at the same time.

I only checked Horse framework in the past and do not have deep knowledge about it. It was not to my taste. I like MARS-Curiosity more.

I do not know how you coded Horse side of the things. I do know for sure that Horse is a multi-threaded framework. You should really check your codes on the Horse side.

[Andre Capelli 0]

2 hours ago, ertank said:

I only checked Horse framework in the past and do not have deep knowledge about it. It was not to my taste. I like MARS-Curiosity more.

I do not know how you coded Horse side of the things. I do know for sure that Horse is a multi-threaded framework. You should really check your codes on the Horse side.

I think it's better to recreate after reading this.