Shira

Nevermind that, I had some conflict with another package where a component is using TWSocket, I worked around it then it installed fine.

15 minutes ago, KodeZwerg said:

Are you aware about that you can add images right here as attachment like I show you when I open your links?

 

Edited

I haven't been able to install the new release under Delphi 11, either the 8.70 or the daily snapshots,  always getting this error about OverbyteIcsHtmlPars, installing the VclFmx group:

if I try the VCL-only group, only a few components get installed:

2 minutes ago, Angus Robertson said:

You are assuming DocBegin is only called for a successful 200 request, when it is called otherwise.  Not sure why you want a download bar, they were useful 20 years ago with slow downloads, or for very large files, but 5MB comes down in a couple of seconds in the modern world. 

That url is just an example, another exe my app downloads is around 70MB, not everyone has fiber optic.

As you say 1.1 works fine, the problem is the double DocBegin trigger, shouldn't it happen only when the actual file starts?

7 minutes ago, FPiette said:

You should make sure ContentLength value is suitable for the ProgressBar Max property. If the content length is unknown, then the value returned is -1. And this is definitely not an ICS bug: A server may not advertise the length of the content it will send to the client. It is your responsibility as client side developer to take this kind of perfect correct issue.

 

That's correct, but the underlying issue comes from DocBegin triggering inbetween the 302 before the redirection starts, which I don't think it should.

Please note I'm not asking for help, criticizing or demanding a fix in any way, I just found what I deemed to be a possible bug so I reported on it for discussion. I have been using your ICS suite for around 20 years and love it 🙂

I found that when this triggers, AMax = -1

procedure TProgressBar.SetParams(AMin, AMax: Integer);
begin
  if AMax < AMin then
    raise EInvalidOperation.CreateFmt(SPropertyOutOfRange, [Self.Classname]);

39 minutes ago, Angus Robertson said:

It is unlikely to be related to chunking, this fails before the body is processed.  You can not disable chunking at the client, only the server.

 

All I need is the URL that fails, not demos. 

 

Angus

 

Sent you in a PM

Ok here's the actual issue: with 1.1 DocBegin triggers twice, the first time, before the 302 redict occurs, has ContentLenght set to -1 since the real file is not being sent yet.

Quote

 

-- Sending Header --
Method: GET

GET /downloads/2319 HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Referer: ***
Accept-Encoding: gzip, deflate
User-Agent: ***
Host: ***
Cache-Control: no-cache

-- Header Received --
Status: 302

HTTP/1.1 302 Found
Date: Sun, 18 Oct 2020 11:16:40 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: ***
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0
set-cookie: ***
location: *** (real file path)
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
cf-request-id: ***
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?lkg-colo=40&lkg-time=1603019800"}],"group":"cf-nel","max_age":604800}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: ***

 

OnDocBegin triggers here before the 302 when it shouldn't

HttpCli1.ContentLength = -1

then it continues to the redirection:
 

Quote

 

-- Sending Header --
Method: GET

GET ***(real file) HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Referer: ***
Accept-Encoding: gzip, deflate
User-Agent: ***
Host: ***
Cache-Control: no-cache

-- Header Received --
Status: 200

HTTP/1.1 200 OK
Date: Sun, 18 Oct 2020 11:16:40 GMT
Content-Type: application/x-executable
Content-Length: 5286400
Connection: keep-alive
Set-Cookie: ***
etag: ***
last-modified: Fri, 09 Aug 2019 19:30:44 GMT
accept-ranges: bytes
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
cf-request-id: ***
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?lkg-colo=40&lkg-time=1603019800"}],"group":"cf-nel","max_age":604800}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: ***

 

HttpCli1.ContentLength = 5286400

** ReqDone: Get, Status = 200

 

9 minutes ago, Kas Ob. said:

These symptoms might mean the ICS implementation have problem processing chunked encoding.

 

Additional information: i can't find the blog posts from CF now, where they did declared that they deliberately use strict HTTP standard to amplify and produce the most common mistakes in HTTP implementation, i saw it with the old SecureBlackBox (v15), where accessing CF API with compression and chunked encoding generate wrong result on client side, eg. insert irregular chunk sizes which does not change the outcome.

 

@Shira Does disabling chunk encoding support help ?

 

 

Disabling GZip (httpoEnableContentCoding) produces the same results. How do you disable chunk encoding in HttpCli?

Further testing, encapsulating that line with Try Except ... makes it work, and does not trigger an exception

  Try
    ProgressBar1.Max := HttpCli1.ContentLength;
  Except
    On E:Exception do begin
      Memo1.Clear;
      Memo1.Lines.Add(E.Message);
    end;
  End;

but running the app with the debugger on (F9) does trigger here:

procedure TProgressBar.SetParams(AMin, AMax: Integer);
begin
  if AMax < AMin then
    raise EInvalidOperation.CreateFmt(SPropertyOutOfRange, [Self.Classname]);

with "TProgressBar property out of range"

In both 1.0 and 1.1 cases the ContentLength property has the same value when DocBegin triggers.

After further debugging it gets even stranger:

I found this code is causing the 404, so I thought probably the SendMessage in TProgressBar.SetParams is causing this due to the way WSocket works with messages, similar to how Application.ProcessMessages can cause these issues.

procedure TForm1.HttpCli1DocBegin(Sender: TObject);
begin
  ProgressBar1.Max := HttpCli1.ContentLength;
end;

but then this simple change fixes it'

procedure TForm1.HttpCli1DocBegin(Sender: TObject);
begin
  ProgressBar1.Max := HttpCli1.ContentLength + 1;
end;

this also works

  ProgressBar1.Max := 1;
  ProgressBar1.Max := ProgressBar1.Max + HttpCli1.ContentLength;

but this fails...

  ProgressBar1.Max := HttpCli1.ContentLength;
  ProgressBar1.Max := ProgressBar1.Max + 1;

but then.. directly setting a random value works:

  ProgressBar1.Max := 123;

which kinda invalidates the problem being related to the ProgressBar SendMessage.

The main factors still remain through all this, HTTP/1.0 works in all cases, HTTP/1.1 fails 

😵

I can provide a working demo to reproduce this if you need.

Edit: using a custom progressbar without SendMessage to set its Max property also works.

EDIT: real issue identified here: 

Hi, I encountered a weird issue with the THttpCli component:

My app downloads a file from my website through a php script that redirects to the actual file. I recently set my website to use CloudFlare and the following happens:

- HttpCli set to use HTTP 1.0: works fine both with cloudflare and the original website

- HttpCli set to use HTTP 1.1: works fine my website, but when routing through CF it results in a 404 error (HttpCli1.StatusCode on HttpCli1RequestDone), even though the headers return 302

These are the headers returned by CloudFlare (some data removed):

Quote

HTTP/1.1 302 Found
Date: Sat, 17 Oct 2020 18:23:33 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=*****; expires=Mon, 16-Nov-20 18:23:32 GMT; path=/; domain=.******; HttpOnly; SameSite=Lax
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0
set-cookie: wp_dlm_downloading=****; expires=Sat, 17-Oct-2020 18:24:33 GMT; Max-Age=60; path=/; domain=*******; HttpOnly
location: ********* (real file URL after the redirection)
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
cf-request-id: *******
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?lkg-colo=40&lkg-time=1602959013"}],"group":"cf-nel","max_age":604800}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: *******

4 minutes ago, Kas Ob. said:

That is not an issue with browser, on contrary browser is right here, the second answer here in SO explain it :

https://stackoverflow.com/questions/1547899/which-characters-make-a-url-invalid

 

 

And from https://tools.ietf.org/html/rfc3986#section-3.5

 

I know, that's why I quoted it, I meant to say the issue is due to how the browser (properly) processes that. I don't know what Imgur is thinking removing the PIN method auth, making life even harder for desktop apps. Now I can only embed a browser window to have the user enter their user/pass (beats the whole purpose of OAuth + other issues) or use an app protocol for the redirection (another set of issues).

Hi, nevermind that, it seems to be an "issue" with the browser:

http://localhost:8080/test?aaa=1#bbb=2&ccc=3

Quote

 

The response_type Parameter token: This authorization flow will directly return the access_token and refresh_token via the redirect URL you specified during registration, in the form of hash query string parameters. Example: http://example.com#access_token=ACCESS_TOKEN&token_type=Bearer&expires_in=3600

The code and pin response types have been deprecated and will soon no longer be supported.

Imgur returns an access token to your application if the user grants your application the permissions it requested. The access token is returned to your application in the fragment as part of the access_token parameter. Since a fragment (the part of the URL after the #) is not sent to the server, client side javascript must parse the fragment and extract the value of the access_token parameter.

 

Hi, I'm using the WebSrv for Imgur's API and catch the tokens after launching the auth URL, so Imgur redirects to localhost with all the params.. but WebSrv can't catch them all because there is a #

Imgur redirection; http://localhost:7710/myapp?state=auth#access_token=....&expires_in= etc

then WebSrv's OnSimpWebSrvReq only gets "state=auth" and omits the rest.