Location change bug with fragment

[HTMLValidator.com 6]

Trying to download a URL with TSslHttpCli.
 

URL is: https://www.htmlvalidator.com/help.php?h=aria-atomic&m=6

 

Which redirects to: https://www.w3.org/TR/wai-aria-1.1/#aria-atomic

 

Which results in a "HTTP/1.1 400 Bad Request".

 

I think the 400 is because TSslHttpCli is not removing the fragment (#aria-atomic) when it is requesting the redirected URL.

 

Real browsers work fine.

 

Is this a bug?

Edited July 10, 2021 by HTMLValidator.com

[Angus Robertson 577]

Neither Firefox nor ICS redirect to w3.org, using the ICS Frame Browser sample:

 

FrameBrowser Get: https://www.htmlvalidator.com/help.php?h=aria-atomic&m=6

[0] > GET /help.php?h=aria-atomic&m=6 HTTP/1.1
[0] < HTTP/1.1 302 Found
[0] < Location: help.php?m=2&h=html
Redirected to: https://www.htmlvalidator.com/help.php?m=2&h=html
[0] > GET /help.php?m=2&h=html HTTP/1.1
[0] < HTTP/1.1 200 OK

 

Angus

[HTMLValidator.com 6]

Sorry about that.

 

Please try https://www.htmlvalidator.com/help.php?h=aria-atomic&m=6 instead of https://www.htmlvalidator.com/help.php?h=aria-atomic&amp;m=6

The "&amp;" should be "&". Not sure if that was my mistake or something trying to be smart and converting the "&" to "&amp;" when I copy and pasted the link.

 

Thanks.

Edited July 10, 2021 by HTMLValidator.com

[Angus Robertson 577]

It would help if your application had better logging, you would then have seen the real error:

 

www.w3.org SSL Handshake Failed - error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed, State: error

 

This only happens if ICS is set to check the certificate chain, which is normal, but means you need to keep the root store up to date to allow verification to work.  I updated the ICS root stores last week but not in SVN yet, and the root certificate 'USERTrust RSA Certification Authority' seems to be different to the one the site is using, I'll look into this next week.

 

Angus

 

 

[HTMLValidator.com 6]

I am not sure I understand.

 

I have never used any certificate checking.

If I request https://www.w3.org/TR/wai-aria-1.1/#aria-atomic directly, then it works fine. If there is a redirect to that URL, then I get a 400 error.... I'm thinking because it's not removing the fragment part (#aria-atomic).

 

This is what my log looks like:

Proxy> No proxy being used
Request> GET /help.php?h=aria-atomic&m=6 HTTP/1.0
Request> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Request> Connection: Keep-Alive
Request> User-Agent: CSS HTML Validator Professional (https://www.htmlvalidator.com/)
Request> Host: www.htmlvalidator.com
Request> 
SslHandshakeErrCode> 0 (0 if no error)
SslHandshakeRespMsg> SSL Connected OK with TLSv1.3, cipher TLS_AES_256_GCM_SHA384, encryption AESGCM(256), message auth AEAD
Response> HTTP/1.1 302 Found
Response> Date: Sat, 10 Jul 2021 18:53:46 GMT
Response> Server: Apache
Response> Referrer-Policy: no-referrer-when-downgrade
Response> X-Content-Type-Options: nosniff
Response> X-Frame-Options: SAMEORIGIN
Response> X-Xss-Protection: 1; mode=block
Response> Content-Security-Policy: 
Response> Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Response> Upgrade: h2
Response> Connection: Upgrade, Keep-Alive
Response> Location: https://www.w3.org/TR/wai-aria-1.1/#aria-atomic
Response> Vary: User-Agent
Response> Content-Length: 0
Response> Keep-Alive: timeout=5, max=100
Response> Content-Type: text/html; charset=UTF-8
Location> Changing to: https://www.w3.org/TR/wai-aria-1.1/#aria-atomic
Request> GET /TR/wai-aria-1.1/#aria-atomic HTTP/1.0
Request> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Request> Connection: Keep-Alive
Request> User-Agent: CSS HTML Validator Professional (https://www.htmlvalidator.com/)
Request> Host: www.w3.org
Request> 
SslHandshakeErrCode> 0 (0 if no error)
SslHandshakeRespMsg> SSL Connected OK with TLSv1.3, cipher TLS_AES_128_GCM_SHA256, encryption AESGCM(128), message auth AEAD
Response> HTTP/1.1 400 Bad Request
Response> date: Sat, 10 Jul 2021 18:53:47 GMT
Response> last-modified: Thu, 04 Jun 2020 15:34:04 GMT
Response> etag: "420-5a743dfdcf300"
Response> accept-ranges: bytes
Response> content-length: 1056
Response> content-type: text/html; charset=iso-8859-1
Response> x-backend: ssl-mirrors
Response> strict-transport-security: max-age=15552000; includeSubdomains; preload
Response> content-security-policy: upgrade-insecure-requests
Response> connection: keep-alive

-- Request Done --
StatusCode   = 400
ReasonPhrase = Bad Request
Time         = 516 ms

 

Edited July 10, 2021 by HTMLValidator.com

[Angus Robertson 577]

You were correct, a fragment/anchor/bookmark in the URL should never be sent to the server, it is purely an instruction to the browser on how to display the page. 

 

The ICS unit did attempt to remove # during relocation, but relocation has many different paths in the code, and it was not always being done. 

 

I've corrected in my master, won't be in SVN for a day or two, but in the latest SVN version add these lines to the function EncodePathOnly:

 

        I := Pos('#', OldPath);
        if I > 1 then
           Result := Copy (OldPath, 1, I - 1);

 

My issue with the SSL/TLS certificate was me using an old sample application.

 

Angus

[HTMLValidator.com 6]

Great, thank you for the fix.

[Angus Robertson 577]

The relocation fix is now in SVN. 

 

The repository server is working fine, but wiki.overbyte.eu with the overnight zip has been down for two days, probably due to the flooding in Liege, but ICS can also be downloaded from https://www.magsys.co.uk/delphi/magics.asp

 

Angus