TNetHTTPClient and StatusText not being "OK"

[alank2 5]

I am using TNetHTTPClient and I am getting back a status code 200 from the request.  When I look at the value in StatusText however, it is empty instead of the OK I expected.

 

I did the same request with postman and looking at the raw status line the OK is present.

 

Has anyone else experienced this?

 

I did look in System.Net.HttpClient.Win.pas and see that it does a:

    Result := ReadHeader(FWRequest, WINHTTP_QUERY_STATUS_TEXT);

 

edit: this was with 11.3

Edited Tuesday at 03:20 PM by alank2

[rvk 40]

23 minutes ago, alank2 said:

I did look in System.Net.HttpClient.Win.pas and see that it does a:

    Result := ReadHeader(FWRequest, WINHTTP_QUERY_STATUS_TEXT);

So, what happens if you (debug)trace into that ReadHeader?

On what line does it 'crap' out?

 

Do you get a ERROR_WINHTTP_HEADER_NOT_FOUND in that function?

Do your response-headers contain a Content-Length?

If not... is your input/response chunked?    (that could be the cause of HEADER_NOT_FOUND)

 

Is this happening with one certain URL or with all/more URLs?

 

[Remy Lebeau 1413]

1 hour ago, rvk said:

So, what happens if you (debug)trace into that ReadHeader?

On what line does it 'crap' out?

One thing to note - ReadHeaders() calls WinHttpQueryHeaders(), but WinHttpQueryHeadersEx() is documented as not accepting WINHTTP_QUERY_STATUS_TEXT, so maybe Microsoft made an update that affects WinHttpQueryHeaders() similarly and didn't document it? Just guessing.

1 hour ago, rvk said:

Do your response-headers contain a Content-Length?

If not... is your input/response chunked?    (that could be the cause of HEADER_NOT_FOUND)

What does that have to do with retrieving the status text?  It doesn't come from those headers.

[alank2 5]

I added the pas files to my project (C++) and stepped to ReadHeader - for some reason I can't evaluate variables properly - if I try to look at LSize even after stepping LSize:=0, it is NOT zero. 

 

After calling WinHttpQueryHeaders, it shows as zero.  GetLastError is ERROR_INSUFFICIENT_BUFFER and it gets by that test.  It does not get by the     "if Length(Result) > 0 then " test however making me think that the SetLength() did not change the length?  Oddly with the debugger on the "if Length(..." line it now shows the LSize as 274, but the if length test fails and it never calls WinHttpQueryHeaders the second time.

 

 

 

Edited Tuesday at 05:06 PM by alank2

[Remy Lebeau 1413]

13 minutes ago, alank2 said:

I added the pas files to my project (C++) and stepped to ReadHeader

You should not need to add the .pas file, just enable Debug DCUs instead, then you can step into the RTL's source code.

13 minutes ago, alank2 said:

for some reason I can't evaluate variables properly - if I try to look at LSize even after stepping LSize:=0, it is NOT zero. 

Are you compiling for 64bit? Do you have all patches installed? Inspecting Delphi variables in C++ is a known issue.

13 minutes ago, alank2 said:

After calling WinHttpQueryHeaders, it shows as zero.  GetLastError is ERROR_INSUFFICIENT_BUFFER and it gets by that test.  It does not get by the     "if Length(Result) > 0 then " test however making me think that the SetLength() did not change the length?

Hard to say without knowing what the true value of LSize is.  Sounds more like LSize may be 0-2 bytes to begin with.

13 minutes ago, alank2 said:

Oddly with the debugger on the "if Length(..." line it now shows the LSize as 274

That is pretty large for a 200 reply.  Most servers just send an "OK" text.  What does the raw response line actually look like on the wire?

13 minutes ago, alank2 said:

but the if length test fails

That should not be possible.  If the specified length is greater than 0 then memory is guaranteed to be allocated or an exception thrown on failure.

 

Personally, I would not have coded it the way they did.  If WinHTTP asks for a buffer allocation then I would just make the 2nd API call unconditionally after allocating the requested memory.  It looks like they are trying to ignore the case where the text is empty (just a null terminator), in which case they should have tested the requested LSize before allocating the memory.  But that is just me.

13 minutes ago, alank2 said:

That attachment can't be found.

[alank2 5]

It is 32 bit.

 

Removing the two .pas modules and turning on debug dcu's fixes the issue with inspecting LSize.  It gets cleared properly now and returning from WinHttpQueryHeaders leaves LSize at 0 which is why it isn't allocating it or retrieving it.

 

The status line in the raw is:

HTTP/1.1 200 OK

 

If it makes any difference, I am sending multipart/form-data.

 

[rvk 40]

38 minutes ago, alank2 said:

Removing the two .pas modules and turning on debug dcu's fixes the issue with inspecting LSize.  It gets cleared properly now and returning from WinHttpQueryHeaders leaves LSize at 0 which is why it isn't allocating it or retrieving it.

But what is the result of that call?

Is it ERROR_WINHTTP_HEADER_NOT_FOUND or something else? 

 

[alank2 5]

The original call to WinHttpQueryHeaders has a LSize==0, and it returns with ERROR_INSUFFICIENT_BUFFER.

 

They intentionally call with a 0 to get the size.

 

It skips by "if GetLastError <> ERROR_INSUFFICIENT_BUFFER then" and moves to SetLength, but now my LSize is 16579768, so clearly the debugger is back to not working right again.  Moving to the CPU I can see that is loads EDX with a value 2 which is the size of OK.  Then it does a SHR edx,1 and changes the value from 2 to 1.  Then a DEC edx which changes it from 1 to 0. 

 

So the mystery here is WHY this line:

    SetLength(Result, LSize div SizeOf(Char) - 1);

 

is dividing by sizeof(Char) which is apparently 2 and then subtracting 1.

 

edit: looking at WinHttpQueryHeaders , it shows:

 

[in, out] lpdwBufferLength

Pointer to a value of type DWORD that specifies the length of the data buffer, in bytes

 

Since it is dealing with wide strings, 2 could be a zero terminator for the wide string and it thinks there is nothing to return.  Why would this function not see the "OK" in the http response?

Edited Tuesday at 07:32 PM by alank2

[rvk 40]

6 minutes ago, alank2 said:

So the mystery here is WHY this line:

    SetLength(Result, LSize div SizeOf(Char) - 1);

 

is dividing by sizeof(Char) which is apparently 2 and then subtracting 1.

 

Because the resulting string only needs (LSize div 2) - 1 characters.

LSize is the size in bytes. Result is an array of widechars/unicodechars which contain 2 bytes.

So if LSize is 6 bytes (OK#0#0 in unicode) you only need 2 character string.

6 div 2 = 3 - 1 is 2 for OK.

 

Quote

If the function succeeds, lpdwBufferLength specifies the length of the string, in bytes, minus 2 for the terminating null.

https://learn.microsoft.com/en-us/windows/win32/api/winhttp/nf-winhttp-winhttpqueryheaders

 

Edited Tuesday at 07:27 PM by rvk

[Remy Lebeau 1413]

Just now, rvk said:

LSize is the size in bytes. Result is an array of widechars/unicodechars which contain 2 bytes.

So if LSize is 6 bytes (OK#0#0 in unicode) you only need 2 character string.

More like (O#0K#0#0#0) if you are looking at raw bytes.  But yes, it should be 6 bytes (2 wide chars + 1 wide null terminator) since the Unicode version of WinHTTPQueryHeaders() is being called.

Just now, rvk said:

6 div 2 = 3 - 1 is 2 for OK.

Correct.  And since the Result is a String, the SetLength() will actually allocate memory for a null terminator even though it is not being requested in the size.  So, the code should be calling SetLength(2) for 'OK' but it will allocate memory for 3 Chars, enough for WinHTTPQueryHeaders() to fill in.

[alank2 5]

I'm not making a lot of sense out of this windows function.

 

Why would it return a size that doesn't include the terminating null \x00 character?   When it returns the buffer, does it not null terminate the string?

 

I'm not sure this is the issue as I modified the LSize to be 200 and let it make the second call and it still returned an empty string.

 

edit: I'm back to why would WinHttpQueryHeaders return a proper 200 string that gets turned into an integer just fine, but not the OK part.

 

Is there any way to access the actual response as bytes to see if it matches my postman test?

 

Edited Tuesday at 08:19 PM by alank2

[Dmitry Arefiev 103]

Can you share the URL and request ?

Edited Tuesday at 08:33 PM by Dmitry Arefiev

[alank2 5]

Unfortunately not.

[Remy Lebeau 1413]

2 hours ago, alank2 said:

Why would it return a size that doesn't include the terminating null \x00 character?   When it returns the buffer, does it not null terminate the string?

It does.  When it is called the 1st time with a nil buffer and 0 size, it returns the actual buffer size that is needed, which would include room for the null terminator.  On the 2nd call, when the buffer is actually filled in, it returns the number of bytes actually written into the buffer, not counting the null terminator, but the actual null terminator is written.  A lot of Win32 APIs work this way.

Quote

I'm not sure this is the issue as I modified the LSize to be 200 and let it make the second call and it still returned an empty string.

Then either the "OK" text is not actually on the response, or the API is broken.

Quote

edit: I'm back to why would WinHttpQueryHeaders return a proper 200 string that gets turned into an integer just fine, but not the OK part.

That would imply a bug in the API that would need to be reported to Microsoft.

Quote

Is there any way to access the actual response as bytes to see if it matches my postman test?

The API doesn't provide the complete raw response, only pieces of it.  You would have to use a packet sniffer like Wireshark or Fiddler to capture the complete raw response.

Edited Tuesday at 10:28 PM by Remy Lebeau

[rvk 40]

12 hours ago, alank2 said:

edit: I'm back to why would WinHttpQueryHeaders return a proper 200 string that gets turned into an integer just fine, but not the OK part.

Without knowing what it actually returns, this is hard to diagnose.

 

You said the LSize was 274. So what's in Buf when the second call to WinHttpQueryHeaders returns?

 

[alank2 5]

3 hours ago, rvk said:

Without knowing what it actually returns, this is hard to diagnose.

 

You said the LSize was 274. So what's in Buf when the second call to WinHttpQueryHeaders returns? 

 

I think this was not real, but part of the LSize isn't the right value because of a debugger issue.

 

14 hours ago, Remy Lebeau said:

The API doesn't provide the complete raw response, only pieces of it.  You would have to use a packet sniffer like Wireshark or Fiddler to capture the complete raw response. 

Wouldn't it be encrypted if I captured it with wireshark (https)?

[Remy Lebeau 1413]

2 hours ago, alank2 said:

Wouldn't it be encrypted if I captured it with wireshark (https)?

Yes, unless you have access to the server's private key. Which is why I also mentioned Fiddler, which acts as a proxy and can capture HTTPS. 

[alank2 5]

This is a cut from the postman console which shows what the request returns (at least from postman):

 

Content-Length: 126768
 
----------------------------446648698401590502853205
Content-Disposition: form-data; name="select"
Static Data
----------------------------446648698401590502853205
Content-Disposition: form-data; name="fileData"; filename="file.json"
<file.json>
----------------------------446648698401590502853205
Content-Disposition: form-data; name="id"
1234567
----------------------------446648698401590502853205--
 
HTTP/1.1 200 OK
Date: Wed, 04 Dec 2024 18:55:06 GMT
Content-Type: application/json; charset=utf-8

[rvk 40]

1 minute ago, alank2 said:

Content-Length: 126768

Have you tried a smaller (or simpler) POST to that page?

 

Have you tried other URLs?

 

Do you only have this (no OK) with that URL?

 

Do you also have this only with POST or also with GET to that URL?

 

[alank2 5]

It is not consistent - I have another API that I use in the same service and it returns OK just fine, but if I do a GET @ https://google.com, it will not return OK.

 

I then tried a new application (not service) with the same code (GET @ https://google.com), it works fine and returns OK.

I rebuilt the project on 10.3.3, 11.3, and 12.2, and all of them return an empty string instead of OK.

 

I wonder if it being a service has something to do with it.

 

Next up I'll build a new service project service and see if the issue occurs with that.

[rvk 40]

11 minutes ago, alank2 said:

but if I do a GET @ https://google.com, it will not return OK.

[...]

I wonder if it being a service has something to do with it.

Could be that it's only a problem in service.

Try to create a simple test project. If you can consistently let it fail, we can test it too.

And if it's only in service that's also an indicator for which you can search.

 

[alank2 5]

I created a new 10.3.3 service application and put the code in - it does NOT return OK.

 

The same code in a standalone application does return OK.

 

I'll come up with a sample project I can post to see if others can reproduce the issue.

[Dmitry Arefiev 103]

What is your Windows version ?

[alank2 5]

Tested it on 10 and 11 same result.

[Remy Lebeau 1413]

Do you have the same problem if you use the WinInet API instead of the WinHTTP API?

 

In any case, at this point does it really matter if the status text is being returned or not?  It is arbitrary text determined by the server, so you shouldn't rely on it for any kind of processing logic, only for logging at best.  What is important is whether the status code works or not, and it sounds like it does.

Edited 4 hours ago by Remy Lebeau