TNetHTTPClient and StatusText not being "OK"

[alank2 5]

I am using TNetHTTPClient and I am getting back a status code 200 from the request.  When I look at the value in StatusText however, it is empty instead of the OK I expected.

 

I did the same request with postman and looking at the raw status line the OK is present.

 

Has anyone else experienced this?

 

I did look in System.Net.HttpClient.Win.pas and see that it does a:

    Result := ReadHeader(FWRequest, WINHTTP_QUERY_STATUS_TEXT);

 

edit: this was with 11.3

Edited 17 hours ago by alank2

[rvk 34]

23 minutes ago, alank2 said:

I did look in System.Net.HttpClient.Win.pas and see that it does a:

    Result := ReadHeader(FWRequest, WINHTTP_QUERY_STATUS_TEXT);

So, what happens if you (debug)trace into that ReadHeader?

On what line does it 'crap' out?

 

Do you get a ERROR_WINHTTP_HEADER_NOT_FOUND in that function?

Do your response-headers contain a Content-Length?

If not... is your input/response chunked?    (that could be the cause of HEADER_NOT_FOUND)

 

Is this happening with one certain URL or with all/more URLs?

 

[Remy Lebeau 1405]

1 hour ago, rvk said:

So, what happens if you (debug)trace into that ReadHeader?

On what line does it 'crap' out?

One thing to note - ReadHeaders() calls WinHttpQueryHeaders(), but WinHttpQueryHeadersEx() is documented as not accepting WINHTTP_QUERY_STATUS_TEXT, so maybe Microsoft made an update that affects WinHttpQueryHeaders() similarly and didn't document it? Just guessing.

1 hour ago, rvk said:

Do your response-headers contain a Content-Length?

If not... is your input/response chunked?    (that could be the cause of HEADER_NOT_FOUND)

What does that have to do with retrieving the status text?  It doesn't come from those headers.

[alank2 5]

I added the pas files to my project (C++) and stepped to ReadHeader - for some reason I can't evaluate variables properly - if I try to look at LSize even after stepping LSize:=0, it is NOT zero. 

 

After calling WinHttpQueryHeaders, it shows as zero.  GetLastError is ERROR_INSUFFICIENT_BUFFER and it gets by that test.  It does not get by the     "if Length(Result) > 0 then " test however making me think that the SetLength() did not change the length?  Oddly with the debugger on the "if Length(..." line it now shows the LSize as 274, but the if length test fails and it never calls WinHttpQueryHeaders the second time.

 

 

 

Edited 15 hours ago by alank2

[Remy Lebeau 1405]

13 minutes ago, alank2 said:

I added the pas files to my project (C++) and stepped to ReadHeader

You should not need to add the .pas file, just enable Debug DCUs instead, then you can step into the RTL's source code.

13 minutes ago, alank2 said:

for some reason I can't evaluate variables properly - if I try to look at LSize even after stepping LSize:=0, it is NOT zero. 

Are you compiling for 64bit? Do you have all patches installed? Inspecting Delphi variables in C++ is a known issue.

13 minutes ago, alank2 said:

After calling WinHttpQueryHeaders, it shows as zero.  GetLastError is ERROR_INSUFFICIENT_BUFFER and it gets by that test.  It does not get by the     "if Length(Result) > 0 then " test however making me think that the SetLength() did not change the length?

Hard to say without knowing what the true value of LSize is.  Sounds more like LSize may be 0-2 bytes to begin with.

13 minutes ago, alank2 said:

Oddly with the debugger on the "if Length(..." line it now shows the LSize as 274

That is pretty large for a 200 reply.  Most servers just send an "OK" text.  What does the raw response line actually look like on the wire?

13 minutes ago, alank2 said:

but the if length test fails

That should not be possible.  If the specified length is greater than 0 then memory is guaranteed to be allocated or an exception thrown on failure.

 

Personally, I would not have coded it the way they did.  If WinHTTP asks for a buffer allocation then I would just make the 2nd API call unconditionally after allocating the requested memory.  It looks like they are trying to ignore the case where the text is empty (just a null terminator), in which case they should have tested the requested LSize before allocating the memory.  But that is just me.

13 minutes ago, alank2 said:

That attachment can't be found.

[alank2 5]

It is 32 bit.

 

Removing the two .pas modules and turning on debug dcu's fixes the issue with inspecting LSize.  It gets cleared properly now and returning from WinHttpQueryHeaders leaves LSize at 0 which is why it isn't allocating it or retrieving it.

 

The status line in the raw is:

HTTP/1.1 200 OK

 

If it makes any difference, I am sending multipart/form-data.

 

[rvk 34]

38 minutes ago, alank2 said:

Removing the two .pas modules and turning on debug dcu's fixes the issue with inspecting LSize.  It gets cleared properly now and returning from WinHttpQueryHeaders leaves LSize at 0 which is why it isn't allocating it or retrieving it.

But what is the result of that call?

Is it ERROR_WINHTTP_HEADER_NOT_FOUND or something else? 

 

[alank2 5]

The original call to WinHttpQueryHeaders has a LSize==0, and it returns with ERROR_INSUFFICIENT_BUFFER.

 

They intentionally call with a 0 to get the size.

 

It skips by "if GetLastError <> ERROR_INSUFFICIENT_BUFFER then" and moves to SetLength, but now my LSize is 16579768, so clearly the debugger is back to not working right again.  Moving to the CPU I can see that is loads EDX with a value 2 which is the size of OK.  Then it does a SHR edx,1 and changes the value from 2 to 1.  Then a DEC edx which changes it from 1 to 0. 

 

So the mystery here is WHY this line:

    SetLength(Result, LSize div SizeOf(Char) - 1);

 

is dividing by sizeof(Char) which is apparently 2 and then subtracting 1.

 

edit: looking at WinHttpQueryHeaders , it shows:

 

[in, out] lpdwBufferLength

Pointer to a value of type DWORD that specifies the length of the data buffer, in bytes

 

Since it is dealing with wide strings, 2 could be a zero terminator for the wide string and it thinks there is nothing to return.  Why would this function not see the "OK" in the http response?

Edited 13 hours ago by alank2

[rvk 34]

6 minutes ago, alank2 said:

So the mystery here is WHY this line:

    SetLength(Result, LSize div SizeOf(Char) - 1);

 

is dividing by sizeof(Char) which is apparently 2 and then subtracting 1.

 

Because the resulting string only needs (LSize div 2) - 1 characters.

LSize is the size in bytes. Result is an array of widechars/unicodechars which contain 2 bytes.

So if LSize is 6 bytes (OK#0#0 in unicode) you only need 2 character string.

6 div 2 = 3 - 1 is 2 for OK.

 

Quote

If the function succeeds, lpdwBufferLength specifies the length of the string, in bytes, minus 2 for the terminating null.

https://learn.microsoft.com/en-us/windows/win32/api/winhttp/nf-winhttp-winhttpqueryheaders

 

Edited 13 hours ago by rvk

[Remy Lebeau 1405]

Just now, rvk said:

LSize is the size in bytes. Result is an array of widechars/unicodechars which contain 2 bytes.

So if LSize is 6 bytes (OK#0#0 in unicode) you only need 2 character string.

More like (O#0K#0#0#0) if you are looking at raw bytes.  But yes, it should be 6 bytes (2 wide chars + 1 wide null terminator) since the Unicode version of WinHTTPQueryHeaders() is being called.

Just now, rvk said:

6 div 2 = 3 - 1 is 2 for OK.

Correct.  And since the Result is a String, the SetLength() will actually allocate memory for a null terminator even though it is not being requested in the size.  So, the code should be calling SetLength(2) for 'OK' but it will allocate memory for 3 Chars, enough for WinHTTPQueryHeaders() to fill in.

[alank2 5]

I'm not making a lot of sense out of this windows function.

 

Why would it return a size that doesn't include the terminating null \x00 character?   When it returns the buffer, does it not null terminate the string?

 

I'm not sure this is the issue as I modified the LSize to be 200 and let it make the second call and it still returned an empty string.

 

edit: I'm back to why would WinHttpQueryHeaders return a proper 200 string that gets turned into an integer just fine, but not the OK part.

 

Is there any way to access the actual response as bytes to see if it matches my postman test?

 

Edited 12 hours ago by alank2

[Dmitry Arefiev 101]

Can you share the URL and request ?

Edited 12 hours ago by Dmitry Arefiev

[alank2 5]

Unfortunately not.

[Remy Lebeau 1405]

2 hours ago, alank2 said:

Why would it return a size that doesn't include the terminating null \x00 character?   When it returns the buffer, does it not null terminate the string?

It does.  When it is called the 1st time with a nil buffer and 0 size, it returns the actual buffer size that is needed, which would include room for the null terminator.  On the 2nd call, when the buffer is actually filled in, it returns the number of bytes actually written into the buffer, not counting the null terminator, but the actual null terminator is written.  A lot of Win32 APIs work this way.

Quote

I'm not sure this is the issue as I modified the LSize to be 200 and let it make the second call and it still returned an empty string.

Then either the "OK" text is not actually on the response, or the API is broken.

Quote

edit: I'm back to why would WinHttpQueryHeaders return a proper 200 string that gets turned into an integer just fine, but not the OK part.

That would imply a bug in the API that would need to be reported to Microsoft.

Quote

Is there any way to access the actual response as bytes to see if it matches my postman test?

The API doesn't provide the complete raw response, only pieces of it.  You would have to use a packet sniffer like Wireshark or Fiddler to capture the complete raw response.

Edited 10 hours ago by Remy Lebeau