Pointer casting with NativeUint

[duzzell 0]

Hello

Using Delphi 12.1, I have a library that under Win64 generates runtime access violations when NativeUInt is used to cast between pointers and integers. This has not generated errors under Win32:

 

Result := Pointer(NativeUInt(FElements)+(NativeUInt(Index)*ElementSize));

This code looks fine to me and others, so my first question is why is the error happening now? I understand that in Delphi 12+ NativeInt (and probably NativeUInt) cannot be used for overloading, but that does not mean to me that they can't be used for casting.

I can fix code like the above by making it explicit:

{$IFDEF CPU64BIT}
  Result := Pointer(UINT64(FElements)+(UINT64(Index)*ElementSize));
{$ELSE}
  Result := Pointer(UINT32(FElements)+(UINT32(Index)*ElementSize));
{$ENDIF CPU64BITS}

The problem here is that there's a lot of this type of casting in the library and I'll never be sure I've changed all of them. I took a hint from this SO post (https://stackoverflow.com/questions/7630781/delphi-2007-and-xe2-using-nativeint) and added code to the .inc file included in every .pas file:

{$IFDEF CPU64BITS}
    type
        NativeInt = INT64;
        NativeUInt = UINT64
{$ELSE}
    type
        NativeInt = INT32;
        NativeUInt = UINT32;
{$ENDIF CPU64BITS}

However, the inc file is included in each .pas file ahead of the interface uses clause. It appears that in Delphi type declarations are not allowed ahead of the uses clause.

So my second question is whether there is a way I can fix this problam without having to edit every single file in the library. It seems to me like I must have missed a switch or directive that would do what I need.

Thanks

[David Heffernan 2345]

The original code looks fine. But you only showed an excerpt. Show a reproduction of an error and then somebody can tell you what is actually going on. Don't fix things until you understand them.

[FPiette 381]

It is advised to use UIntPtr to cast a pointer to an unsigned integer. Note that System.pas make UIntPtr an alias for NativeUInt. This means it is not your problem.

I suggest you build a very small (preferably console mode) sample program which reproduce the issue so that we can reproduce your error, understand what is wrong and tell you what to do.

Edited August 24 by FPiette

[Remy Lebeau 1392]

3 hours ago, duzzell said:

Using Delphi 12.1, I have a library that under Win64 generates runtime access violations when NativeUInt is used to cast between pointers and integers. This has not generated errors under Win32:

Result := Pointer(NativeUInt(FElements)+(NativeUInt(Index)*ElementSize));

 

What is FElements declared as?  If you can enable pointer arithmetic on it (if not already), then the code becomes much much simpler:

Result := @FElements[Index];
// or:
Result := FElements+Index;

Even if the ElementSize is dynamic at runtime, you can utilize a PByte cast instead of a NativeUInt cast:

Result := PByte(FElements)+(Index*ElementSize);

 

Edited August 24 by Remy Lebeau

[duzzell 0]

Here are a few longer excerpts. These errors are occurring so often that I think there must be something in my setup that is causing them.

The errors have been of similar format:

    Project raised exception class $C0000005 WITH MESSAGE c0000005 ACCESS_VIOLATION.

1)

{$IFDEF FPC} {$PUSH}
  {$WARN 4055 off : Conversion between ordinals and pointers is not portable}
  {$WARN 4056 off : Conversion between ordinals and pointers is not portable}
{$ENDIF}
function ZRawToUnicode(const S: RawByteString; const CP: Word): UnicodeString;
begin
  if Pointer(S) = nil
  then Result := ''
//DU
{$IFDEF CPU32BITS}
  else Result := PRawToUnicode(Pointer(S), PLengthInt(UInt32(S) - StringLenOffSet)^{$IFDEF WITH_TBYTES_AS_RAWBYTESTRING}-1{$ENDIF}, CP);
{$ENDIF CPU32BITS}
{$IFDEF CPU64BITS}
  else Result := PRawToUnicode(Pointer(S), PLengthInt(UInt64(S) - StringLenOffSet)^{$IFDEF WITH_TBYTES_AS_RAWBYTESTRING}-1{$ENDIF}, CP);
{$ENDIF CPU64BITS}
//  else Result := PRawToUnicode(Pointer(S), PLengthInt(NativeUInt(S) - StringLenOffSet)^{$IFDEF WITH_TBYTES_AS_RAWBYTESTRING}-1{$ENDIF}, CP);
end;
{$IFDEF FPC} {$POP} {$ENDIF}

2)

//DU
{$IFDEF CPU32BITS}
    procedure DataEvent(Event: TDataEvent; Info: Integer); override;
{$ENDIF CPU32BITS}
{$IFDEF CPU64BITS}
    procedure DataEvent(Event: TDataEvent; Info: Int64); override;
{$ENDIF CPU64BITS}
//    procedure DataEvent(Event: TDataEvent; Info: NativeInt); override;

//DU    
{$IFDEF CPU32BITS}
procedure TZAbstractRODataset.DataEvent(Event: TDataEvent; Info: Integer);
{$ENDIF CPU32BITS}
{$IFDEF CPU64BITS}
procedure TZAbstractRODataset.DataEvent(Event: TDataEvent; Info: Int64);
{$ENDIF CPU64BITS}
//procedure TZAbstractRODataset.DataEvent(Event: TDataEvent; Info: NativeInt);
var I, j: Integer;
begin
  inherited DataEvent(Event, Info);
  if Event = deLayoutChange then
    for i := 0 to Fields.Count -1 do
      for j := 0 to high(FieldsLookupTable) do
        if (FieldsLookupTable[j].Field = Fields[i]) and (FieldsLookupTable[j].DataSource = dltResultSet) then begin
          FResultSetMetadata.SetReadOnly(FieldsLookupTable[j].Index, Fields[i].ReadOnly or not (pfInUpdate in Fields[i].ProviderFlags));
          FResultSetMetadata.SetSearchable(FieldsLookupTable[j].Index, (pfInWhere in Fields[i].ProviderFlags));
        end;
end;

3)

{$IFDEF FPC} {$PUSH}
  {$WARN 4055 off : Conversion between ordinals and pointers is not portable}
  {$WARN 4056 off : Conversion between ordinals and pointers is not portable}
{$ENDIF}
function TZKeyAndFunctionPairList.Get(Index: NativeInt): PZKeyAndFunctionPair;
begin
  {$IFNDEF DISABLE_CHECKING}
  if NativeUInt(Index) > Capacity then            //excluded due to IFNDEF
    Error(SListIndexError, Index);                    //excluded due to IFNDEF
  {$ENDIF DISABLE_CHECKING}
  Result := Pointer(NativeUInt(FElements)+(NativeUInt(Index)*ElementSize));
end;
{$IFDEF FPC} {$POP} {$ENDIF}

function TZFunctionsList.FindByKeyAndName(const aKey : Cardinal; const aName: string): Integer;
var
  I: NativeInt;
  KeyAndFunctionPair: PZKeyAndFunctionPair;
begin
  Result := -1;
  for I := 0 to FFunctions.Count - 1 do begin
    KeyAndFunctionPair := FFunctions.Get(I);
    if aKey = KeyAndFunctionPair.Key then                        <== AV stop here, caused by Get() I think
      if aName = KeyAndFunctionPair.Value.Name then begin
        Result := I;
        Break;
      end;
  end;
end;

4)
{There's a question about character size here, but I want to know why
my fix allows the code to run}

{$IFDEF FPC} {$PUSH} {$WARN 4055 off : Conversion between ordinals and pointers is not portable} {$ENDIF}
function SQLQuotedStr(Src: PWideChar; Len: LengthInt; Quote: WideChar): UnicodeString; overload;
var
  P, Dest, PEnd, PFirst: PWideChar;
begin
  Dest := Nil;
  P := Src;
  PEnd := P + Len;
  PFirst := nil;
  while P < PEnd do begin
    if (P^=Quote) then begin
      if Dest = nil then
        PFirst := P;
//DU
{$IFDEF CPU32BITS}
        Inc(UInt32(Dest));
{$ENDIF CPU32BITS}
{$IFDEF CPU64BITS}
        Inc(UInt64(Dest));
{$ENDIF CPU64BITS}
//      Inc(NativeUInt(Dest));
    end;
    Inc(P);
  end;

...

[Remy Lebeau 1392]

I see a number of things in this code that shouldn't be using raw pointer manipulations in the first place.  For instance, there is no reason for ZRawToUnicode() to dig into the internals of a RawByteString just to get its length.  Simply use Length(S) instead, eg:

function ZRawToUnicode(const S: RawByteString; const CP: Word): UnicodeString;
begin
  if Pointer(S) = nil then
    Result := ''
  else
    Result := PRawToUnicode(Pointer(S), Length(S){$IFDEF WITH_TBYTES_AS_RAWBYTESTRING}-1{$ENDIF}, CP);
end;

For that matter, the RTL in both Delphi and FPC already carries a codepage inside of RawByteString and knows how to convert RawByteString to UnicodeString using that codepage, so code like this really shouldn't be needed in the first place.  But, your IFDEF suggests RawByteString may be an alias for TBytes, so only in that case this kind of code would be needed.

 

I already commented earlier on the code in TZKeyAndFunctionPairList.Get().  However, you still haven't shown what FElements is, or how it's being managed.

 

Lastly, what is the point of the Dest variable in SQLQuotedStr()?  It is not being used for anything meaningful in this code snippet.  Assuming this code is not the full implementation and Dest is actually written to in code not shown, Dest doesn't appear to point at valid memory, so writing to it will likely crash.  You can't increment a nil pointer, and even so, you are incrementing it only 1 byte at a time, not 1 WideChar at a time.

Edited August 25 by Remy Lebeau

[duzzell 0]

Remy, your comment about RawByteString possibly being redelared made me look for other things I wouldn't have thought of, and in a used file I found this line:

  NativeInt             = Integer;

That would explain why the errors occur ontly for win64.

 

Even though this is the same technique as in the fix I attempted, the whole idea of redeclaring a known type is so bizzare to me that I didn't look to see whether it had already be done elsewhere in the code.

 

Thanks a million, Remy. You make me so glad I posted the question.

 

[dummzeuch 1505]

1 hour ago, duzzell said:

  NativeInt             = Integer;

Even though this is the same technique as in the fix I attempted, the whole idea of redeclaring a known type is so bizzare to me that I didn't look to see whether it had already be done elsewhere in the code.

That code was probably written for a Delphi version that did either not have a NativeInt type (Delphi 5 or earlier) or possibly declared it incorrectly. The latter was the case for Delphi 6 to 2007.

Edited August 26 by dummzeuch

[duzzell 0]

I've been using pre-release versions of the code and different versions of Delphi were handled by deeply nested defines that did not seem quite right to me. I'm using the release version now, the defines work correctly, and I had to undo some of the changes I had made. Also, the new syntax highlighting in Delphi 12 that reflects the current state of  defines was a big help.

[Remy Lebeau 1392]

6 hours ago, dummzeuch said:

The latter was the case for Delphi 6 to 2007.

And even though Native(U)Int existed starting in Delphi 2007, they were buggy up to Delphi 2010.

[Stefan Glienke 2002]

Somehow I cannot believe that ZEOS did not properly support 64bit until now - you must have been using some old code

[dummzeuch 1505]

On 8/26/2024 at 5:23 PM, Remy Lebeau said:

And even though Native(U)Int existed starting in Delphi 2007, they were buggy up to Delphi 2010.

In which way were they buggy? I'm only aware of the wrong declaration of NativeInt as Int64 in Delphi 7 to 2007. These compilers were all 32 bit only, so NativeInt should have been Integer. That was fixed in Delphi 2009.

Edited September 13 by dummzeuch