How to iterate a TDictionary using RTTI and TValue

[John Kouraklis 94]

Hi,

I've got a class and it has a property TDictionary<..,..>.

 

I want to use RTTI to access the key and value of the dictionary.

 

I am able to get the property but I can't figure out how to access the key and values.

 

I use this approach:

var
 genericDictionary: TDictionary<TKey, TValue>;
 propValue: TValue;
 ...
   propvalue.ExtractRawDataNoCopy(@genericDictionary);
   ---> Here I can see in the debugger that genericDictionary has the items and the correct count
   

Then, I get AV when I try to iterate the Keys because TKey is an abstract class.

 

Anyone can help?

 

Thanks

[Remy Lebeau 1436]

I imagine trying to access the content of a TDictionary via RTTI will be very difficult, if not down-right dangerous due to the extra work needed to deal with managed types and such.  You would be better off simply retrieving the TDictionary object pointer from the property (ie, using "propvalue.AsType<TDictionary<TKey, TValue>>" or "propvalue.AsObject as TDictionary<TKey, TValue>") and then access its content via normal TDictionary methods, not via RTTI at all.

Edited March 17, 2020 by Remy Lebeau

[John Kouraklis 94]

Hi Remy,

I tried that (.AsType....) before as well; I run to the same problem of using TKey which is an abstract class although I can see in the debugger that the object points to the content of the dictionary.]

 

 

[Remy Lebeau 1436]

15 minutes ago, John Kouraklis said:

I tried that (.AsType....) before as well; I run to the same problem of using TKey which is an abstract class although I can see in the debugger that the object points to the content of the dictionary.]

Then please show your actual code.  It sounds like you are trying to instantiate TKey objects, which you should not be doing at all.  You don't need to create a TKey object in order to enumerate a TDictionary.  Use the TDictionary.Keys and TDictionary.Values properties, or the TDictionary.GetEnumerator() method.

[John Kouraklis 94]

I found out that TKey in my code is declared (for unexplained reasons) in Data.DBXCommon with is irrelevant. Maybe that is the cause of the AVs I get.

 

But now I don't seem to be able to declare TKey as data type although I include System.Generics.Collections. How do I declare it in a unit? It is originally declared using $HPPEMIT (Generics.Collections, line 750)

[John Kouraklis 94]

Ok, now the more important problem as stated in the beginning. In this code, genDict throws an exception although propValue is correct

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  System.Generics.Collections,
  System.Rtti;

type
  TObjDic = TObjectDictionary<string, string>;

  TMClass = class
  private
    fMC: TObjDic;
  public
    constructor Create;
    destructor Destroy; override;
    
    property MC: TObjDic read fMC write fMC;
  end;

constructor TMClass.Create;
begin
  inherited;
  fMC:=TObjDic.create;
end;

destructor TMClass.Destroy;
begin
  fMC.Free;
  inherited;
end;

var
  mc: TMClass;
  ctx: TRttiContext;
  rType: TRttiType;
  rProp: TRttiProperty;
  propValue: TValue;
  genDict: TObjectDictionary<string, TValue>;

begin
  mc:=TMClass.Create;
  rType := ctx.GetType(mc.ClassInfo);
  rProp:=rType.GetProperty('MC');
  // In this code, this may seem redundant but I use this code is part of
  // bigger codebase
  propValue:=rProp.GetValue(mc);
  genDict:=nil;
  genDict:=propvalue.AsObject as TObjectDictionary<string, TValue>;
  if Assigned(genDict) then
  begin
    for var str in genDict.Keys do
    begin
      // For demo
      var f:=str;
    end;
    genDict.Free;
  end;
  mc.Free;
end.

 

[Remy Lebeau 1436]

17 hours ago, John Kouraklis said:

Ok, now the more important problem as stated in the beginning. In this code, genDict throws an exception although propValue is correct

You have a mismatch in your code:

Quote

type
  TObjDic = TObjectDictionary<string, string>;

...
    
var
  ...
  genDict: TObjectDictionary<string, TValue>;

begin
  ...
  genDict:=propvalue.AsObject as TObjectDictionary<string, TValue>;
  ...

 

Do you see it?  String <> TValue.  You need to fix your declaration of the genDict variable to match the TObjDic type.  In fact, why are you not using TObjDict itself in the declaration of genDict?

type
  TObjDic = TObjectDictionary<string, string>;

...
    
var
  ...
  genDict: TObjDic;

begin
  ...
  genDict := propvalue.AsObject as TObjDic;
  ...

 

Edited March 18, 2020 by Remy Lebeau

[John Kouraklis 94]

The whole code is part of a serialiser, so the dictionary type is unknown to the RTTI code.

 

In the code above, I replaced TKey with string because I couldn't find the declaration of TKey.

[Remy Lebeau 1436]

4 hours ago, John Kouraklis said:

The whole code is part of a serialiser, so the dictionary type is unknown to the RTTI code.

Generics are not easy to serialize manually because of the nature of their dynamic types.  You will likely have to resort to something like the following (untested, but should give you an idea of what is involved):

var
  mc: TMClass;
  ctx: TRttiContext;
  rType: TRttiType;
  rProp: TRttiProperty;
  rMethod: TRttiMethod;
  rKeyField, rValueField: TRttiField;
  propValue, methValue, genKey, genValue: TValue;
  genDict, genDictEnum: TObject;
  genPair: Pointer;
begin
  mc := TMClass.Create;
  try
    {
    if Assigned(mc.MC) then
    begin
      for var pair in mc.MC do
      begin
        // use pair.Key and pair.Value as needed ...
      end;
    end;

    which is actually this behind the scenes ...

    if Assigned(mc.MC) then
    begin
      var enum = mc.MC.GetEnumerator;
      while enum.MoveNext do
      begin
        var pair = enum.Current;
        // use pair.Key and pair.Value as needed ...
      end;
    end;
    }

    rType := ctx.GetType(mc.ClassInfo); // rType = TMClass
    rProp := rType.GetProperty('MC');
    propValue := rProp.GetValue(mc);
    genDict := propValue.AsObject;

    if Assigned(genDict) then
    begin
      rType := rProp.PropertyType; // rType = TObjDict
      rMethod := rType.GetMethod('GetEnumerator');
      methValue := rMethod.Invoke(genDict, []);
      genDictEnum := methValue.AsObject;

      rType := rMethod.ReturnType; // rType = TDictionary<TKey, TValue>.TPairEnumerator 
      rMethod := rType.GetMethod('MoveNext');
      rProp := rType.GetProperty('Current');

      rType := rProp.PropertyType; // rType = TPair<TKey, TValue>
      rKeyField := rType.GetField('Key');
      rValueField := rType.GetField('Value');
    
      methValue := rMethod.Invoke(genDictEnum, []);
      while methValue.AsBoolean do
      begin
        propValue := rProp.GetValue(genDictEnum);
        genPair := propValue.GetReferenceToRawData;
        genKey := rKeyField.GetValue(genPair); // genKey.TypeInfo = TypeInfo(string)
        genValue := rValueField.GetValue(genPair); // genValue.TypeInfo = TypeInfo(string)
        // use genKey and genValue as needed ...
        methValue := rMethod.Invoke(genDictEnum, []);
      end;
    end;
  finally
    mc.Free;
  end;
end.

Quote

In the code above, I replaced TKey with string because I couldn't find the declaration of TKey.

Because it doesn't exist.  For a dictionary, TKey and TValue are Generic parameters of the TDictionary class.  They are not standalone concrete types, like you are thinking of.

Edited March 18, 2020 by Remy Lebeau

[John Kouraklis 94]

@Remy Lebeau Wow....Amazing and complicated stuff. So I really need to drill down to the enumerator to get hold of the pairs.

 

Thanks a lot for this!!!

 

You mentioned that it might be unreliable to do this with generics. Do you think the app may crash for more complex dictionaries?

 

22 minutes ago, Remy Lebeau said:

Quote

In the code above, I replaced TKey with string because I couldn't find the declaration of TKey.

Because it doesn't exist.  For a dictionary, TKey and TValue are Generic parameters of the TDictionary class.  They are not standalone concrete types, like you are thinking of.

And out of curiosity, how does the compiler recognise TKey then? It must be from the HPPEMIT directive, right?

[Remy Lebeau 1436]

2 hours ago, John Kouraklis said:

@Remy Lebeau Wow....Amazing and complicated stuff. So I really need to drill down to the enumerator to get hold of the pairs.

Yes, if you can't specify the actual TKey/TValue Generic parameters up front at compile-time when defining how you access the dictionary object from the Sytem.Rtti.TValue record.  In which case, you can't use a standard "for..in" loop to enumerate the dictionary.  All the necessary type information is lost once you access the dictionary object via the System.Rtti.TValue record if you can't type-cast the object.  So yes, this is the only way I can think to enumerate the dictionary object using only RTTI and not caring what the actual Generic parameters are.  I mean, if you really needed the RTTI for those types, you could take the dictionary object's ClassName() string, parse the typenames between the angle brackets, and resolve them using TRttiContext.FindType(), but that won't help you to enumerate the dictionary at all since you still need access to its enumerators (either via its GetEnumerator() method or its Keys+Values properties).

Quote

You mentioned that it might be unreliable to do this with generics. Do you think the app may crash for more complex dictionaries?

I really couldn't say.  Probably not, if you use the RTTI correctly and handle the enumeration correctly.  Hopefully the RTTI and Sytem.Rtti.TValue record will handle the complexities for you.

Quote

And out of curiosity, how does the compiler recognise TKey then?

It doesn't, outside of the implementation code for the TDictionary class, where the TKey and TValue Generic parameters have meaning.  Outside of the class, they don't exist.  They are part of the class type itself, for instance TDictionary<string, int> and TDictionary<int, char> are separate and distinct types with no relation to each other.

Quote

It must be from the HPPEMIT directive, right?

No.  {$HPPEMIT} has no effect whatsoever on how the Delphi compiler processes code.  {$HPPEMIT} merely outputs an arbitrary line of text to a C++ .hpp header file, if one is being generated while the Delphi compiler is parsing the code.

 

Curious - what version of Delphi are you using? I have RTL sources up to XE3, and there are no {$HPPEMIT} statements at all in System.Generics.Collections.pas in those versions, so it must be something added in more recent versions.

Edited March 19, 2020 by Remy Lebeau

[John Kouraklis 94]

Remy, thanks for the explanation and for sharing your knowledge 👍. I admit that even if I had gone all the way down to retrieve the enumerators I would never have thought to Invoke them.

 

I use 10.3.3 and the first instance of TKey in System.Generics.Collections is this:

------> This is line 748
  (*$HPPEMIT END OPENNAMESPACE*)
  (*$HPPEMIT END '  template<typename TKey, typename TValue>'*)
  (*$HPPEMIT END '  inline __fastcall TPair__2<TKey, TValue>::TPair__2(const TKey AKey, const TValue AValue) :'*)
  (*$HPPEMIT END '                                            Key(AKey), Value(AValue)'*)
  (*$HPPEMIT END '  {}'*)
  (*$HPPEMIT END CLOSENAMESPACE*)
  [HPPGEN(HPPGenAttribute.mkNonPackage)]
  TPair<TKey,TValue> = record
    Key: TKey;
    Value: TValue;
    constructor Create(const AKey: TKey; const AValue: TValue);
  end;

  // Hash table using linear probing
  TDictionary<TKey,TValue> = class(TEnumerable<TPair<TKey,TValue>>)
  private type
    TItem = record
      HashCode: Integer;
      Key: TKey;
      Value: TValue;
    end;
    TItemArray = array of TItem;

 

[Remy Lebeau 1436]

In that TPair record, TKey and TValue are Generic parameters, they belong to the TPair type itself.  In C++, that TPair record type gets renamed to TPair__2 because of the Generic parameters.  When the Delphi compiler generates a C++ .hpp file, it will declare a constructor inside that type.  But since the type is renamed for C++, the HPPEMITs are being used to provide an implementation for that constructor using the new name.

 

I have never seen the [HPPGEN] attribute before, though.  It is not documented.

Edited March 19, 2020 by Remy Lebeau

[John Kouraklis 94]

Thanks Remy--Always good to learn

[Andrea Magni 75]

Hi @John Kouraklis and @Remy Lebeau, thanks for this topic that helped me too.

Just wanted to add I think we should free the enumerator object (returned by TDictionary<>.GetEnumerator).

I've used this approach in MARS REST library, as you can see here:

https://github.com/andrea-magni/MARS/blob/41fd78e5416e38fad0f6a0451402a10824aa62c4/Source/MARS.YAML.ReadersAndWriters.pas#L222

 

Thanks again you both.

 

Andrea

[Remy Lebeau 1436]

9 hours ago, Andrea Magni said:

Just wanted to add I think we should free the enumerator object (returned by TDictionary<>.GetEnumerator).

Asked and answered on StackOverflow:

Do I need to free the enumerator returned by GetEnumerator?

 

Edited November 2, 2021 by Remy Lebeau

[darnocian 87]

I wish RTTI embedded the generic type parameters and the TRttiType included a way to extract  them.

 

I've often had to rely on specific use cases and knowing that I can extract the types easily from known method return types as highlighted by the enumerator / pair example above...

 

I've had times where I needed some really generic stuff and had to split the TWhatever<T> as a string and resolve the generic parameter however it is defined recursively -  which is a bit annoying.

[Andrea Magni 75]

14 hours ago, Remy Lebeau said:

Asked and answered on StackOverflow:

Do I need to free the enumerator returned by GetEnumerator?

 

 

I saw that too. I meant in the code above a genDictEnum.Free is missing somewhere before mc.Free call.

 

Thanks