Another option might be to use tagged pointers.
Normally, objects are aligned in memory in such a way that certain bits in object pointers are always zero, so those bits can be repurposed if you are careful.
For instance, if you limit your integer values to 31 bits (x86) or 63 bits (x64), you can use an unused bit in an object pointer to flag whether the pointer holds an integer value vs an object address, and just mask off the bit when extracting the value.
For example:
// this assumes objects are never stored at an odd-numbered memory address..
var intValue: Integer := ...;
Objects[c,r] := TObject((NativeUInt(intValue) shl 1) or $1);
...
var objValue: TObject := ...;
Objects[c,r] := objValue;
...
if (NativeUInt(Objects[c,r]) and $1) <> 0 then
begin
// is an integer...
var intValue := Integer(NativeUInt(Objects[c,r]) shr 1);
...
end
else
begin
// is an object...
var objValue := Objects[c,r];
...
end;
...
if (NativeUInt(Objects[c,r]) and $1) = 0 then
Objects[c,r].Free;