That is a very dangerous approach. Don't do it that way. First, there is no guarantee that accessing the ClassName on an invalid object will raise an exception. And second, reading from invalid memory may cause other side effects (ie, page faults, etc).
Since your integers are very small, then you could simply look for integers first, and treat higher values as objects since they should never reside at such low memory addresses, eg:
const
MaxObjIntValue = 8;
...
var value := NativeInt(Objects[c,r]);
if (value >= 0) and (value <= MaxObjIntValue) then
begin
// is an integer, use value as needed...
end
else
begin
// is an object, use TObject(value) as needed...
end
...
if NativeInt(Objects[c,r]) > MaxObjIntValue then
Objects[c,r].Free;