TJson - Strip TDateTime property where value is 0?

[Lars Fosdal 1792]

Consider this pseudo code

uses
  Rest.Json;

TDateClass = class
private
  FHasDate: TDateTime;
  FNoDate: TDateTIme;
public
  constructor Create;
  property HasDate: TDateTime read FHasDate write FHasDate;
  property NoDate: TDateTime read FNoDate write FNoDate;
end;

constructor TDateClass.Create;
begin
  HasDate := Now;
  NoDate := 0;
end;

var
  Json: String;
  DateClass: TDateClass;
begin
  DateClass := TDateClass.Create;
  Json := TJson.ObjectToJsonString(Self, [joIgnoreEmptyStrings, joIgnoreEmptyArrays, joDateIsUTC, joDateFormatISO8601]);

which results in the Json string looking like

{
              "HasDate":"2019-02-14T06:09:00.000Z",
              "NoDate":"1899-12-30T00:00:00.000Z", 
}

while the ideal result would be

{
              "HasDate":"2019-02-14T06:09:00.000Z",
}

Q:Is there a way to make TDateTime properties with a zero value be output as an empty string - and hence be stripped?

[Attila Kovacs 629]

Kinda. Convert it first with ObjectToJsonObject then remove the time 0 pairs then generate the string.

Or create a json wrapper for TDateClass  where HasDate and NoDate are strings. In this case you have to take care of the Date-Format. 

 

btw. "1899-12-30T00:00:00.000Z" is a valid date. 🙂

Edited March 27, 2019 by Attila Kovacs

[Lars Fosdal 1792]

Thanks! Neither suggestion is "perfect", but that gives me something to work with.

 

I wonder how expensive it would be to do suggestion 1 for every structure that I use?

 

There are supposed to be ways to plug in validators and converters for TJson & family, but the documentation is non-existent, and I don't want to take time to reverse engineer it all.

[Uwe Raabe 2057]

4 minutes ago, Lars Fosdal said:

There are supposed to be ways to plug in validators and converters for TJson & family,

There is a possibility, but it would be difficult to apply that behavior on any TDateTime field to be converted, which could as well be a pretty unwanted side effect. Instead you can put an attribute to each field being treated this way.

 

What you need here are two classes (well, one would do, but the second one simplifies things a lot). The first one is the interceptor:

type
  TSuppressZeroDateInterceptor = class(TJSONInterceptor)
  public
    function StringConverter(Data: TObject; Field: string): string; override;
    procedure StringReverter(Data: TObject; Field: string; Arg: string); override;
  end;

function TSuppressZeroDateInterceptor.StringConverter(Data: TObject; Field: string): string;
var
  ctx: TRTTIContext;
  date: TDateTime;
begin
  date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
  if date = 0 then begin
    result := EmptyStr;
  end
  else begin
    result := DateToISO8601(date, True);
  end;
end;

procedure TSuppressZeroDateInterceptor.StringReverter(Data: TObject; Field, Arg: string);
var
  ctx: TRTTIContext;
  date: TDateTime;
begin
  if Arg.IsEmpty then begin
    date := 0;
  end
  else begin
    date := ISO8601ToDate(Arg, True);
  end;
  ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, date);
end;

The second one is a special attribute:

type
  SuppressZeroAttribute = class(JsonReflectAttribute)
  public
    constructor Create;
  end;

constructor SuppressZeroAttribute.Create;
begin
  inherited Create(ctString, rtString, TSuppressZeroDateInterceptor);
end;

Now you can decorate your class fields like this:

type
  TDateClass = class
  private
    [SuppressZero]
    FHasDate: TDateTime;
    [SuppressZero]
    FNoDate: TDateTime;
  public
    constructor Create;
    property HasDate: TDateTime read FHasDate write FHasDate;
    property NoDate: TDateTime read FNoDate write FNoDate;
  end;

As I mentioned earlier, you can omit the new attribute and use the JsonReflectAttribute directly, but that is a bit cumbersome:

type
  TDateClass = class
  private
    [JsonReflect(ctString, rtString, TSuppressZeroDateInterceptor)]
    FHasDate: TDateTime;
    [JsonReflect(ctString, rtString, TSuppressZeroDateInterceptor)]
    FNoDate: TDateTime;
  public
    constructor Create;
    property HasDate: TDateTime read FHasDate write FHasDate;
    property NoDate: TDateTime read FNoDate write FNoDate;
  end;

 

[Lars Fosdal 1792]

@Uwe RaabeThat is beautiful!  But - where the heck is the documentation for this!?

[Uwe Raabe 2057]

AFAIK there is none.

[Sherlock 663]

17 minutes ago, Lars Fosdal said:

@Uwe RaabeThat is beautiful! 

Decorating almost always is. That is one of the most awesome features of Python, simple use of decorators.

[Lars Fosdal 1792]

At least that confirms that I am not blind. EMBT has fumbled on the doc here.

Too bad as I have I wanted to use a custom converter to take a Json stream that have a mixed type list and convert it to/from a single object in Delphi, but I didn't have the patience and aptitude to dig that deep. 

 {

  "list": [

    5,

    "thing",

    {"object":"type"}

  ]

}

 

It is the Message structure from the Google+Exporter Json
https://docs.google.com/document/d/1gOYJe61sI1GbO9qpFZJtwY3vdsZalsxtPgUnB2cvzAw/edit

[Attila Kovacs 629]

Too bad you can't register a converter for simple types.

[Uwe Raabe 2057]

44 minutes ago, Attila Kovacs said:

Too bad you can't register a converter for simple types.

You can, but only when creating the TJSONMarshal instance yourself. The convenient class methods ObjectToJsonString and ObjectToJsonObject abstract this away, so by using these methods you lose the flexibility.

 

Actually I prefer registering converters per instance instead of globally, so I think in principle it is done right. It definitely lacks documentation and sometimes I wish they had avoided private methods in favor of protected virtual ones.

[RussellW 3]

Slightly off topic. Using a Date of 0 is fine in certain circumstances but sometimes you might need that actual date rather than meaning that no date is set. Developer Express have a NullDate constant which is -700000 ... very useful

[Bill Meyer 337]

1 minute ago, RussellW said:

Slightly off topic. Using a Date of 0 is fine in certain circumstances but sometimes you might need that actual date rather than meaning that no date is set. Developer Express have a NullDate constant which is -700000 ... very useful

A completely unrealistic negative number which is easily recognized -- would  be the sort of flag I would use. Although, in my work, all meaningful dates will be later than 1950, so not really a problem.

[Uwe Raabe 2057]

15 minutes ago, RussellW said:

Developer Express have a NullDate constant which is -700000

There is a similar constant in Vcl.WinXCalendars.pas, albeit only private.

 

I recently had a case where that 0 DateTime value was used instead of NULL in a database. As long as you are keen to update all your customers databases, you better stay with that value and its meaning.

[Lars Fosdal 1792]

Thanks guys, Good observations.

The dates in question are set to whatever FireDAC returns from a SQL Server datetime field, which typically appears to be 0 for NULL.

The set date range is always in the future, relatively speaking. If the date is less than today, it is an expired dairy product by definition. We don't sell those 😉 

 

I'll change the requirement for blank dates to be <= 1.0 to be on the safe side.

[stijnsanders 35]

In case anyone would be interested. Because when working with Variants with ADO, and because I really dislike long lists of overloads, I've created my own JSON handler fully based on Variants (and arrays of Variants, and Variants holding IUnknown derivatives...):

https://github.com/stijnsanders/jsonDoc#jsondoc

And Variant has a Null value as well, which could solve this problem. (And also a varDate VarType, but I just notice now that my jsonDoc converts from varDate to JSON, but not correctly back, hmm.. but like Attila Kovacs says above, it's open source so you could just add a type handler any way you like)

[Uwe Raabe 2057]

3 hours ago, Lars Fosdal said:

The dates in question are set to whatever FireDAC returns from a SQL Server datetime field, which typically appears to be 0 for NULL.

Indeed, when you ask for the date you will get 0 even when the DB field is NULL:

function TDateTimeField.GetAsDateTime: TDateTime;
begin
  if not GetValue(Result) then Result := 0;
end;

Unfortunately you cannot distinguish that from a valid date (unless you narrow the range for those). I would prefer when the above function would actually return a non-valid date in that case, but I guess that would break compatibility.

 

Meanwhile, the trick is not to ask for a date in the first place when the field is NULL. To store such an invalid date into a TDateTime field the NullDate approach is quite valid. Actually any value below -DateDelta ( = 693594) would do, as it results in a non-positive Date part of a TTimeStamp and makes DecodeDateFully return false. The chosen value of NullDate = -700000 ist just a bit easier to remember.

[Lars Fosdal 1792]

I wonder when we will get nullable types. That will be awesome for DB related code.

[Attila Kovacs 629]

On 3/27/2019 at 4:54 PM, Uwe Raabe said:

You can, but only when creating the TJSONMarshal instance yourself.

Barely passed 6 months and I'm sitting here with dates like "2019-10-27T22:48:20+0100" from a php API, which are ISO8601 dates, but Delphi can't handle it because of the missing colon in the time zone offset......

I've tried everything I can think of to hook the conversion to TDateTime, globally (not on field basis) but I failed.

The corresponding methods are not virtual, Reverter can be regged only on classes, interceptors only on field basis.

Maybe it's too late and I'm missing something, @Uwe Raabe, how can I register a reverter for TDateTime?

(Creating the marshal objects is not a problem at all, I'm already doing it for other reasons.)

 

 

[Uwe Raabe 2057]

Can you provide a minimal example showing the problem?

[Lars Fosdal 1792]

@Attilla, from the theorist side
if a time zome colon is mandatory in a ISO8601 date, then the problem is really that php API, isn't it?
While if a colon is optional, the Delphi implementation of the conversion seems lacking, hence a QP would be a starting point.

From the more practical side

- Is using class based Json conversion out of the question?

- Would pre-processing be an option to correct for the format deviation, f.x. by reg.exp searching for the dddd-dd-ddTdd:dd:dd+dddd and injecting the missing colon

[Attila Kovacs 629]

@Lars Fosdal Thx for the input, I wanted to create something generic with the less effort, I'll go with Uwe's example above and decorate the fields for this PHP API for now. Strange, that I'm the only one who facing this, but the DecodeISO8601Time() conversion in System.DateUtils is not ISO8601.

 

[Uwe Raabe 2057]

The colon in the time zone offset seems to be optional, while the Delphi implementation requires it. I suggest creating a QP entry with a test case showing the problem. A failing test case often increases the probability for a quick fix.

[Lars Fosdal 1792]

 

 

 

 

1 hour ago, Attila Kovacs said:

@Lars Fosdal Thx for the input, I wanted to create something generic with the less effort, I'll go with Uwe's example above and decorate the fields for this PHP API for now. Strange, that I'm the only one who facing this, but the DecodeISO8601Time() conversion in System.DateUtils is not ISO8601.

 

I guess we've avoided the problem since we only have used Zulu timestamps.

              "departureDate":"2019-10-28T06:00:00.000Z",
 

The problem is at the bottom of procedure DecodeISO8601Time in  System.DateUtils-

That is where the assumption is made that there will always be a time separator if there is a minutes section

 

      AHourOffset  := StrToInt(LOffsetSign + GetNextDTComp(P, PE, InvOffset, TimeString, 2));
      AMinuteOffset:= StrToInt(LOffsetSign + GetNextDTComp(P, PE, '00', STimeSeparator, True, True, InvOffset, TimeString, 2));

 

Please make a QP.