Transform string into TDateTime

[Alberto Paganini 3]

I would like to replace JsonStringToDateTime with JsonStringToDateTime2 in my code because it is significantly faster.

Here below a small example

program MyTest;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  madExcept,
  madLinkDisAsm,
  madListHardware,
  madListProcesses,
  madListModules,
  System.SysUtils,
  Soap.XSBuiltIns;

const
  gNullDate: TDateTime = 2 + 365;


{*****************************************************************************************}
function JsonStringToDateTime(S: widestring): TDateTime;
{*****************************************************************************************}
begin
  result := gNullDate;

  with TXSDateTime.Create() do
  begin
    try
      XSToNative(S);
      result := AsDateTime;
    finally
      Free;
    end;
  end;
end;
{*****************************************************************************************}

function JsonStringToDateTime2(S: widestring): TDateTime;
{*****************************************************************************************}
var
  Dummy: string;
begin
  try
    Result := EncodeDate(StrToInt(Copy(S, 1, 4)), StrToInt(Copy(S, 6, 2)), StrToInt(Copy(S, 9, 2)));
    Result := EncodeTime(StrToInt(Copy(S, 12, 2)), StrToInt(Copy(S, 15, 2)), StrToInt(Copy(S, 18, 2)),
      0);
  except
    result := gNullDate;
  end;
end;

var
  TimeResult: TDateTime;
  a: TDateTime;
  b: TDateTime;
  I: Integer;
  hh, mm, ss, ms: Word;

begin
  a := Now;
  for i := 0 to 5000 do
  begin
    TimeResult := JsonStringToDateTime('2021-02-11T02:25:00.000Z');
  end;
  b := Now - a;
  DecodeTime(b, Hh, mm, ss, ms);
  Writeln('JsonStringToDateTime sec:' + IntToStr(ss) + ' msec:+' + IntToStr(ms));

  a := Now;
  for i := 0 to 5000 do
  begin
    TimeResult := JsonStringToDateTime2('2021-02-11T02:25:00.000Z');
  end;
  b := Now - a;
  DecodeTime(b, Hh, mm, ss, ms);
  Writeln('JsonStringToDateTime2 sec:' + IntToStr(ss) + ' msec:+' + IntToStr(ms));

  Readln;
end.

 

Can you see any drawbacks in JsonStringToDateTime2 ?

 

Many thanks

Alberto

Edited February 10, 2021 by Alberto Paganini

[Attila Kovacs 629]

Yes, remove "Dummy", and maybe Result := Result + Encodetime()?

Also, you are ignoring time zones, but it is not necessary a problem for you.

 

 

Edited February 10, 2021 by Attila Kovacs

[Alberto Paganini 3]

Quote

Yes, remove "Dummy",

Yes, sorry I forgot to remove it.

 

Quote

and maybe Result := Result + Encodetime()?

Any particular reason for that? I am not trying to be funny here. I just want to understand why this is better.

 

Quote

Also, you are ignoring time zones, but it is not necessary a problem 

Yes, not a problem in my case.

 

 

[Henk Post 3]

3 hours ago, Alberto Paganini said:

Any particular reason for that? I am not trying to be funny here. I just want to understand why this is better

Otherwise you neglect the date part calculated in the previous line.

[Guest]

7 hours ago, Alberto Paganini said:

const   gNullDate: TDateTime = 2 + 365;

One doubt here:

• What makes this a "null date"?

 

in my RAD 10.3.3 VCL test, I have a valid value to datetime!

uses
  System.DateUtils;

const
  lMyNullDateTime: TDateTime = (2 + 365);

procedure TForm1.Button1Click(Sender: TObject);
begin
  Memo1.Lines.Clear;
  Memo1.Lines.Add(System.DateUtils.DateTimeToMilliseconds(lMyNullDateTime).ToString);
  Memo1.Lines.Add(FormatDateTime('dd/mm/yyyy hh:mm:ss:zzz', lMyNullDateTime));
end;
---------------------------
1) 59958230400000
2) 01/01/1901 00:00:00:000
------------------------------------------

const
  lMyNullDateTime: TDateTime = 0;
...
59926521600000
30/12/1899 00:00:00:000

What is the point here?

 

Edited February 11, 2021 by Guest

[Guest]

Will be that exist a var type "TDateTime" really "NULL" ?

In Object Pascal (Delphi) I think that not!

 

uses
  System.DateUtils;

{ const } var
  lMyNullDateTime: TDateTime; // = (0);

procedure TForm1.Button1Click(Sender: TObject);
begin
  { NullStrictConvert:
    Specifies whether Null conversion is an error.
    -----------------------------------------------
    NullStrictConvert determines the outcome of attempts to convert Null variants to other types.
    If NullStrictConvert is true (default), attempting to convert a Null variant raises a EVariantTypeCastError,
    unless the conversion is to a custom variant that defines a conversion from Null. If NullStrictConvert is false,
    then conversion from Null follows the following rules:
  }
  // to avoid a exception here!
  System.Variants.NullStrictConvert := false; // True =  EVariantTypeCastError
  //
  // at general, it's acceptable, but generate a "exception" because it's expected a "Double" value! not a Null state!
  lMyNullDateTime := null;
  //
  Memo1.Lines.Clear;
  Memo1.Lines.Add(System.DateUtils.DateTimeToMilliseconds(lMyNullDateTime).ToString);
  Memo1.Lines.Add(FormatDateTime('dd/mm/yyyy hh:mm:ss:zzz', lMyNullDateTime));
  //
  Memo1.Lines.Add('VarType(lMyNullDateTime) = ' + VarType(lMyNullDateTime).ToString); // 7 = TDateTime
end;

59926521600000
30/12/1899 00:00:00:000
VarType(lMyNullDateTime) = 7

hug

Edited February 11, 2021 by Guest

[Fr0sT.Brutal 900]

12 hours ago, Alberto Paganini said:

Can you see any drawbacks in JsonStringToDateTime2 ?

Sure. SIX Copy calls. Six IntToStr calls. Argument is not "const".

[haentschman 92]

[OT]

Imho is the reason the "with"... With ist like 90's.

[/OT]

[Attila Kovacs 629]

@emailx45 Why are you concerned about what value he uses for zero date? There must be a reason for that, for example the REST endpoint he talking to does not accept dates before 1.1.1901. I don't understand your rage nor your comic book.

[Guest]

@Attila Kovacs  dont worry about me.

follow your way! it doesnt matter your opnion about my posts, as you know!

read more about "comic books" and maybe you create a Disney World, DC Comics, and others - this give much money, did you know?

 

hug

Edited February 11, 2021 by Guest

[Alberto Paganini 3]

 

@emailx45

 

Quote

 

One doubt here:

• What makes this a "null date"?

 

That is technically correct, there is no "null date" in Delphi.

The software I am working on should receive dates >01/01/1901 only. If something goes wrong the procedure assigns 01/01/1901 to Result. Somewhere else the application checks Result and if it is is 01/01/1901 treats this accordingly.

I should have renamed it something like gOutOfScopeDate

 

[Alberto Paganini 3]

8 hours ago, Fr0sT.Brutal said:

Sure. SIX Copy calls. Six IntToStr calls. Argument is not "const".

I was not able to think of anything better in order to transform parts of a string into numbers and then put everything into a TDateTime.

Is there an alternative to that, considering that performance is important here?

[Guest]

Quote

I should have renamed it something like gOutOfScopeDate

do it what is better for you!

Quote

Is there an alternative to that, considering that performance is important here?

You dont need create new functions before search on your IDE resources!

 

• here, im using RAD Studio 10.3.3 Arch (Rio) and System.DateUtils.pas unit for DateTime support!

 

implementation

{$R *.dfm}

uses
  System.DateUtils; { for test DateTime info/convertions etc... }

var
  lMySF: TFormatSettings;
  //
  lMyDateTimeInString: string = '31/12/2021 16:32:10.650'; // datetime ok
  // lMyDateTimeInString: string = '32/13/2021 16:32:10.650'; // datetime wrong! for test!
  //
  lMyDateTimeDefaultOnErrors: TDateTime = 0; // for your "NULL" datetime use!!!

function fncJSONStringToDateTime(lJSONString: string): TDateTime;
var
  lMyDateTimeConverted: TDateTime;
begin
  //
  result := 0; // or NULL using define "before" ... System.Variants.NullStrictConvert := false; // True =  EVariantTypeCastError
  try
    // for example: one or other!!!
    //
    // lMyDateTimeConverted := StrToDateTime(lMyDateTimeInString, lMySF); { SysUtils } // here, we can have a "AV"
    //
    ShowMessage(lMySF.LongTimeFormat);
    //
    lMyDateTimeConverted := StrToDateTimeDef(lMyDateTimeInString, lMyDateTimeDefaultOnErrors, lMySF); { SysUtils } // DONT cause "AV"!
    //
    result := lMyDateTimeConverted;
  except // unnecessary if using "StrToDateTimeDef(...)"
    on E: Exception do
    begin
      // if necessary, show the message or re-raise the exception!!! you should decide what do it here...
      // ... or, for example, "EXIT" and get out from this procedure... I dont know what you prefere or need!
    end;
  end;
  //
  // if above, dont cause any error, then, this is not necessary here!
  //
  // some System.DateUtils functions: more on HELP SYSTEM
  //
  // all ready for you in this unit:
  // ---------------------------------------
  // System.DateUtils.EncodeDateTime(const AYear, AMonth, ADay, AHour, AMinute, ASecond, AMilliSecond: Word): TDateTime
  //
  // System.DateUtils.DecodeDateTime(const AValue: TDateTime; out AYear, AMonth, ADay, AHour, AMinute, ASecond, AMilliSecond: Word
  //
  // if System.DateUtils.IsValidDateTime(const AYear, AMonth, ADay, AHour, AMinute, ASecond, AMilliSecond: Word) then ....;
  // etc..
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  Memo1.Lines.Clear;
  //
  Memo1.Lines.Add(                                      { }
    DateTimeToStr(                                      { dont show "milliseconds" = .zzz - then use FormatDateTime() for example }
    fncJSONStringToDateTime(lMyDateTimeInString), lMySF { }
    )                                                   { }
    );
  //
  Memo1.Lines.Add(                                      { }
    FormatDateTime('dd/mm/yyyy hh:mm:ss.zzz',           { }
    fncJSONStringToDateTime(lMyDateTimeInString), lMySF { }
    )                                                   { }
    );
  //
  Memo1.Lines.Add(                               { }
    System.DateUtils.MilliSecondOf(              { }
    fncJSONStringToDateTime(lMyDateTimeInString) { }
    ).ToString                                   { }
    );
end;

initialization

lMySF                  := TFormatSettings.Create('pt-BR'); // this not use "FREE" to release memory!
lMySF.LongTimeFormat   := 'hh:mm:ss.zzz'; // <---- "." = DecimalSeparator for Brazil
lMySF.DecimalSeparator := '.';            // conform your Regional Setting on system!

finalization

end.

hug

Edited February 12, 2021 by Guest

[Guest]

About use of "WITH", really, it's not good pratice as general rule!

• Only in specific case, you can use it, for sure!
  • In all rest, dont use! Mainly, "with" into "with"!
• Then, your code is more readable for you or anyone!

[Fr0sT.Brutal 900]

15 hours ago, Alberto Paganini said:

That is technically correct, there is no "null date" in Delphi.

The software I am working on should receive dates >01/01/1901 only. If something goes wrong the procedure assigns 01/01/1901 to Result. Somewhere else the application checks Result and if it is is 01/01/1901 treats this accordingly.

Why not just use TDateTime(0)?

15 hours ago, Alberto Paganini said:

I was not able to think of anything better in order to transform parts of a string into numbers and then put everything into a TDateTime.

Is there an alternative to that, considering that performance is important here?

Well, first you should check and ensure that date parsing is the reason of slowdown in your workflow. There won't be much sense in extreme optimization of this routine if you have only 1 date value in 1Mb file.

Then, avoiding string copy is the must-do when reaching the best performance. In your case for such short and fixed-length numbers there's no necessity in Copy

  SetLength(s, 2);
  s[1] := src[1];
  s[2] := src[2];
  IntToStr(s);
  s[1] := src[3];
  s[2] := src[4];
  IntToStr(s);
  ...

Going deeper,

var s: String;
    pSrc, pDest: PChar;
begin
  SetLength(s, 2);
  pSrc := Pointer(src);
  pDest := Pointer(s);
  pDest^ := pSrc^;
  (pDest+1)^ := (pSrc+1)^;

gives 2280% (!) perf gain over Copy when running in a loop (not an strictly clean test though because I allocated a string once and looped 30k times, but you have 5 2-char parts so reusing still makes sense).

 

Further, instead of filling string + IntToStr, you could just do

`DatePart := CharToDigit(Str[1])*10 + CharToDigit(Str[2])`

where CharToDigit is *inline* function that does

`Result := Ord(aChar) - Ord('0')`

Of course, some validations should be added but that's the idea.

[Guest]

3 hours ago, Fr0sT.Brutal said:

Of course, some validations should be added but that's the idea

a simple "StrToDateTime() or StrToDateTimeDEF( <<secure way for result a DateTime anyway>> )" from Delphi is enought for any test!

if necessary, as support, use "System.DateUtils" unit on "uses" and all procedures/functions all ready for use:

Quote

...
  System.DateUtils.DayOf(<<TDateTime>>) = word
  System.DateUtils.MonthOf(<<TDateTime>>) = word
  System.DateUtils.YearOf(<<TDateTime>>) = word
  System.DateUtils.HourOf(<<TDateTime>>) = word
  System.DateUtils.MinuteOf(<<TDateTime>>) = word
  System.DateUtils.SecondOf(<<TDateTime>>) = word
  System.DateUtils.MilliSecondOf(<<TDateTime>>) = word

  ... between many other - all ready for use!

 

hug

[Fr0sT.Brutal 900]

1 hour ago, emailx45 said:

a simple "StrToDateTime() or StrToDateTimeDEF( <<secure way for result a DateTime anyway>> )" from Delphi is enought for any test!

...but not an option if performance is important

[Guest]

hilarius!!! the Embarcadero should know about this "performanceeeeeeeee" 

[Alberto Paganini 3]

Quote

Well, first you should check and ensure that date parsing is the reason of slowdown in your workflow. 

One important part of the application I am working on is to retrieve data from a service provider as often as possible.

The function JsonStringToDateTime is called tens of thousands of times from several processes in a few minutes in order to parse the data provided therefore, the faster the function the more times the application can retrieve data.

This is one of the most "popular" functions but there are others similar and called as much as this one and I will look into these too.

 

I understand this is just a quick fix, the next step would be to refactor the logic of the application in order to retrieve data and parse it in a better way (maybe parse only the necessary data) but that would require more time and I would like to see some results in the short run.

However, for the records here the three versions with the usual quick test. 

program SoapTest;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  madExcept,
  madLinkDisAsm,
  madListHardware,
  madListProcesses,
  madListModules,
  System.SysUtils,
  Soap.XSBuiltIns;

const
  gOutOfScopelDate: TDateTime = 2 + 365;


{*****************************************************************************************}
function JsonStringToDateTime(S: widestring): TDateTime;
{*****************************************************************************************}
begin
  result := gOutOfScopelDate;

  with TXSDateTime.Create() do
  begin
    try
      XSToNative(S);
      result := AsDateTime;
    finally
      Free;
    end;
  end;
end;
{*****************************************************************************************}

function JsonStringToDateTime2(S: widestring): TDateTime;
{*****************************************************************************************}
begin
  try
    Result := EncodeDate(StrToInt(Copy(S, 1, 4)), StrToInt(Copy(S, 6, 2)), StrToInt(Copy(S, 9, 2)));
    Result := Result + EncodeTime(StrToInt(Copy(S, 12, 2)), StrToInt(Copy(S, 15, 2)), StrToInt(Copy(S,
      18, 2)), 0);
  except
    result := gOutOfScopelDate;
  end;
end;


{*****************************************************************************************}
function JsonStringToDateTime3(Src: widestring): TDateTime;
{*****************************************************************************************}

  function CharToDigit(aChar: Char): Integer; inline;
  begin
    Result := Ord(aChar) - Ord('0');
  end;

var
  S: string;
  pSrc, pDest: PChar;
begin
  SetLength(S, 8);
  pSrc := Pointer(Src);
  pDest := Pointer(S);

  try
  //date
    pDest^ := pSrc^;
    (pDest + 1)^ := (pSrc + 1)^;
    (pDest + 2)^ := (pSrc + 2)^;
    (pDest + 3)^ := (pSrc + 3)^;
    (pDest + 4)^ := (pSrc + 5)^;
    (pDest + 5)^ := (pSrc + 6)^;
    (pDest + 6)^ := (pSrc + 8)^;
    (pDest + 7)^ := (pSrc + 9)^;
  {(*}
  Result := EncodeDate(      // year
    CharToDigit(S[1]) * 1000 + CharToDigit(S[2]) * 100 + CharToDigit(S[3]) * 10 +CharToDigit(S[4]),
    CharToDigit(S[5]) * 10 +CharToDigit(S[6]), //month
    CharToDigit(S[7]) * 10 +CharToDigit(S[8])); //date
  {*)}
  //time
    pDest^ := (pSrc + 11)^;
    (pDest + 1)^ := (pSrc + 12)^;
    (pDest + 2)^ := (pSrc + 14)^;
    (pDest + 3)^ := (pSrc + 15)^;
    (pDest + 4)^ := (pSrc + 17)^;
    (pDest + 5)^ := (pSrc + 18)^;
 {(*}
  Result:=Result+EncodeTime(
    CharToDigit(S[1]) * 10 +CharToDigit(S[2]),  //hh
    CharToDigit(S[3]) * 10 +CharToDigit(S[4]),  //mm
    CharToDigit(S[5]) * 10 +CharToDigit(S[6]), //ss
    0//ms
    );
 {*)}
  except
    result := gOutOfScopelDate;
  end;
end;

const
  aTimes = 5000;

var
  TimeResult: TDateTime;
  a: TDateTime;
  b: TDateTime;
  I: Integer;
  hh, mm, ss, ms: Word;

begin
  a := Now;
  for i := 0 to aTimes do
  begin
    TimeResult := JsonStringToDateTime('2021-02-11T02:25:00.000Z');
  end;
  b := Now - a;
  DecodeTime(b, Hh, mm, ss, ms);
  Writeln('JsonStringToDateTime sec:' + IntToStr(ss) + ' msec:' + IntToStr(ms));

  a := Now;
  for i := 0 to aTimes do
  begin
    TimeResult := JsonStringToDateTime2('2023-03-13T02:25:00.000Z');
  end;
  b := Now - a;
  DecodeTime(b, Hh, mm, ss, ms);
  Writeln('JsonStringToDateTime2 sec:' + IntToStr(ss) + ' msec:' + IntToStr(ms));

  a := Now;
  for i := 0 to aTimes do
  begin
    TimeResult := JsonStringToDateTime3('2023-03-13T02:25:00.000Z');
  end;
  b := Now - a;
  DecodeTime(b, Hh, mm, ss, ms);
  Writeln('JsonStringToDateTime3 sec:' + IntToStr(ss) + ' msec:' + IntToStr(ms));

  Readln;
end.

In all honesty, I didn't think that replacing Copy would make a big difference, obviously, I was wrong.

I was under the impression that StrToInt could be replaced but I didn't know how to do.

The improvement that the changes suggested by @Fr0sT.Brutal make is very big! Thank you.

 

I hope I have understood all the suggestions.

 

Many thanks

Alberto

 

Edited February 13, 2021 by Alberto Paganini

[Attila Kovacs 629]

@Alberto Paganini You need const in the parameter list,  also measure with TStopwatch and not with Now(), and try not to use local string variable, like:

 

function JsonStringToDateTime(const Src: string): TDateTime;
const
  ofs = Ord('0');
var
  pSrc: PChar;
  Time: TDateTime;
begin
  if Length(Src) < 19 then
    Exit(gOutOfScopelDate);

  pSrc := Pointer(Src);
  if not TryEncodeDate( //
    ((Ord((pSrc)^) - ofs) * 1000) + ((Ord((pSrc + 1)^) - ofs) * 100) + ((Ord((pSrc + 2)^) - ofs) * 10) + (Ord((pSrc + 3)^) - ofs), //
    ((Ord((pSrc + 5)^) - ofs) * 10) + (Ord((pSrc + 6)^) - ofs), //
    ((Ord((pSrc + 8)^) - ofs) * 10) + (Ord((pSrc + 9)^) - ofs), Result) then
    Exit(gOutOfScopelDate);

  if not TryEncodeTime( //
    ((Ord((pSrc + 11)^) - ofs) * 10) + (Ord((pSrc + 12)^) - ofs), //
    ((Ord((pSrc + 14)^) - ofs) * 10) + (Ord((pSrc + 15)^) - ofs), //
    ((Ord((pSrc + 17)^) - ofs) * 10) + (Ord((pSrc + 18)^) - ofs), //
    0, //
    Time) then
    Exit(gOutOfScopelDate);

  Result := Result + Time;
end;

 

[Fr0sT.Brutal 900]

On 2/13/2021 at 3:49 PM, Alberto Paganini said:

I hope I have understood all the suggestions.

Not completely. Copying with pointers was my option if using IntToStr (because it requires strings). When you implement inttostr manually, there's no need in copying - just convert the chars pointers point at.

Also, try-except blocks add significant boilerplate and should be avoided in perf-critical routines. TryEncode* are the right way to do.

 

On 2/13/2021 at 3:49 PM, Alberto Paganini said:

One important part of the application I am working on is to retrieve data from a service provider as often as possible.

The function JsonStringToDateTime is called tens of thousands of times from several processes in a few minutes in order to parse the data provided therefore, the faster the function the more times the application can retrieve data.

Well, that's your vision from the point of app's logic. But you should trust numbers only - because conversion operations could take just a small part of the whole cycle, f.ex., receiving + parsing + preprocessing + handling result. As a very simple test you could measure perf with empty handling phase (just receiving) and take this number as an ideal. Then add your logic and see what has changed. Then use  your optimized functions and check again.

Btw, now that you're not protected from invalid data by Copy+IntToStr, don't forget to check input. If source string has expected length, if chars belong to expected range and so on.

On 2/13/2021 at 9:02 PM, Attila Kovacs said:

try not to use local string variable

I see no problems here