TJSONObject.Format bug

[David Schwartz 426]

It seems if you have quoted strings that contain things like "... 100% satisfaction ..." or "... a 10% discount..." in an input string, the call to doc.Format here:

 

    doc := TJSONObject.ParseJSONValue(the_resp) as TJSONObject;
    JSON_mmo.Text := doc.Format(DEF_INDENT);  // <-----

 

chokes on the "% <space> <next char>" sequence saying it's not a valid format.

 

Is there a way to escape these % chars or do something that doesn't require deleting them?

[Lars Fosdal 1792]

Not sure why it would barf on %, but you can replace % with \u0025 (Unicode escape)
See also
https://stackoverflow.com/questions/19176024/how-to-escape-special-characters-in-building-a-json-string/27516892

 

What happens if you put

a double %%    

or %25 (Similar to \u0025)

or \%
 

[David Schwartz 426]

I dunno why it would barf either, except when I went through the raw response text with an editor and replaced all '%' characters with ' pct' the problem disappeared.

 

Maybe it's actually using the normal Format method there? (I have not dug into it.)

[Lars Fosdal 1792]

That sounds likely.  I stumbled on similar problems with formatting of json strings in other languages when I googled.

[Anders Melander 1783]

From what I can see, by reading the source, the only place SysUtils.Format is called is in EJSONParseException.Create.

 

TJSONObject.Format ends up calling TJSONString.ToChars to encode the chars and I can't see anything there that would choke on %.

 

Regardless, it would be trivial for you to look at the call stack when the exception occurs to determine where SysUtils.Format is called from.

[Der schöne Günther 316]

I don't know how to reproduce. This here runs fine:

 

program Project1;

{$APPTYPE CONSOLE}

uses System.JSON;

const
	the_resp = '{"msg": "you get ... 100 % satisfaction"}';
	the_resp2 = '{"msg": "you get \"... 100 % satisfaction\""}';
	DEF_INDENT = 2;
var
	doc: TJSONObject;
	formatted: String;
begin
	doc := TJsonObject.ParseJSONValue(the_resp) as TJSONObject;
	formatted := doc.Format(DEF_INDENT);
	WriteLn(formatted);

	doc := TJsonObject.ParseJSONValue(the_resp2) as TJSONObject;
	formatted := doc.Format(DEF_INDENT);
	WriteLn(formatted);
end.

 

[Dave Nottage 557]

6 hours ago, David Schwartz said:

It seems if you have quoted strings that contain things like "... 100% satisfaction ..." or "... a 10% discount..." in an input string, the call to doc.Format here:

I used @Lars Fosdal's code:

https://github.com/DelphiWorlds/Kastri/blob/672522381e8fe192f562cdcb95b9f473b08fc0b6/Core/DW.REST.Json.Helpers.pas#L77

Works better than the Format method, for me 🙂

 

[David Schwartz 426]

On 7/19/2021 at 3:25 AM, Dave Nottage said:

I used @Lars Fosdal's code:

https://github.com/DelphiWorlds/Kastri/blob/672522381e8fe192f562cdcb95b9f473b08fc0b6/Core/DW.REST.Json.Helpers.pas#L77

Works better than the Format method, for me 🙂

 

This looks great, except I can't get the compiler to find the Tidy method:

 

      var JO := TJSONObject.ParseJSONValue(google_srch.TheResponse) as TJSONObject;
      JSON_mmo.Text := JO.Tidy( google_srch.TheResponse );

[dcc32 Error] Main_form.pas(533): E2003 Undeclared identifier: 'Tidy'

I have the unit in the Uses clause and even included it in the project.

[Dave Nottage 557]

10 hours ago, David Schwartz said:

This looks great, except I can't get the compiler to find the Tidy method:

Tidy is a method of TJson (via a class helper), not TJsonObject. It's also a class method, so you don't need an instance of anything to call it

[David Schwartz 426]

8 hours ago, Dave Nottage said:

Tidy is a method of TJson (via a class helper), not TJsonObject. It's also a class method, so you don't need an instance of anything to call it

Yes, I know that. Here's the problem:

 

implementation

{$R *.dfm}

uses StrUtils, Math, ShellApi, DW.REST.Json.Helpers;

The file is in the same folder as the other project files.

I even included the unit in the project. So there's no reason the compiler shouldn't be able to find this method.

-------------------

      JSON_mmo.Text := Tidy( google_srch.TheResponse );
[dcc32 Error] Main_form.pas(533): E2003 Undeclared identifier: 'Tidy'

 

      JSON_mmo.Text := TJSON.Tidy( google_srch.TheResponse );
[dcc32 Error] Main_form.pas(533): E2003 Undeclared identifier: 'TJSON'      
 

      JSON_mmo.Text := JO.Tidy( google_srch.TheResponse );
[dcc32 Error] Main_form.pas(533): E2003 Undeclared identifier: 'Tidy'

--------------------

 

FWIW, I get red squigglies on a numerous classes and methods that the compiler things are not defined, but it compiles just fine and runs.

 

(This is D10.4.2.)

Edited July 21, 2021 by David Schwartz

[Lajos Juhász 293]

2 hours ago, David Schwartz said:

    JSON_mmo.Text := TJSON.Tidy( google_srch.TheResponse );
[dcc32 Error] Main_form.pas(533): E2003 Undeclared identifier: 'TJSON'      

A quick find in files returned that TJSon is implemented in the REST.Json.pas and it is not in your uses.

[David Schwartz 426]

On 7/21/2021 at 2:48 AM, Lajos Juhász said:

A quick find in files returned that TJSon is implemented in the REST.Json.pas and it is not in your uses.

 

EDIT:

 

Actually, it's implemented in DW.REST.Json.Helpers. I cannot find any other reference to Tidy in the code base that I just pulled down from github.

 

  TJsonHelper = class helper for TJson
  public
    class function FileToObject<T: class, constructor>(out AObject: T; const AFileName: string;
      AOptions: TJsonOptions = [joDateIsUTC, joDateFormatISO8601]): Boolean; static;
    class procedure SaveToFile(const AObject: TObject; const AFileName: string); static;
    /// <summary>
    ///   Replacement functions for TJson.Format, which does not format correctly
    /// </summary>
    /// <remarks>
    ///   Overload that takes a string parameter has been modified so that it should also work for
    ///      JSON which is not generated by TJsonValue.ToString
    ///   Please use this function with CAUTION!! It still may not cater for all situations, and is still not perfect
    /// </remarks>
    class function Tidy(const AJsonValue: TJsonValue; const AIndentSize: Integer = 2): string; overload; static;
    class function Tidy(const AJson: string; const AIndentSize: Integer = 2): string; overload; static;
  end;
. . .
// Now based on: https://pastebin.com/Juks92Y2 (if the link still exists), by Lars Fosdal
class function TJsonHelper.Tidy(const AJson: string; const AIndentSize: Integer = 2): string;
const
  cEOL = #13#10;
var
  LChar: Char;
  LIsInString: boolean;
  LIsEscape: boolean;
  LIsHandled: boolean;
  LIndent: Integer;
begin
  Result := '';
  LIndent := 0;
  LIsInString := False;
  LIsEscape := False;
  for LChar in AJson do
  begin
    if not LIsInString then
    begin
      LIsHandled := False;
      if (LChar = '{') or (LChar = '[') then
      begin
        Inc(LIndent);
        Result := Result + LChar + cEOL + StringOfChar(' ', LIndent * AIndentSize);
        LIsHandled := True;
      end
      else if LChar = ',' then
      begin
        Result := Result + LChar + cEOL + StringOfChar(' ', LIndent * AIndentSize);
        LIsHandled := True;
      end
      else if (LChar = '}') or (LChar = ']') then
      begin
        Dec(LIndent);
        Result := Result + cEOL + StringOfChar(' ', LIndent * AIndentSize) + LChar;
        LIsHandled := True;
      end;
      if not LIsHandled and not LChar.IsWhiteSpace then
        Result := Result + LChar;
    end
    else
      Result := Result + LChar;
    if not LIsEscape and (LChar = '"') then
      LIsInString := not LIsInString;
    LIsEscape := (LChar = '\') and not LIsEscape;
  end;
end;

end.

 

Edited July 23, 2021 by David Schwartz

[David Schwartz 426]

Ok, I figured it out.

 

Thanks!

 

Edited July 23, 2021 by David Schwartz