Firedac FDBatchMove.Execute error when last line in double-quote CSV file does not contain linefeed

[egnew 3]

I am using a TFDMemTable to import text data.  When processing comma delimited files with double-quoted values an error occurs when the last line in the file does not have a line feed.

 

To reproduce this problem create a text file with the following values in notepad:

"name","date","amount"
"Alpha","1/1/2021","100"
"Bravo","1/2/2021","200"
"Charlie","1/3/2021","300"

 

Make sure you save the file with the final cursor at the end of the "Charlie" line.

The following error will occur on BatchMove.Execute;

 

 

The error does not occur if double-quote characters are not used.

The error does not occur if you add a line feed so the cursor is on the line after "Charlie" when the file is saved.

 

myTable := TFDMemTable;
myReader := TFDBatchMoveTextReader;
myWriter := TFDBatchMoveDataSetWriter;
myMover := TFDBatchMove;

 

procedure Test (const p_FileName: String);
var                                                                               // --- Added in attemp to bypass the issue

  I: Integer;                                                                   // --- Added in attemp to bypass the issue
begin
  myReader.FileName := p_FileName;
  for i := 0 to myReader.Datadef.Fields.Count-1 do    // --- Added in attemp to bypass the issue
    myReader.Datadef.Fields.DataType := atString;   // --- Added in attemp to bypass the issue
  myReader.DataDef.WithFieldNames := True;
  myWriter.DataSet := myTable;
  myWriter.Optimise := False;
  myMover.Reader := myReader;
  myMover.Writer := myWriter;
  myMover.GuessFormat;
  myMover.Analyze := [taDelimSep, taHeader, taFields];
  myMover.AnalyzeSample := 50;
  myMover.Execute;

end;

The files being processed are automatically obtained from other systems.    We cannot control the presence of the linefeed without modifying the received file.  Can you please advise on how to handle this issue?

[Guest]

if cannot control it, dont problem!

just load on Memo or any other target... and save it with your necessity. 

if need add a CR+LF for example.

 

Another way, more simple: using TString class

 

loading on a TSTRING class or sub-classes, you can take each line and analize it... you see?

 

a long line with all data with ",;.Chars etc...

later, u can divide it using same tecnich, on another Tstring, for example, BUT NOW, using another DELIMITER CHAR, you see my idea?

it's simple way if well done.

 

this if you need analize the file before process it, else, try just load and save in a new file with you changes, at the end, use BATCHMOVE

 

hug

Edited February 19, 2021 by Guest

[egnew 3]

Thanks for your solutions but I am asking how to get the FDBatchMove to properly handle the situation.

 

[Dmitry Arefiev 101]

This is a bug in TFDBatchMoveTextReader. To fix it:

• open FireDAC.Comp.BatchMove.Text.pas
• locate there lines (around line # 1750):

        until FEof or lEOL or not lInDelim and (FBuff[iLen] = DataDef.Separator);
        if not FEof then
          Dec(iLen);

• replace with:

        until FEof or lEOL or not lInDelim and (FBuff[iLen] = DataDef.Separator);
        if not FEof or not lEOL then
          Dec(iLen);

• add FireDAC.Comp.BatchMove.Text.pas to your project, or include path to this unit to project->options Search Path, or to tools->options Library Path

PS: Best will be to report this issue to quality.embarcadero.com

[Attila Kovacs 629]

1 hour ago, Dmitry Arefiev said:

PS: Best will be to report this issue to quality.embarcadero.com

This raises a question for me. 🤔

[egnew 3]

Thanks for your help.  The issue is resolved.

I reported the bug and your solution:

 https://quality.embarcadero.com/browse/RSP-32465

[weirdo12 19]

FYI, this issue remains unresolved in 10.4.2. I added some more details to the report today.