Jump to content
Sign in to follow this  
egnew

Firedac FDBatchMove.Execute error when last line in double-quote CSV file does not contain linefeed

Recommended Posts

I am using a TFDMemTable to import text data.  When processing comma delimited files with double-quoted values an error occurs when the last line in the file does not have a line feed.

 

To reproduce this problem create a text file with the following values in notepad:

"name","date","amount"
"Alpha","1/1/2021","100"
"Bravo","1/2/2021","200"
"Charlie","1/3/2021","300"

 

Make sure you save the file with the final cursor at the end of the "Charlie" line.

The following error will occur on BatchMove.Execute;

 

image.png.23dca7f8b2182af3d5366b93b41edbd4.png

 

The error does not occur if double-quote characters are not used.

The error does not occur if you add a line feed so the cursor is on the line after "Charlie" when the file is saved.

 

myTable := TFDMemTable;
myReader := TFDBatchMoveTextReader;
myWriter := TFDBatchMoveDataSetWriter;
myMover := TFDBatchMove;

 

procedure Test (const p_FileName: String);
var                                                                               // --- Added in attemp to bypass the issue

  I: Integer;                                                                   // --- Added in attemp to bypass the issue
begin
  myReader.FileName := p_FileName;
  for i := 0 to myReader.Datadef.Fields.Count-1 do    // --- Added in attemp to bypass the issue
    myReader.Datadef.Fields.DataType := atString;   // --- Added in attemp to bypass the issue
  myReader.DataDef.WithFieldNames := True;
  myWriter.DataSet := myTable;
  myWriter.Optimise := False;
  myMover.Reader := myReader;
  myMover.Writer := myWriter;
  myMover.GuessFormat;
  myMover.Analyze := [taDelimSep, taHeader, taFields];
  myMover.AnalyzeSample := 50;
  myMover.Execute;

end;


The files being processed are automatically obtained from other systems.    We cannot control the presence of the linefeed without modifying the received file.  Can you please advise on how to handle this issue?

Share this post


Link to post
Guest

if cannot control it, dont problem!

just load on Memo or any other target... and save it with your necessity. 

if need add a CR+LF for example.

 

Another way, more simple: using TString class

 

loading on a TSTRING class or sub-classes, you can take each line and analize it... you see?

 

a long line with all data with ",;.Chars etc...

later, u can divide it using same tecnich, on another Tstring, for example, BUT NOW, using another DELIMITER CHAR, you see my idea?

it's simple way if well done.

 

this if you need analize the file before process it, else, try just load and save in a new file with you changes, at the end, use BATCHMOVE

 

hug

Edited by Guest

Share this post


Link to post

Thanks for your solutions but I am asking how to get the FDBatchMove to properly handle the situation.

 

Share this post


Link to post

This is a bug in TFDBatchMoveTextReader. To fix it:

  • open FireDAC.Comp.BatchMove.Text.pas
  • locate there lines (around line # 1750):
        until FEof or lEOL or not lInDelim and (FBuff[iLen] = DataDef.Separator);
        if not FEof then
          Dec(iLen);
  • replace with:
        until FEof or lEOL or not lInDelim and (FBuff[iLen] = DataDef.Separator);
        if not FEof or not lEOL then
          Dec(iLen);
  • add FireDAC.Comp.BatchMove.Text.pas to your project, or include path to this unit to project->options Search Path, or to tools->options Library Path

PS: Best will be to report this issue to quality.embarcadero.com

  • Like 3
  • Thanks 1

Share this post


Link to post
1 hour ago, Dmitry Arefiev said:

PS: Best will be to report this issue to quality.embarcadero.com

This raises a question for me. 🤔

Share this post


Link to post

FYI, this issue remains unresolved in 10.4.2. I added some more details to the report today.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×