Jump to content

John Terwiske

  • Content Count

  • Joined

  • Last visited

Community Reputation

1 Neutral

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. John Terwiske

    InterBase or Firebird?

    For all new work that does not require a server DB I use SQLite.
  2. John Terwiske

    Firedac Sqlite bug?

    I've reported this as a bug: https://quality.embarcadero.com/browse/RSP-31670
  3. John Terwiske

    Firedac Sqlite bug?

    Odd that there would be a regression in 10.3.3. I'm looking forward to 10.4.1, most especially for the ability to non-statically link to SQLite library.
  4. John Terwiske

    Firedac Sqlite bug?

    Thanks for the confirmation in 10.3.3.
  5. John Terwiske

    Firedac Sqlite bug?

    In Delphi Rio 10.3.3 (I've not tested this on other versions) there seems to be a parsing bug with this recursive CTE: WITH RECURSIVE example1_cte (Id, No, ParsedText, Unparsed) AS ( SELECT Id, 0 as No, '' as ParsedText, TextToParse || ',' as TextToParseWithTrailingComma FROM test1 UNION ALL -- "ALL" is not strictly necessary in this example SELECT Id, No+1, -- increment the counter for this row SUBSTR(Unparsed, 0, INSTR(Unparsed, ',') ), SUBSTR(Unparsed, INSTR(Unparsed, ',') + 1) FROM example1_cte WHERE Unparsed <> '' ) -- SELECT from the recursive cte SELECT Id, No, TRIM(ParsedText) as ParsedText FROM example1_cte WHERE ParsedText <> '' ORDER BY Id ASC, No ASC; When executed a Firedac error message appears: [FireDac][Phys][SQLite] Error: no such table column: example1_cte.Id. And yet, if the line below the UNION ALL is changed to SELECT Id+0, (I.e. just adding a zero to Id), the error message goes away, and the CTE works correctly. btw, this query works fine without the "+0" in other SQLite query processors (SQLiteStudio). I think this is a bug, but cannot locate it in JIRA. Attached is a quick and dirty example project. (Change the FDQuery1 to remove the "+0" to see the error.) I don't have 10.4 so not sure this still exists. Project1.dpr Project1.dproj Unit1.dfm Unit1.pas
  6. Thank you. It would appear that whatever issue I was having with unicode is no longer a problem with the Delphi zip code/classes. Your solution works with all of the non-spanning zip files I've got on hand for testing. It's about 6% slower vs. the third party solution I'm currently using, and this matters with these files that are in the 2Gbyte range. I'll look into this some more... someday. 🙂
  7. Yes, but I don't see how this extracts a single file to a stream of some sort. And extracting to a file on disk makes these useless for my purposes. It's almost there, but I think this is why I started using the third party years ago. Well, that and at the time as I recall there was a problem with unicode characters in the items within the zip file (memory fades on this though.)
  8. What if there are multiple files within the Zip archive? I need to extract a specific item within a Zip file. I'm using a third party component that works like this (streamlined version): (n.b. MyUnArchiver is a third party zip component on a Delphi Form) MyUnArchiver.FileName := 'D:\testdata\largefile.zip'; // the zip file on disk MyUnzippedStream := TStringStream.Create; // I use StringStream for processing, but any stream works here try MyUnArchiver.OpenArchive; MyUnArchiver.ExtractToStream('archive item name', MyUnzippedStream); // name of the given archive item within the zip file MyUnZippedStream.Position := 0; // the extraction will set the position to end of stream, so this is necessary // process MyUnZippedStream .. finally MyUnzippedStream.Free; end; I'd like to reduce my third party dependencies, but it's a "maybe later" priority. The third party component (and I am not affiliated with them) is called ZipForge.
  9. For me, right now, git is it. However, for really big teams and for those that must still support older versions (remember when that was a thing?) Clearcase worked the best (about 200 developers back then, mostly C, some C++, some Delphi) supporting multiple versions and all of it tied in to bugs and design. The one I didn't much care for was Continuus, but that was mostly the user interface (the idea was great though.) I never used Starteam from Borland, but I know folks who did and liked it.
  10. John Terwiske

    FastMM5 vs. inbuilt Delphi 10.3.3 memory manager

    Here are some results comparing the Delphi built-in memory manager and the one from FastMM5. Please see my earlier post in this thread. Indeed, the performance is much improved with multi-threading under FastMM5 vs. the in-built Delphi MM. Although a non-threaded single pass comparison was more favorable to the built-in memory manager for 10.3.3, the more real-world testing is significantly more favorable to FastMM5. This simple test consists of reading 50 of the vcf files I mentioned in my earlier post. Each of them is about 48 Mb and uses a threaded approach with 12 physical cores on the test machine, together with the OmniThreadLibrary Parallel.ForEach paradigm. These times are for the 64 bit executable (slightly faster with 32 bit exe, as expected) Delphi 10.3.3 built-in Memory Manager: 21929 milliseconds FastMM5: 4762 ms For FastMM5, there is an approximate 1.5% decrease in time taken using the OptimizeForSpeed strategy.
  11. John Terwiske

    FastMM5 vs. inbuilt Delphi 10.3.3 memory manager

    Yes, all true.
  12. John Terwiske

    FastMM5 vs. inbuilt Delphi 10.3.3 memory manager

    I checked and no. However, your idea reminded me that FastMM_SetOptimizationStrategy(mmosOptimizeForSpeed) is there as an option. I'm reworking my test launch a threaded version that will work with a few hundred of these files and I'll compare FastMM5 and the Delphi memory manager.
  13. John Terwiske

    FastMM5 vs. inbuilt Delphi 10.3.3 memory manager

    Yes, I'm working to incorporate this "reading" stage into a pipeline using a threaded approach with OmniThreadLibrary. That's why I'm looking at FastMM5. I suppose I'll need to create a test that launches several "readers" for a few hundred of these files to see improvement with FastMM5. My shortcut of just trying one file is not sufficient for testing that.. darn it. (Btw, the search string only occurs once in the file, Memo1.Lines.Add just happens once. The vcf file format is well defined in the genetic analysis domain; this "#C" is the a way of identifying column headings/field names used for the remainder of the file).
  14. Hello, I'm a bit confused about my code with regard to FastMM5. FastMM5 appears to be 15 percent slower than the regular in-built (FastMM?) memory manager that comes with Delphi 10.3.3. This matters, as I need to process thousands of these files, and it's been years since I've worked with Delphi. Hopefully, there's something I'm forgetting to do, and folks here can help. Here's the code: procedure TForm1.Button2Click(Sender: TObject); var StreamToReadFrom: TStreamReader; LineCounter: Integer; MyString: string; MyUnzippedStream: TBufferedFileStream; //TReadOnlyCachedFileStream<== faster than TBufferedFileStream, but still slower under FastMM5; MyStopwatch: TStopWatch; begin Memo1.Clear; LineCounter := 0; MyStopwatch := TStopwatch.Create; MyStopwatch.Start; // The file D:\variants.vcf is approximately 48.4 million bytes with 202K varying length lines of text that is not unicode MyUnZippedStream := TBufferedFileStream.Create('D:\variants.vcf', fmOpenRead); try StreamToReadFrom := TStreamReader.Create(MyUnZippedStream, False); try StreamToReadFrom.Rewind; while not StreamToReadFrom.EndOfStream do begin MyString := StreamToReadFrom.ReadLine; if AnsiStartsStr( '#C', MyString ) then Memo1.Lines.Add( MyString ); Inc(LineCounter); end; finally StreamToReadFrom.Free; end; finally MyUnZippedStream.Free; MyStopwatch.Stop; end; Memo1.Lines.Add( 'Lines read: ' + LineCounter.ToString ); Memo1.Lines.Add( 'Milliseconds: ' + mystopwatch.ElapsedMilliseconds.ToString ); end; This code is a highly simplified portion of a stage used in a complicated pipeline. This is fairly fast as it is in just one thread, but a 15% slowdown under FastMM5 for thousands of these files is not really something I was looking to see.