Jump to content


Popular Content

Showing content with the highest reputation on 02/01/20 in all areas

  1. Arnaud Bouchez

    Reading large UTF8 encoded file in chunks

    There is very fast line feed search, using proper x86_64 SSE assembly, checking by 16 bytes per loop iteration, in our SynCommons.pas: function BufferLineLength(Text, TextEnd: PUTF8Char): PtrInt; {$ifdef CPUX64} {$ifdef FPC} nostackframe; assembler; asm {$else} asm .noframe {$endif} {$ifdef MSWINDOWS} // Win64 ABI to System-V ABI push rsi push rdi mov rdi, rcx mov rsi, rdx {$endif}mov r8, rsi sub r8, rdi // rdi=Text, rsi=TextEnd, r8=TextLen jz @fail mov ecx, edi movdqa xmm0, [rip + @for10] movdqa xmm1, [rip + @for13] and rdi, -16 // check first aligned 16 bytes and ecx, 15 // lower 4 bits indicate misalignment movdqa xmm2, [rdi] movdqa xmm3, xmm2 pcmpeqb xmm2, xmm0 pcmpeqb xmm3, xmm1 por xmm3, xmm2 pmovmskb eax, xmm3 shr eax, cl // shift out unaligned bytes test eax, eax jz @main bsf eax, eax add rax, rcx add rax, rdi sub rax, rsi jae @fail // don't exceed TextEnd add rax, r8 // rax = TextFound - TextEnd + (TextEnd - Text) = offset {$ifdef MSWINDOWS} pop rdi pop rsi {$endif}ret @main: add rdi, 16 sub rdi, rsi jae @fail jmp @by16 {$ifdef FPC} align 16 {$else} .align 16 {$endif} @for10: dq $0a0a0a0a0a0a0a0a dq $0a0a0a0a0a0a0a0a @for13: dq $0d0d0d0d0d0d0d0d dq $0d0d0d0d0d0d0d0d @by16: movdqa xmm2, [rdi + rsi] // check 16 bytes per loop movdqa xmm3, xmm2 pcmpeqb xmm2, xmm0 pcmpeqb xmm3, xmm1 por xmm3, xmm2 pmovmskb eax, xmm3 test eax, eax jnz @found add rdi, 16 jnc @by16 @fail: mov rax, r8 // returns TextLen if no CR/LF found {$ifdef MSWINDOWS} pop rdi pop rsi {$endif}ret @found: bsf eax, eax add rax, rdi jc @fail add rax, r8 {$ifdef MSWINDOWS} pop rdi pop rsi {$endif} end; {$else} {$ifdef FPC}inline;{$endif} var c: cardinal; begin result := 0; dec(PtrInt(TextEnd),PtrInt(Text)); // compute TextLen if TextEnd<>nil then repeat c := ord(Text[result]); if c>13 then begin inc(result); if result>=PtrInt(PtrUInt(TextEnd)) then break; continue; end; if (c=10) or (c=13) then break; inc(result); if result>=PtrInt(PtrUInt(TextEnd)) then break; until false; end; {$endif CPUX64} It will be faster than any UTF-8 decoding for sure. I already hear some people say: "hey, this is premature optimization! the disk is the bottleneck!". But in 2020, my 1TB SSD reads at more than 3GB/s - https://www.sabrent.com/rocket This is real numbers on my laptop. So searching at GB/s speed does make sense. We use similar techniques at https://www.livemon.com/features/log-management With optimized compression, and distributed search, we reach TB/s brute force speed.
  2. Dave Nottage

    Android Service using local sqlite DB

    That's just the timer. Battery life will be affected by the settings used with the actual location services, as determined by MonitoringDistance and MonitoringInterval. There's some information here about the best strategies to use: https://developer.android.com/guide/topics/location/strategies
  3. Remy Lebeau

    Delphi 10.3.3 - Indy from 0ba2688b Git "Hello World"

    Yes, that is what it does. The OnConnect, OnDisconnect, OnExecute, and OnException events are called in the context of the connection thread, yes. That is a bit broad to answer. It really depends on how and where they are being used. I'm assuming you are referring to performing per-thread preparations, right? Then it comes down to whether you are using Indy's default 1-thread-per-connection model, or using thread pooling instead. If the former, then the server's OnConnect and OnDisconnect events will usually suffice. If you do not assign any component to the server's Scheduler property, a TIdSchedulerOfThreadDefault component is created and assigned for you automatically when you activate the server. TIdSchedulerOfThreadDefault creates a new thread per connection, and frees each thread when the connections are closed. Thus the server's OnConnect and OnDisconnect events are called only once per thread. If the latter, then as connections are closed and threads get reused, the OnConnect and OnDisconnect events can be called for multiple connections on the same thread, so that wouldn't really be a good place to perform per-thread initializations/cleanups (per-connection, yes). A better option (at least for now) is to derive a new class from TIdThreadWithTask and have it override the virtual BeforeExecute() and AfterExecute() methods to perform per-thread initializations/cleanups as needed, and then you can explicitly assign a TIdSchedulerOfThreadPool component to the server's Scheduler property, and assign your Task class to the TIdSchedulerOfThreadPool.ThreadClass property. Technically, you could take the same Task approach with TIdSchedulerOfThreadDefault, too. But that would generally be overkill, but usable.