Jump to content

FredS

Members
 View Profile  See their activity

  • Content Count

    408

  • Joined

  • Last visited

  • Days Won

    4

Posts posted by FredS

  1. Posted

    42 minutes ago, Alexander Elagin said:

    assign myself somewhere

    I'm also Primary contact and there has never been another, the email hasn't changed from Godzilla, yet nada..

     

    Not that I was interested but some bot shouldn't know that 🙂

    • Like 1

  4. Posted

    12 hours ago, Sherlock said:

    what code should have is a comment explaining the why

    Having come back to code from last century I now comment things that may require additional time to read.
    Even just a one liner helps when you scan code. I also limit the use of 'With', had some of that a few levels deep before 🙂

     

  6. Posted

    11 minutes ago, Lars Fosdal said:

    Every bloody time

     

    In Godzilla and beyond, at least for me.. Occasionally I run Rio when there are a lot of compiler directives in a file.

    Rio is a better Notepad++ than Berlin but eventually you need to run the code and it all ends by closing Rio and re-opening Berlin..

     

     

    • Like 1

  8. Posted

    14 minutes ago, pyscripter said:

    CPU usage is reported OK

     

    There are multiple issues, first and foremost is the response time for spawning threads.

    Then there are all those 'undocumented Features' new and old..

     

    Remove the TMonitor in your Load and get rid of the Sum because the only reason to call that is to tax the system.
    Maybe switch to x64 and run that again, at least the time will increase in x64 and allow for more threads to be added.

     

    Don't forget to wait for them to idle out ~10 seconds then run two more times to see if it breaks with negative numbers, which should also give you increased times.

     

    See the negative idle in Rio 10.3.3:

     

  9. Posted · Edited by FredS
    added TThreadPoolStatsHelper

    46 minutes ago, pyscripter said:

    following results for multiple runs

    CPU/Avg should be showing 100% else the lock is most likely throttling that or your implementation of formatted is off.

     

    46 minutes ago, pyscripter said:

    The condition FWorkerThreadCount < FMinLimitWorkerThreadCount does not appear to make sense.

    I believe the point is to spawn MinLimitWorkerThreadCount ASAP and to make those available regardless of CPU load.

     

    Also the last [two] tests must be after the pool has cleared, usually 10 secs.

     

    Should have been included with the original code: 

    //MMWIN:CLASSCOPY
unit _MM_Copy_Buffer_;

interface

type
  TThreadPoolStatsHelper = record helper for TThreadPoolStats
    function Formatted: string;
  end;


implementation


function TThreadPoolStatsHelper.Formatted: string;
begin
  Result := Format('Worker: %2d, Min: %2d, Max: %2d, Idle: %2d, Retired: %2d, Suspended: %2d, CPU(Avg): %3d, CPU: %3d',
                   [self.WorkerThreadCount,
                    self.MinLimitWorkerThreadCount, self.MaxLimitWorkerThreadCount,
                    self.IdleWorkerThreadCount, self.RetiredWorkerThreadCount, self.ThreadSuspended,
                    self.AverageCPUUsage, self.CurrentCPUUsage]);
end;

end.

     

  10. Posted

    6 hours ago, GeMeMe said:

    Could you share

     Knew that was coming 🙂

     

    This is a monumental task because some patches made it into Tokyo and I merged them while another I applied.
    I can't exactly publish System.Threading which alters some things dramatically.

     

    Will attach the RSP-11267 patch since it was applied over the unaltered Berlin Threading unit.
    Will add my notes so you get an idea of the changes and limitations. Will add the simple changes which alter the constants used.
    Not sure how I can deal with some of the core changes though..

     

    Tokyo never made it onto my development machine, that merge was simply an easy way to create an intermediate file that should have been used to update Rio.

    But in Rio System.Threading was altered so that 1.5 years after its original release we still have to deal with posts like this one.. needless to say I never made that merge..

      

    
interface
{$REGION 'History'}
//  earlier     - Modified as per RSP-11267
//  09-Jan-2019 - Changed MonitorThreadDelay to 100 to be more responsive, if we are taking a clients system
//                to 95% CPU or higher shouldn't we be able to adjust that without a long pause
//   		      	- Changed NumCPUUsageSamples to NumCPUUsageSamplesShort (1000ms) and NumCPUAvgSamples (5000ms)
//              - Changed Retirement Delay to 500ms Idle already can take 80-320 seconds
//              - Allow Parking/Suspending of unlimited threads, this is key when another application starts to
//                chew up resources while tasks have already been spawned
//              - Fixed CPU usage, was turned off after the last Queued task began work
//              - Fixed a silly timeout patch that should have used the NoWorker flag instead
//                Be Aware this may not be a perfect solution if more than one pool is created.
//                An Assertion has been added to highlight this possible issue
//  10-Jan-2019 - Added RunningThreadCount
//              - Calling CreateMonitorThread always now; We want a Singleton MonitorThead regardless of the amount of Tasks requested
//              - Added Output2Debug function
//  20-Jan-2019 - Merged non Tokyo specific changes from Tokyo
//              - There are Tokyo specific changes that must be done in a new File
{$ENDREGION}
{$I ..\..\MatchDelphi.inc}
{$TYPEDADDRESS OFF} // required for Resource String Pointers
{$SCOPEDENUMS ON}
{-$DEFINE OUTPUT_DEBUG_STRING} // Activate to show Output2Debug Messages
{$IFDEF DEBUG}
  {$MESSAGE HINT 'Altered System.Threading tested for Single/Default TThreadPool Only'}
{$ENDIF}
        /// <summary-Fred>
    ///   Changed MonitorThreadDelay to 100 to be more responsive,
    ///   if we are taking a clients system to 95% CPU
    ///   or higher shouldn't we be able to adjust that
    /// </summary>
    MonitorThreadDelay = 100 {500};
    /// <summary-Fred>
    ///   We keep two Averages now, one of 5 secs for standard output as before
    ///   and another for 1 sec used to calculate Suspensions/Parking
    /// </summary>
    NumCPUUsageShortSamples = 1000 div MonitorThreadDelay;
    NumCPUAvgSamples        = 5 *  NumCPUUsageShortSamples;
    /// <summary-Fred>
    ///   SuspendInterval is the value used to limit the number of threads that
    ///   can be Parked, IMO this should reflect MonitorThreadDelay because that
    ///   is the time it takes to recalculate the CPU Averages
    /// </summary>
    SuspendInterval = MonitorThreadDelay; // Interval to use for suspending work in worker threads
    /// <summary-Fred>
    ///   Twice MonitorThreadDelay seems to work well
    /// </summary>
    SuspendTime = MonitorThreadDelay * 2; // Time to spend in SuspendWork;
    RetirementDelay = 5000; // Delay interval for retiring threads

    and finally the output of the original test: 

    Altered Threading Berlin x64:

Before: Worker:  0, Min:  4, Max: 100, Idle:  0, Retired:  0, Suspended:  0, CPU(Avg):   0, CPU:   0
After:  Worker:  4, Min:  4, Max: 100, Idle:  3, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:37.3397033

Before: Worker:  4, Min:  4, Max: 100, Idle:  4, Retired:  0, Suspended:  0, CPU(Avg):   7, CPU:   3
After:  Worker:  5, Min:  4, Max: 100, Idle:  4, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:36.4708368

Before: Worker:  5, Min:  4, Max: 100, Idle:  5, Retired:  0, Suspended:  0, CPU(Avg):   7, CPU:   0
After:  Worker:  6, Min:  4, Max: 100, Idle:  5, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:36.2496669

Before: Worker:  6, Min:  4, Max: 100, Idle:  6, Retired:  0, Suspended:  0, CPU(Avg):  35, CPU:   0
After:  Worker:  7, Min:  4, Max: 100, Idle:  6, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:36.2084714

Before: Worker:  7, Min:  4, Max: 100, Idle:  7, Retired:  0, Suspended:  0, CPU(Avg):   7, CPU:   0
After:  Worker:  8, Min:  4, Max: 100, Idle:  7, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:36.8463683

Before: Worker:  0, Min:  4, Max: 100, Idle:  0, Retired:  0, Suspended:  0, CPU(Avg):   3, CPU:   0
After:  Worker:  4, Min:  4, Max: 100, Idle:  3, Retired:  0, Suspended:  1, CPU(Avg): 100, CPU: 100
Finished in 00:00:37.0737835

     

    1 - RSP-11267.patch

    • Thanks 2

  11. Posted · Edited by FredS
    Added Berlin findings, RSP-24570

    8 hours ago, GeMeMe said:

    Finished in 00:00:09.5530676

    Then there is that undocumented 'Feature' where after you wait for termination your elapsed time goes from ~1s to ~81s: 

    PPL Test ---------------
Before: Worker: 0, (Min: 4..Max: 100), Idle: -9, Retired: 0, Suspended: 0, CPU(Avg): 5, CPU: 2
Finished in 00:00:01.6734632
After:  Worker: 4, (Min: 4..Max: 100), Idle: -5, Retired: 0, Suspended: 0, CPU(Avg): 20, CPU: 100
------------------------

After waiting for threads to terminate:


PPL Test ---------------
Before: Worker: 4, (Min: 4..Max: 100), Idle: -5, Retired: 0, Suspended: 0, CPU(Avg): 1, CPU: 1
Finished in 00:00:30.7872728
After:  Worker: 4, (Min: 4..Max: 100), Idle: -3, Retired: 0, Suspended: 0, CPU(Avg): 20, CPU: 50
------------------------
PPL Test ---------------
Before: Worker: 4, (Min: 4..Max: 100), Idle: -3, Retired: 0, Suspended: 0, CPU(Avg): 1, CPU: 0
Finished in 00:01:21.6815195
After:  Worker: 4, (Min: 4..Max: 100), Idle: -2, Retired: 1, Suspended: 0, CPU(Avg): 20, CPU: 100
------------------------

    I ran this in Berlin and it appears to work fine.

    But I made changes to System.Threading, both to achieve near 100% CPU usage as well as applying some of the patches submitted by those reporting to QC..

     

    Then there is the issue that in Win64 this takes over 37 seconds, RSP-24570 is for CBuilder but its the same in Berlin and Rio 10.3.3

  14. Posted · Edited by FredS

    4 hours ago, Stefan Glienke said:

    compare different hash algorithms

     

    I haven't looked at Fastcode for ages, today I tried out the 'System.Generics.FastDefaults' unit only to find that the implementation of Murmur3 has changed.

    The latest one actually did NOT work at all for me...
     

    The one used in my tests is from the Initial commit: 

    Alpha version 0.1

    https://github.com/JBontes/FastCode/commit/7c47518e93870aaf155e70b6f43e3ba1dfb93898

     

    Edit:

    Turned out that Berlin needed a restart, results of the same test (NewM3 being the one in the current master): 

    Four Million (Murmor3,PreAlloc): 03.124-1398
Four Million (NewM3,PreAlloc):   03.078-7503

     

  16. Posted

    7 hours ago, Stefan Glienke said:

    or we are talking apples and orange

    How come I always get homework when I come here 🙂

     

    The code used was originally done to test a home rolled implementation of 'AddIfNotExists' in Berlin.
    In short that particular test won't do you any good.

     

    But here are the steps used:

    1. Turned OFF Overflow and Range checking in xxHash32
    2. Changed the test from using the default Jenkins hash to use (that) xxHash32.
    3. This was done by copying the TDelegatedEqualityComparer used with Murmor3 and changing a single line: 
      Result := Integer(TxxHash32.CalculateHash32(PChar(Value)^, Length(Value) * CharSize));
// 
Result := Murmor3(PChar(Value)^, Length(Value) * CharSize, 0);

       

    4. The test itself is based on a TDictionary<string,integer> where the string is a GUID appended with the for loop value.
      This was originally done since one version of Murmor had issues with collisions when the first part of the string was identical.

    That's it, as before: your mileage may vary..

     

  19. Posted · Edited by FredS
    added link to Murmor3 source

    7 hours ago, Kas Ob. said:

    xxHash https://github.com/Xor-el/xxHashPascal

    Just ran a TDictionary comparison using Murmor3 v TxxHash32.CalculateHash32: 

        One Million (xxHash,PreAlloc):   01.446-8681
    One Million (Murmor3,PreAlloc):  00.743-3334

    Four Million (Murmor3):          04.895-7561
    Four Million (Murmor3,PreAlloc): 03.130-7519
    Four Million (xxHash,PreAlloc) : 04.491-7058

    Your mileage may vary..

  22. Posted

    2 hours ago, David Schwartz said:

    TMyClass = type TAnotherClass;

     

    This treats TPassword like a string yet in Berlin+ you can add a unique Record Helper for TPassword. 

    TPassword = type string;

// Example of a Helper for TPassword
  TPasswordHelper = record helper for TPassword
    function CheckPwdComplexity(const MinPwdLen: Word; const Level: TPwdComplexity): boolean;
    function ContainsLower(const Required : Cardinal = 1): boolean;
    function ContainsSpecial(const Required : Cardinal = 1): boolean;
    function ContainsUpper(const Required : Cardinal = 1): boolean;
    function Secure(const ProtectWith: TSecureStringProtection = ssFast): ISecureString;
    function BCrypt: TBCrypt; inline;
  end;
    • Like 2
    • Thanks 1

  25. Posted · Edited by FredS

    8 minutes ago, Mark Williams said:

    He doesn't mention if that is on Windows 10 or not

    On Windows 10 1903.18362.476.

     

    Also your code doesn't `CoUninitialize` if IIDL is nil.

    ..and what does GetLastError say when you get IIDL=nil?

     

    • Like 1
×