Register COM Object for create process/Fightiing AntiVirus

[RTollison 0]

Not sure how to go about this but in my in-house program, not for client use, i create a process to run a command for each file i process. varies depending on selection. 1 - 2500. However, the antivirus has to scan/test each create process that is run in a thread. so if i say spin up 50 threads at a time to process all 2500 files it takes forever, 12-14 hours. I worked with our IT dept to get the command that is running cleared and my program for spinning up the threads to run it. But I watch the task manager details and every time i spin up a thread then the AntiVirus inspects it, so what used to take like 10 minutes is now hours long. after begging an IT to whitewash the create process (solid no way on that) she suggested i look into creating a com object that they might whitewash to see (testing phase) if it would be bypassed by the AV. we don't know if it will help. i said it sounded like what my program is doing now with the exe. but IT will not move on this until i try a COM object that just runs the command (hopefully) without using create process. maybe use the CMD package added in the delphi add-ins.

any help/ideas would be greatly appreciated, even the crazy ones. "Have you ran check disk to see if there are any problems?" nothing changed...

[Remy Lebeau 1264]

Creating a COM object to run a process won't bypass the AV if it is looking for process creations.  That will just add more overhead to the app that is wanting to create the processes.  Unless the AV ignores COM objects that run elevated, which would be pretty risky.  Besides, it's not really possible for an AV to determine whether a process creation is occurring inside a COM object vs an application anyway.  I think your IT is acting stupid, but that is just my opinion.

 

Do you really need to run an external process on every file?  What is the new process doing exactly that you can't do directly inside your own app with an equivalent library?

[RTollison 0]

sorry should have added more details as usuall. the "files" are cobol programs that have to be compiled into a cobol object so the cobol runtime can use them. our programs are broken down into sub-folders say "AA" thru "ZZ".  rather than compile every "file" every time, i wrote an app that allows me to select which folders to compile. only ones that have changes or the once a month all "files" compiled. originally i used the "WinExec32V2" to spin up a process for every file in the group. this was definitely the wrong approach when compiling all files. So i added threading and a thread limiter. then i could spin up say 50 threads with a different file in each. this would then only take about 10 minutes for all files. Then the thycotic AV came along and now this threading process takes about 12 hours for all files. the compiler program has been whitewashed. i can run it manually and the AV doesn't nag me about are you sure its safe prompts. it did before getting it whitewashed. but the thycotic AV seems to snag/delay the threads and i am trying to past it somehow. like have it check the first one and deam them all safe from there. long shot i know, but i really hate the once a month process because my pc comes to a crawl all day long. so we (myself and IT) thought/hoped that if i had a registered object that thycotic would not keep checking it over and over. i have been tinkerying with TDOSCommand to see if thycotic would let a direct command run without checking every one of them. got interupted for the past 2 days on other projects hopefully wednesday will be a good day for changes/testing.

Edited October 18, 2023 by RTollison

[Kas Ob. 91]

5 hours ago, RTollison said:

sorry should have added more details as usuall.

Yes and thank for expansion on that.

 

Just an idea to entertain, how about QEmu as emulator running old Windows or may be just a local Hyper-V hosting the compiling process, or even ship the need-to-be-compiled files to another server/PC compile then return the result.

 

A question on the side doesn't the compiler have batch mode ? also what type of the harddrive are you using (HDD/SSD)? 

[mvanrijnen 121]

Try to:

Add sourcecode folders as exclusion for AV.

Add your programs own work/temp folder as exclusion for AV.

If the compiler creates executables (or object files), make sure these are linked/copied into a folder checked with the AV. 

 

[Fr0sT.Brutal 897]

It's important to determine which exact action triggers slowdown: createprocess itself or checking of produced files. You can check this by just running the compiler in many threads with '-?' parameter.

Anyway what about drive speed? Maybe try to move to SSD or even memdrive?

[RTollison 0]

this compiler does not have batch mode (acu cobol) everything is in a whitewashed folder, IT said that anything in C;\coboldev would not be scanned (we made sure that the spelling was exactly the same) and everything involved here is in the coboldev folder. hard drive is SSD. Like i stated the only variance here is the AV. when i start running my program for all with say 250 threads, i watch task manager and i see my ccbl32.exe start up but then i see thycotic going nuts on cpu usage. i told IT no you not excluding that folder because everything of mine is in that folder. after much back and forth IT thought that the create process for the ccbl32 was doing something else that the AV was trying to monitor.

[Fr0sT.Brutal 897]

What happens if you run compiler with dummy arg (-?) ?

What happens if you run sane number of threads (10)? No sense in hundreds of threads, they only waste resources and won't give you speedup

[RTollison 0]

C:\AcuCobol_925>ccbl32 -help
COMPILER: ccbl [options] source_file

-v    Verbose compiler                 -w    Suppress warnings
-e    Error output to named file       -o    Name object file
-x    Ignore CBLFLAGS environment      -a    Show all warnings
-$    HP e3000 conditional compilation

                         *** Listing Options ***

-Li   Print general information        -Lo   Name listing file
-Lf   Print full listing               -Lx   Print extended statistics
-Ls   Print symbol table               -Ll   Set page length
-Lw   Wide listing format              -Lc   Cross-reference listing
-Lp   Print preprocessed source        -La   Split out 'cautions'

                        *** Compatibility Modes ***

-Cr   RM/COBOL(tm) compatibility       -Ci   ICOBOL(tm) compatibility
-Cb   Default organization is BINARY   -Ce   Use source/copy name extension
-Cf   Use fixed-length records         -Ca   Use ANSI ACCEPT and DISPLAY
-Ck   Indexed keys in SELECT order     -Cm   SELECT allows IBM COBOL syntax
-C3   Version 1.3 compatibility        -C4   Version 1.4 compatibility
-C5   Version 1.5 compatibility        -C##  Version #.# compatibility
-Cv   IBM DOS/VS Cobol compatibility   -Cp   HPe3000 Cobol compatibility
-Cg   Ignore conditional compilation

                           *** Video Options ***

-Vh   Default intensity is high        -Vl   Default intensity is low
-Va   Alternate ACCEPT intensity       -Vq   Quiet ACCEPT statements
-Vc   Imply CONVERT on numeric ACCEPT  -Vd   Imply CONVERT on numeric DISPLAY
-Vx   Always allow exception keys      -Vb   Treat BLANK LINE as BLANK EOL
-Ve#  Set EXCEPTION mode               -Vu   Imply UPDATE on ACCEPT
-Vi   Use item color on erase          -Vg#  Do not propagate COLOR

                         *** Data Item Options ***

-Db   Treat COMPUTATIONAL as BINARY    -D2   Treat COMPUTATIONAL as COMP-2
-D6   Treat unsigned PACKED as COMP-6  -D1   Put PIC 99 binary in one byte
-Ds   Imply SIGN IS SEPARATE phrase    -D5   Treat BINARY as COMP-5
-Da#  Align data on #-byte boundaries  -Dm   Minimize binary data size
-Dl#  Set SYNC boundary limit to #     -Dy   Imply SYNC for all binary data
-Dz   Relax size-checking rules        -Di   Initialize WORKING-STORAGE
-Df   COMP-1/COMP-2 are FLOAT/DOUBLE   -Dw#  Set maximum target word-size
-DCa  Use ACUCOBOL numeric formats     -DCi  Use IBM numeric formats
-DCm  Use Micro Focus numeric formats  -DCn  Use NCR numeric formats
-DCb  Use MBP numeric formats          -DCr  Use Realia numeric formats
-DCv  Use VAX/COBOL numeric formats    -Dx#  Set size of index data items
-D7   Micro Focus binary data size     -Dd31 Allow 31 digits in numeric items
-Dv=# Default value of data            -De   Assume linkage is not aligned
-Dq   Treat QUOTE as APOSTROPHE

--fastRefMod             Use optimized method for reference modification
--fpRounding=[OSVS|VSC2] Emulate OSVS or VSC2 style rounding
--lastWSDataSeg=#        Number of data segments used for WORKING-STORAGE
--noAlignLit             Do not use default algorithm for aligning literals
--noTrunc                Binary items not trimmed to picture on store
--truncANSI              Don't truncate stores in COMP-5

                         *** Debugging Options ***

-Ga   Include everything               -Gy   Include symbols (-Zs)
-Gd   Include source (-Zd)             -Gl   Include line numbers
-Gs   Include extra symbol info        -Gz   Prevent program from being debugged

                       *** Miscellaneous Options ***

-Zi   Imply IS INITIAL PROGRAM         -Zn   Turn off local optimizer
-Za   Check array bounds at runtime    -Zg   Enable segmentation
-Zc   Optimize for space over speed    -Zz   Treat spaces as zero in numbers
-Zr0  Disallow recursive PERFORMS      -Zl   Create large-model program
-Zy   Use 4-digit year in DAY/DATE     -Zf   Create #define file
-Z3   Restrict code to version 1.3     -Z4   Restrict code to version 1.4
-Z5   Restrict code to version 1.5     -Z##  Restrict code to version #.#
-Zw   Prepare for workbench import     -Zm   Include ADDRESS OF memory sizes
-Ze   Include XML symbol table

--acceptRefresh          Use current value of variable in screen ACCEPT
--arithmeticVSC2         Truncate according to the rules of VS COBOL II
--binaryMath (--bin)     Use binary math when able to (default for 7.3+)

--brand nnn              Brand the object with serial number nnn
                                                                      --decimalMath (--dec)    Always use decimal math package (pre-7.3 default)
--newARC                 Use relaxed Abbreviated Combined Relation syntax
--relaxRBUsing           Allow radio button USING to be indexed data items

                          *** Mapping Options ***

-Mp   List all paragraph names         -Mr   List all reserved words
-Ms   List all opened source files     -Mv   List all variable names
-Mo   Specify an output file           -Ml   List names in lowercase
-Mu   List names in uppercase          -Mm   List names in mixed case
-Mc   List all control properties      -Mw   List screen section info
-Me   List all entry points            -Mx   Create XCDs

                           *** File Options ***

-Fs   Implied START TRANSACTION        -Ft   Transact all file operations
-Fl   Single lock mode default         -Fo   Set directory for XFD files
-Fx   Create XFDs for indexed files    -Fa   Create XFDs for all file types
-F#   Create version n XFDs [n=3,4,5]  -Fc   Preserve case of field names
-Fe   Create old format XFD files      -Fv   Include comments in XFD files
-Fp   ALL files are OPTIONAL           -Fm   Default lock mode is manual
-Fn   No file sharing by default

--fileAssign=[DYNAMIC/EXTERNAL] Specify default file assignment
--fileIDSize=[2/4/8]            Specify size of file handles passed to COBOL

                          *** Source Options ***

-Sa   Force ANSI source format         -St   Force terminal source format
-Si   Include lines based on pattern   -Sx   Exclude lines based on pattern
-S#   Set tabs every # columns         -Sr   Use RM/COBOL compatible tabs
-Sd   Include debugging lines ('D')    -Sp   Specify COPY search path
-Sc   Specify code system              -Sl   Force long line source format

                          *** Warning Options ***

-Wa   Warn on misaligned using         -Wl   Warn on non-01 level using
-Wr   Warn on 01 level redefines too long

                        *** Object-Code Options ***

-n                       Generate native code for host
--ia-32, --intel         Generate Intel IA-32 code
--pa_risc, --pa          Generate PA-RISC 1.0 32-bit code
--pa_risc_2.0, --pa2     Generate PA-RISC 2.0 64-bit code
--powerpc, --ppc         Generate PowerPC 32-bit code
--powerpc_64, --ppc64    Generate PowerPC 64-bit code
--power                  Generate POWER-compatible 32-bit code
--sparc                  Generate SPARC 32-bit code
--sparc_v9               Generate SPARC 64-bit code
--noInlineCall           Turn off inline CALL optimization

                      *** ESQL Pre-compiler options ***

-Pc   Connect to database to check SQL -Pd   Specify database for connection
-Pi   Specify SQL include directory    -Pk   Specify keyword set
-Pp   Specify database password        -Pr   Relaxed syntax checking
-Ps   Precompile and compile           -Pu   Specify database user name
-Pw   List keywords                    -Pv   Relax rules for VARCHAR items
-Pg   Preprocess using next argument   -Pe   Include ESQL in listing & debug

 Additionally, the following commands may be used (as described above):
       -e filename; -Sa; -Sd; -Si; -St; -Sx

 The output will be sent to stdout unless otherwise specified

                      *** Reserved Words ***

-Rr   Suppress RM/COBOL(tm) words      -Rv   Suppress VAX/COBOL(tm) words
-Ra   Suppress ACUCOBOL(tm) words      -R8   Suppress 1985-standard words
-Rs   Suppress Screen Section words    -Rw   Suppress individual word
-Rc   Change reserved word             -Ri   Suppress ICOBOL(tm) words
-Rx   Ignore any particular word       -Rn   Make synonym for reserved word
-R2   Suppress 2002-standard words

                      *** Error options ***

-Qm   Max error count                  -Qp   Skip to next period on error

                      *** Conditional compile options ***

                                                                      /CONSTANT name value                   /[no]directive

                      *** Internal table options ***

-Td # Identifier and statement table   -Te # Subscript statement table

                      *** Interoperability options ***

--javaclass              Generate a .java class to call this COBOL program
--javamain               Generate a .java main to call this COBOL program
--netdll                 Generate a .net DLL to call this COBOL program
--netexe                 Generate a .net EXE to call this COBOL program

INFORMATION: cblutil -info object_files ...

LIBRARIAN: cblutil -lib [options] object_files ...

-o    Name new library file            -v    Verbose librarian
-r    Remove object files if okay

CODE TRANSLATION: cblutil -native [options] object_files ...

-o                       Name output file
-v                       Verbose translator
-Zc                      Optimize for space over speed
-Zn                      Turn-off optimizer
--ia-32, --intel         Generate Intel IA-32 code
--pa_risc, --pa          Generate PA-RISC 1.0 32-bit code
--pa_risc_2.0, --pa2     Generate PA-RISC 2.0 64-bit code
--powerpc, --ppc         Generate PowerPC 32-bit code
--powerpc_64, --ppc64    Generate PowerPC 64-bit code
--power                  Generate POWER-compatible 32-bit code
--sparc                  Generate SPARC 32-bit code
--sparc_v9               Generate SPARC 64-bit code

RUNTIME: runcbl [options] [ object_file [parameters] ]

-d    Run with debugger                -r    Run debugger from script
-e    Direct error output to file      -ee   Redirect stderr to file
-w    Suppress warning errors          -s    Prevent user aborts (safe)
-c    Name configuration file          -l    List configuration file
-v    Print version number             -#    Set switch to ON
-i    Set input to file                -o    Set output to file
-a    Name assembly library            -y    Name COBOL library
-x    Show extended error messages     -h    Ignore hangup
-b    Background processing

[mvanrijnen 121]

i think @Fr0sT.Brutal meant, to run it without real comping, so you can see if delay is in starting the Exe itself of the compiling process.

 

[RTollison 0]

Sorry if you just type the ccbl32 with anything other than -help or with all the actual parameters it just throws up the -help option. no compiling. 

in my threading the default number is 10 but i wanted to see what was going on so i ran it up to 50 and that was when i noticed the thycotic.exe was going nuts as well. 

since creating a COM or whatever seems out of line i can tinker with process and TDOSCommand to create a fake threading. since the doscommand has an onterminated events then i can just spin up that many processes and let them run and decrease a counter when done. i should be able to make that work.

[Fr0sT.Brutal 897]

33 minutes ago, RTollison said:

if you just type the ccbl32 with anything other than -help or with all the actual parameters it just throws up the -help option. no compiling.

Yes! And will it cause slowdown if you run 50 of such processes in threads?

Quote

in my threading the default number is 10 but i wanted to see what was going on so i ran it up to 50 and that was when i noticed the thycotic.exe was going nuts as well. 

So when there are 10 threads, things do work?

Edited October 18, 2023 by Fr0sT.Brutal

[RTollison 0]

at the normal 10 threads it takes the 12-14 hours. the thycotic jumps to like 80% of cpu and stays around that percentage throughout the process. so it brings my system to a crawl because overall 100% of cpu stays pegged. from a cmd prompt if i execute the compile command with all the parameters it takes less than 1 second for most cobol programs a few take 3-4 seconds (lots bigger programs). i put 52 compiles into a batch file and they completed fairly quickly(< 1 minute). but those same programs in my mass compile program took 6 miniutes to finish. so for whatever the reason thycotic goes nuts and eats a bunch of cpu time which slows down everything. maybe i will just create a batch file with everything to compile with all the proper parameters and then just run the batch file.  another code change but will see what happens. that would be faster overall but would go back to 1 compile command at a time. but still better than the 12 hours

 

[Fr0sT.Brutal 897]

Yeah, probably AV considers cmd.exe as trusted so doesn't check carefully what exactly it does.

You can try to run cmd once and feed commands to it via STDIN just like if they were entered in console. Use pipes for this.

This sample works in console as expected:

>(echo dir && echo dir) | cmd

You don't even have to check STDOUT - just write to STDIN all the commands in queue. cmd will block the pipe when busy

Edited October 19, 2023 by Fr0sT.Brutal

[JonRobertson 39]

On 10/18/2023 at 11:32 AM, RTollison said:

i put 52 compiles into a batch file and they completed fairly quickly(< 1 minute). but those same programs in my mass compile program took 6 minutes to finish

I came across this while searching for something completely different.  Although an old thread, I have a suggestion (if you have not already tried):

 

Change your program to create .bat files in C:\coboldev to do a batch compile, say 50 compiles per batch file. Launch cmd.exe and pass it the name of the batch file. Monitor for completion, as you mentioned previously. Or have cmd.exe exit upon batch completion (cmd.exe /c comp01.bat) and wait for the created process to end. Perhaps have 10 cmd.exe processes running simultaneously performing batch compiles. And delete each batch file once the process completes.

 

This may solve the slowness cause by the AV scanner, if it is having fits about separate processes being created for each launch of ccbl32, but not when launched several times by a single process.

 

You could also try using a language/tool more powerful than cmd batch, such as PowerShell or Python, to prompt which folders to compile and launch ccbl32 for each file. Or roll your own CI by comparing timestamps of the source against the most recent compiled module. Or try a tool like Jenkins. Lots of options.