David Schwartz

I'm looking more closely at the Omni Thread Library ... it seems to have more to offer than the what's in Delphi. Just no documentation or help files.

Well, thanks for the effort. 🙂

I don't think you're understanding the question. When you do most DB operations, there's an OnxxxCompleted event that's called. With tasks, there's ... nothing. They just stop. When each task finishes, I want to update the line in the ListView that says it's Finished and show the Elapsed Time. Not when the entire batch is finished. When each one is finished. WaitForAny just says at least one of them has finished, not which one. WaitForAll says they've ALL finished. How do I know when each one finishes, without polling their current state?

I just pulled these two out of the air. Clearly you have more considerations than you're sharing. You said you're "monitoring" perhaps 1000s of nodes. How? In real-time? Are they being sent automatically into a central place in a big stream or do you have to poll each one? Are you needing to buffer them or save them, or just sample them and toss them aside? Or are they being dropped into a database for later analysis? Is there a time-frame on the analysis window? Also, what's the interface? You mention low-level hardware peeking and poking. You might be doing that with 1000's of devices scattered around a lab, but not if they're scattered around the city, state, country, or the world. So how is the data being delivered? There's a guy who posted something recently and was wondering if he could use Delphi for it. He has some remote sensor devices that wake up every hour, measure the temperature, humidity, wind speed and direction, take a photo, put them into an email and send it off, then they go back to sleep. They only look for remote commands when they wake up briefly. (It turned out they're solar-powered devices scattered all over SW Australia.) That's a lot different than having things that you can interact with at any time and make explicit requests of. His only real requirement was, can I read email and extract the data from the messages? He didn't even say that, but that's what it came down to. You could be using the exact same sensors he was using and sending all of your data back as emails as well. Pretty much anything can read emails and extract data from them. So why limit yourself to just two languages and platforms? (He kept coming back to using "email", and I was really puzzled why. I got him to send me info on the remote sensing devices, and they gave an option to send data via MQTT or email, and they chose email. Well, ok then. Email it is.) I built a system once where we had a box that received weather satellite images. It was sent to a Centronics parallel port intended to be connected directly to a printer. That's how the images were delivered. I had to hook that to a parallel interface, read the data, re-rasterize it, convert the data to a bitmap image, and save it into a DB. We also got text data from an RS-232 port on the device; we had to parse it, analyze it, and put it into the DB as well. (Weather data published by the US National Weather Service is very cryptic, if you've never seen it.) When we printed the satellite images at full resolution (as received) on a laser printer, they were about 1" square, which wasn't very useful as-is. ALL of the data was valid for a limited period of time. The text data contained start and end times for the vaidity of most of the messages; the images were sent in fixed time slots and their validity depended on when they were sent. Once data was expired, we deleted it. We needed to run GIS queries and pull data out of the DB based on a flight plan, not the data. At the time, I used C++, but it could have just as easily been done with Dephi or C#. There's a LOT of stuff you aren't saying here that makes it nearly impossible to answer your question with any precision. But in general, both choices are likely to be sufficient, in absense of any further details.

On a RPi or Arduino?

Take a look at what TMS Software offers in their FNC library. That may give you some ideas as well. 🙂

I wasn't aware of that, but every time I've asked I've been told it's extremely difficult. I'll watch it and see what I've missed. I used to do embedded stuff in C / C++ (Borland C++ to be exact) but never anything with Pascal. Delphi is designed to make programming Windows very easy. Embedded systems require a lot of scaffolding that Dephi doesn't really offer, mainly because you don't need it for Windows. If you stick to non-GUI apps, then it might work fine. I didn't abandon C/C++ embedded programming to get into Delphi so I can double the amount of work I needed to do. 🙂 However, as remote services become increasingly popular, they start to resemble distributed embedded systems. So I envision a time in the not too distant future where it will be fairly simple to create a REST-based service in Delphi that can be stuffed into a RPi and similar hardware with very little effort. It's easy to deply stuff to Windows boxes, not so much with Linux yet. And you still need a lot of RAM and storage. The problem with Dephi and Linux is all of the various platforms that need to be targeted. Maybe it would be easier to treat these the way I did way back in the day and build an image that included the entire OS along with the app code. You burned that into ROM or EEPROM and then turned the machine on and tested it. Why not do that with Delphi instead of having to worry about targeting an EXE to some pre-existing Linux distro that may or may not have all of the required libraries and support stuff needed? Just build an entire image, upload it, then reboot the target machine.

Tools are tools. Say a little bit more about the target environment and what you need to work with. You cannot use Delphi to program, say, Raspberry Pi's and small embedded controllers. TMS has a way to control RPi's that may or may not fit for your needs (they support MQTT as well). BLE support is built into Delphi. If you have a headless embedded system that runs Windows, you can use Delphi to build stuff that can be controlled remotely, much like REST-based services. If you have 1000+ remote "smart sensors" that are sending data back to a data collection server, then Delphi can be good for managing the collected data and as a "Dashboard" of sorts to control everything. But maybe MatLab or R would be best for your needs. So far you haven't really said what your requirements are.

Switch to VirtualBox perhaps?

I'm trying again with this, and am still getting the same errors. There is a flag in compiler settings that is checked globally. (See attached image that shows it's checked.) The FAQ and several other places say this is related to the compiler thinking that I'm using Runtime Packages, which I AM NOT. (See second attached image below.) The HELP info on it just says this: To alleviate the problem, it is generally easiest to turn on the $IMPORTEDDATA switch and recompile the unit that produces the error. The problem is, I'm not sure what unit is producing the error. I'm doing a BUILD ALL. It's just a compile error: [dcc32 Error] E2201 Need imported data reference ($G) to access 'IsMultiThread' from unit 'FastMM4' I put this at that at the top of my .dpr file, and the top of the main .pas file, made no difference. (*$IMPORTEDDATA ON*) <<---- unit u0; I set this option in the .inc file, which should declare the needed options in the FastMM4 unit: {Set this option when you use runtime packages in this application or library. This will automatically set the "AssumeMultiThreaded" option. Note that you have to ensure that FastMM is finalized after all live pointers have been freed - failure to do so will result in a large leak report followed by a lot of A/Vs. (See the FAQ for more detail.) You may have to combine this option with the NeverUninstall option.} {$define UseRuntimePackages} {-----------------------Concurrency Management Options------------------------} {Enable to always assume that the application is multithreaded. Enabling this option will cause a significant performance hit with single threaded applications. Enable if you are using multi-threaded third party tools that do not properly set the IsMultiThread variable. Also set this option if you are going to share this memory manager between a single threaded application and a multi-threaded DLL.} {$define AssumeMultiThreaded} Again, made no difference. It simply refuses to build. Both 4.992 and 4.993. I've never had a problem before Delphi 10.4.2.

I'm not sure what this has to do with the 486, but I would not use that solution as it can be interrupted and its state changed. The whole purpose of critical sections is that they extend a simple atomic operation to span several instructions by using a protocol that can be safely interrupted. The only other option is to disable interrupts, but even then the CPU usually has a "high-priority interrupt" that can still interrupt the execution flow. This is exactly how some of the hacks worked that used the look-ahead cache on Intel's CPUs a few years back. We were creating something that had no OS support. Turbo Pascal (and later Delphi) are designed to work inside of DOS and/or Windows. Both of these OS's supports critical sections, but they're only valid for stuff running entirely within the OS. When you're reaching out across a bus (or, today, the internet) to an asynchronous process in an unknown environment, all bets are off. Anybody who remembers the TSRs that ran in DOS knows they did not run within the OS context (even if you consider DOS an actual OS). Nor did most of the device drivers for DOS, or even Unix for that matter. We were working with a fully interrupt-driven OS where everything was managed within the OS. What this project did was gave us a way to extend a reliable communication channel across a data bus, sometimes through shared memory, with a peer on the other side, without having to know anything about what was running on the other side. We actually based it on the first 3 layers of the ISO networking model. By the time the 80486 family hit the scene, the boards using smaller, slower CPUs were retired, and the hardware was updated to make this easier. I'd left the company by that point. (laid-off)

Thanks for the correction. I guess it was the 8008 with the multiplexed 8-bit Addr bus. It has been a long time since I looked at any of that stuff. The 8085 was a bit odd. It had a bunch of additional instructions added to it that were more-or-less duplicated on the Z80, but at the last minute they decided not to publish them. They were implemented in the first couple of production runs, but then they were removed. So only a couple of new instructions showed up on it that were mainly for controlling something.

This is a bit OT, but I'm curious about something. TMS WebCore now has a Miletus option that lets you build apps that run on Raspberry Pi hardware. I don't have a complete picture of what the Miletus wrapper does, but it's very similar in function to what Electron does. My notion of it is that it's not much more than a web server that serves up an app written in Delphi that has been translated to javascript -- you have an HTML page that has a SCRIPT tag in the header that loads it into the browser where it runs. If you did this more conventionally, it would be something like, say, a python scriopt running in a REST service that was controlled by a remote app. I don't know if you'd need a separate REST server running with a Miletus-hosted WebCore app. But I guess the part in the browser communicates with something on the other end that allows you to "write-thru" to control things on the device. A use-case that springs to mind most immediately for me is using it to implement some kind of "admin" panel for whatever is running on the device. These devices are generally headless anyway, so it solves that problem. Eg, you could have an app written in whatever that controls something, and you might want a UI-based admin panel to adjust settings. This would be perfect for that purpose. (Think of connecting to your router or printer directly to adjust settings on them.) If the device in question has BT or a WiFi connection, then it could communicate with and control other remote devices, and also control stuff directly using its built-in I/O pins. I'm just curious what other ideas for use-cases others might have for controlling IoT devices via a remote app like this running in a web browser? Note that the "admin" aspect also implies (to me) using it to control things in your home that IoT is more known for, so you could build an entire control panel for all of the "smart" devices in your home. What else?

Well, if you look at everything Delophi produces as a "rich UI" and dismiss using Delphi to build even "dumb forms" then I guess your "classic nginx+PHP+user-side static pages" is going to look much simpler. I built this in WebCore in about an hour. It's far from what I'd consider a "rich UI" although there's a lot that could be done to spiff it up. It's not the kind of thing I'd use WebCore + Miletus to build for an IoT device, but it shows the power available using Delphi to build browser-based web apps. http://bestkeywordmixer.com The JS file for this is 1.8 MB in size (debug version), which is comparable to the size of the standalone Delphi EXE. The Miletus core is just shy of 8MB. Their example demo app is a weather thing that displays the temperature, humidity, and barometric pressure on your device. They built an iOS app to display it in an iPhone to demonstrate the REST API, although it would work just as well running it in the browser. I don't know ... maybe it could be used to remotely turn some lights on and turn up the heat in the house as you're leaving work to come home, then open the garage door as you approached. I don't find that terribly useful. However, if you're doing data collection and/or running some kind of equipment from remote devices, it could be very helpful to allow you to manage them remotely.

The web-app runs as a js script inside the browser. If you close the browser then reopen it, or navigate away, it's gone. Nothing is persistent. And nothing is running on the site where it was loaded from. However, there does seem to be a back-door they provide on the server it came from that's embedded in the Miletus platform that lets the app running remotely connect with resources on the server. I imagine they will eventually open it up, but for now it's not. They're still probably making changes that they don't want to proliferate by opening it up early. I don't know diddly about Electron other than some stuff I read that indicates it's extremely bloated with lots of stuff that's not needed all the time. Miletus was inspired by Electron, but is apparently quite slimmed-down. Very innovative IMHO!

I'm not all that familiar with JSON, but looking at this and what you're saying, I'm wondering if this shouldn't be structured as an array? "a_lbl1": [ {"leads": [{json}]}, {"leads": [{json}]}, {"leads": [{json}]} ]

It's just a payload loaded by the HTML page -- a chunk of javascript and maybe some images and icons, which could just as easily be hosted elsewhere if need be. Funny that I don't see comments like this about the plethora of js libs that average web pages haul in just to make pretty buttons and combo-boxes. Not sure the difference between a "complex UI" and something less complex is going to be significant given the amount of scaffolding needed to host the VCL. That issue aside, it still doesn't answer my question about what use-cases you can think of for this technology on IoT devices.

Sorry, I don't see any. Bruno said it's a hair under 8MB. But I don't know if that includes the code needed to emulate the VCL library.

fixed. thanks. Valid question, but this isn't my product. They built it because their users asked for it. Maybe we're saying similar things. I'm curious what use-cases there are besides admin panels. That said, if you're proficient in Delphi, it may be much faster to build something in Delphi than a static web site builder. And if there's any logic required behind the scenes, then the complexity of the traditional approach just doubled or tripled because you'll need to use another language (eg, js, python, php) for that purpose.

I'm reading this debate and it triggers a lot of memories from a project I led in the first job I had out of college. I also searched for that TInterlocked.CompareExchange function and found quite an interesting discussion on SO that reminded me that we're really lucky there was a bug in Intel's 80286 chip that killed their protected-mode OS efforts and actually derailed their whole segmented memory architecture. I hope some of you enjoy this. For the record, this all happened in the 1979-86 time-frame. There's also something useful here related to this thread. ================================================================ In my first job out of college, I was hired to work at an Intel facility that created and built their line of Single Board Computers (SBCs) that used their MultiBus backplane. (It was an industrial design; most people are familiar with the analogous but much simpler S-100 bus that IBM introduced in their PCs.) I was assigned a simple project one day: to write a driver for their 8-bit real-time embedded OS (RMX-80) that ran on an SBC with an 8-bit CPU (8080 or 8085) and talked with another SBC that simply had four UARTs on it (UARTs are 8-bit serial communication devices, aka "comm ports") used for RS-232 connections. The code itself had to fiddle with things via the IN and OUT instructions because Intel's chips infamously did not do memory-mapped IO like Moto's 6800, or the 6500 used by Woz in the first Apple computer. I talked with a hardware guy and was assured this was safe to do because the MultiBus ensured one clock cycle atomicity. So we could have an 8080 or 8085 board with any number of these 4-port serial controllers plugged in, and run IN and OUT (single-byte I/O) instructions to get and fetch data from/to each UART on each serial card using different 'ports' and there would be no need to deal with locks. YAY! One weekend, some guy in Marketing started noodling around and made a grid with all 27 of the boards we made and that were in development listed along the X and Y axes, and in the intersecting cells he made an 'X' if it made sense for the two boards to communicate. Then he decided it might be a Good Idea to make this a more generic solution and took it to a Product Planning Committee. A couple of weeks later, I was informed that my "simple" project had just been considerably expanded. It was renamed "Multibus Message Exchange" or "MMX" for short. And yes, if "MMX" seems familiar, it was recycled years later for something completely different. The original thing got absorbed into their OS designs and eventually became obsolete. Here's where it gets interesting (and quite boring if you're not familiar with computer hardware)... We had SBCs that had a variety of CPUs on them: 8080, 8085, 8086, 8088, 80286 (in development) and 80386 (still being defined). Each CPU has two busses: an Address bus and a Data bus. The 8080 and 8085 had 8-bit busses for both. The 8086 and 8088 had a 16-bit Addr bus; the 80286 had a 20-bit Addr bus; and the 80386 was planned to have a 32-bit Addr bus. Meanwhile, the 8080, 8085, and 8088 had an 8-bit Data bus; the 80286 had a 16-bit Data bus; and the 80386 had a 32-bit Data bus. The Multibus had both 16-bit Data and Addr busses. With the pending design of the 80386, they widened the Multibus to support 32-bit Data and Addr busses. Some of the boards also had 8048 MPUs on them (the chips that were in early PC keyboards) that were also 8-bit devices, eventually replaced with 8051 chips. Also, some had "shared memory" on them, which was a big block of memory where the address on the outside looking in (from another board with a larger address space) was almost always different than what the code running on that board's CPU saw on its end. If you're keeping score of combinations, we're at a pretty big number for such a simple idea. This thing that started out as a simple little project now had to support the ability to send messages from one OS on one CPU to a possibly different OS on a different CPU; sometimes using shared memory that had a different address range on either side; where they both could have different sized data and address busses; and the MultiBus backplane itself could not be locked for more than one clock cycle. They were working on a "smart 4-port UART" board that had an 8085 controlling the four UARTs and 16 KB of shared memory. The 8085 had a 16-bit (64KB) address space, but the OS ran in a ROM that was hardwired to the first 16KB because when you grounded the RESET pin momentarily, it reset the CPU and set the IP to start executing at address 0. Needless to say, when you had a 16-bit or 32-bit data bus on one side and the other only had an 8-bit data bus, this caused a problem because the data could only be seen one byte at a time on one side, but it was 2 or 4 bytes wide on the other. It was also impossible to pass pointers between boards with CPUs on both sides. This made memory-mapped I/O highly problematic, which I think is one reason Intel went with the IN and OUT instructions instead. This mess was due in part to the fact that the MultiBus only had one clock cycle atomicity. We could not write 16-bit or 32-bit values to a shared memory address and ensure they would be received without getting corrupted because the shorter receiving side had to do multiple reads, which toggled the MultiBus' clock with each read. (Until this point, the engineers had no reason to design an 8-bit board to read 16- and 32-bit wide data and multiplex it into a series of 8-bit data bytes.) <sigh> If anybody remembers the old "small", "medium", "large", and "compact" memory models in old Intel compilers, this is part of what those designations were intended to address. It was a frigging nighmare. Those model designations first showed up while I was working on this project, and we got an updated compiler with a note about this apologizing that it was the only way to deal with the segmented memory model in the 80286 other than forcing everybody to use the same model, which was totally unworkable. We BEGGED the engineers in charge of the Multibus to add an explicit LOCK signal to it so boards with mis-matched data and/or address busses could have something to ensure that an INTEGER of ANY WIDTH could be transferred atomically across that damned bus that they loved to extoll was "the most advanced bus in the industry!" We succeeded in getting the Product Planning Committee to take it up for discussion, and they agreed, but they chose not to go back and fix any existing boards -- it would only be implemented going forward. Meaning we were stuck having to deal with that mess on all of the existing board combinations they decided needed to be supported. Ugh. --------------------------------------- I did a ton of research and stumbled upon something that someone at IBM had just published in one of their journals that seemed to anticipate our exact situation. It was a short article that simply explained a reliable way of building a lock or semaphore without any hardware support. All it required was an atomic TestAndSet that worked on both sides with at least one bit. We had atomic byte-sized TestAndSet operations that worked everywhere, so that's what we used. TestAndSet reads the value of a memory location and returns its value, and writes a 1 into that location, in a single atomic uninterruptible action. Most semaphores issue a lock, try to grab a flag, then either proceed or release the lock and try again. In this case, a lock isn't needed. It's done through a protocol that everybody follows. You do a TestAndSet, and if the value returned is 1 then you sleep for a bit and retry until you get it. If you get a 0, then you do it again on a second location. If you get back a zero the second time, then you have a green light to proceed. If not, you clear the first flag, sleep a bit, then try again with the first flag. When you're done, you clear the second flag, then clear the first flag. For us, this was like cutting the Gordian Knot! (The paper went into a bunch of details about why this works realiably. I don't recall them.) ========================================================================= PS: if you're read this far, I'll let you in on a little secret related to this that very few people know about the history of the 80286 and the bug that kept it from working in protected mode... The 80286 architecture was created by some guys from Honeywell that were trying to build a simplified version of their Multics secure computing system. Part of the magic was that the OS had some support in the hardware. Our OS guys worked closely with the chip designers to ensure that the 80286 chip supported all of the magic stuff that the OS needed to be secure. They had a big meeting and everybody signed off on the chip design and the chip guys went off to build it while the OS guys started working on the fancy new OS. I was hired a few months after that meeting to work on that very OS. Our hardware guys had been working in parallel to design a MultiBus SBC that they could drop the 80286 chip into and it should just work. For new chip designs, Intel would first create an In-Circuit Emulator (ICE) that they'd use for hardware testing. We got some of them and they worked properly as expected. But when we finally got the first prototype chips, they failed some tests. We sent the results back to the chip guys and were told "we're working on it" for months. We got back a couple more iterations of 286 chips and they fixed some problems, but one seems to persist. Apparently, after the final meeting and agreement on the chip design, the chip guys built the ICE units based on that design, but then decided to optimize something in the microcode and made some changes in the on-chip wiring that simplified a bunch of other logic, and they never ran it by the OS team. They also didn't update the ICE unit's design to reflect these changes, believing they were simple "refactorings" that would have no side-effects. They kept dodging the OS team's attempts to meet and help with this. When the OS team finally got their hands on the T-Spec, they went ballistic, because the chip guys basically broke the security model with their "refactoring". Seems that what they did caused the CPU's Stack PUSH instruction to be non-atomic when the chip was running in "protected mode". Meaning that if an interrupt came in while an address was being pushed onto the stack, it could be interrupted after the segment register (16 bit base index) was pushed onto the stack, but not the 16 bits (offset part) of the IP -- because they came from different registers. The result was that when the interrupt popped the return address off the stack, it got the segment register value, but the next word was NOT the IP address. Obviously, it went off into the weeds and typically generated an address fault. Unfortunately, they decided they couldn't fix it without delaying the chip's release by 6 months or so. That was getting too close to the release of the next chip, the 80386, so they decided to accelerate the release of the 80386 instead and wave-off use of "protected mode" in ALL 80286 designs. (Customers were not happy!) In fact, IBM and some other companies (besides us) had been working very diligently to build a secure OS that was supposed to run on the 80286 in "protected mode". But the 286 never worked in "protected mode" because they never fixed this problem. When we released the 80286 chip, IBM and others sold an upgraded OS that ran on it, but not what they had planned. The 80386 was released earlier, but nobody ran it in "protected mode" either because that required the segmented memory model which everybody HATED. The 80386 was supposed to have a bunch of new features, and only ended up with a few in order to get it out the door faster. But the one that captured the industry's attention the most was its support for a "flat memory model". That was the point where Intel threw in the towel and shifted entirely to a "flat memory model", doing away with all of those crazy "small", "medium", "large", and "compact" compiler models and all of that nonsense that came with a segmented addressing scheme. (Never mind that it was actually modeled after IBM's mainframes, which is one reason IBM loved its design.) Ok, now get back to work!

Brain fart. Sorry. 😕

why the ARM version? I mean ... why not the Intel version? I'm curious if there's much of a performance difference.

I have a list of REST API queries going to the same service and I want to be able to run them all in parallel. I've been told there shouldn't be any problem running up to 100 or so in parallel on this service. Are the Delphi TRESTClient / TRESTRequest / TRESTResponse components thread-safe? If so, can I create just one TRESTClient and then multiple Request/Response pairs linked to it? Or do I need duplicate Client objects as well? If not, is there something better to use? I'm guessing I'd run each synchronously in its own thread and wait until they all returned to continue, because the TRESTRequest does not appear to have an async option, right? If the data collected in each thread is in a TStringlist, then is it safe to save it to disk using a unique name with sl.SaveToFile(...)? (There's no contention for the files until after they've all completed.)

I notice that OmniThreadLibrary 3.07.9 (10/26/2021) has added this: [HHasenack] Added Cancel and IsCancelled to IOmniParallelTask.

I was mainly referring to the Delphi IDE, which only runs under Windows. The other stuff from Embt doesn't interest me. They can't deliver a stable IDE in Windows; I certainly don't trust that their cross-platform tools are any better. Not hitching my wagon to them. No thanks! Personally, I'm far more interested in using something like TMS WebCore for cross-platform development than native tools at this point, mainly because I've been watching how slowly Delphi as been getting updated and the inevitable bugs that show up after any major changes. TMS is way more responsive, and javascript seems to be far more stable running in all web browsers than trying to keep up with every update of mobile operating environments. Yes, it might be a bit more limited, but I'd rather invest in a more stable and consistent UX/UI than constantly dealing with what Delphi's native platforms offer. There was a point in my life where I *LOVED* working with beta copies of software and chasing down weird issues, but not any longer. I want to just get in, get something done, and move on. I guess I have come to value getting stuff out of my head as fast as possible without having to take a bunch of detours to deal with potholes in the road. My memory is getting flakey, I have trouble remembering names of things, and each of those detours takes me longer to regain focus when I get back to where I was.