Collect JSON ajax data from a local zigbee web server

[dan27125 0]

Greetings from St Louis, Missouri USA

 

I'm using and learning C++ Builder 11 professional.  I want to make a windows app that displays telemetry data from a zigbee gateway on the local internal subnet.  It sends periodic telemetry base64 data via (I'm guessing...) jason/ajax.  The zigbee gateway has a web page "http://" + ip + "/debug.html" that displayes the zigbeeData in a scrolling window. That scrolling windows is the only data that I want to to capture.  

 

I'm wondering if there are a few drop in components I can use to setup / collect json/ajax pairs from a web server on the local subnet ?  

What is the easiest way to setup a pipe to collect this (presumably JSON) zigbeeData ?  

What Rad Studio components should I look for ?

 

I know how to process the base64 data.  I wrote the embedded zigbee code which does that part.  This "zigbee gateway" I have all the source code for it.  Its a microchip ethernet gateway with a mini TCP stack.  Its handing it off to a mini web server that I did not write. Attached is ajax.xml, debug.html (what I did not write) and a picture of a web browser displaying the base64 strings in a scrolling window.   I just need help with how to use C++ Builder 11 professional to it collect it from "http://" + ip + "/debug.html" 

I'm a C/C++ software engineer with many years experience doing low level C/C++ programing.  I've been using Borland,Inprise,Radstudio C++ for high level programming and then lots of embedded compilers for embdedded apps.  I'm very familar with TCP and berkley sockets.  I don't know much about Web gets, puts, json or java. I presume some sort of layer uses a TCP port to send/receive packets.   I know how to find the local IP address of the gateway using Dns GetHostName API. I presume the IP will be "pluged" in to something. Hopefully a RAD Studio component that I drop on a form, define the json fields/pairs and then set the ip address.  Do you have any suggestions, tips recommendations ?

Best regards

Dan

ajax.xml

debug.htm

[dan27125 0]

Update:  I'm the person that posted this.   I'm not sure how to update my post so I'm replying to myself for now....  I just initially learned about the existence of the Rest API and Rest Debugger.  I tried using the Rad Studio Rest Debugger.  I put in the local URL (the same you plug into a browser to view the data).  I ran execute and gets back some information.  It doesn't display the data.  It doesn't seem to be in json format.  Obviously I'm new to using Rest and this Rest Debugger.  Maybe I'm not using the right way?  Maybe someone with more experience could look at the two files I uploaded ajax.xml and debug.htm?  Tell me what format the data is?  From a web browser It appears as a scrolling window.  I wonder if the web browser as a debug or information window that will tell me about the format being sent and displayed?   

[Brian Evans 83]

Postman is a popular tool to call and test REST APIs during development. It is not specific to Delphi. Once you know the calls and responses from the APIs you can work on accessing them in Delphi. 

 

To see the underlying REST API used by the web page use developer mode/tools in your browser. Refresh the page with developer tools open and check the network tab to see the requests and responses made. 

[Remy Lebeau 1260]

@dan27125 I commented earlier on your same question on StackOverflow:

https://stackoverflow.com/questions/77868076/collect-json-ajax-data-from-a-local-zigbee-web-server

 

Now that I see this post, the StackOverflow post appears to be a direct copy/paste of this post? That would explain why it mentions "Attached is ajax.xml, debug.html" without actually providing those files.  It is generally frowned upon by internet etiquette to post the same question in multiple forums at the same time. It shows a lack of patience on your part, and a disrespect for people who might take time out of their day to answer.

 

That being said, now that I see the relevant data files are on this post and not the StackOverflow post, I can fill in some gaps from my StackOverflow comment.

 

The HTML is running a client-side JavaScript that uses a timer to invoke the browser's built-in AJAX client at regular intervals to send an HTTP request to download the XML file, parse the XML, and inject the results into the HTML elements using the browser's DOM. There is no streaming involved at all, each data update is an independent HTTP request.

 

In C++Builder, you would simply do the same thing. Run a timer that uses any HTTP client you want (TIdHTTP, T(Net)HTTPClient, libcurl, etc) to download the XML from the appropriate URL, and then you can use any XML library, such as (T|I)XMLDocument, to parse the XML and do what you need with the data values.

Edited January 24 by Remy Lebeau

[dan27125 0]

Remy ...Good news ... I made a test function on a button to pole once that works.  It uses IdHTTP and  XMLDocument.  

 

Ok now can you direct me on the "VCL thread safe" way to create a child thread that always runs in the background? 

Puts the results in thread safe container class (stdlist ?) for the apps main thread to pole and process?

 

I know how to write non VCL classic win32 thread safe code so one thread waits for event then puts the results in a shared buffer.  I have my own derived stdlist template that adds a windows critical section semaphore to block during the moment of insert and delete 

However that was how things were done 20 years ago.   What I don't know is how to be "thread safe" from within the VCL.  Furthermore I want to learn to write C++ Builder code style that might be portable to other platforms.   What is the proper way for a VCL main thread to create a child thread and then later on if necessary how does the main thread terminate the child thread  if it is still running?   Any links or tips to help me learn would be appreciated.

 

I'd like to follow these learning examples but I'm getting an error when trying to include thread and mutex as follows 

#include <thread>

#include <mutex>

 

I'm using Embarcadero® C++Builder 11 Version 28.0.48361.3236.   I have a subscription and can update it if necessary...

What silly thing am I doing wrong that I cant include thread or mutex in these examples ?

 

https://learncplusplus.org/what-is-a-data-race-in-multi-threading-c-apps-and-how-to-avoid-it/

https://blogs.embarcadero.com/learn-powerful-multi-threading-operations-with-c-builder/I

 

 

Also... below is a paste of the sample working code that collects the data from the zigbee gateway.  

 

 

void __fastcall TForm1::Button1Click(TObject *Sender)
{

    String s =  "http://192.168.1.123/ajax.xml";

    UnicodeString us = IdHTTP1->Get(s);
    XMLDocument1->LoadFromXML(us);
    const _di_IXMLNode resp = XMLDocument1->ChildNodes->FindNode("response");

    if (resp == NULL)
    {
        Memo1->Lines->Text = "Response Null";
        return;
    }

       const _di_IXMLNode zigbeeData = resp->ChildNodes->FindNode("zigbeeData");
    if (zigbeeData!=NULL)
    {
        String s;
        s = zigbeeData->Text;
        s += "SUCCESS\n";
        Memo1->Lines->Text = s;
    }

}
 

Edited January 25 by dan27125

[Roger Cigol 88]

If you want to write portable C++ that will compile outside of VCL then you have to avoid all VCL classes (eg String).

If you just want the thread implementation to be able to compile outside of VCL then avoid VCL thread classes (which do work for C++ Builder of course) and use std::thread from the C++ STL.

(std::thread also works fine for RAD Studio 11 (and 12 of course)).

 

Not sure why your two include statements should cause problems. I use exactly this in code, regularly.

 

does 

#include <string>

or

#include <vector>

 

also throw up compile time errors?

 

 

[Roger Cigol 88]

Ah! an idea - assuming you are building a 32 bit app:

Are you using the Borland "Classic" (aka - very old C++ standard) compiler?

menu: Project | Options | C++ Compiler - make sure the "use classic borland compiler" is set to "false" (this will then use the clang32 compiler. It's slower to build. 

[Remy Lebeau 1260]

On 1/25/2024 at 6:58 AM, dan27125 said:

Ok now can you direct me on the "VCL thread safe" way to create a child thread that always runs in the background?

There are countless examples of that online all over the place.  Start with the TThread class, overriding its virtual Execute() method, and use its Synchronize() or Queue() method when you need to communicate with the main UI thread.

On 1/25/2024 at 6:58 AM, dan27125 said:

Puts the results in thread safe container class (stdlist ?) for the apps main thread to pole and process?

Seems like you already know how to do that.

On 1/25/2024 at 6:58 AM, dan27125 said:

I know how to write non VCL classic win32 thread safe code so one thread waits for event then puts the results in a shared buffer.  I have my own derived stdlist template that adds a windows critical section semaphore to block during the moment of insert and delete

So, what's stopping you from using that?  That would be a perfectly valid approach.

On 1/25/2024 at 6:58 AM, dan27125 said:

However that was how things were done 20 years ago.

And?  It still works.

On 1/25/2024 at 6:58 AM, dan27125 said:

What I don't know is how to be "thread safe" from within the VCL.

All that means is that code which touches VCL controls needs to run in the context of the main UI thread.  There are many different way to do that.  One way is TThread::Synchronize()/TThread::Queue().  Another way is using window messages (for example, wParam and lParam can hold pointers to data).  Another way is using a thread-safe container and having the main thread poll it periodically, such as in a timer.  Or any combination of these approaches (for example, putting data in a thread-safe container, then posting a message to poll the container).

On 1/25/2024 at 6:58 AM, dan27125 said:

Furthermore I want to learn to write C++ Builder code style that might be portable to other platforms.   What is the proper way for a VCL main thread to create a child thread and then later on if necessary how does the main thread terminate the child thread  if it is still running?   Any links or tips to help me learn would be appreciated.

Are you planning on writing portable code in C++Builder only, or do you need to support other C++ compilers?  If you stick with just C++Builder then there is no reason not to use what it provides, like TThread (which does run on all supported platforms).  But, if you want to support other compilers, then you have to stick with only the standard C++ library and 3rd party cross-platform libraries.

On 1/25/2024 at 6:58 AM, dan27125 said:

I'd like to follow these learning examples but I'm getting an error when trying to include thread and mutex as follows 

#include <thread>

#include <mutex>

Those headers were introduced in C++11. Are you using one of the clang-enhanced C++ compilers that support C++11?  The "classic" Borland compiler only supports a small subset of C++11 language features, but no C++ standard library features.  So make sure you are using a compiler that supports what you want.

[dan27125 0]

Good news.  Thanks to your help I have it working.  I also found a few other older posts from other people, including xml api syntax.    This is a worker thread on a sleep timer using TIdHTTP and _di_IXMLDocument.    This HTTP XML polling app "seems" to be working fine.   But now I'd like to ask a few follow up questions.

1) I found some old indy documentation.  IndyDocs_10.1.5.0_HtmlHelp  Is that the latest Indy DOCS ?

2) Indy TIdHTTP class operator new and delete i.e  Worker child thread construction and destruction.  Is it OK use the new and delete operators with the TIdHTTP class ? Is new and delete operator like below OK ?
void scope_use_example_TIdHTTP()
{
    TIdHTTP *http = new TIdHTTP(NULL);
    delete http;
}

3) XML API usage, scope. For TXMLDocument it looks like you can't / don't use C++ new/delete operators with TXMLDocument. 

Embarcadero is/was recently having some Radstudio web documentation problems being down or slow.  I found some posts on this forum.  

It looks like for dynamic TXMLDocument such as a worker thread you use interface_cast like

"_di_IXMLDocument xml = interface_cast<Xmlintf::IXMLDocument> (new TXMLDocument(NULL))"

 

 

Trying to post a reply with code I'm getting *** Forbidden. Message seems to be spam. ***.  So I'm going to put the code in a separate attachment as  example1.cpp

 

 

 

example1.cpp

Edited January 30 by dan27125
Error message "Forbidden. Message seems to be spam."

[dan27125 0]

Reply to Roger Cigol

 

<< Ah! an idea - assuming you are building a 32 bit app:

<< Are you using the Borland "Classic" (aka - very old C++ standard) compiler?

<< menu: Project | Options | C++ Compiler - make sure the "use classic borland compiler" is set to "false" (this will then use the clang32 compiler. It's slower to build. 

 

I just checked and in fact found use classic Borland compiler was enabled.   I turned it off.  But the app wont build.  So something needs to be fixed/changed.   That switch was left on because when turned off, I get linker errors. I'm getting tons of compiler errors with it turned off.   Its time for me to fix this. I'm using C++ When the app was created 9 years ago, it borrowed some old non VCL classic win32 code that was written using Borland C++ from 1996.  I had some custom templates I made in the 1990s for safe win32 threads that use shared memory.   I don't use it anymore.  Also, this app is a bootloader server for a number of embedded cpus.  9 years ago There was allot of "paranoia" to keep everything on a byte in order and reuse code to have less  things go wrong.  From below link errors I can see there are old unwanted conditionals somewhere in the project file that I cant figure how to find and delete.  For example Project | Options, Shared options vs non shared. And then you have to toggle between debug and non debug mode.  I can see things in there that the IDE wont wont me to edit / remove.   I need to clean up the project file.   I wonder if it cab be edited as a text or an XML file.  So I can root out old stuff.  Ha ha "_USE_OLD_RW_STL" is a container from the 1990s and the classic. I don't need that anymore but I cant see where that is in the project file.  

 

 

bcc32c command line for "ttywin.cpp TFIFO.CPP W_IGS1.cpp thread.cpp netdef.cpp about.cpp debug.cpp ig-ini.cpp tcomport.cpp IGS-0.cpp TCP.cpp Tconfig1.cpp StyleDialog1.cpp ig2-ini.cpp InfoDialog1.cpp Config21.cpp vclstuff.cpp Base64Console1.cpp TINIFILE.CPP InverterConsole1.cpp tinverter.cpp zpanserver2.cpp mesgbox.cpp IGS-Base64.cpp zpanserver1.cpp VidFields.cpp zpanserver3.cpp W-IGS-2.cpp timespan.cpp WIFI_Console1.cpp Xmodem1.cpp vclthread.cpp ETRXList1.cpp Pic24HexFile.cpp em250_proto_ascii2.cpp InverterConsole_P24.cpp LLC_Temp.cpp DATETIME.CPP teui64.cpp WIFI_Console_EBL1.cpp HttpDownload1.cpp Update1.cpp tfirewall.cpp WIFI_Console_EBL2.cpp InverterConsole2.cpp tzigbee.cpp zpanserver4.cpp tinverter telemtry.cpp EthTcpGateway1.cpp REALTIME.CPP"
  c:\program files (x86)\embarcadero\studio\22.0\bin\bcc32c.exe -cc1 -D NDEBUG -D _USE_OLD_RW_STL -D TCP_IGS -D _USE_OLD_RW_STL -D Uses_2022_NID -D 
  FRAMEWORK_VCL -output-dir ..\IG -I "c:\program files (x86)\embarcadero\studio\22.0\Lib" -I "c:\program files (x86)\embarcadero\studio\22.0\include" 
  -I C:\PVKEY\PVKEY-DEV-6\IGS\2024-01-30\W-IGS-V253-03 -I "c:\program files (x86)\embarcadero\studio\22.0\include\windows\vcl" -isystem "c:\program 
  files (x86)\embarcadero\studio\22.0\include" -isystem "c:\program files (x86)\embarcadero\studio\22.0\include\dinkumware64" -isystem "c:\program 
  files (x86)\embarcadero\studio\22.0\include\windows\crtl" -isystem "c:\program files (x86)\embarcadero\studio\22.0\include\windows\sdk" -isystem 
  "c:\program files (x86)\embarcadero\studio\22.0\include\windows\rtl" -isystem "c:\program files (x86)\embarcadero\studio\22.0\include\windows\vcl" 
  -isystem "c:\program files (x86)\embarcadero\studio\22.0\include\windows\fmx" -isystem C:\Users\Public\Documents\Embarcadero\Studio\22.0\hpp\Win32 
  -isystem C:\Users\Public\Documents\Embarcadero\Studio\22.0\hpp\Win32 -fcxx-exceptions -fborland-extensions -nobuiltininc -nostdsysteminc -triple 
  i686-pc-windows-omf -fpack-struct=1 -emit-obj -mrelocation-model static -masm-verbose -ffunction-sections -fexceptions -fseh -mstack-alignment=16 
  -fno-spell-checking -fno-use-cxa-atexit -fno-threadsafe-statics -x c++ -std=c++17 -O2 -O2 -fmath-errno -tM -tW --auto-dependency-output  -tWM -wdef 
  -wstl -wamp -wuse -Q -6 -wcln -Vx -a1 -wnod -wamb -wstu -wasm -r- -xp -k -Ve -r ttywin.cpp TFIFO.CPP W_IGS1.cpp thread.cpp netdef.cpp about.cpp 
  debug.cpp ig-ini.cpp tcomport.cpp IGS-0.cpp TCP.cpp Tconfig1.cpp StyleDialog1.cpp ig2-ini.cpp InfoDialog1.cpp Config21.cpp vclstuff.cpp 
  Base64Console1.cpp TINIFILE.CPP InverterConsole1.cpp tinverter.cpp zpanserver2.cpp mesgbox.cpp IGS-Base64.cpp zpanserver1.cpp VidFields.cpp 
  zpanserver3.cpp W-IGS-2.cpp timespan.cpp WIFI_Console1.cpp Xmodem1.cpp vclthread.cpp ETRXList1.cpp Pic24HexFile.cpp em250_proto_ascii2.cpp 
  InverterConsole_P24.cpp LLC_Temp.cpp DATETIME.CPP teui64.cpp WIFI_Console_EBL1.cpp HttpDownload1.cpp Update1.cpp tfirewall.cpp WIFI_Console_EBL2.cpp 
  InverterConsole2.cpp tzigbee.cpp zpanserver4.cpp "tinverter telemtry.cpp" EthTcpGateway1.cpp REALTIME.CPP 

 

 

[Remy Lebeau 1260]

3 hours ago, dan27125 said:

I found some old indy documentation.  IndyDocs_10.1.5.0_HtmlHelp  Is that the latest Indy DOCS ?

Probably, yes.  Indy's documentation hasn't been updated in a VERY LONG time.  Mainly because the system used to generate the documentation at the time no longer exists, so all of that content would have to be migrated to a completely new system, and noone has had the time to do that migration.

 

On the other hand, Indy's public interface hasn't changed TOO much in a very long time, either.  Certainly it's had plenty of new features/interfaces added over the years, but they haven't been added to the documentation.

3 hours ago, dan27125 said:

Indy TIdHTTP class operator new and delete i.e  Worker child thread construction and destruction.  Is it OK use the new and delete operators with the TIdHTTP class ? Is new and delete operator like below OK ?

Yes, in fact it is REQUIRED to do so, because TIdHTTP derives from Delphi's TObject class, which MUST be constructed only in heap memory.  On the C++ side, you can (and should) use a smart pointer class like std::unique_ptr to manage the 'new'/'delete' calls for you.  But you CAN use them manually if you choose to.

3 hours ago, dan27125 said:

For TXMLDocument it looks like you can't / don't use C++ new/delete operators with TXMLDocument.

You CAN, but you have to very VERY CAREFUL with it.

 

If you construct the TXMLDocument object with a non-null Owner, you can safely use 'delete' to destroy the object.

 

But, if you construct the TXMLDocument object with a null Owner, that will enable the object's reference counting semantics, so DO NOT use 'delete' on it!  You MUST use its AddRef()/Release() methods instead.  The smart pointer _di_IXMLDocument class will handle that for you.

3 hours ago, dan27125 said:

Embarcadero is/was recently having some Radstudio web documentation problems being down or slow.  I found some posts on this forum.

They are still having server problems with other services, but DocWiki is working again, yes.

3 hours ago, dan27125 said:

It looks like for dynamic TXMLDocument such as a worker thread you use interface_cast like

"_di_IXMLDocument xml = interface_cast<Xmlintf::IXMLDocument> (new TXMLDocument(NULL))"

I have never heard of interface_cast before.  That is a new one for me.

 

A better option is to use the standalone NewXMLDocument(), LoadXMLDocument(), or LoadXMLData() function instead of 'new'ing the TXMLDocument class directly.  They return a properly prepared ownerless/ref-counted _di_IXMLDocument, eg:

_di_IXMLDocument xml = NewXMLDocument();

 

Edited January 30 by Remy Lebeau