David Schwartz

No. I've exhausted the time allotted to investigate this, and there's no budget for 3rd-party commercial subscriptions. What used to be pretty easy up to about a year ago or so is much, much more difficult today. The only real HTML tags that appear useful as possible "landmarks" for identifying different sections of content are the Header tags (H1, H2, etc.). An e-Reader wouldn't have any problem rendering text-to-speech because it does not need to "recognize" the sections. It just parses through it and performs TTS as it encounters things, adapting to the various tags it finds as needed. It doesn't need to find structure or parse out pieces of content. I don't know how feasible it is to extract CSS attributes along the way that might influence how it renders things -- it's not like you can "hear colors" other than having it report the start and end of a hex code or maybe a recognizable color name, eg., "blue". It's pretty evident that the DIVs and SPANs around the content are sprinkling in quite a bit of visual stuff that would just be a ton of noise if it was all vocalized by a TTS system -- italics, bolds, colors, (sub)headings, etc. There are ad blocks that can be inserted between meaningful sections, and they "look" nearly the same except for one little visual tag that's presented as a CSS attribute that has a random classid. If I had another week to spend on this, I might come up with something that works, but at this point, my time budget is exhausted.

very carefully. They're getting harder and harder to upgrade. The thinner the cases get, the more stuff is soldered in place. The newest ones are not upgradeable from what I'm told. Do some research ... look for SSDs on eBay specifically for MacBook Pros and you can learn a lot from the vendors who sell them. I have a mid-2014 MBP and these devices can have their RAM and SSD upgraded simply by removing the bottom, but you need special tools. The problem is, some machines can't be upgraded past a certain point. Mine came with 1TB of SSD, but I can't upgrade it to 2TB, although some mid-2014 models can. I also can't expand my RAM past 16GB. There are vendors who sell upgraded equipment on eBay. I bought a Late 2014 Mac Mini that was upgraded to 2TB of SSD and 16GB of RAM for $1100. I thought it was the same as my MBP, but it turns out the i7 chip is only a dual-core not quad-core. Apple didn't put quad-core i7's in Mac Minis until the newer generation they announced last fall. A new Mac Mini similarly configured but with a 3.2GHz 8-core i7 is about $2300.

Uhhh ... no, it's far from unique. Just something I guess you've never considered. Look, Apple has a discount program at most colleges, and many schools are REQUIRING students to have a laptop. So when the inevitable 15% or so of students drop-out every semester, a lot of them decide to sell their nearly-new equipment, usually at a pretty decent discount because they don't know much about that stuff. Many of them are like a friend I've got who has a saying: "A fast nickel is better than a slow dime" and they're hurting for cash, so they post their stuff up on eBay at ridiculous discounts. If you're quick, you can score some AMAZING deals! For example, my sister wanted a 12.9" iPad Pro and asked me to see what I could find for her. I scoped out eBay and found one that the owner said was 6 months old, barely used, with AppleCare. The description was quite vague, and when I asked her to clarify something, she just wasn't sure. The price was $475, which was a pretty good deal for a 32GB WiFi-only device with nothing else. I've seen similar things with cracked screens sell for this much, and I was surprised that it hadn't been snatched up in over 24 hours. When it arrived, I freaked-out! It came with a $150 case, 128GB of memory, an Apple Pencil, original boxes, and had cellular service built-in. I could have flipped it that day for $750! My sister said she wanted a WiFi only device, and I said I'd be happy to trade it for mine (the specs she asked for) but she decided to think it over. The next day she told me her partner saw it and immediately took it over to the Verizon store and activated it on their account. They LOVE it! (I should have opened the box and looked at it BEFORE I saw my sister and just given her mine instead; she'd have been quite satisfied.) That may seem like a fluke, but there are plenty of folks who simply don't know what they've got, and a LOT of them own Apple equipment -- because they DON'T HAVE TO KNOW! My brother has three kids, one has been in and out of college for 7 years; one just graduated last year; and one is about to go into college. He called me in total frustration 2-1/2 years ago and said his oldest daughter just had her HP laptop die for the umpteenth time and he just can't afford to keep replacing the damn things. He asked what I'd suggest, and I told him the same thing I've said numerous times in the past: get her a used MacBook! Of course he hemmed and hawed and kept saying how he can't afford it, and I just said again, "Well that's what I'm going to tell you every time you ask." I'm not sure exactly what happened after that, but somehow one of them came across a 6-mo old MBP from another student and picked it up at about 25% discount off of list price. It had AppleCare. I've asked several times how it's holding up, and at first my brother was hesitant ... "so far so good, but if the damn thing breaks, I'm going to be really pissed." Here we are 2-1/2 years later and guess what? The other HP and Dell his other two kids had both broke AGAIN, and they didn't honor their warranty deals for obscure reasons. But that MBP has only had a couple of minor problems: something on the kbd broke and he had some other weird problem with the OS, and both times they took it to the Apple Store and they fixed things right up. He was so amazed that he told me he will never buy another PC again, only Apple hardware. I've heard this same story over and over and over from so many people.... like a manager at a place I was working said he had been a loyal Dell users for years, but his kids kept breaking them, and they seemed to be breaking with increasing frequency. One Monday he said he was fed up and went to the Apple Store and bought 5 (yes, 5) brand new MBPs for his family, and said as soon as they can get switched off of their Dells, he was going to donate the Dells to some charity. My brother is the kind of guy who NEVER buys anything NEW. So he actually enjoys hunting down good deals on used Macs! Kinda weird, I know. But they're not that hard to come by if you know where to look. My first iMac I bought new at a Thanksgiving sale at Best Buy and it was discounted by several hundred dollars. I've never bought a new Mac since, although I do buy new iPads from T-Mobile (my cell phone carrier) b/c I can get 24 months same-as-cash with no credit app from them. My fist MBP I got with a Apple-branded BarclayCard that gave me 5% cashback for purchases made through Apple with 24 months same-as-cash payments, and I bought it from their refurb store at a 15% discount. You just have to know how to find good deals on this stuff I guess.

Thanks for all the side-chatter, but none of it is helpful. I was merely passing along the results of what I found. I'm not running Linux; I'm not getting captchas doing this manually, and I have no reason to believe I will get them using similar timings with a different means of doing the data entry; I'm only interested in the first page of results; and in summary, y'all are making stuff up that is completely irrelevant to my present needs. All I'm focusing on is scraping one page of results, and that's all that's up for discussion. But thanks for the insights.

I'm just stating my experience; I guess you're free to call it whatever you want. As I said, YMMV. Also, it seems you don't track resale prices of Apple products as closely as I do, so of course you're going to think whatever I'm saying is dubious. This is why there are people in the world on both sides of the climate issue. The unquestionable fact is ... the average level of CO2 in the air has been steadily rising for decades. The debate is ... so what? The only thing most people will react to is when living beings start dying off of hypoxia due to too much CO2 and not enough O2 in the air. But the plants and animals will have died off long before that, so I guess we'll never really know until it's far too late.

I don't know why people sell their computers, they don't say if they're developes or not. But depending on the time of year and other tidbits of stuff, it's obvious that there are lots of college students who sell their nearly-new laptops for whatever reasons they might have. I don't care what their reason is, but MBPs consistently fetch considerably higher prices vs. stuff you can buy at Costco and BestBuy at deep discounts. Why buy a used PC when you can get a new one for not that much more that has a full warranty? It's just not worth the risk IMHO. A year-old MBP might go for $2000 while a similarly configured Dell or HP might cost $400. The biggest cost factor in MBPs is their SSD. You can get a 256GB or 512GB model and upgrade it to 1TB or 2TB way cheaper than buying one preconfigured that way. PCs are 100% commodity-based pricing except for the latest-and-greatest CPUs, RAM, and SSD.

Ok, Mac vs. PC.... I don't understand why people think Apple's are "too expensive". If you had a car that broke down as often as PCs do but it was 1/3 the cost of one that NEVER breaks down, but the TCO of the cheaper one exceeded the TCO of the more expensive one, you think you're saving money??? Maybe it can't start every 3rd say, or it locks up while updating itself and doesn't give you an option. What if you were on your way to a big meeting and it locked up? Just stopped in the middle of the road for 15 minutes while it updated its firmware? You'd say, "oh, but it was so much cheaper than the ones that DON'T lock up or need to be fixed regularly!" TCO. You need to look at TCO, not just what you paid for it originally. The hardware is functionally equivalent for most intents and purposes between the two. But everybody knows a new Mac will cost 2x-3x what an equivalent PC will cost. Over their lifetime, the Mac will give you fewer problems and cost you less after factoring in the wasted time and expenses that PCs incur. Anyway, I don't buy Mac hardware NEW. I get it either from the Apple Refurbished Store, or on eBay. I don't worry about the price differential for a couple of reasons: (1) Apple HW holds its value over time, PCs don't; (2) with one exception, the Apple HW I've had has outlasted every PC I've ever had. The exception was an original 24" iMac with Intel CPU from 2005ish -- it's graphics card died not long after the AppleCare expired and they wanted more to replace it than the unit was worth. I was able to sell it without HDD and RAM for nearly $400 anyway. Their LCD panels (removed from the case) were selling used on eBay for nearly $300 all by themselves! In 2013 or so, I got a fully-loaded 2012 MBP at the refurb store for $2600 ($3300 MSRP), used it daily, and sold it 2 years later for $2300. You could say it cost me $300 for 2 years of use! How is that "more expensive" than a similar PC that would have cost me closer to $1000 over the same time period because it lost its value so fast? After that, I picked up a 2014 MBP that was a little beefier for $2400. I've still got it today, and I could sell it on eBay for $1200-$1400. That's like $200/yr. (I got the battery replaced earlier in the year for $179. That's all it has cost me since I got it, other than AppleCare, which I never needed to use.) Every PC laptop I've owned has had batteries die within 18 months. The only problem I've had with my MBP was earlier this year -- the battery started bloating up, and as I said, it cost me $179 to get it replaced. They actually removed the logic board and display from my laptop and swapped-in an entire bottom half. So it's "half-new". 🙂 I could probably get my $179 back with a higher selling price selling on eBay! I've never seen a PC that didn't lose 70% of it's value in 2 years. When I need one, I only buy them from Best Buy, and I get the ones that customers return after big holiday sales that are marked down another 25%. As for the software -- MacOS vs. Windows -- I'm fed up with Microsoft's attitude that they've turned Windows into a "subscription-driven" model. I'll open up a laptop and it will start installing who-knows-what. It starts without asking me, locks me out of the machine for 5-30 minutes, and usually resets things that take me days to fix. Also, you pretty much need to run some kind of anti-virus on PCs, and I regularly deal with Windows popping-up asking about this or that threat. This isn't needed on Macs, as far as I'm concerned. I do my personal Delphi development inside of a VM on my Mac -- either my MBP or a Mac Mini. I don't run AV or anything in the VM, and it runs faster than on any Windows laptop I've had. PCs w/Windows are cheaper out the door, but over time they depreciate to the point of being worthless in 18-24 months, and they waste a HUGE amount of time. They also seem to need batteries replaced pretty regularly. Macs are cheaper in the long-run, mostly because they do NOT depreciate the way PCs do, and I've had very few problems with them. But I'm aware there are people in the world who buy one car after another and every one is a lemon, just like there are people who buy anything from Apple, say, and they think they're lemons as well. I can't help that. I don't think dying batteries in PCs means they're lemons, but the fact that they lose their value so damn quickly is a good indicator that they're Really Poor Investments. But as they say, YMMV. I regard Macs as simply a far better investment than PCs. And since I can use either one to do whatever I need, I prefer to buy the one with lower TCO. I know some folks claim that's some kind of "blind religion", but I guess they'd also call Warren Buffett a religious zealot because this is how he buys things as well.

I didn't know you could install MacOS in a VM. And I thought it's only supposed to be run on Apple hardware.

Well, if you ever get around to dealing with your audio file playback problem again, keep us apprised of your progress.

I have used Delphi on Macs since the first iMacs were released that used Intel's chips in 2004 or so. I first used Parallels and stuck with it for several years. I don't care for their support; I usually need problems solved NOW, not in a day or two. The Parallels support team is located in India, and it takes 12 hours minimum to get a reply from where I live, and they rarely understand what I'm referring to. They have frequently asked me to diagnose the problem myself or send them a copy of the VM. Yeah, right. I don't think so. I got tired of having to pay them a fee to upgrade every time MacOS was updated, and switched to VMWare Fusion. That worked very well, and they offer free updates each time MacOS is updated. They don't add unnecessary crap just to force an annual upgrade on you the way Parallels does. (None of the features added to Parallels over the years were ever of any value to me.) A couple of years ago I switched to VirtualBox, and I'm very happy with it. No more upgrade fees, and it does get updated far more frequently than the other two. I have also tried different configurations of things, including keeping the source files on the Mac's HD and compiling everything inside the VM. That seems like a Good Idea, until you realize it slows things down by about an order of magnitude due to inefficiencies of going through the firewall between the VM and the host. I strip out or disable all anti-virus and scanner software from Windows in the VM, and as a result it usually runs noticeably faster than a similarly-configured native Windows machine. Why would anybody want to buy a high-powered machine then deliberately set it up to run slower for no particular benefit? I don't check email or surf the internet from within my VM. I even download most files from the host side. In all the years I've been working with Delphi inside of a VM without any sort of anti-virus or other similar stuff, I've only once encountered an infection of some kind. It was very easy to deal with. I simply deleted the VM, reloaded the most recent backup from by backup drive, turned it on, and copied back the latest source files from my host. And that's how simple it should be to deal with ransomware viruses, too. It's really easy to copy your files from inside the VM to a folder on the host when you finish your work for the day. Or check everything into a git repo on the host or in the cloud. Or ... or ... or ... there are lots of options that don't slow down your work or create a problem. I suspect your need to double-build is due to some kind of file locking or delays that occur between the VM and the host. Perhaps a temp file isn't being deleted in a timely fashion. The semantics of a Unix file system and the Windows file system are not 100% compatible. The way you're using things, you're exacerbating any subtle incompatibilities that may exist, not to mention slowing down your builds significantly. Honestly, I have never run into that double-build problem.

property editors ... there are ways to disable them. I was trying to help you solve your audio playing problem. Instead, it looks like you're more interested in discussing politics of how libraries are packaged. If you have some beefs about a library, you need to take them up with the library's vendor. I can't help you with that stuff. And I have no problem with it either.

You can use the OpenWire GUI tool it installs in the IDE to wire-up your data flows, or you can do it in code. Trust me when I say that you do NOT want to do it manually in code! But hey ... it'll solve your problem.

Sorry, but this is the first time anybody has suggested that there could be a timing issue at work here, and your example certainly wouldn't have uncovered it. I assumed that the OnNavigateComplete2 actually meant that the processing was "complete". I've never had a need to use OnDocumentComplete before, and I've never even seen it used in any examples. But now that you mention it, it makes sense. So I modified my code to call that instead of OnNavigationComplete2 and it seems to be returning what I'm looking for. YES! THANK YOU!

It's not just Google. More and more sites are doing stuff like this. What nobody seems to want to admit is, all the assertions people keep make about outerHTML returning what the user sees is basically bunk. There is user-visible text being rendered inside of this control, and outerHTML is not providing access to it in all cases. The question remains ... how can you access the content that's being rendered for viewing by the user? Up until this point the stock reply is to use outerHTML. Now that we've established it doesn't work in all cases, what DOES work instead? This is a technical question. I don't care about the politics.

To save you the time of adding a line of code to run a query, here's what outerHTML gives you based on what Google returns for "dentist in baltimore". Feel free to strip out the javascript and show me the text that resembles what the user sees when this query is run directly on google.com. <html lang="en" itemtype="http://schema.org/SearchResultsPage" itemscope=""><head><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>dentist in baltimore - Google Search</title><script nonce="Xp+ZL1qXiueWiJzqZqCD8g==">(function(){window.google={kEI:'Wk5UXdCiNs_7-gTLw6jgBQ',kEXPI:'31',authuser:0,kscs:'c9c918f0_Wk5UXdCiNs_7-gTLw6jgBQ',kGL:'US',kBL:'d31E'};google.sn='web';google.kHL='en';google.jsfs='Ffpdje';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.time=function(){return(new Date).getTime()};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d="",f=google.ls||"";e||-1!=b.search("&ei=")||(d="&ei="+google.getEI(c),-1==b.search("&lei=")&&(c=google.getLEI(c))&&(d+="&lei="+c));c="";!e&&google.cshid&&-1==b.search("&cshid=")&&"slh"!=a&&(c="&cshid="+google.cshid);a=e||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+d+f+"&zx="+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};google.arwt=function(a){a.href=document.getElementById(a.id.substring(1)).href;return!0};(function(){var g=this||self,k=Date.now||function(){return+new Date};var t={};var v=function(b,d){if(null===d)return!1;if("contains"in b&&1==d.nodeType)return b.contains(d);if("compareDocumentPosition"in b)return b==d||!!(b.compareDocumentPosition(d)&16);for(;d&&b!=d;)d=d.parentNode;return d==b};var w=function(b,d){return function(a){a||(a=window.event);return d.call(b,a)}},B=function(b){b=b.target||b.srcElement;!b.getAttribute&&b.parentNode&&(b=b.parentNode);return b},C="undefined"!=typeof navigator&&/Macintosh/.test(navigator.userAgent),D="undefined"!=typeof navigator&&!/Opera/.test(navigator.userAgent)&&/WebKit/.test(navigator.userAgent),E={A:1,INPUT:1,TEXTAREA:1,SELECT:1,BUTTON:1},aa=function(){this._mouseEventsPrevented=!0},F={A:13,BUTTON:0,CHECKBOX:32,COMBOBOX:13,GRIDCELL:13,LINK:13,LISTBOX:13,MENU:0,MENUBAR:0,MENUITEM:0,MENUITEMCHECKBOX:0,MENUITEMRADIO:0,OPTION:0,RADIO:32,RADIOGROUP:32,RESET:0,SUBMIT:0,SWITCH:32,TAB:0,TREE:13,TREEITEM:13},G={CHECKBOX:!0,OPTION:!0,RADIO:!0},H={COLOR:!0,DATE:!0,DATETIME:!0,"DATETIME-LOCAL":!0,EMAIL:!0,MONTH:!0,NUMBER:!0,PASSWORD:!0,RANGE:!0,SEARCH:!0,TEL:!0,TEXT:!0,TEXTAREA:!0,TIME:!0,URL:!0,WEEK:!0},ba={A:!0,AREA:!0,BUTTON:!0,DIALOG:!0,IMG:!0,INPUT:!0,LINK:!0,MENU:!0,OPTGROUP:!0,OPTION:!0,PROGRESS:!0,SELECT:!0,TEXTAREA:!0};var I=function(){this.h=this.a=null},K=function(b,d){var a=J;a.a=b;a.h=d;return a};I.prototype.g=function(){var b=this.a;this.a&&this.a!=this.h?this.a=this.a.__owner||this.a.parentNode:this.a=null;return b};var L=function(){this.i=[];this.a=0;this.h=null;this.j=!1};L.prototype.g=function(){if(this.j)return J.g();if(this.a!=this.i.length){var b=this.i[this.a];this.a++;b!=this.h&&b&&b.__owner&&(this.j=!0,K(b.__owner,this.h));return b}return null};var J=new I,M=new L;var P=function(){this.o=[];this.a=[];this.g=[];this.j={};this.h=null;this.i=[];O(this,"_custom")},ca="undefined"!=typeof navigator&&/iPhone|iPad|iPod/.test(navigator.userAgent),Q=String.prototype.trim?function(b){return b.trim()}:function(b){return b.replace(/^\s+/,"").replace(/\s+$/,"")},da=/\s*;\s*/,ia=function(b,d){return function(a){var c=d;if("_custom"==c){c=a.detail;if(!c||!c._type)return;c=c._type}var e;if("click"==c&&(C&&a.metaKey||!C&&a.ctrlKey||2==a.which||null==a.which&&4==a.button||a.shiftKey))c= "clickmod";else{var f=a.which||a.keyCode||a.key;D&&3==f&&(f=13);if(13!=f&&32!=f)f=!1;else{var l=B(a),h=(l.getAttribute("role")||l.type||l.tagName).toUpperCase(),m;(m="keydown"!=a.type||!("getAttribute"in l&&!((l.getAttribute("type")||l.tagName).toUpperCase()in H)&&"BUTTON"!=l.tagName.toUpperCase()&&!l.isContentEditable)||a.ctrlKey||a.shiftKey||a.altKey||a.metaKey||(l.getAttribute("type")||l.tagName).toUpperCase()in G&&32==f)||((m=l.tagName in E)||(m=l.getAttributeNode("tabindex"),m=null!=m&&m.specified),m=!(m&&!l.disabled));m?f=!1:(l="INPUT"!=l.tagName.toUpperCase()||l.type,m=!(h in F)&&13==f,f=(0==F[h]%f||m)&&!!l)}f&&(c="clickkey")}h=a.srcElement||a.target;f=R(c,a,h,"",null);a.path?(M.i=a.path,M.a=0,M.h=this,M.j=!1,l=M):l=K(h,this);for(;m=l.g();){m=e=m;var n=c;var p=m.__jsaction;if(!p){var u;p=null;"getAttribute"in m&&(p=m.getAttribute("jsaction"));if(u=p){p=t[u];if(!p){p={};for(var x=u.split(da),ea=x?x.length:0,y=0;y<ea;y++){var r=x[y];if(r){var z=r.indexOf(":"),N=-1!=z,fa=N?Q(r.substr(0,z)):"click";r=N?Q(r.substr(z+1)):r;p[fa]=r}}t[u]=p}m.__jsaction=p}else p=ha,m.__jsaction=p}m=p;"clickkey"==n?n="click":"click"!=n||m.click||(n="clickonly");n={m:n,action:m[n]||"",event:null,s:!1};f=R(n.m,n.event||a,h,n.action||"",e,f.timeStamp);if(n.s||n.action)break}f&&"touchend"==f.eventType&&(f.event._preventMouseEvents=aa);if(n&&n.action){if(h="clickkey"==c)h=B(a),h=(h.type||h.tagName).toUpperCase(),(h=32==(a.which||a.keyCode||a.key)&&"CHECKBOX"!=h)||(h=B(a),l=(h.getAttribute("role")||h.tagName).toUpperCase(),h="BUTTON"==l?!0:!(h.tagName.toUpperCase()in ba)||"A"==l||"SELECT"==l||(h.getAttribute("type")||h.tagName).toUpperCase()in G||(h.getAttribute("type")||h.tagName).toUpperCase()in H?!1:!0);h&&(a.preventDefault?a.preventDefault():a.returnValue=!1);if("mouseenter"==c||"mouseleave"==c)if(h=a.relatedTarget,!("mouseover"==a.type&&"mouseenter"==c||"mouseout"==a.type&&"mouseleave"==c)||h&&(h===e||v(e,h)))f.action="",f.actionElement=null;else{c={};for(var q in a)"function"!==typeof a[q]&&"srcElement"!==q&& "target"!==q&&(c[q]=a[q]);c.type="mouseover"==a.type?"mouseenter":"mouseleave";c.target=c.srcElement=e;c.bubbles=!1;f.event=c;f.targetElement=e}}else f.action="",f.actionElement=null;e=f;b.h&&(q=R(e.eventType,e.event,e.targetElement,e.action,e.actionElement,e.timeStamp),"clickonly"==q.eventType&&(q.eventType="click"),b.h(q,!0));if(e.actionElement){if(b.h)"A"!=e.actionElement.tagName||"click"!=e.eventType&&"clickmod"!=e.eventType||(a.preventDefault?a.preventDefault():a.returnValue=!1),b.h(e);else{if((q= g.document)&&!q.createEvent&&q.createEventObject)try{var A=q.createEventObject(a)}catch(la){A=a}else A=a;e.event=A;b.i.push(e)}if("touchend"==e.event.type&&e.event._mouseEventsPrevented){a=e.event;for(var ma in a);k()}}}},R=function(b,d,a,c,e,f){return{eventType:b,event:d,targetElement:a,action:c,actionElement:e,timeStamp:f||k()}},ha={},ja=function(b,d){return function(a){var c=b,e=d,f=!1;"mouseenter"==c?c="mouseover":"mouseleave"==c&&(c="mouseout");if(a.addEventListener){if("focus"==c||"blur"==c|| "error"==c||"load"==c)f=!0;a.addEventListener(c,e,f)}else a.attachEvent&&("focus"==c?c="focusin":"blur"==c&&(c="focusout"),e=w(a,e),a.attachEvent("on"+c,e));return{m:c,l:e,capture:f}}},O=function(b,d){if(!b.j.hasOwnProperty(d)){var a=ia(b,d),c=ja(d,a);b.j[d]=a;b.o.push(c);for(a=0;a<b.a.length;++a){var e=b.a[a];e.g.push(c.call(null,e.a))}"click"==d&&O(b,"keydown")}};P.prototype.l=function(b){return this.j[b]};var V=function(b,d){var a=new ka(d),c;a:{for(c=0;c<b.a.length;c++)if(S(b.a[c],d)){c=!0;break a}c=!1}if(c)return b.g.push(a),a;T(b,a);b.a.push(a);U(b);return a},U=function(b){for(var d=b.g.concat(b.a),a=[],c=[],e=0;e<b.a.length;++e){var f=b.a[e];W(f,d)?(a.push(f),X(f)):c.push(f)}for(e=0;e<b.g.length;++e)f=b.g[e],W(f,d)?a.push(f):(c.push(f),T(b,f));b.a=c;b.g=a},T=function(b,d){var a=d.a;ca&&(a.style.cursor="pointer");for(a=0;a<b.o.length;++a)d.g.push(b.o[a].call(null,d.a))},Y=function(b,d){b.h=d;b.i&& (0<b.i.length&&d(b.i),b.i=null)},ka=function(b){this.a=b;this.g=[]},S=function(b,d){for(var a=b.a,c=d;a!=c&&c.parentNode;)c=c.parentNode;return a==c},W=function(b,d){for(var a=0;a<d.length;++a)if(d[a].a!=b.a&&S(d[a],b.a))return!0;return!1},X=function(b){for(var d=0;d<b.g.length;++d){var a=b.a,c=b.g[d];a.removeEventListener?a.removeEventListener(c.m,c.l,c.capture):a.detachEvent&&a.detachEvent("on"+c.m,c.l)}b.g=[]};var Z=new P;V(Z,window.document.documentElement);O(Z,"click");O(Z,"focus");O(Z,"focusin");O(Z,"blur");O(Z,"focusout");O(Z,"error");O(Z,"load");O(Z,"change");O(Z,"dblclick");O(Z,"input");O(Z,"keyup");O(Z,"keydown");O(Z,"keypress");O(Z,"mousedown");O(Z,"mouseenter");O(Z,"mouseleave");O(Z,"mouseout");O(Z,"mouseover");O(Z,"mouseup");O(Z,"paste");O(Z,"touchstart");O(Z,"touchend");O(Z,"touchcancel");O(Z,"speech");(function(b){google.jsad=function(d){Y(b,d)};google.jsaac=function(d){return V(b,d)};google.jsarc=function(d){X(d);for(var a=!1,c=0;c<b.a.length;++c)if(b.a[c]===d){b.a.splice(c,1);a=!0;break}if(!a)for(a=0;a<b.g.length;++a)if(b.g[a]===d){b.g.splice(a,1);break}U(b)}})(Z);window.gws_wizbind=function(b){return{trigger:function(d){var a=b.l(d.type);a||(O(b,d.type),a=b.l(d.type));var c=d.target||d.srcElement;a&&a.call(c.ownerDocument.documentElement,d)},bind:function(d){Y(b,d)}}}(Z);}).call(this);(function(){var b=[];google.jsc={xx:b,x:function(a){b.push(a)},mm:[],m:function(a){google.jsc.mm.length||(google.jsc.mm=a)}};}).call(this);(function(){google.c={};(function(){var f=window.performance;var g=function(a,b,c){a.addEventListener?a.addEventListener(b,c,!1):a.attachEvent&&a.attachEvent("on"+b,c)};google.timers={};google.startTick=function(a){google.timers[a]={t:{start:google.time()},e:{},m:{}}};google.tick=function(a,b,c){google.timers[a]||google.startTick(a);c=void 0!==c?c:google.time();b instanceof Array||(b=[b]);for(var e=0,d;d=b[e++];)google.timers[a].t[d]=c};google.c.e=function(a,b,c){google.timers[a].e[b]=c};google.c.b=function(a){var b=google.timers.load.m;b[a]&&google.ml(Error("a"),!1,{m:a});b[a]=!0};google.c.u=function(a){var b=google.timers.load.m;if(b[a]){b[a]=!1;for(a in b)if(b[a])return;google.csiReport()}else google.ml(Error("b"),!1,{m:a})};google.rll=function(a,b,c){var e=function(d){c(d);d=e;a.addEventListener?a.removeEventListener("load",d,!1):a.attachEvent&&a.detachEvent("onload",d);d=e;a.addEventListener?a.removeEventListener("error",d,!1):a.attachEvent&&a.detachEvent("onerror",d)};g(a,"load",e);b&&g(a,"error",e)};google.aft=function(a){a.setAttribute("data-iml",google.time())};google.startTick("load");var h=google.timers.load;a:{var k=h.t;if(f){var l=f.timing;if(l){var m=l.navigationStart,n=l.responseStart;if(n>m&&n<=k.start){k.start=n;h.wsrt=n-m;break a}}f.now&&(h.wsrt=Math.floor(f.now()))}}google.c.b("pr");google.c.b("xe");}).call(this);})();</script></head></html>

Thanks, you created a trivial example to prove YOUR case, not what I'm dealing with. BRAVO! Meanwhile, did you try it with a Google query, or from one of the news sites? If you did, you'd see what I'm pointing at. A guy on SO boldly asserted, "Just strip out the SCRIPT tags and parse the BODY." Yeah, well, what do you do when a site returns <HTML><HEAD>...</HEAD></HTML> with NO BODY TAG? It's just a big chunk of <SCRIPT>...</SCRIPT> tags in the HEADer. But, hey, your example works fine. Unfortunately, that's not what Google or these sites are returning. I understand very clearly how this is supposed to work IN THEORY. In PRACTICE ... you're going to have to actually run one of these (really simple) queries and see what I'm seeing, or you're going to keep implying that I'm hallucinating. Run a Google query for something simple like "dentists in baltimore" and see what your approach yields. Because that's what I'm dealing with and what I'm asking about. Clearly, nobody who's responded to my questions has bothered to look at what these sites are actually serving up these days!

Try it on a Google search or on some news sites, and let me know what you find.

What I can say with certainty is that every documented method I've used for extracting content from some of these web pages returns big chunks of JS code that's not readable to humans, even as the web browsers I'm extracting it from do, in fact, display a human-readable web page. As far as how or when that happens, I don't know exactly. That's why I'm asking here. TWebBrowser controls display these pages correctly, or people would not be using said controls, because among other things, Google searches would display as nonsense. My question is, what Delphi code is needed to extract the ultimately rendered HTML that exists inside of said controls and is made visible to users? BTW, I'm sure this sort of page encoding does, indeed, make it harder to crawl and scrape using existing tools. I think that's part of its intended use in many cases.

I need to parse a Google search result page. I'm getting the data into a TWebBrowser, and am using DIHTMLParser, but there's a problem. I found several possible solutions, but Google has put a bunch of javascript into the raw HTML page that's returned, so if you parse that page, you don't get much. I found someone on SO who suggested grabbing the IHTMLDocument2.body as an IHTMLElement, but it's nil. Perhaps it returns an IHTMLElementCollection in .all now, but the example code was from 2013, and I can't find any examples that go this direction in Delphi. (Does the .body element just contain what's in the BODY part of the HTML page?) I'm working with D10.2.2, in case it matters. And, yes, DIHTMLParser guys have a Google parser plugin that was last updated in 2018, but it doesn't work now. How can I get the HTML elements that are actually displayed in the TWebBrowser's window after the javascript has finished executing? I'm not finding anything in my searches that addresses this. (BTW, what's visible in TWebBrowser is the same as what shows up in Chrome.)

I've reposted this from a different perspective because people seem more fixated on the fact that I'm trying to parse a Google page or get search results than what the actual problem is. Please respond to my new question instead.

Yes, I'm specifically looking at Google. I'm wanting to automate a specific process that is currently being done manually that looks at several things that show up on the first page of a Google search result. A lot of folks wanting this info currently outsource it to people in Asia.

I have a general question that has nothing to do with ordering pizzas, per se. It's about designing a mobile app that could have a range of adaptability to current inventory states on the back-end. Think about an app you might build to order a pizza. There might be some combos that consist of a predefined set of toppings, and an option for a "made-to-order" selections. So you've probably got one tab with a list of different combo pizzas, that's likely a bunch of radio buttons (ie., pick a combo) on it. Then you probably have a separate tab with a bunch of check-boxes on it so you can pick whatever toppings you want individually. The crust will always have a few radio buttons (thin crust, thick crust, deep dish, stuffed crust), and the base topping might as well (tomato sauce vs. a white sauce, say). Now let's say you're designing a JSON/REST-based service to handle these orders on the back-end. I have a few questions about how folks here would approach this. First, would you define a class to represent all of the possible options in an order (ie, TPizzaOrder)? If so, how would you define it? Second, generally speaking, how would you structure the data being sent to the server? (It would seem that a class could have its own JSON I/O methods, making this really easy.) Finally, individual ingredients might come and go, depending on availability and popularity. Would you design the app's interface to load or update a list of currently available ingredients from the server each time it ran? New ingredients would need to have more data downloaded; otherwise, availability would be indicated by a bunch of booleans. The first time the app is run, someone would register, and then it might need to initialize with all of the current ingredients. (If you didn't do this, then the app would need to be updated every time the list of available options changes.) Oh, and there's current pricing data... I know a lot of this might sound pretty obvious, but I've seen apps with far fewer options that make very little effort to accomodate the variations, and I think the vendors are limiting themselves because the programmers aren't thinking about the problem clearly. (Or they're just afraid of the apparent complexity of the problem?)

Continued searching uncovered this: https://stackoverflow.com/questions/4082966/what-are-the-alternatives-now-that-the-google-web-search-api-has-been-deprecated Note: The Google Web Search API has been officially deprecated as of November 1, 2010. It will continue to work as per our deprecation policy, but the number of requests you may make per day will be limited. Therefore, we encourage you to move to the new Custom Search API. This seems to addresses the bigger question and tells me I'm not the only person stumped by this issue. This post says at the bottom: The search quality is much lower than normal Google search (no synonyms, "intelligence" etc.) It seems that Google is even planning to shut down this service completely. And some comments include these warnings: This is why Google claims that the search results are different support.google.com/customsearch/answer/141877?hl=en Mainly: Using specified sites (does not apply here), no social or personalized or real time results – MFARID Apr 27 '14 at 17:41 I tried it but it doesn't work now. I asked to look in the entire web for suunto ambit watch, but I got no results (I searched in the public URL that I got) – Dejell Feb 11 '15 at 20:0 WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" – Bryan Larsen Aug 11 '15 at 14:50 "On April 1, 2017, Google will discontinue sales of the Google Site Search. All new purchases and renewals must take place before this date. The product will be completely shut down by April 1, 2018." – Gajus Mar 7 '17 at 15:52 Also, it looks like a bunch of 3rd-party vendors have sprouted up who offer APIs for regular web searches. SerpWow looks interesting. Methinks if it's as simple as hooking into a standard Google API, there wouldn't be a lively 3rd-party market offering the same thing, or info on hacking the only existing API that seems to come close.

I'm not so sure ... this is a structural change. It was the same for a decade before it changed, and then nearly another decade. A REST API would be better, but only as long as it has sufficient info on it that would allow one to effectively create the same result that a normal query produces. I don't need to interact with anything, just pull off a few bits of data from each entry. And if there's a map at the top then I need to read the links that show up below it. The only API I found was for scanning URLs, not running general queries. But they've got so damn many APIs that it can be hard to find things if you don't know what they're probably called.

I'd like to have a check-list, but I want the user to be able to edit its contents easily, mostly by adding new entries and maybe reordering them. (I'm using checkboxes to avoid deleting things that may be used sometimes but not always. It's not going to be common to delete things in this case.) The way I've done this in the past, and how I've usually seen it done, is to somehow overlay a memo and a checklistbox of some sort, maybe using a page control. If you want to edit it, the data is loaded into a memo and the memo is displayed. When you want to go back to select items, the data is loaded into the checklistbox and it is displayed instead. Most line-oriented controls like TListBox and TListView aren't easy to edit. Memos are what people are most used to. I'm curious if anybody has any other approach that's worth trying?