docwiki.embarcadero.com is not working

[Anders Melander 1783]

20 minutes ago, Lajos Juhász said:

I believe that will be the perfect place to ask!

It's also the perfect place to get a noncommittal answer. I'll give you the answer right now so you don't have to wait:

Quote

We're aware of the problem. Finding a solution has very hight priority and we're working round the clock to get it resolved. Слава Україні!

Okay, maybe not that last part but that would at least have meaning.

[Mike Torrettinni 198]

10 hours ago, dummzeuch said:

The person responsible is sitting in Australia, the access to the server is only available during business hours in California and needs the approval from somebody in Europe. And the server does not have a reset button because that would be a security risk. 😉

If this is the case, this looks more and more like international failure 😉

[David Heffernan 2345]

19 hours ago, Mike Torrettinni said:

If this is the case, this looks more and more like international failure 😉

It isn't the case. Why are you speculating on something about which you have zero information?

[DJof SD 13]

I thought he was just continuing a joke.

[David Heffernan 2345]

59 minutes ago, DJof SD said:

I thought he was just continuing a joke.

I can't see a joke here. Maybe I should just mute the topic because there is nothing more to say. 

[DJof SD 13]

36 minutes ago, David Heffernan said:

I can't see a joke here. Maybe I should just mute the topic because there is nothing more to say. 

NP. A check of a number of BM's in my browser shows that clicking refresh enough times does eventually result in the desired page shown correctly.

 

I guess the quip about the support chain starting in Oz then continuing to CA and finally Europe wasn't a joke then. Oh well. My bad.

[jeroenp 26]

20 hours ago, DJof SD said:

NP. A check of a number of BM's in my browser shows that clicking refresh enough times does eventually result in the desired page shown correctly.

This is true: some of the time it is up.

 

Over the last 24 hours downtime is "more" than 55% (not the kind of SLA I would be satisfied with, but hey: I'm not Embarcadero IT), see the saved status at https://archive.ph/2HXRI:

Quote

56.631%

Last 24 hours

From the history on that page (or browsing the live status at https://stats.uptimerobot.com/3yP3quwNW/780058619), you can see that both uptime and downtime periods vary widely between roughly 5 and 90 minutes.

To me that sounds like intermittently failing or underdimensioned hardware.

 

Hovering over the red bars you see the uptime during the weekend (especially on Sunday) is better than during weekdays. Speculating further, this could have to do with access rates being less during these days.

 

The non-public status on the UptimeRobot maintenance page shows this graph which has better resolution than the public status page:

 

 

Response times are about twice as large as my blog on WordPress.com, see https://stats.uptimerobot.com/AN3Y5ClVn/778616355

I have also modified my docwiki http check to use deeper page (instead of the home page). That monitor is at https://stats.uptimerobot.com/3yP3quwNW/780058617

 

Let's see what the monthly average will be some 30 days from now.

 

--jeroen

[CyberPeter 11]

It's an absolute disgrace.  They really don't care do they ?  Plenty of bugs reported already on this issue in https://quality.embarcadero.com (also by me).

They tend to get closed as 'resolved' and duplicate.  Must be good for the stats.

I don't know what's worse, not being able to fix something like this or totally ignoring the community and not communication about it at all.

[Mike Torrettinni 198]

5 minutes ago, CyberPeter said:

They really don't care do they ? 

I assume they would care very much as soon as one (or a few) of the big customers would say to fix it before they pay their yearly fee.

 

8 minutes ago, CyberPeter said:

totally ignoring the community and not communication about it at all.

Eh, no big bucks in community itself.

 

Probably many of you guys who are part of bigger teams (2 developers and up 🙂 ) can say that rarely the whole team cares about stuff like that.

100% of my team is not happy about stuff like that is not shy about voicing it. But, my team is just me.

 

If one of the bigger teams would voice dissatisfaction to their bosses, as a whole team, perhaps they can move some stuff.

[Dalija Prasnikar 1396]

6 hours ago, CyberPeter said:

It's an absolute disgrace.  They really don't care do they ?  Plenty of bugs reported already on this issue in https://quality.embarcadero.com (also by me).

They tend to get closed as 'resolved' and duplicate.  Must be good for the stats.

I don't know what's worse, not being able to fix something like this or totally ignoring the community and not communication about it at all.

Issue needs to be reported only once. Adding duplicate reports does not help in solving issue faster.  When there is a duplicate report of any issue those are closed as duplicates to enable tracking status in one point not all over the place. This has nothing to do with statistics, this is basic QA workflow.

 

It would be nice to have official statement, but there is not much anyone here can do about it.

[A.M. Hoornweg 144]

15 hours ago, jeroenp said:

This is true: some of the time it is up.

 

Over the last 24 hours downtime is "more" than 55% (not the kind of SLA I would be satisfied with, but hey: I'm not Embarcadero IT), see the saved status at https://archive.ph/2HXRI:

From the history on that page (or browsing the live status at https://stats.uptimerobot.com/3yP3quwNW/780058619), you can see that both uptime and downtime periods vary widely between roughly 5 and 90 minutes.

To me that sounds like intermittently failing or underdimensioned hardware.

 

 

 

No it is not.

 

Guys, just take a look at the PHP exception trace.  More specifically, the line that reads  "LoadBalancer->getConnection(0,Array,false)", that's where the shit hits the fan.

 

That line triggers a "connection refused" exception. One of the servers which the load balancer can choose from in this array is inaccessible. 

Not all of them are inaccessible because when you press F5 often enough in your browser the load balancer will sooner or later hit one that works. 

 

So this is a simple web site management issue.  The person in charge should simply remove broken db servers from the list which the load balancer uses to choose from.

 

 

So... Why the heck does Idera not manage this properly?  

 

 

 

 

 

 

 

 

 

[Uwe Raabe 2057]

1 hour ago, A.M. Hoornweg said:

Why the heck does Idera not manage this properly?

Indeed, that seems to be more disturbing than the outage itself.

[dummzeuch 1505]

Maybe @A.M. Hoornweg should send an invoice to Emarcadero (Idera) for diagnosing the problem. That might get their attention more than us mere customers complaining.

[Vandrovnik 214]

I suppose it will be more difficult to find information on docwiki.embarcadero.com using Google, because Google Bot probably most of the time also sees just errors...

[DJof SD 13]

Maybe it'll be fixed after Wednesday. See: https://register.gotowebinar.com/register/7497661470554577420

[jeroenp 26]

7 hours ago, A.M. Hoornweg said:

 

No it is not.

 

Guys, just take a look at the PHP exception trace.  More specifically, the line that reads  "LoadBalancer->getConnection(0,Array,false)", that's where the shit hits the fan.

 

That line triggers a "connection refused" exception. One of the servers which the load balancer can choose from in this array is inaccessible. 

Not all of them are inaccessible because when you press F5 often enough in your browser the load balancer will sooner or later hit one that works. 

 

So this is a simple web site management issue.  The person in charge should simply remove broken db servers from the list which the load balancer uses to choose from.

 

That last sentence might not be completely true.

Based on the MediaWiki 1.31.1 source code, I drafted a blog post yesterday. It is not published yet as I ran into some WayBackMachine and Archive Today trouble: they are both slow and the Archive Today redirect mentioned at https://blog.archive.today/post/677924517649252352/why-has-the-url-archive-li-changed-to coincided with HTTP-302 redirect loops on my side.

 

This was my conclusion in the blog post is this:

Quote

Failing method is called by the class starting at [Wayback/Archive] .../mediawiki/blob/1.31.1/includes/cache/localisation/LCStoreDB.php - Line 28

Some of the requests succeed, so there seem to be three possibilities:

1. Sometimes the load balancer cannot get a database connection at all
2. Sometimes the load balancer gets a valid connection and that connection then fails returning a query
3. Sometimes the load balancer gets a valid connection and that connection succeeds returning a query

That their largest and most important site is still failing and there is no communication from Embarcadero either on social media or 3rd party forums (they do not have their own forums any more) is inexcusable.

So it might be that it is not a single point of failure, and even underdimensioned might not cut it.

 

Then to your next excellent question:

7 hours ago, A.M. Hoornweg said:

So... Why the heck does Idera not manage this properly?  

I have been wondering about this since like forever. Even in the Borland days, uptime was often based on fragility and this has not improved much.

 

Having no status page is also really outdated. Looking at some TIOBE index languages surrounding Delphi:

 

- https://status.rubygems.org/

- https://cran.r-project.org/mirmon_report.html

- https://status.mathworks.com/

 

[jeroenp 26]

6 hours ago, Vandrovnik said:

I suppose it will be more difficult to find information on docwiki.embarcadero.com using Google, because Google Bot probably most of the time also sees just errors...

 

This is in part why Meik Tranel offered to host an export of the database. Another part that it is driving people nuts that the only reasonable search index will point to (sometimes cached) pages, but one cannot access them.

 

Hopefully:

- not all Embarcadero docwiki database servers have issues

- there is recent a back-up that can be restored if they have

 

Fingers crossed....

 

 

[Remy Lebeau 1396]

11 hours ago, A.M. Hoornweg said:

Guys, just take a look at the PHP exception trace.  More specifically, the line that reads  "LoadBalancer->getConnection(0,Array,false)", that's where the shit hits the fan.

It is not just the load balancer that is having issues.  For example, trying to access several pages today, I'm running into a new kind of error:

Quote

Wikimedia\Rdbms\DBQueryError from line 1457 of /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: SELECT lc_value FROM `lib_sydney_en_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'deps' LIMIT 1
Function: LCStoreDB::get
Error: 1146 Table 'wikidb.lib_sydney_en_l10n_cache' doesn't exist (10.50.1.120)

 

Backtrace:

#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1427): Wikimedia\Rdbms\Database->makeQueryException(string, integer, string, string)
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1200): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1653): Wikimedia\Rdbms\Database->query(string, string)
#3 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/database/Database.php(1479): Wikimedia\Rdbms\Database->select(string, string, array, string, array, array)
#4 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(52): Wikimedia\Rdbms\Database->selectField(string, string, array, string)
#5 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(412): LCStoreDB->get(string, string)
#6 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(458): LocalisationCache->isExpired(string)
#7 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(334): LocalisationCache->initLanguage(string)
#8 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(371): LocalisationCache->loadItem(string, string)
#9 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LocalisationCache.php(292): LocalisationCache->loadSubitem(string, string, string)
#10 /var/www/html/shared/BaseWiki31/languages/Language.php(3177): LocalisationCache->getSubitem(string, string, string)
#11 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(352): Language->getMagic(MagicWord)
#12 /var/www/html/shared/BaseWiki31/includes/MagicWord.php(280): MagicWord->load(string)
#13 /var/www/html/shared/BaseWiki31/includes/parser/Parser.php(4848): MagicWord::get(string)
#14 /var/www/html/shared/BaseWiki31/extensions/TreeAndMenu/TreeAndMenu_body.php(24): Parser->setFunctionHook(string, array)
#15 /var/www/html/shared/BaseWiki31/includes/Setup.php(948): TreeAndMenu->setup()
#16 /var/www/html/shared/BaseWiki31/includes/WebStart.php(88): require_once(string)
#17 /var/www/html/shared/BaseWiki31/index.php(39): require(string)
#18 {main}

 

[jeroenp 26]

12 hours ago, Remy Lebeau said:

It is not just the load balancer that is having issues.  For example, trying to access several pages today, I'm running into a new kind of error:

...

Error: 1146 Table 'wikidb.lib_sydney_en_l10n_cache' doesn't exist (10.50.1.120)

 

Whoa, that at first very much confused me thinking there was a data integrity error, but then after realising lib_sydney_en_l10n_cache - Google Search didn't return results I reproduced it with a different one taking some 10 seconds for the request to even get displayed:

Error: 1146 Table 'wikidb.rad_xe8_en_l10n_cache' doesn't exist (10.50.1.120)

Besides the very long response time (how slow can a database lookup be?), look at the table names:

• lib_sydney_en_l10n_cache
• rad_xe8_en_l10n_cache

They have two different prefixes (lib and rad) and two different product codes (sydney and xe8).

This looks like a a setup with each product version having at least two different Wikimedia databases each having an l10n_cache table (and likely copies being made for each new product version, which I can understand from a versioning perspective) all integrated in one documentation site.

 

Searching for [Wayback/Archive] l10n_cache - Google Search resulted in [Wayback/Archive] Manual:l10n_cache table - MediaWiki (which describes the table for all ranges of Mediawiki versions) and a whole load of pages with various circumstances in which people bump into missing this table.

Then I looked at the status monitor [Wayback/Archive] Docwiki https - EmbarcaderoMonitoring - docwiki https where it looks someone started working on it almost 15 hours ago:

Response Time Last 2 days

5012.65ms / 10372.00ms / 214.00ms

Avg. response time / Max. response time / Min. response time

Recent events

Down for 14 h, 40 min

The reason is Internal Server Error.500

Details:The server encountered an unexpected condition that prevented it from fulfilling the request.

March 7, 2022, 17:57 GMT +00:00

Running again

March 7, 2022, 17:46 GMT +00:00

I really really hope they know what they are doing, as right now the databases don't look well and things have not improved for more than 15 hours (I was interrupted while writing this reply).

 

--jeroen

 

[A.M. Hoornweg 144]

Maybe these servers were outsourced to a country now suddenly under embargo and Idera has been locked out ?  

[tgbs 14]

On 3/4/2022 at 1:25 PM, Vandrovnik said:

... and these are silently ignored, I guess 😞

 

[RSP-37138] Product documentation has been unavailable for several days - Embarcadero Technologies

Internal status: Validation. How much time does it need just to validate this problem?

 

This too

https://quality.embarcadero.com/browse/RSP-37004

Edited March 8, 2022 by tgbs

[jeroenp 26]

38 minutes ago, A.M. Hoornweg said:

Maybe these servers were outsourced to a country now suddenly under embargo and Idera has been locked out ?  

That chance is slim because 

DNS records for docwiki.embarcadero.com — NsLookup.io

A records

	IPv4 address
	204.61.221.12

 

QUASAR DATA CENTER, LTD.

Location	Houston, Texas, United States of America
AS	AS46785
AS name	QUASAR DATA CENTER, LTD.

 

 

The blog post I wrote thanks to loads of input in this thread: The Delphi documentation site docwiki.embarcadero.com has been down/up oscillating for 4 days is now down for almost a day. « The Wiert Corner – irregular stream of stuff

[Mike Torrettinni 198]

I guess it's all business as usual, they are improving it:

 

 

So, no more complaining, let's give them time to finish it.

[David Heffernan 2345]

12 minutes ago, Mike Torrettinni said:

I guess it's all business as usual, they are improving it:

No, this isn't business as usual. The broken docwiki is not because they are improving it. I know that they are working on a new system, but the old one broke on its own accord. At least that's what I understand from a PM from Marco.

[Mike Torrettinni 198]

16 hours ago, Mike Torrettinni said:

let's give them time to finish it.

until 11.1.

Edited March 9, 2022 by Mike Torrettinni