FYI - Several Embarcadero services are currently unavailable

[Dalija Prasnikar 1341]

I am sure that everyone here has everything in duplicate, so when something dies they can replace it in an instance with minimal interruptions, especially if it dies over the weekend during the huge cold spell. Stuff happens, no matter how small or big the company is. Do I need to remind you that on one instance Facebook folks had to literally break into their server room because they got locked out of it. 

 

I am not going to defend the lack of official communication, this is something that needs to change, but this is also not something that people on lower hierarchy levels (meaning the ones that blog and communicate with us) can do on their own.

[Lars Fosdal 1732]

I agree. You can't blame the client facing people.

The management needs a good kick in the shins, though.

 

[Joseph MItzen 243]

Don't worry; I figured out where they can get a replacement server!

 

 

[David Heffernan 2300]

1 hour ago, Dalija Prasnikar said:

I am sure that everyone here has everything in duplicate, so when something dies they can replace it in an instance with minimal interruptions, especially if it dies over the weekend during the huge cold spell.

If I'm running a company like Emba, then I'm not putting production servers on physical hardware with single points of failure.

 

1 hour ago, Dalija Prasnikar said:

Do I need to remind you that on one instance Facebook folks had to literally break into their server room because they got locked out of it. 

I mean, perhaps there are some incompetent people at Facebook too. I wonder if they learnt from this and changed.

[Dalija Prasnikar 1341]

Just now, David Heffernan said:

If I'm running a company like Emba, then I'm not putting production servers on physical hardware with single points of failure.

Who said that it is server that failed?

 

When workers cut my phone cable I was without access to Internet it for three days (that was long before broadband Internet was available). There is just some stuff where you cannot possibly have redundancy, or the cost would just be a way too much. It is not like they are running mission critical infrastructure, so they need to be prepared for absolutely every scenario. And even then, there are things that might happen outside of their (or anyone's) control.

 

Not everything that ever happens is result of some incompetence.

 

As far as Facebook is concerned you can read it here https://en.wikipedia.org/wiki/2021_Facebook_outage What this page does not cover is breaking into the server room, but that was something one of people involved posted on Twitter at the time. The problem was that entering the server room also required authentication which was not working because their whole network infrastructure was down. I have no idea what they have or have not learned from that incident. 

[Vandrovnik 202]

1 hour ago, Dalija Prasnikar said:

Who said that it is server that failed?

Nobody - but they don't say anything.

And since they have outages (QC, Getit) frequently, I'm willing to believe they have a lot of room for improvement.

[Dalija Prasnikar 1341]

16 minutes ago, Vandrovnik said:

Nobody - but they don't say anything.

This is why I said that this was not a software problem, but a hardware problem. Beyond that anything anyone says is pure speculation.

[Brian Evans 83]

Embarcadero doesn't do clustering - just look at Interbase and RAD Server. The result is poor availability of services. While a bit complex Windows Server Failover Clustering with a cluster aware database provides solid availability during both maintence and hardware failures. 

 

This downtime might be a needed dose of reality - claiming "Enterprise" services with such poor availability is not a good look. 

[Lajos Juhász 239]

The Quality Portal is back online. Docwiki is still have issues, it starts loading (blue background) then:

 

#0 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1134): Wikimedia\Rdbms\Database->reportConnectionError('No route to hos...')
#1 /var/www/html/shared/BaseWiki31/includes/libs/rdbms/loadbalancer/LoadBalancer.php(749): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /var/www/html/shared/BaseWiki31/includes/GlobalFunctions.php(2801): Wikimedia\Rdbms\LoadBalancer->getConnection(0, Array, false)
#3 /var/www/html/shared/BaseWiki31/includes/cache/localisation/LCStoreDB.php(45): wfGetDB(-1)

[Celso Henrique 8]

 

[Dalija Prasnikar 1341]

https://blogs.embarcadero.com/we-are-experiencing-a-hardware-outage/

[Lars Fosdal 1732]

Six Sigma and Five Nines opportunity lost 😛

[David Heffernan 2300]

55 minutes ago, Dalija Prasnikar said:

https://blogs.embarcadero.com/we-are-experiencing-a-hardware-outage/

"This is caused by a hardware outage at one of our data centers. RAD Studio 12 download volume spikes tasked our infrastructure more than expected. " 

 

Why would it spike weeks after release of the software?? 

[Brandon Staggs 185]

Does anyone have an insight on what keeps a company like Embarcadero using their own servers for this kind of stuff rather than something like AWS or Azure? (This is not me making a judgment, just genuine curiosity.) 

[Uwe Raabe 1890]

Neat: https://www.idera.com/ppc/uptime-infrastructure-monitor/it-performance-monitoring

[Lajos Juhász 239]

34 minutes ago, Uwe Raabe said:

https://www.idera.com/ppc/uptime-infrastructure-monitor/it-performance-monitoring

 

Nice idea, maybe they should by a licence.

[Vandrovnik 202]

1 hour ago, David Heffernan said:

"This is caused by a hardware outage at one of our data centers. RAD Studio 12 download volume spikes tasked our infrastructure more than expected. " 

 

Why would it spike weeks after release of the software?? 

And how could spikes in download even destroy a server (hardware)...

[Uwe Raabe 1890]

2 minutes ago, Vandrovnik said:

And how could spikes in download even destroy a server (hardware)...

I guess that is just a coincidence. Given the number of outages before, it probably is that piece of hardware dying since months.

[Rollo62 502]

Good to hear that D12 is so successful that datacenters starts boiling

Move on like that  

 

[wright 2]

1 hour ago, Brandon Staggs said:

Does anyone have an insight on what keeps a company like Embarcadero using their own servers for this kind of stuff rather than something like AWS or Azure? (This is not me making a judgment, just genuine curiosity.) 

Why would embarcadero use their competitor's servers?
AWS   => AMAZON 
Azure => Microsoft (MS Visual Studio) 

[Brandon Staggs 185]

7 minutes ago, wright said:

Why would embarcadero use their competitor's servers?
AWS   => AMAZON 
Azure => Microsoft (MS Visual Studio) 

I was not aware that Embarcadero competes with AWS or Azure for cloud services. Do you mean their parent company Idera? They offer cloud services? (Then why wouldn't they use that and let the cloud service provider handle the virtualization and hardware?)

 

If you mean that Embarcadero competes, in general, with those companies, I don't see the relevance... Delphi is a Windows-only IDE, so they have to use Windows so they already use Microsoft licenses and services, and they also provide components for using AWS in Delphi, though I have never used them, so that doesn't seem like a deal-breaker.

 

Again, I am just curious. I asked if anyone here as insight on why they use in-house servers, and you suggest it has to do with not using competitors' services. How do you know that is their reason? Besides, they could use another cloud provider besides those.

Edited January 17 by Brandon Staggs

[Stefan Glienke 1905]

Did you know that LEGO and Michelin are competitors? They both manufacture tires...

[Joseph MItzen 243]

2 hours ago, Brandon Staggs said:

Again, I am just curious. I asked if anyone here as insight on why they use in-house servers

Like a lot of stuff at this company, things are probably done a certain way just because they've always been done a certain way by the same people who have worked there since the 80s or 90s.

[Joseph MItzen 243]

10 hours ago, Dalija Prasnikar said:

It is not like they are running mission critical infrastructure, so they need to be prepared for absolutely every scenario

I would think delivering software and information to customers would be mission-critical infrastructure. Honestly, if you're in charge of a data center, redundancy and failover are basic elements of the job, like making sure a bank account isn't overdrawn is to accounting. Netflix periodically shuts down 10% of its infrastructure to be sure their system is resistant to failure!

 

Just like the old forum used to be made of custom code held together with wire and chewing gum and managed by a volunteer employee in their spare time, it sounds like Embarcadero has one on-premises server for all the Delphi stuff with no failover, geographically-separated redundancy, etc. Given the rock-bottom price of hardware today in addition to the ready availability of cloud virtualization, there's no excuse for not having a physical or virtual standby. On top of that, server problems aren't a sudden development like your severed cable example. Drawing upon what Uwe said, it seems reasonable to conclude they just kept ignoring recurring problems or beating it with a stick until a component failed completely, at which point they ordered a replacement.

[Attila Kovacs 617]