OT: Facebook, Instagram, WhatsApp down

[Docbop]

Normally wouldn't mention this, but surprised they've been down for hours now. After working as SysAdmin for large companies I can imagine the nightmare of Pointy Headed Bosses (PHB) running around clueless and think yelling and constantly asking "what's the ETR???". I remember when at Earthlink we had a outtage not on the internet, but on the customer website. One PHB was running into the server room every three minutes asking for an ETR (estimate time to repair). He got so bad we locked him out of the server room so we could actually work on the problem.

 

 

But being as big as Facebook and Instagram are really surprised they are still off the air this long. Going to be interesting to find out what the problem was when this is over.

[SamuelBLupowitz]

Apparently the DNS records for all three apps are mysteriously gone, as well -- "facebook.com" was shown as available for sale this afternoon. So this is not an accident. I bet Mark Zuckerberg is having a very bad day.

[El Lobo]

Going to be interesting to find out what the problem was when this is over.

If you ever find out, would you let us know in this thread? Curious.

[ABECK]

When the envelop from the hosting company says "Final Notice", you gotta pay.

[ABECK]

Apparently the DNS records for all three apps are mysteriously gone, as well -- "facebook.com" was shown as available for sale this afternoon. So this is not an accident. I bet Mark Zuckerberg is having a very bad day.

First the whistleblower, now this. Finding it hard not to enjoy it a little.

[Winston Psmith]

A hack this big, it's not some guy eating corn chips in his basement, looking for an exploit, this took an organization. I would not be surprised to find some intelligence agency behind it.

[El Lobo]

"The reason for the outage was not immediately clear. However, multiple security experts pointed to a Domain Name System problem as a possible culprit. A spokesperson for the company did not immediately respond to a request for comment from CNN Business.

The outage came the morning after "60 Minutes" aired a segment in which Facebook whistleblower Frances Haugen claimed the company is aware of how its platforms are used to spread hate, violence and misinformation, and that Facebook has tried to hide that evidence. Facebook has pushed back on those claims.

The interview followed weeks of reporting about and criticism of Facebook after Haugen released thousands of pages of internal documents to regulators and the Wall Street Journal. Haugen is set to testify before a Senate subcommittee on Tuesday."

https://www.cnn.com/2021/10/04/tech/facebook-instagram-whatsapp-outage/index.html

[El Lobo]

Finding it hard not to enjoy it a little.

This is kinda fun.

[OB Dave]

Joe broke it.

[SamuelBLupowitz]

Finding it hard not to enjoy it a little.

This is kinda fun.

Yes, my glee over this enormous evil tech conglomerate getting kicked in the proverbial nuts is helping balance out the jitters I am feeling from not being able to compulsively check the social media apps that I am clearly addicted to.

[Stokely]

Probably would have bothered me at some point, when I went to do my every-six-minute check on what I was missing out (that was the figure put on it by a study I read about the anxiety these apps cause people).

 

Ditched it in 2016 after too much election nonsense, after a weaning-off period my stress levels went way down. And I'm not putting (as much) money in Zuck's pocket, the ****.

[drawbars]

Would love to ditch it, but the local patrons won't even bother checking my website. "That's OK, but where's your Facebook update?"

[Stokely]

Oh no doubt, if I needed to use it for business I would. At least that way though the "mode" of use would be very targeted and purposeful, no politics or looking at "news" feeds etc

[AnotherScott]

There was an outage earlier in the day, and an explanation was posted (copied below, found via google cache)... but things seem to have gotten worse since then. It was up for a while in the interim.

 

========================================

 

More Details on Today's Outage

 

Facebook Engineering

 

Early today Facebook was down or unreachable for many of you for approximately 2.5 hours. This is the worst outage we"ve had in over four years, and we wanted to first of all apologize for it. We also wanted to provide much more technical detail on what happened and share one big lesson learned.

 

The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration values ended up causing much more damage than it fixed.

 

The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store. This works well for a transient problem with the cache, but it doesn"t work when the persistent store is invalid.

 

Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

 

To make matters worse, every time a client got an error attempting to query one of the databases it interpreted it as an invalid value, and deleted the corresponding cache key. This meant that even after the original problem had been fixed, the stream of queries continued. As long as the databases failed to service some of the requests, they were causing even more requests to themselves. We had entered a feedback loop that didn"t allow the databases to recover.

 

The way to stop the feedback cycle was quite painful - we had to stop all traffic to this database cluster, which meant turning off the site. Once the databases had recovered and the root cause had been fixed, we slowly allowed more people back onto the site.

 

This got the site back up and running today, and for now we"ve turned off the system that attempts to correct configuration values. We"re exploring new designs for this configuration system following design patterns of other systems at Facebook that deal more gracefully with feedback loops and transient spikes.

 

We apologize again for the site outage, and we want you to know that we take the performance and reliability of Facebook very seriously.

[Docbop]

Here's what Slashdot has on the screwage at FB, IG, WA.

 

>>>>

Facebook -- and all the major services that Facebook owns -- are down today. ArsTechnica:

We first noticed the problem at about 11:30 am Eastern time, when some Facebook links stopped working. Investigating a bit further showed major DNS failures at Facebook: "Google anycast DNS returns SERVFAIL for Facebook queries; querying http://a.ns.facebook.com directly times out."

 

The problem goes deeper than Facebook's obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services -- which are hosted on Amazon rather than being internal to Facebook's own network -- were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services' load balancers were reachable, the application servers that should be feeding the load balancers were not. A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. With no BGP routes into Facebook's network, Facebook's own DNS servers would be unreachable -- as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR.

 

<<<<<

[ABECK]

Did Zuck take his toys and go home over frustration with his playmates?

[Stokely]

Won't someone think of all the alternate reality misinformation bubbles put at risk by this...actual reality might flood the vacuum!

[Rod S]

Apparently the DNS records for all three apps are mysteriously gone, as well -- "facebook.com" was shown as available for sale this afternoon. So this is not an accident. I bet Mark Zuckerberg is having a very bad day.

First the whistleblower, now this. Finding it hard not to enjoy it a little.

Having worked in multinationals that have had a share of hate in the past, I'll probably eat my words.

 

But... Schadenfreude at its best...

 

I do use WhatsApp fairly frequent during the day for quick communications (business and personal), so it's been a bit annoying.

[El Lobo]

Would love to ditch it, but the local patrons won't even bother checking my website. "That's OK, but where's your Facebook update?"

^ You know, that's the thing. I get a solid 50% (if not more) of gig inquires on FB messenger. In spite of (most) people having my email and cell #.

Bands I'm in don't use web sites anymore; everything comes through Facebook. I email a long list of friends and fans, but significant numbers use my FB posts to know about gigs. I can manage a temporary outage. Not much happens on Monday. But if FB stays down for a long time, I'll have to find another way to communicate.

[ElmerJFudd]

Could this be related to their dns servers hosting the dns zone for 'domain.com' and then putting them on 'domain.com'? Wouldn"t it make sense to have another domain so if it goes down you are not locked out?

 

https://arstechnica.com/information-technology/2021/10/facebook-instagram-whatsapp-and-oculus-are-down-heres-what-we-know/

[ElmerJFudd]

Oh my, interesting reaction attached below.

[RABid]

Just tried to log on with iPhone. Nope. Ipad. No. MacBook. Again, no. Thought the Big Z finally made good on his promise to block Apple users.

[KuruPrionz]

[AnotherScott]

I kind of hope it stays down for, oh, maybe a week or so. Enough to show people that, yes, they can indeed live without facebook.

[hardware]

Housekeeping chores, nothing more.

 

Regulators and politicians are owned by big tech investors. We"ll get a show, some grandstanding, and the overdue market correction.

 

I didn"t even notice what happened until my X frantically called me freaked out that her WiFi stopped working.

 

As if I could treat her trauma?

[Mighty Motif Max]

Instagram in particular is a big thing for younger people and also is growing for business promotional purposes, not to mention that a lot of social media use is for staying in touch with people who you might not be geographically around all the time. Granted, there is sometimes also the negative aspect of people having to give the appearance that everything in their lives is perfect all the time, but (IMO) that's not so much the fault of a platform existing as much as it is a reflection of larger social issues that this is the symptom of.

 

You can live without it but at least with younger folks they don't really think about any other way to communicate - phone calls are hardly a "thing" for anything social, emails are only for "official" stuff, etc. It's more of an entire societal shift. And if this outage ends up spanning a long amount of time like some here would like, it's annoying when you might, say, be someone like me where you're geographically isolated from people and it's not "standard" to have any other contact info for people and actually were waiting on a response for something important. And thus it isn't as easy to get that if the one method you did have for contact is gone too.

 

A day or two? Eh, not the end of the world. Gone for, say, a week, two weeks, etc? Depends...as an example I'm very heavily involved in the dog competitive sports world and generally you would post wins on that weekend; that's pretty much expected. If your business gets a lot of engagement from social media, as many do now, this could also be an issue. If there was anything time-sensitive that depended on that platform, say, a concert livestream, that would also be bad.

 

OTOH it's less vicious arguments in the comments for the general population too.

[TommyRude]

Facebook is down but Keyboard Corner is up.

 

All is right in the world.

[GovernorSilver]

^ You know, that's the thing. I get a solid 50% (if not more) of gig inquires on FB messenger. In spite of (most) people having my email and cell #.

 

Bands I'm in don't use web sites anymore; everything comes through Facebook. I email a long list of friends and fans, but significant numbers use my FB posts to know about gigs. I can manage a temporary outage. Not much happens on Monday. But if FB stays down for a long time, I'll have to find another way to communicate.

 

I've been hearing similar thoughts by other working musos as well. The realization of how much professional musicians depend on Facebook and Instagram to maintain an online presence for maintaining relationships with fans and each other, attracting new fans, marketing their products and services, etc.

 

There's also Soundcloud and Youtube but both platforms have weaknesses in the communication aspect compared to FB and IG

 

Hopefully this incident will encourage working musos to get websites set up again, even if the web page is just a collection of links to their FB, IG, Twitter, etc. Just some centralized location that fans know to go to, that's separate from the other platforms.

[Winston Psmith]

FWIW, I still can't log on.

 

I signed on for the two-factor identification a while back, where they send a 6-digit code number to my phone; can't get a confirmation #, and can't get any help from FB.

 

I think that once I do manage to log on again, I'm just going to delete my page and account.

[Stokely]

They make that sort of difficult. You can disable your account but if you happen to go back to it, bam it's up again, no longer disabled.

 

I fixed that by removing my hundreds of friends, which took work (and letting any actual friends know my contact info), then changing my name. You can only change your name once every six months or so. Kind of like salting the earth to avoid the temptation to get back on there, no way I'm going to go through all that work to add people back in!