Last week, Verizon caused a major BGP misroute that took large chunks of the Internet, including CDN company Cloudflare, partially down for a day. This week, the rest of the Internet has apparently asked Verizon to hold its beer.
Cloudflare went down again for half an hour yesterday, and this time, it was the company's own fault—we're still waiting on a full post-mortem, but the short version is that a firewall regular expression rule targeting malicious Javascript spiked the firewalls' CPU usage, crippling throughput and causing widespread HTTP 502 errors. Microsoft's Office365 also seems to have experienced a multi-hour partial outage yesterday, with the service working over some ISPs and routes but not others for about four hours.
Facebook and its properties WhatsApp and Instagram have suffered widespread outages relating to image display for most of today. The problem seems to be bad timestamp data being fed to the company's CDN in some image tags; when I looked into the broken images littering my own Facebook timeline, I discovered different timestamp arguments embedded in the same URLs. Loading an image from fbcdn.net with bad "oh=" and "oe=" arguments—or no arguments at all—results in an HTTP 403 "Bad URL timestamp".
If you dig into the code, Facebook refers to the same image URL with different arguments in its anchor tags and its image tags; in the posts with broken images, the URLs in the anchor tags work, but the ones in the image tags don't. (On the plus side, now Read More – Source
[contf] [contfnew]
Ars Technica
[contfnewc] [contfnewc]