"Show me the incentives and I'll show you the outcome"
With Facebook products, you see outages in December and May. Why? Those are the last few weeks to complete your project before your performance review. Miss this window and you find yourself in career trouble. Facebook employees put a tremendous amount of pressure on themselves during these months.
I would not be surprised if this outage was caused by a bad pull request related to a new project.
Thanksgiving and Christmas are then some of the quietest times in terms of outages. Of course, the interval right after then is usually worse, since all of the backlogged changes from the holidays land at once.
There is a period just before the freeze which can also get a bit hectic, but we should be well beyond that point for most relevant services by now. Don't know the exact dates since I left in September. But your point stands: holiday freezes exist and are taken seriously.
Depending on your current level, this can be the same thing (Facebook has an "engineering progression" - there's a time frame within which you're expected to be promoted to a certain level).
You go through PSC as if you are the needed level.
So you're calibrated as if you are a 5. But if you're still a 4 because you're not delivering as a 5... then you'll fail to meet expectations, naturally. The rest sort of takes care of itself.
Pro tip: ask them for the severance instead of the PIP, and GTFO.
One or the other, I think they usually try to put you on a PIP before firing though. It can feel like an up or out kind of culture, but the timelines are fairly reasonable IMO - e.g. they give you 5 years to progress from an entry-level SWE to mid/senior-level SWE (L3 -> L5).
In FAANG for typical SWEs it is rare for to outright get fired for a mistake. How you handle the mistake matters a lot though.
Usually if things are less than perfect during performance reviews, you just don't get promoted.
i'm not necessarily familiar with all the services (or the specific dependencies) on that list, but a lot of it will be a knock on effect from e.g: fb login not working as expected
People will very often see it as "service x with fb login is broken"
I work in one of these companies and I can confirm that is a network outage and even our internal tools hosted in our datacenter (so no dependencies with fb) are down
nope. proper and full implementation of BCP38, MANRS, full RPKI validation of all IP ranges and such is a long distance away.
within trusted groups of ISPs where the admins all know each other, in certain specific geographic regions, it's better than others.. I would say that the number of ISPs I could call "fully compliant" with current best practices, in the Portland-Seattle-Vancouver area, is higher than in many other parts of the world.
Anecdote: I was surprised to find they honour TTLs on their DNS service. Everyone else I have access to seems to return the new IP almost immediately, but shaw waits it out.
ARIN, RIPE and APNIC publish a great deal of useful documentation on how to set up BGP stuff properly in the modern era. And information for how to fully implement dual stack with ipv6.
As for other current best practices there's really a very great variety of disciplines involved in ISP operations, depending on what type of ISP it is. What's relevant for a GPON FTTH last mile provider might not be as relevant for a colocation/dedicated server ISP. Or for a WISP. My recommendation on that would be to pick the subject to research and make inquiries in the special interest groups dedicated to that (for instance, two way satellite, or automated billing/provisioning systems for hosting companies, or CRM systems, or whatever).
To be fair, BGP only runs between large network operators. The security / trust of routing agreements is mostly solved through business deals and contracts.
Every time this happens I have a mental metaphorical image of somebody tripping over the $5 power strip that has the single point of failure route-server or route-reflector plugged into it.
Any info on what network/operator/operator/etc is suffering the issue? (just so people know about it if it's public, i don't see anything on HN at least)
Root Cause Analysis has determined that solar flares, which come from the Sun, are the reason any life exists on Earth. Without the sun, no technical evolution would have occurred and no outage in Facebook's infrastructure would have happened.
This red "major outage" status has been reported on their dashboard since at least Dec 2, when an (unrelated to messenger) api bug was introduced that I've been waiting on a fix for.
It has been surprisingly slow to get a fix or an update.
Depending on the company, "major" can be used to mean a number of things, such as visible and painful customer impact, or significant revenue impact.
Based on the description of the outage (issues serving new ads to Messenger), it reads like the latter.
> We are currently investigating an issue where creating new ads with app_destination as MESSENGER results in the following error ...
Admittedly if it does have significant revenue impact, I'm surprised that change hasn't been rolled back or otherwise mitigated in the week+ that the incident's been open. Or maybe it has, and nobody remembered to close the incident.
It looks like they've had issues for a while now. 'December 2nd' ? "
Invalid Page Welcome Message for Messenger Destination Ads"
- The thought of 'ads' in a private messaging app repulses me. I don't use any of these three apps. Mostly since they all require you hand over your mobile phone number. Which then becomes a very powerful 'foreign key'/unique identifier to so many other third or fourth-party marketing databases which might also have it.
I know Telegram is less secure and Secret Chats are an opt-in. Security-wise Signal would be better... but:
Signal experience isn't that good. Alien from the platform they run on. For example, no system recommendations to send this person a message or share something with them through signal.. Telegram is more a platform native and integrated into the system.
Also there is the issue of messages arriving late (not minutes but hours late, sometimes days late)
I can share to Signal or contacts-via-Signal, and have been for years... I think this is a you problem.
Signal is honestly fantastic as a SMS-replacement app. It sends Signal messages to contacts with Signal and normal SMS to contacts without (Telegram might do the same thing, being able to set it as your default SMS app, I'm not sure).
In fact phone operators for a while tried to ban whatsapp here, SMS prices in Brazil are just nuts (it is normal for an SMS cost fifty cents or so, sometimes even more, a full SMS conversation can cost you several days worth of food).
Everyone here just uses whatsapp, because of how ridiculous SMS prices are.
It must be a market difference. Here in north america, SMS is the "default" way you message new contacts, and the majority of chats I have with people are over SMS. SMS is free and unlimited on virtually every phone plan available.
Yeah I see the statement a lot on international forums like HN and Reddit of "SMS is still a thing? No one uses it!" But it's highly cultural. I only use WhatApp for two friends in Europe. Most of my friends here in the US use Facebook Messenger for our group chats. My iPhone-using friends use iMessage, but if I'm communicating with an Android-using friend, it's just plain SMS. I prefer more robust platforms but I also send hundreds of plain SMS messages each month. It's just easier than trying to figure out if they're on WhatsApp, Messenger, Signal, Discord, Telegraph, Hello/Allo/Duo/etc... everyone has SMS and everyone has a phone number so it's the lowest friction way to communicate.
That's what I love about these forums, is getting the cultural experiences of other people and finding out they're not always the same as your experiences. Very eye-opening.
Exactly. Signal appears in Android's share feature. I have a friend group on it where we send each other funny images and I never had any problems sharing to the group from Chrome or apps.
WhatsApp secret conversations don't carry either over multiple devices if you don't activate sync with google drive (which kind of defeats the purpose of encryption anyway).
I just got onto Telegram this week and was pleasantly surprised to see completely native apps across iOS, iPadOS and a full featured native Mac OS X app.
I wish Signal didn't require a phone number, otherwise everything else about it is nice. I don't want to be required to have a phone number, let alone provide it to them.
Oh, don't worry the Windows client is still crap as well. A few months ago I couldn't even launch from the Start menu and install was hanging until I found some obscure reference to deleting registry keys, a process which I haven't had to do for years.
Whatsapp is extremely dominant in my country, but I keep encouraging friends and acquaintances to install Telegram. As far as I know, there won't be anything preventing FB from geoblocking European users when the company eventually gets in some serious trouble, like it just did in the US.
Because few people care about privacy but many care about UI, UX & features. Telegram is better there than any other chat app by a susbstantial margin.
This, unfortunately. Matrix (Element) is ever so sloowly getting there, it's slightly better than Signal now in some areas and worse in others, but neither (nor whatsapp or any other messenger) come close to Telegram.
Wire is also a good contender but they seem to have found that if you don't sell user data, there is little money to be made from personal accounts and are focusing more on business accounts. You can still register and use it as an individual, though (and I continue to do so). It has more features than Signal and isn't funded by Facebook so that's a plus, but the UX is slightly worse than Signal.
Makes you wonder what Telegram is going to make money with though. It's a huge liability for me, just so damn convenient... Need to find a better alternative.
They're totally different apps! Signal's focus is 1-to-1 encrypted chats. Telegram's focus is broadcasting messages and group chats. One of them is a secure messenger, the other is a social media network on top of a messaging app.
Because Signal is virtually unknown here. Telegram has a small, but growing following. I'm aware that Telegram uses homegrown encryption which is not ideal.
In the same way that FB, Whatsapp, and pretty much every other app works. I doubt there are any chat apps that don't encrypt between the client and server.
IMHO, it is NOT encrypted, unless the service provider (also can't read your message). A promise from a Russian based company (the Facebook of Russia) doesn't put any assurances for me.
We can't just assign new definitions to encryption. It's encrypted in such a way that a person watching the wire can't see the contents of the message. If you don't want the message on the server, use a secret e2e chat. They are very up front and clear on this matter.
For the vast majority of my messages, I'm fine with them living on their servers. If I need to send credentials to my family, I can use an e2e channel. For me, it's a step up from hangouts and definitely better than whatsapp or facebook messenger.
I acknowledge that it is not the standard tech definition. From a user perspective, 'Encryption' ought to mean only me and the person/people whom I communicate with can read the message.
The founder of Telegram, Pavel Durov, was also _the original creator_ of VK. Some say “he's the Mark Zuckerberg of Russia”, an unfair comparison if you'd ask me. While both Pavel and Mark have built successful social platforms, that's where the similarities ends.
Because he refused to comply with government demands to identify activists using VK, Durov was personally ousted from the company (and self-exiled from Russia altogether).
While refreshing my Durov trivia, I found he published a text a couple days ago, and this following passage just had a good vibe;
“I focused on what I enjoyed most – creating social platforms that (hopefully) bring good to humanity. I spent most of my personal funds on Telegram for people to enjoy a free service that strives for perfection.”[1]
why Signal and not wire? (no sim card needed), No need to allow access to your address book to make group chat. Works in browser, ios, android, desktop app. Using signal protocol. And you can draw in it.
why wire and not lockdown?
why lockdown and not matrix?
why matrix and not IRC on a self-hosted server?
why IRC and not HF packet radio
why HF packet radio and not OTP notes on flashpaper delivered by carrier pigeon?
this is the current problem i have with chat. there are too many options.
I've been using Wire for a few years and the experience is solid. No need for phone numbers anymore, just @handle, and it does multi-device encryption and backup. Its search is very fast, even for large number of messages. Good support for attachments and superb audio and video.
On the flip side, it's not very customizable and doesn't have a lot of the more "modern" UX goodies like emoji reactions or threads. But it's overall very good and I see no reason to change.
"geoblocking European users" is already happening. They recently announced some major API changes on the messenger platform that only affect user's from Europe or businesses based in Europe and gave developers about 2 weeks to make changes https://developers.facebook.com/docs/messenger-platform/euro...
Sorry if my wording was unclear. The US part only referred to getting in trouble.
If you wonder why I think they may geoblock European users, it's mostly because that's how many American companies are dealing with GDPR (especially retail businesses, but also major media outlets).
The subject of the antitrust suits filed against Facebook came up in a Facebook comment thread for me yesterday.
Some guy piped up and stated, essentially, that "Zuckerberg is too rich and too big and nothing will happen.". He evidently had never heard of Standard Oil or Ma Bell.
Standard Oil was broken up over 100 years ago and Bell System took 70 years of antitrust complaints (the last case ran through courts for over a decade!) and even then Bell eventually caved and broke themselves up.
Never say never and all that...but it's not like past precedence has shown us the courts are trigger happy when it comes to breaking up monopolies.
As of 12:31 PM eastern on December 10, the status page does show "Major Outage," but it had been showing that related to a totally separate Messenger issue since the 2nd.
There are hundreds of facebook alternatives websites already and dozens of Instagram and Messenger and WhatsApp. I guess tiktok will literally replace Instagram very soon (not because of this outage, but it will help).
There are options sure but on my friends case I can see people aren't so eager to leave the most popular applications/networks, the privacy argument is often shunned with classic "I don't have anything to hide" line and often one is "I won't be installing another app on my x/y/z".
But Telegram still seems to be most popular second choice after Messenger and/or WhatsApp - I've seen recently two people joining it out of sudden (perhaps messenger gained traction because of Belarus situation); personally that's my default instant messenger nowadays, while Signal is the backup one. I'd like to move everyone to element/matrix but that's rather impossible to achieve...
Signal can also function as SMS client on Android, so it's easy to communicate with those who have Signal or use plain SMS; besides additional hardening against SMS based exploits.
In Europe, Whatsapp is ubiquitous. FB Messenger and iMessage are pretty rare here.
Edit: looks like I was wrong. I'm in the Netherlands. In the Netherlands and our neighbouring countries, Whatsapp is ubiquitous, so I presumed the rest of Europe was the same. I guess this proves that assumption is the mother of all failures.
I'd say >50% of smartphone users have Whatsapp in the UK and >50% of smartphone users have FB messenger, of course quite a few have both. I prefer Whatsapp, but contacting a small or large business like a pub or takeaway often works surprisingly well with FB.
@ThePadawan Got it now! WhatsApp did a great job in tricking users with their interface. No other pun intended, but it fits perfectly in the context: "What's the App?"
I remember I missed a few calls from my mom recently, and she asked me if I noticed her missed calls. I didn't have any missed calls as she was calling me through WhatsApp, but in such a way that she thought is the "regular" way of calling. I explained, now she knows. Not sure where this is going...
In India, WhatsApp is the gateway to Internet for millions as many of its users wouldn't have ever opened a browser or received an email.
WhatsApp new version updates gets prime time News spot in even prominent, respected News channel.
Social structures are being built on WhatsApp here, Relatives get offended when you don't join their WhatsApp groups, people don't believe when you say that you don't use WhatsApp, Package disptach details from eCommerce arrive through WhatsApp(no permissions asked) and of course spam arrives through WhatsApp(Local shops wish for your birthday because you signed up for that damn discount card 10 years ago).
A lot more than with SMS. You can send someone an SMS but they are not going to respond because either they cost money or they have some tiny allowance of them.
The point is that the penetration of WhatsApp is way higher than the penetration of unlimited SMS plans. So whatsapp has broader reach.
SMS only works if you live in a region where unlimited SMS plans are standard AND you don't have any friends outside of that region OR WhatsApp doesn't have market share there. Basically only US.
Eh, most messengers use phone number as the primary identifier these days. Whether I use Telegram, Signal or WhatsApp, I add people using their phone number. In the last 3 years or so I can think of one exception where someone preferred Discord but I still got their phone number as a "backup".
I find it funny how every single service builds out a community infrastructure, complete with messaging for near-synchronous and more async communication.
I've always been a fan of distributed resiliency but when it comes to basic communications, I don't have the time or motivation to hunt down the service and method of communication within that service some person chooses to use. I much prefer ubiquitous communication, at least from a user interaction stand-point.
Older messenger applications often supported handfuls to dozens of popular communication systems so you could interact with one touch point/interface, and coukd still benefit from the fact the marketplace was forcing some competition while your communication wouldn't be strangled by some single private entity's policies or actions.
For myself and others in my social circle (mid 30's Americans), we'll often comment on something in each others' Instagram Stories and have a casual chat. Usually these are quick conversations, and longer ones are on some other pure messaging platform.
I used igdm.
Until I stopped using ig itself, several weeks ago.
Like I did FB many, many years ago.
I didn't close down the accounts either. I just stopped using them.
Stopping that, and a good clear out of bookmarks [deleted], mail-subscriptions etc has forced me to restart my onlininess (save for very basic email).
Feel much better, and I'd like to start the next year 'clean'.
Ok, so instead of breaking you up into three companies, we'll break you up into 4. The fourth can own the messaging infrastructure and license it out to anyone interested.
I can find a legal solution to your technical blockades. Be careful what you ask for.
This is fun to read but also implies that we have representatives competent enough to even understand things at this level. Maybe they don't have to understand it, I doubt they even consult someone who could help them understand this.
I laughed at this because I thought the same thing.
Conspiratorial or no, that line of argument won't fly. Saying everything is so integrated that it can't possibly be decentralized is basically proof of monopoly.
It would also be a hilarious argument to make after previously convincing the European Commission in 2014 that there is no technical way to integrate Facebook Messenger and WhatsApp user accounts. [0]
I think “technical” here just means something like: one uses phone numbers and the other uses emails. The current technical solution doesn’t make it straightforward to merge. Evolving the technical solution would require product and user facing changes. And probably a lot of edge cases
I am not sure why a shared messaging infrastructure implies merging accounts is easy/possible. It's fully possible to have messages pass on a common infrastructure between different entities that correspond to messenger/whatsapp accounts.
I'm not saying I necessarily believe facebook here, but I don't understand how they are contradicting themselves here.
> Saying everything is so integrated that it can't possibly be decentralized is basically proof of monopoly.
That doesn't make any sense to me. I would expect different applications from the same company to share the same underlying infrastructure. Whether or not they are a monopoly is orthogonal to the issue of integration.
At the far end of this reasoning three separate companies that all use AWS share the same underlying infrastructure and all go down when AWS does. It doesn't mean they're not independently managed.
And the same could be said for the electrical grid, the post services, roads, etc. I agree that infrastructure and monopoly are two very distict concepts.
> Saying everything is so integrated that it can't possibly be decentralized is basically proof of monopoly.
This doesn't matter from the standard of antitrust law. What most fail to grasp is that being a monopoly is not illegal. It's only illegal to use monopoly power to harm consumer welfare.
In this case integration between services is a pretty strong defense against this. Facebook can argue that consolidating the infrastructure allows them to deliver each product at a lower cost and/or higher quality than if it was served independently.
If the court accepted those facts, then Facebook would have a ironclad argument against being broken up under the consumer welfare standard.
> > Saying everything is so integrated that it can't possibly be decentralized is basically proof of monopoly.
> This doesn't matter from the standard of antitrust law. What most fail to grasp is that being a monopoly is not illegal. It's only illegal to use monopoly power to harm consumer welfare.
Citation needed? I thought, in the US at least, that the government will go after you if you use your monopoly powers to enter into other markets. For instance Google surfacing its own, other products as the top results in search.
It depends on how the Justice department feels, to some extent. GM in the 50s and 60s actually kept an eye on their market share, and sometimes would kill/cutback projects or products that may have threatened going over a certain percentage (around 60%, IIRC.) At that time merely going over that number would have been enough to start an investigation.
Some would argue though that back then DOJ 'cared' more.
The summary is that the Chicago school of Economics promoted the idea that in a competitive market, predatory pricing is not possible. Thus, when you see an industry structure emerge (e.g. duopoly), you should assume that it is an efficient competitive outcome. After this, instead of focusing on market share percentages, antitrust authorities began to focus primarily on consumer harm (which is much more challenging to show in court e.g. for predatory pricing, you cannot only argue that your competitor has very low prices right now, you also have to prove that they intend to raise prices once they achieve monopoly power).
Jurisprudence also changed on the topic since then. In particular, Robert Bork published the legal theory of consumer welfare in 1978. SCOTUS essentially adapted that wholesale into case law starting with Reiter v. Sototone.
Given the current makeup of the court, it seems extremely unlikely that the justices would overturn this precedent.
The biggest challenge for the antitrust lawyers in this case will be to outline consumer harm. In more traditional cases harm is measured in dollars. Here it will have to be measured in privacy, free speech, etc. And that’s a much harder case to make. I even wonder if it’s possible to argue that services for which you don’t pay (Fb, Insta, and Whatsapp) and that are not essential can be considered monopolistic.
Yes that was a defense. Explorer.exe and IE were intimately tied together. Remember “active desktop”? You could make your Windows desktop background a web page. Crazy times...
With Wallpaper Engine or similar programs you can still do it. It's no longer a Windows feature, but they have enough API surface to make it possible for other software to implement.
Bare in mind back then was before "Web 2.0", there wasn't AJAX, responsive web design and any of the other technologies we now take for granted. And to add to the woes, most computers still weren't powerful enough to handle running Active Desktop. It was a huge resource drain.
Some of the other desktop integration features of Internet Explorer 4 were pretty nice though.
These days most desktop environments allow for desktop widgets and usually there's a web view widget amongst them. So you can still have your web desktop if you wanted. At least now computers are powerful enough, and the frontend tools are useful enough, that there is some arguable benefit.
I don't think MS said it was bundled in the kernel, but it is true that parts of IE are tied into the OS (or at least used to be) in the form of mshtml and jscript.
Actually you could, as long as that browser used the Trident engine. Microsoft could have easily shipped a dedicated update application that's just a simple "browser" locked to the windows update website.
At that point (around the time of the acquisition), I guess one could say there was significant overlap between FB and IG users, and that FB had more users than IG.
Server side was a few python components with lots of tricks in Postgres for scaling purposes. You didn't have any real choices for what language you used in iOS
Yes, that's what I meant. The point I was trying to make was that just knowing server side scripting was a necessary but not sufficient condition to create a hit photo sharing app at that time, because there weren't many options on the camera at the time, and the filters were what drove the growth at that time.
FYI that's exactly what they did when they broke up Ma Bell into the 7 RBOC's. Same company, same massive infrastructure. They owned everything and the government was still able to break them up.
the funny thing is that this actually proves the opposite. The consumer overall would obviously be way better off with three redundant and independent systems that don't go down at the same time.
"Just like many independent companies use the same computing infrastructure (AWS) the three unrelated services can be spun off and pay FB Infra to use the messaging platform!"
That would be a terrible strategy. Imagine how many people who have only used Facebook messenger between them are switching to another messaging platform now; some of them may like the alternative better.
With Facebook products, you see outages in December and May. Why? Those are the last few weeks to complete your project before your performance review. Miss this window and you find yourself in career trouble. Facebook employees put a tremendous amount of pressure on themselves during these months.
I would not be surprised if this outage was caused by a bad pull request related to a new project.