For those not involved in the Haskell community, Simon Marlow worked full time on the GHC compiler and specifically run-time system for many years. Along with Simon Peyton-Jones, he's huge in the Haskell world. Marlow also wrote the very excellent "Parallel and Concurrent Programming in Haskell" book.
Facebook also employs Bryan O'Sullivan, an epic Haskell library writer (Aeson, Attoparsec, Text, Vector, and on and on http://hackage.haskell.org/user/BryanOSullivan). Bryan also co-authored the "Real World Haskell" book.
So Facebook has hired two prolific Haskellers and probably others I don't know about.
It really shows what an investment they are willing to make in the language. They aren't just training up internal engineers, they're bringing in some of the best in the community.
I'm hugely excited that Facebook is making this investment and giving some of the developments back to the community. There are many smaller companies that would be interested in Haskell but don't have the resources or expertise to tackle some of these complicated problems.
Facebook is blazing the trail for production Haskell and the rest of us can follow.
That's a bit sad a great language like Haskell (I haven't personally used it but heard great things about it) is used for a hippie thing such as Facebook. Facebook doesn't even have much of a future, its a hippie fad just like Myspace was, eventually they'll get bored and find some new hippiespace social network: Snapchat, Whatsapp, whatever.
You know, you're not alone in thinking along these lines, but, if you would have just put a wee bit more effort into your wording, you could have made a good point out of it. Too bad you kept it even way below average Facebook post quality.
Oh and seeing how I'm actually responding now anyway, there's one more thing: use of the word "Hippie" does not actually qualify as name calling, you do know that, right? I do hope so; you see, it is mostly used in the cartoon Southpark, by a kid who is both the biggest bully of them all, and the most pathetic one as well. The ancipitality is really obvious, so I'm curious: was the satirical context so lost on you that you decided to copy this cartoon character's behaviour verbatim instead? Because in that case I think you could actually gain one or two social skills from Facebook! Chop chop, be a good lad now and give 'er a try, eh?
All correct, but I don't think Bryan O'Sullivan does much haskell these days. He's the manager of some sort of "Developer productivity" team. The Hack programming language came out of his team [1] (tho he wrote a blog post to contrast all the media coverage and say "my people did it, not me! [2]); he spoke about his activity at the last F8 [3]. But he does teach Haskell at Facebook after work :) [4]
I've worked with him to resolve some defects in attoparsec the last few months and I can tell you he is still writing lots of Haskell. He's just in maintenance mode and probably will not start new libraries any time soon.
Bos manages the internal tooling team at Facebook. It is little surprise that Facebook's internal tooling is acquiring a functional programming flavor.
I dislike the underlying premise, the adverts, and (especially) the "real names" policy.
But... between great bits of Open Source like React, cool infrastructure projects like this, and a technical culture which seems a whole lot more open than many other big companies, it's getting kind-of hard to go on hating. Walk back a bit from the obsession with open plan offices, and I might just cave...
> But... between great bits of Open Source like React, cool infrastructure projects like this, and a technical culture which seems a whole lot more open than many other big companies, it's getting kind-of hard to go on hating. Walk back a bit from the obsession with open plan offices, and I might just cave...
Wholeheartedly agree with you comment, Facebook's engineering blog is truly humbling.
There is not a day where I do not try to think of something that would both make the Internet more decentralized, anonymous and secure while being extremely profitable. The latter is crucial because capitalism would make the change viral and being profitable would let me attract talent.
In short, make something that people want AND that promotes values I believe in.
Facebook could easily promote more privacy and anonymity while remaining profitable. It could have been a non-profit and aimed to make money without compromising privacy. It wouldn't be as profitable but that's a different story.
I would be genuinely interested in knowing how they could have scaled organizationally without money from investors who would have pushed them to make a profit.
This. From my experience at Facebook and Google, FB's open space is way more quiet than cubicles at Google. People in the other cubicle feel like the others don't hear them (they do).
Quietest office? More quite than a proper office, with a door, with just you and you computer? Not being snarky, just trying to understand if I understood that correctly?
This is an amazing effort, implementing ApplicativeDo and using Haxl for automatic batching and concurrency, doing code hot-swap in a compiled language, developing per-request automatic memoization, finding a aeson performance bug, translating C++ from/to haskell to do partial marshalling of data, implementing allocation limits to ghc threads, creating a shared c++ library to bundle the c++ dependencies in ghci for interactive coding, killing two ghc bugs, and more...
and in the end producing a reliable scalable solution.
It's definitely not just a "We're really happy with Haskell here at FB" post. This probably explains the trollish comments elsewhere to the tune of "Haskell without Simon+bos /= scalable".
It is a little intimidating, though. I think the perception is "Haskell is hard enough, AND I have to be a C++/GHC internals expert too?" The hard truth is you have to be this level of expert at something to achieve robustness at industrial scale.
Simon's comment in this thread hints at some of the power of Haskell alone:
The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.[1]
This pretty much sums up appdev in Haskell. Engineer (or employ) amazing abstractions with no funny business (hello Spring!) and no loss in safety or expressiveness.
I'm really happy this team exists, this is the kind of work that will take Haskell into the mainstream. Kudos!
Thanks for the overview Simon, great to hear about the use of Haskell at scale. At CircuitHub we use Haskell to build our entire web app, Haskell is great for most tasks these days.
Did you use a particular framework? I've been greatly enjoying my 4th attempt at learning Haskell (I think only now am I seasoned enough to really get it), and my general proof that I know something and can use it usefully is to develop a web app (as all apps are now anyways). However, I didn't find anything which looked like it had majority community buy-in unlike Rails/Sinatra/Flask et al.
Happstack, Snap, and Yesod are popular. I'm partial to Happstack because it doesn't use any wonky extensions like template Haskell. It's just a very straightforward, very fast webserver. It's also very agnostic of how you actually render your HTML, whether it's static serving or dynamic generation with Blaze-builder or something.
Template Haskell and QuasiQuoters are optional in Yesod too. The project scaffold you get when you run `yesod init` uses TH and QQ by default, but you don't have to use the scaffold:
For my side project, I use yesod/websockets for the server and haxe for the client. I feel ok with that choice instead using hpaste. Though eventually I would like to see a complete haskell stack.
@Lewissham: about the fourth attempt, it just so happens that this was my third attempt at getting something that was right for me as well as maintaining the typesafety. I do realize that code is still not as seamless as I would like. It still lets me understand the overall roadmap that the code needs to take to be usable/maintainable etc.
Yesod is awesome I can't see myself going back to less typesafe database schema tools. The choice I made seemed like a safer path, perhaps not ideal.
The application that I am trying to put together is a realtime websockets application. Therefore, I am using yesod/websockets for the server. For the client, I chose haxe simply because of the targets available making it simpler to write plugins.
On one of my irc chats or somewhere, I did hear good things about haste, though it is primarily a javascript library, therefore I plan to evaluate it later.
About the client: it is a single page app to prevent any page refreshes and I am a bit sold on the frp (using promHx). I guess I should have elaborated a bit more about this in my earlier comment, because my choices are not indicative of anything amiss in yesod.
Elm was my first choice for a different side project. I went away because then elm did not have, or it was my lack of knowledge,support for crud widgets. I miss the elm signal approach in haxe for one thing. Here is the link to my repo so things are a bit concrete https://github.com/asm-products/ccar-websockets.git . Warning: this is still work in progress and there are way too many issues before I can even release this. The elm repo is here: https://github.com/dservgun/elm_projects .
Edit: added some repos to set a context. The original comment about 4th attempt hit home as the latest repo is my 4th(I am going to quickly lose count) attempt to achieve at a model that might work/scale. There are still some issues that I would like to solve: whether to use code generators or not. Code gens will save me some typing, but I found myself tweaking printfs or equivalents to fit it into a narrow model.
Edit II: Elm, iirc, is being supported by prezi that also happens to be using haxe quite actively. So I felt a bit more reassured about using haxe.
I will have to take a look if fponticelli: https://twitter.com/jdegoes/status/471319014614892544 , thinks it is. My app is not a pure web app, it has some client components such as excel plugins that need to talk to the server. I was therefore looking for something that I can port to other targets with some support from the language, which is where imo haxe/nekovm seem to shine. Yes, there may be some parts missing from haxe, though, it seemed to work ok for my needs and if its good for salesforce or tivo, I thought I may not be a trailblazer afterall.
Has the book caught up with the software? Last I checked, the book was written in terms of obsolete APIs that didnt match the yesod-geneterated app boilerplate, and Snoyman said that he didn't have time yet to update the book.
I have the book print version and that certainly is outdated. But the online version isn't outdated (mostly). The current version of Yesod[0] is 1.4 and according to the book site it targets 1.4. That being said contributing to any of Michael Snoyman's (main yesod developer) project (including the book) is easy and fun. He is very responsive and friendly. :)
I have been using this site for (yesod 1.4) [http://www.yesodweb.com/book/introduction] and I found the documentation current. Is there something that doesn't compile?
Even though it was at one point a physical book, I think, it's really a regularly updated documentation for the library. Sometimes it's not fully updated because it's written by one guy (Michael Snoyman, who also leads Yesod), but it always gets fully updated for the new version. It's not like a print book where you have to wait for the 2nd edition to come out.
|We implemented automatic memoization of top-level computations using a source-to-source translator. This is particularly beneficial in our use-case where multiple policies can refer to the same shared value, and we want to compute it only once. Note, this is per-request memoization rather than global memoization, which lazy evaluation already provides.
I would like to know more about this. What is a request exactly? An API call? If so, when an existing policy is changed, do the memoization tables have to change as well? How are the memoization tables shared? If this is running on a cluster, I would imagine that lookups in a memoization table could be a bottleneck to performance.
For example, let's say that one of the things you want to compute is the number of friends of the current user. This value is used all over the codebase, but it only makes sense in the context of the current request (because every request has a different idea of "the current user"). So this is a memoized value, even though in the language it looks like a top-level expression.
Memoization only stores results during a request. It starts empty at the beginning of the request and is discarded at the end, and it is not shared with any other requests. It's just a map that's passed around (inside the monad) during a request.
Thanks for the response. Just trying to expand my brain here =), so I have a followup question.
I always thought of memoization as storing the parameters to, and result of, a function call in a memotable. Doing some quick research, I came across this definition of memoization from NIST that sounds more general "Save (memoize) a computed answer for possible later reuse, rather than recomputing the answer." What I understand from what you said is that when a request is processed, it produces a map that is passed around for the duration of the request.
The memo table (map) is a bit of state that is maintained throughout the request's lifetime. When we compute a memoized value, it is inserted into the map, and if we need the value again we can just grab it from the map instead of recomputing it.
The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.
I am under the impression that a large part of engineering effort at established companies go into porting existing components to a deemed to be more appropriate language for that task.
Is it plain impossible to pick the best fit language without implementing a solution in the first place and fleshing out the requirements and challenges that specific to the problem space? Or do the problems evolve fast enough that no matter how well you design the system, it will need to be deprecated once in a few years?
It's rarely "just" a port. It's usually because the new language has better characteristics. These days porting can be a very gradual, as-needed process - partly because of thrift, another Facebook tool!
One of the big parts of becoming a professional for me was accepting that code has a lifecycle; code is written to make the business money at the time, but it's entirely normal for it to change and die as time goes on.
(That said, you should just write everything in Haskell and then you won't have these problems. When was the last time you saw a company port code away from Haskell?)
It's tricky picking a "best fit language" when you're not sure at project inception what you're fitting it to. Say, for instance, that you're developing v. 1.0 of an application in a new application space. Your CEO catches wind that another company is looking to enter that same space. She is also adjusting requirements to find a business model that works. You would choose a development stack that allows rapid prototyping and refactoring.
Once you win user-share in the space the application starts having problems with scalability or debugging becomes an issue or a myriad of other problems that come with success. Now you need to consider writing parts (or possibly the whole application) in a development stack that allows greater scalability or has built-in semantics for core functionality in your application, thus allowing less reliance on 3rd party libraries, etc.
The other factor is that the team who authors an application may have a totally different skill set from the team who takes over the mature application. The authors may have been VB developers with good SQL skills. Your current team has more people who have dealt with Haskell in large-scale applications.
There is some parallel to home-building. A homeowner does a room conversion for more space, more privacy, etc. The builders of the home didn't install tracks and sliding-panel walls for better room configuration. At the time, they didn't see a need. The homeowner now has a need, so he goes through a much more expensive (relatively speaking) building effort.
The problem with this is just how often the story seems to involve porting of solutions. Not necessarily refinement or advancement, but pure porting. With lots of debate/fighting on the syntactic properties of the solutions.
All software tends to unmaintainability, even if you don't touch it (libraries change/deprecate etc). Given enough time, the problems that you are experiencing, say, memory allocation in C, get solved by something else, such as garbage collectors, and the amount of time it takes to solve the headache in what you've got is longer than just rewriting into the new language.
What xmonad does actually isn't hotswapping, but exec'ing the new code and simply passing the current state along. Instead they are using GHC's runtime linker to load/unload modules dynamically, in the same fashion as ghci does.
I was wondering that myself. Maybe they will release code examples on how that works. I'm only slightly familiar with the Haskell ecosystem but I have also wondered if it has a plugin like system (e.g. OSGI for Java).
I've had some mild success with https://hackage.haskell.org/package/dynamic-loader (which is basically just a Haskell wrapper around the C-functions in the RTS), but compared to writing normal Haskell, this is playing with fire as you loose all type-checking guarantees.
This is impressive, and in line with Haskell's philosophy to "avoid success at all costs".
As a mere mortal programmer, who knows a little Haskell, my takeaway is that if you want to run Haskell in at web scale for a a large userbase, you need the language's primarily compiler author to help build the application and to modify the Haskell compiler to make it performant. And you also need your team led by a 20-year-veteran Haskell expert who is one of the language's handful of luminaries whonwrotr a plurality of its main libraries. What are the rest of us to do, who aren't at Facebook?
This is an incredibly stupid takeaway. Facebook can hire the best software engineers and computer scientists in the world, so of course they hire such people and these people build stuff at Facebook.
That doesn't mean other people can't use Haskell to build web applications.
> That doesn't mean other people can't use Haskell to build web applications.
Very few companies do, though, so OP's point still stands.
This article would be a much better ad for Haskell if it weren't written by one of GHC's main author. The article also casts doubt on whether Haskell was picked because it's the best tool for the job or because of the team's familiarity with it.
Do you know a better language for implementing an eDSL that needs to be pure? Also familiarity does matter. With the requirement of needing a pure eDSL though I think Haskell would be the easiest path for most teams.
> Haskell's philosophy to "avoid success at all costs".
How? I think you are misunderstanding the philosophy. Like SPJ once explained, its 'avoid (success at all costs)' and not '(avoid success) at all costs'. It basically means, 'don't compromise on your principles in order to achieve success'.
I've always interpreted it as both "(avoid success) at all costs" and "avoid (success at all costs)" - the former as a joke and the latter a serious guideline.
Yes, it might take some effort to scale things across a 1.4 billion users. Now, in what language wouldn't that be a challenge?
I suppose if you're one of the "many" other companies that are approaching 2 billion active users/month, you might need to put some thought into your systems too...
Loading and unloading code currently uses GHC's built-in runtime linker, although in principle, we could use the system dynamic linker.
That they could use the system dynamic linker makes me think they're using some form of relatively basic dlopen/dlsym/call-method procedure, or something along those lines. That's fine, though the use of "hotswapping" evokes the image of some more elaborate DSU mechanism.
The question was serious. Malicious actions could certainly include propaganda operations and indeed there has been a growing investment across social networks to block content specific to certain ideas considered dangerous, disinformation and misinformation.
My guess is actually that, yes, this system has rules (and gets live feeds of them) to block certain political content associated with influence operations. But I was hoping to get a more serious, direct and charitable answer from someone who works on the system.
I'm also waiting for more commentary to this original question. I would wager that no it is not possible to detect propaganda effectively. Effective propaganda notably presents only one side to the story and uses various other tactics to send that message home but detecting one-sided but factual statements is impossible with today's natural language processing tools, as they struggle enough with being able to make sense of anything.
On another note, what happens if Facebook is able to detect "propaganda content" but it only silences propaganda it doesn't like? This is what I originally meant.
It's similar to how propaganda works in US media. It's not the news stories themselves, it's all about the news networks which are told from above which stories to run. I can see Facebook operating in a similar fashion today or in the future.
Definitely not all strategic communications could be detected and blocked. However some research has been tasked on the issue. Take "Antisocial Behavior in Online Discussion Communities" (http://arxiv.org/pdf/1504.00680v1.pdf). This paper, though it terms it 'antisocial', studies online influence operations and is mentioned on NATO's propaganda ("strategic communication") centre for excellence Facebook page (https://www.facebook.com/StratComCOE).
The point of having a rule set that can be pushed to like the system described here is that it can be fed information from a centralized task force. Indeed, these systems are used across industries with patterns and rules to detect malware, hacking, and a host of other activity (using formats such as STIX or TAXI).
It's unlikely that Facebook itself would be trying to determine whether a post to some content is propaganda - it's more likely that their automated system will apply the rules it receives from information defense agencies.
As per your other note, the selective censorship of information is absolutely a form a propaganda of its own. Facebook (and the US government) don't have the best history here and have 100% certifiably censored information to prevent effective organization of (for example) May Day protests. I'm not going to go speak about that at length here - hoping to get some technical clarity from those involved.
I learned a few things from this post outside the usual "technical" explanations:
1. They have CORE Haskell contributors on their payroll to deliver this type of project (what this mean is that no... Haskell isn't any better than other language, it's just that they have people who know Haskell very very deep to the compiler level...)
2. In-house custom language eventually does not scale (the EOL is much much much shorter than other programming languages), plan for that :)
In the rare cases I would ever downvote something, this would be the case.
This is just complete snark, that misunderstands the general thesis of the article (intentionally?).
Having a core language writer on your team by no means implies that the language in itself is unusable without the designer himself. Your comment explicitly mentions this, and you could really generalise this to any language at all.
The fact that they hired a haskell contributor over a PHP or Ruby contributor was perhaps something you should consider. Or, you know, you read the article, where they answer some of these basic assumptions.
> (what this mean is that no... Haskell isn't any better than other language, it's just that they have people who know Haskell very very deep to the compiler level...)
This is extremely presumptuous for a parenthetical. If this is what you believe it merits a comment all to itself, because I don't see how this follows in the least. To rephrase what you're claiming: if a language has a flaw than it can not be any better than any other language.
I don't know if Rete is appropriate. Rete optimizes for a large number of rules and a small amount of data. I used to hack Rete based OPS5 a lot, including adding support for multiple data worlds. If you like Common Lisp, then the OPS5 code base is fun to hack.
The graph is sorted by performance, with the worst performing (not necessarily the most common) on the left. We've also done more profiling and optimisation since we took those measurements.
FXL employed some tricks that were sometimes beneficial, but often weren't - for example it memoized much more aggressively than we do in Haskell. Mostly that's a loss, but just occasionally it's a win. When a profile shows up one of these cases, we can squash it by fixing the original code.
What matters most is overall throughput for the typical workload, and we win comfortably there.
I don't think the graph says which request types are more or less common. The ordering could be arbitrary.
It would be interesting to know why some are slower. Perhaps they require very little processing and so the time becomes dominated by FFI transformations? It would be nice to know!
One of the cases we found while performance profiling was a tradeoff between memory usage and computation completion. On certain requests FXL would use too much memory and be halted by the equivalent to AllocationLimits, while Haxl would happily plow on using less memory and complete the request. When looking at many of those requests in aggregate, the end result would have more successfully completed requests but with longer response times mixed in. Completing more requests was seen as a win over the apparent decrease in throughput.
Facebook censors so much and swaps out links. I can't even post a link to https://scientificamerica.com. Their SSL cert is weak and should not be trusted. It is swapped out for phishing scams.
Facebook also employs Bryan O'Sullivan, an epic Haskell library writer (Aeson, Attoparsec, Text, Vector, and on and on http://hackage.haskell.org/user/BryanOSullivan). Bryan also co-authored the "Real World Haskell" book.
So Facebook has hired two prolific Haskellers and probably others I don't know about.