This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful.
Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming.
This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.
> This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it.
To the forum's credit, the top comment (by u/judofyr) the last time tailscale blogged about their unusal db setup, got it exactly right: https://news.ycombinator.com/item?id=25768042
> Interesting choice of technology, but [tailscale] didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica.
> In particular [tailscale has] designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd. That's quite a brave task considering how many databases fail at Jepsen.
> Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten [tailscale] quite far.
I did commit the cardinal sin of HN metacommentary and referred to the state of the thread at a point in time as if it was a durable property of the thread, which was a mistake I own and take full responsibility for.
As someone who was there to witness it when it happened, that is not the impression I got at all, and I think Dang may have taken the more humble interpretation in that remark.
Lmao Stavros we should share a laptop or something or just speak over the phone the way we run in to each other so often.
Its different reading it at the time because it factors in the context of the community at the time as well as the lived experience of what it was like at the time to be a member or participant in given discussions. With that contrast, you can then get an idea of the intent of those comments beyond just the words, as they are often (and in this case) reinterpreted through the lens of current day contexts.
Example: ask what people think of the hit TV show The Apprentice today in the present, then hop in a delorean and ask someone before the trump presidency or candidacy what they think of the show.
I for one do not think the comments trivialized anything. I see these as inevitable questions that a leader like Drew Houston should be prepared to answer at some point. Think of it this way, would you disqualify me if you were hiring and I asked a similar question at the interview?
I think the underlying actual question for Dropbox to ponder (but perhaps not answer publicly) was "what is your moat?" without the business development language and at least for me that remains an open question.
I remember I was in college when Dropbox started and I thought Dropbox was silly to offer more storage for inviting more users. It was easy to approach people and ask them to try something in college. I think I almost maxed out my invites. I remember feeling I was taking Dropbox to the cleaners...
One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?
> One activity question for HN: why do you think Dropbox continues to exist today while copycats like Barracuda Copy have failed?
FWIW, i've been using Dropbox since practically the day they went online, and have been a subscriber for all but the first few months of that time. They've never once failed me, never been offline when i needed them, never lost anything, they created a Linux sync client out of the box (whereas Google first claimed the would make GDrive for Linux but never did), and it's easy to manage via multiple interfaces (mainly their own web UI and https://rclone.org, which can proxy your dropbox behind an add-hoc FTP server running in your LAN).
They're extremely pricey compared to the same storage in GDrive, but Dropbox has never let me down so i'm in no hurry to drop them.
Why can't a company use cloud? It's not like there's Amazonbox client I could install now and get the same service. Dropbox never was about the storage itself, IMHO - it was about the ease of usage. There were enough options to get storage - cloud or non-cloud - when it came out, but none of them had the easy folder-based sync and sharing (public folder) of Dropbox.
It's not if the company uses cloud or not. It's if they are solving the hard problems or instead, outsourcing the technical challenges to their
cloud provider while charging the user a non justifiable
premium for the intuitive user interface :-) So it's about
the eventual value add and the business case.
The challenging part is to provide the back-end. Get massive storage, with proper checksum integrity, at a cost effective value and with flawless security.
A single command (and minor variations around it):
get's you all that. It can be easily mapped to an intuitive
user interface by a software development boutique in weeks.
And because that is the difficult part, it seems for the
first 8 years of their history that was solved for them by
AWS, according to their own reported history.
Sharing a folder or file from Dropbox is no more
intuitive than using the S3 console. Looking at
the very colorful history of security issues
around Dropbox [1] it's certainly not safer.
So why would a user pay more for storage costs than with
a cloud provider, with notoriously less security? Does a tiny layer on top one or more cloud providers
justify the add-on value for the user or the current
valuation of 9 Billion dollars?
And for what its worth, systems like minio have emerged due to that commercial innovation. AWS has made a lot of investment in that system and will earn perpetual annuities from it, but by creating a defacto protocol we now get many choices that use its API. Its interesting because AWS never was benevolent with their S3 code, they just created a novel and rigorous offering that became worthy of emulation. Dropbox really hasn't done any equivalent contribution. If anything, the open source projects that have emerged to offer self-hosted alternatives to it are more feature-complete and extensible- such as nextcloud/owncloud, pydio, syncthing, etc.
side note: I'm filled with regret for every document that I authored on Dropbox Paper. It didn't last for long, but there was a time that it was the official wiki for a company I worked at. Its not impossible to export, but never with full-fidelity.
> Sharing a folder or file from Dropbox is no more intuitive than using the S3 console.
I've not really used the S3 console but, at least on macOS, sharing a file from Dropbox is as simple as "right-click on the file in Finder, Share..." then enter an email address or create a share link from the popup. Or I can right-click and "Copy Dropbox Link" if the direct link is good enough.
But now you have to know which bucket the file is in, have the AWS CLI tools installed, use Terminal, copy the URL to messages / email, etc.
How is this easier than "right-click, pick a menu option"?
(If I want an expiring Dropbox link, it's there under 'link settings' in the pop-up. Admittedly only has date granularity but still there as an option.)
Again, how is this easier than a single right-click in Finder? Also don't those files need to be public to create a link? You can't create a link to a non-public file which grants access either directly via the link or with a password?
Their fundament is UX, not technology. Their technology could be whatever and users wouldn't care - the users care about syncing their files easily. Few geeks might flame about it on a tech forum, but nothing more.
Not that I like Dropbox too much. I'm going to switch to my self hosted solution. But it's certainly not as easy as using Dropbox, even as someone who uses AWS since 2012 frequently.
People seem really stressed that they've changed databases twice before. Their service never went down and they didn't stop delivering features while all this was happening, so I don't see the big deal. You pick what works at the time, and incrementally evolve as problems and solutions become apparent.
This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then everyone would be like "why didn't you just scp a json file between machines" ;)
Another comment on this thread tried to dunk on them, saying that they'd done this so many times before (JSON to etcd, etcd to sqlite) that they must be getting really good at it. That comment struck me as so close to understanding what they're doing right here, it was painful.
Yeah. Maybe people are overestimating how expensive this kind of change is. We can all imagine being part of a larger org, where one day an All Hands appears on your calendar, and some exec you've never heard of proclaims the beginning of a 2 year multi-team database migration project. So now instead of making nifty software, you are just in 40 hours of meetings a week fighting over small details on this migration nobody wants to do, and everyone is miserable about it. Features stopped getting released, people leave the company, the weekly status meeting never makes the project seem anywhere near completion.
Say "switch databases on a successful project" and I bet people's minds go there. That is indeed expensive and painful, but if you have a small team, it's just being agile.
There is also the "punching up" / sour grapes aspect that can't be ignored. "I would have picked the right database from the start, but nobody's funding my crazy idea." Life just isn't fair, and HN is where you can vent, I guess.
I have the odd feeling that this kind of overestimation comes from the experience that folks tend to have in writing more traditional N-tier applications, especially post-Rails. A certain kind of database-first, ORM-first design philosophy really took hold after that, and it became somewhat common to see incredulity in response to telling someone that you weren't using something like MySQL or Postgres in production.
The kinds of projects that I more generally tend to see make these migrations successfully are the ones that design with in-memory data structures first and then sort out how best to persist that kind of thing to disk after their ___domain model has solidified a little bit.
When I used dynamic languages (Node.js and Ruby) I leaned on the database as the structural source of truth for types—it was the rigor a successful project needs.
But as I’ve been using only statically typed languages the last few years (on the backend), I’ve moved to designing the domains in the code, like you said.
My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack.
I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.
The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters.
The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka).
By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures.
Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters.
You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it).
*This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.
But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month
That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.
> But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month
A k8s cluster for something like that wouldn't cost anywhere that amount, plus you would get the advantage of being able to deploy your app in multiple availability zones. If you don't need k8s features, there are definitely simpler options that will give you redundancy and won't break the bank.
I only used a MacBook cause it was faster to ask IT for one than to have my ticket processed and approved by the cloud and security team which both required a write up on why I need a vm and what justification I have for needing it. Meanwhile IT is like "here you go".
I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.
That's what I mean though. I've never hit those limits. These days it's easy to put together a raid 5 with NVMe disks and you'll quickly reach gigabyte per second write speeds.
One of the largest forums online in 2003 was was ezboard. A billion page views per month. Back then - insane - well that was beyond insane. I started work there as a young idiot. "What db you guys using, mysql?...." long pause... "We use a file system." I dug in. Yep. A fucking file system. Files allowed: simple backups, impossible speed, obvious cache systems, partitioning.
Later on i met the roomate of the author of memcache. Few hundred lines of C. Completely changed the game how caching systems worked. Years later a super senior FB engineer explained to me over a beer - "we basically use mysql as a file store, everything else is in memcache." Good ideas are stupid simple.
nobody goes to these restaurants for their omelets, though at the same time nobody should care how tailscale actually implements things so long as it works well.
imo they’re not solving a notoriously hard problem, they’re realizing that they have the tiniest version of this problem (and that will most likely continue to be true). when that’s the case and you want to prioritize stuff that moves the needle like dev velocity, why do anything less simple? getting hung up on building scalable solutions far beyond what you need is somewhere between a fool and a big co employees game. far too often people introduce complexity way before it’s needed, and that almost always slows down your devs ability to actually fix bugs and make the product more reliable.
They don't, but there are other "finesse" dishes that people do in fact go to restaurants for that are notable for their simplicity, and the fact that you can only pull them off if you have flawless technique. Japanese, in particular, is notorious for this.
There’s a manga about a food critic (Oishinbou [1]), where he takes another food critic to a hole in the wall Tokyo restaurant as an example of elite cooking. The chef-owner brings him his meal.
Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill.
Which is basically also the ending of the Pixar film Ratatouille. The protagonist wows a critic with a simple dish prepared to perfection. It’s a common trope, but also true. “Simple” food prepared well can be much better than a complex dish with many flavors.
Similarly, I’ve also heard (probably from a show like Top Chef, so YMMV) things like a salad or soup are also quick ways to differentiate chefs. In this case, it’s less about technique and more about identifying and mixing flavors.
It works because there is no place to hide. You either have good flavor or not.
my interpretation is that there is a certain class of people who think "if i just apply the process then i will get a satisfactory result", as bob martin does with using TDD to implement a sudoku solver instead of just thinking through the problem and designing a solution up front. i don't think its a particularly great analogy because the bob martin example is about trying to use a "mindless" process to achieve a desired design, not picking an overkill option.
(Not to namedrop, but, uh) I know apenwarr glancingly IRL and he's amazing. A force of nature. When people talk about "10x" programmers I think of him.
A discussion that keeps recurring on HN is how you can get a lot done with simple, boring solutions. This is a perfect example of that in practice. What’s the simplest thing that’ll work? Do that, monitor the solution, and when you start hitting a limitation reevaluate what the next appropriate solution is.
There are plenty of comments here deriding this as insanity when we have established best practices, but “best practices for me are not necessarily best practices thee”.
People aren’t perfect, not every engineer comes fully-formed grokking the industry’s best solutions and their trade offs.
This is what learning looks like. This is what an evolving system looks like. And this is how valuable software gets built every day.
Who gives a fuck about scalability on a poc? “Let’s just put in a json file”. I Love it. Personally I start at the “use SQLite” step and go from there because SQLite is such amazing software. Thanks to this post I’m excited to try out Lifestream.
You might rightly point out that Tailscale isn’t a side-project/poc anymore. But that’s exactly why they’re changing to SQLite and writing this post; this is their team learning to deal with larger problems and changing needs, and they’re sharing that journey with us.
As long as they’re not _ignoring_ serious risks let them experiment. But try not to get too upset if they’re making different risk/effort trade-offs than you would. Even if it’s “objectively” the “wrong” thing.
My rubric for technical sanity is:
- are you taking (and testing) backups?
- is your infra/data secure?
- are you monitoring what’s happening?
- can you recover from catastrophic failure in a acceptable time frame?
- are you meeting your legal obligations?
Cover those needs and you can be forgiven a lot suboptimal/experimental implementation details.
> People aren’t perfect, not every engineer comes fully-formed grokking the industry’s best solutions and their trade offs.
The thing is, David Crawshaw is being extremely honest and humble here in admitting their shortcomings and how they're learning from it... but an entire troupe of Silicon Valley engs seem to have lost sleep over it, for reasons beyond me.
Are these even shortcomings? They made JSON scale for 18 months, which is an achievement. They seamlessly switched to etcd, which is one of the "right things" for distributed state problems. Like us, they discovered that when you adopt a "distributed state" solution, you inherit all the API limitations those things come with because nothing really solves (or can solve) the "multiple writers multiple readers real-time consistency" problem. Nothing broke, but etcd was a pain to work with: anything you store in etcd, the developers that work with it have to understand the rituals of etcd. Wouldn't it be nice to have that just be a SQL database? Oh, look: that's easy to do: use sqlite, and ship WAL segments.
There is negativity on this thread, but it strikes me as incredibly, even embarrassingly, ill-informed.
The question is, is this the simple solution? Perhaps I and others have been spoiled by cloud offerings but having set up RDS once and never having to worry much about the db architecture after that for years as we scale has maybe spoiled us.
I’ll be honest and admit my own bias to cheaper, self-managed tooling. I’ll admit that RDS definitely meets the criteria of simple.
From what I understand RDS works great and you’re happy to pay what feels like the tiny cost…right up until it doesn’t anymore.
You’re paying Amazon a premium for them to manage the tedious stuff. But sometimes dealing with the tedious is what teaches you how the stack works and makes you jmuch more prepared to solve those more complex problems down the road.
The uncomfortable fact is businesses finding themselves in situations where they experience vendor-lock-in just because they’ve let their capability wither away.
Clearly David’s point about avoiding MySQL and PostgreSQL due to fears of “vendor lock-in” resonates with me, even if I personally think that’s not quite where the threat is.
…for Postgres anyway. I’ve heard enough stories that I never want to touch an oracle product in my career.
I favorited your post. It has taken out some insecurities that I had about best practices. I couldn't explicitly formulate them either. Thanks for the (professional/career) advice/therapy :)
They're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fairly hands-off from SQLite's perspective. There's also the SQLite Session Extension [2] that is a built-in way to support generating and applying "patches".
I'm curious how these tools will mature, it seems like a good match for microservices.
The litestream project was created by https://github.com/benbjohnson who wrote https://github.com/boltdb/bolt (a key value store) which has been instrumental (from my point of view) in the Go community as one of the original choices for an embedded database as it was not only fast, but had transactions with stable snapshots.
Bolt author here. Badger is a good database but it's mostly a tradeoff of write versus read throughput, specifically, range queries. Badger is an LSM so it doesn't perform as well (last time I checked) with iterating over ordered key/value pairs. LSMs have bloom filters to speed up point queries so those aren't as much of an issue.
It should be noted that BoltDB has been retired as it suffered from a number of congenital defects. It was implicated in the major Roblox outage that happened a couple months ago, and Consul doesn't use it anymore.
BoltDB author here. Just to clarify, the project was retired because of maintenance burden. CoreOS wanted to make some changes but I didn't have the bandwidth to test and maintain it all. Since this was before Go had good version management, we decided that they could fork the project as "bbolt" and users could move over as needed. They did a good job maintaining it so I eventually archived the original project.
It is strange that, although https://github.com/syndtr/goleveldb is used in the go-ethereum project for years, it is mentioned rarely (comparing to other go dbs).
I watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up the challenge.
Not commenting on tailscale, but for the state of databases.
Sometimes boring is the right choice.
PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted.
If you do this for the money, investing in tried and true (but boring) software should be the default solution.
I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using PostgreSQL and had a good ride.
IMHO MySQL becoming boring (fast, pretty reliable, non-compliant quirks being ironed out, decent backup tool availability, credible admin tools, JSON and CTE support) is one of the most important long term trends on the internet.
I strongly dislike the combination of php and MySQL, from the (recent) experiences I’ve had with it. I don’t see a good reason to pick that as a greenfield stack, but I could be convinced otherwise.
For me it's not necessarily the tech themselves, but the lack of any framework to guide development for the next guy.
All the PHP+MySQL I've worked with has been horribly bespoke and brittle.
Want to significantly change a URL? Need to refactor every path in the .PHP file, because it's importing something from ../.. Or get in the spaghetti business with path aliasing on the webserver.
Want to refactor the database schema? Difficulty: Impossible, because raw SQL strings are scattered everywhere, including on other systems.
Want to add/modify a new internal CRUD form? Gonna be several hours of work to RE what the last guy did to keep things consistent, while tiptoeing into the SQL to not break anything. With a proper framework, this is a 5 minute task and a few LOC into the "admin interface".
It works, and you can do a lot with just a few php files and a basic Linux system, but there are downsides.
All of these problems are created by the developers.
I have created many on-the-fly mini-PHP frameworks to use within existing legacy code bases to avoid all of this.
Easiest way to avoid manually requiring files is to use classes with the auto loader. Done.
The second easiest solution is to have different constants that has the full path to a directory and then just you use the correct constant when doing a require. No relative paths required.
You can even have one require constant per file if that is your game.
Raw sql is fine, just don’t scatter it in every file, instead put sql in specific repositories, e.g ArticleRepository, and do all article queries in there. Now it is easy.
An ORM actually creates more problems when refactoring because entities are spread out everywhere.
If you use something like Phpstorm IDE it can help refactor raw sql too.
Problem is that many developers doesn’t think this thru and even worse they accept the situation and don’t do anything about even if they can. Instead they go to forums and complain. I’m sorry but I have zero tolerance for that.
A decent ORM framework provides all those things out of the box, and I disagree that the ORM creates more problems than its worth. It is a fine tool to accelerate development.
I had a 10 year career doing PHP, in the last year or so I also enjoyed Laravel and built 10 or so sites with it. It did many things right.
Eventually some blade template error resulted in an impossible stack trace due to the insane amount of magic behind the scenes in Laravel, and while debugging that I decided to stop relying on such black magic hack of a framework. Around the same time Go gained in popularity and I have been working with Go since and are very happy with it.
This was Laravel 5 days and I'm sure much has changed.
I completely adored the laracasts made by Jeffrey Way. I wish every framework had a guy like that do produce learning materials.
Frameworks helps developers to move in the right direction, and that is a good thing, however when you have used a framework long enough you start seeing the cracks.
Frameworks solves a general problem, you on the other hand has a specific problem. To be able to accommodate your specific problem and everyone else specific problems frameworks has tendency to heavily rely on magic.
* magic classes - using PHP magic methods for everything, the actual class is usually empty and things happening elsewhere.
* magic files - put a file with the correct name in folder somewhere and things happen
* magic configuration - put a hard coded string in some config file and you change the entire behaviour of the app.
* magic layers - if you follow the execution path with the debugger you spend the majority of time circling around in different layers and very little time in your own code.
All of these things are considered bad practice when doing it in your own code, but for some bizarre reason it is considered good practice in frameworks.
> All of these things are considered bad practice when doing it in your own code, but for some bizarre reason it is considered good practice in frameworks.
I think that’s because that kind of code has a much higher maintenance burden. It’s much more difficult to get up to speed on what’s going on in complex magical code. But you can probably trust a framework that’s achieved critical mass to make reasonable decisions and stay alive for a while.
Laravel has an excellent ORM, a great query builder (I write almost no SQL), an effective database migrations system, a solid class hierarchy (no include statements anywhere), a really useful job queuing system, and it is built around Composer (one of the best package managers anywhere).
Lighthouse is really a first-class, schema-first GraphQL binding with a very logical code interface.
I don't see any code maintenance problems in Laravel that can't be mitigated by the same discipline you need anywhere else.
I don’t have anything objective to back this up, but in my experience it requires a lot more discipline and initial knowledge from the devs. Much of the existing documentation for PHP is just plain wrong, or horrible advice. It’s difficult for new devs to learn these practices. PHP let’s you be ‘lazy’ and hack in shortcuts to solve problems - lots of discipline required there.
If you have a solid team and some people with a good background in the framework and PHP that can mentor effectively, then it probably won’t be bad. It still would not be my first choice though.
It depends. I was in a demo call from Blaze, a CDP (Customer Data Platform), Segment.io competitor. Their underlying db seems to be MongoDB and for a CDP makes a lot of sense. In a CDP you collect all kind of data on n hard ids that identify 1 person interacting with your web assets, schemaless and json first is a lot easier to reason in this context than SQL. This is because you keep enriching your profile with additional attributes over time. Can PostgreSQL do this? Absolutely, but it’s not its main feature.
Things are complex, especially in data, I think it’s important to evaluate use cases rather than going with generic assumptions like “use boring tech”.
Their paid managed service works well for enterprise SaaS companies who need < 3-4 TB of storage max ever per customer cluster and have enterprise scale users (<100k users).
It also works well for early stage Consumer Internet startups that haven't yet achieved huge growth but care a lot about developer productivity while churning features at high velocity.
But it gets blamed for reliability issues suffered by highly successful Consumer Internet companies that have achieved scale and have DAU, MAU in > 10M and have lot more than 4TB of data in a single cluster.
But that's basically a good problem to have at that stage. Usually such a company would have many more problems – monolithic application with monolithic database with huge unmanageable schema and indexes gone wild etc. Usual solution at that stage would be some sort of rearchitecture towards Microservices with multiple specialized databases for different use-cases – usually cloud hosted managed databases with horizontal scalability dedicated for online user-path workloads and separate OLAP tech stack for offline ETL/analytics workloads.
At work I have a multi-petabyte cluster in MongoDB Atlas with no issues. The managed service is fantastic; I'm going to need some citations on how it degrades after a few terabytes. I don't really like MongoDB's document-model-everywhere approach but I have to respect how well their managed service works.
I believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model..
The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated.
A document within Adama (https://www.adama-platform.com/) is basically a giant JSON file held within memory with clients connected via a WebSocket. I'm basically building my own indexing since I want the indexing to be reactive, and I've got exceptionally fun optimization problems.
Litestream, in my opinion, is a great way to get started. Actually, it's beyond fantastic if you maintain the 1 server to 1 database because migrations become so easy. So, I applaud the team for taking this simple approach.
If you have a "single source of truth", sqlite/Litestream seems like a pretty-near-optimal way of taking advantage of the relational model while keeping design simplicity. I don't know what the higher-level architecture of Tailscale is, but we have the same problems; we have a complicated "single source of truth" that takes the form of a Consul cluster, but sqlite makes an absolute ton of sense for us, because we can condense the Consul cluster down to a SQL schema and then ship it around our fleet.
(we don't use Litestream right now; we're pushing the complexity up a layer in our design instead)
A lot of people on this thread are looking down their noses at sqlite, but I think they're kind of beclowning themselves; the unreasonable effectiveness of sqlite has been a meme in infra dev for a couple years now. It's not a new idea. Lots of people are doing stuff like this.
The funniest bit on this thread is the person saying they should use RDS, as if their infrastructure was just a big Rails app.
It would be interesting to known why a standard boring RDS setup wouldn’t solve their problem completely. In fact I would be more interested to understand that than the actual details of sqlite tailing.
(I think the reason they gave was vendor lock-in, but apart from that, I didn’t understand why it wouldn’t be adequate)
One obvious reason not to use RDS is wanting disk-local caches of information replicated from a single leader, rather than having every single machine in your fleet calling out to an external service on every read. That's certainly why we're not considering Postgres in our infrastructure, even though managed Postgres is a product we in fact offer.
We use Postgres! It's the backing state for our API, and an important source of truth in our architecture. Postgres is a great way of serving a GraphQL API. It is not necessarily a good way to back an infrastructure service.
That's still all your infrastructure components calling out to an external service on every read --- and for what advantage?
This isn't an app server; it's an infrastructure component, running (I don't know about Tailscale here, but we use SQLite in similar uses cases) on potentially hundreds or thousands of machines.
No, you don't need to make any network call on every read with listen/notify. It will essentially be a local cache, working the same way as etcd watcher, using one postgres connection per machine.
Really curious…how do you meaningfully overlay any indexes on top of a big-ass JSON file? Technical details are appreciated, and no problem if it’s your secret sauce—just very curious how this is accomplished!
So, a funny thought experiment is what happens when you parse a JSON file at the same time? You also index by primary key (the field name).
So, I mirror this thinking and having be just an object with the keys being the primary key. Then, I simply index all the children by their fields based on insights from the developer via the index keyword.
So, if you have
record R { public int id; client int owner; int age; index age; }
table<R> rows;
then queries for age can be accelerated by the table.
like "iterate rows where age==42" will basically hone in on the bucket of age==42. I currently only index clients by hash and integers.
The critical aspect which makes this work is that I monitor all mutations. When a child object has a field mutated, then it is removed from all indices and placed into an unknown index. Any queries will also consider it as the purpose of queries to simply narrow the field. Once data changes are persisted, the index is updated and items are moved out of the unknown bucket. This works fairly well because the indices are primarily used during the privacy check phase.
The normal way? You can implement whatever kind of index you like — b-tree index, bitmap index, hash index are all useful and conceptually simple if you're familiar with the backing data structures.
For example, if you want to index a "foreign key" id stored in each "record" in a JSON array of objects, you build a hash table from the FK id values to the JSON array indices of the objects that have that id. It can be as stupid simple as an `fk_index = defaultdict(set)` somewhere in your program, to use a Pythonism.
Now when someone wants JSON objects in that array matching a given FK id, they can just O(1) look in the index to know the position of records that match. Much better than an O(N) scan of every item in the array.
Of course you have to to maintain the index as writes to the JSON happen, but that's not bad once you understand how things work. No real secret sauce.
> Honest question: what do the sqlite authors think of litestream?
Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff that's what makes the new read replication work. They ended the call by asking me if there's anything they could help with or anything I needed.
> I guess I'm wondering if it's a recommended/supported mode of operation in sqlite.
I don't think the SQLite team endorses any tooling outside of what they build AFAIK. So in that sense, no, it's not officially supported. However, the API for maintaining checkpointing is publicly available and there are docs for controlling it from outside processes (which is what Litestream does).
> What are the failure cases? How much data can you lose?
Litestream is designed so that it keeps retrying in the event that you can't connect to a replication destination. S3 is pretty reliable so it may be a network outage that could cut you off. You can also enable Prometheus metrics to be reported out of Litestream if you want to add monitoring and alerting.
As far as data loss, by default it's setup to bundle database changes together every 1 second, compress them, and upload them to somewhere like S3. So your window for data loss is 1 second unless S3 goes down.
You can also run regular backups with the SQLite CLI and upload those as a fallback. That's what I typically do since it's really cheap and easy to setup an hourly cronjob and I'm overly paranoid. :)
We didn’t discuss the session extension. That extension solves a different problem of having multiple primaries that need to merge their data. I haven’t used it personally but it looks great for that purpose.
I'm so desperate for a SQL database where I can just put it on a bunch of commodity hardware via Docker, connect them up and never worry about this again. Ideally it'd monitor my queries and create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee
/dream
FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular.
FoundationDB is an Apple thing, isn’t it? Might explain why. It feels kinda like they’ve put it out there and it just hasn’t had anyone with a marketing budget to push it along.
FoundationDB was the first publicly available distributed transactional KV on the market that I can recall. It wasn’t “open” though - always proprietary. They generated a lot of hype and good press for being this NoSQL DB that was all about ACID correctness in the age of “mongo is webscale (by writing to /dev/null)”. They had a demo showing impressive throughout on a low power cluster with no data loss during a power cycle. I think that despite generally impressive tech they had some trouble selling but did end up selling the company to Apple.
I was pondering about using simple decentralized databases that are kept synchronized asynchronously.
I arrived to a design where SQLite databases would be synchronized by Kafka. Kafka is really robust, and has a friendly semantics when configured to be in-order delivery.
The catch is, you don't issue writes to SQLite anymore. You write Kafka messages, and have to prepare to resolve concurrency problems at read+write-to-db time. For example, user registration is not an atomic action anymore - you write a registration attempt message, and then it might race with a concurrent registration attempt from an other node (say due to retries and random load balancing).
You resolve the race in message-reception time (doable due to stable order on multiple reader nodes). I expect this would work nice, but needs you to rethink all actions that used to be synchronous db writes.
I love this. If it works, keep it until shortly before it stops working. The older I get the more love I have for simple solutions that are easy to work with (and change when you've grown out of them).
These days I wince when I hear of overcomplicated solutions for an MVP that "will scale" at a nebulous point in the future, while we're going to be paying the complexity penalty every day for a benefit that may never come.
Funny in here, people praise SQLite but dismiss MySQL when it's tiny bit off the SQL standard and people complain how PHP converts values with best effort guess on different types but not when SQLite does that too.
Don't know why anyone would use SQLite in production except for as some KV store embedded in an app.
Because an SQL engine that defers control over connections, concurrency, storage, and process management to users is extremely valuable for multiple reasons, including simple cases like caches where you'd ordinarily just store things in flat files or BerkleyDB-style database, and complex cases like distributed statekeeping where existing SQL databases aren't going to do anything close to the right thing for your application, but you also don't want to reinvent SQL and query planning and write-ahead-logs to build your complex database.
The vibe I get from this thread is that a lot of people have only experienced SQLite from the vantage point of something like Rails running in a "test" environment, and are used to explaining to people on some PHPbb somewhere that you can't really use SQLite to back a multiuser SAAS app. That's as may be, but when you're building infra components, you control who the "users" are; not everything is a web server taking requests from all comers.
Not for this kind of applications. Same as Java being on 13 billion of devices, it doesn't mean anything if it's some config store underneath some mobile or desktop app.
There are projects you can try new things with and projects that should just work when it's expected for many people to use. Don't push your excitement to the users.
They have been trying to create or reinvent the database rather than using tried and true solutions. The analogue would be rolling your own cryptography.
At first I was also a little confused by this architecture. I think that was because I was missing a clear picture of what they want to do. I think it is:
- one leader that can write to the db
- a ton of other hosts that can read from the "streamed" db and want to do so often/fast
- hosts send their writes (if they have any?) to the leader via some api call (?)
Is that right? If so, the solution makes a ton of sense. "Just use pg/MySQL" in that case would mean hundreds (thousands?) of read replicas, which doesn't sound fun, or every read over the network.
Disclaimer: I hate doing ops, but I’ve been in a position to actively hate doing it fairly regularly for several years of my career. My perspective is as a person who doesn’t want to deal with any of this kind of stuff. So I’ve probably failed to acquire knowledge which would make it less frictionful for me, purely from lack of interest.
RDS has some significant downsides which I would personally consider no go if I were in a position to evaluate it. The one which stands out as particularly painful from my past experience is… it’s excruciatingly slow to provision or make configuration changes. Like lose whole days of work to a few iterations of trial and error slow. Combined with AWS’ sprawling and inscrutable set of authorization and configuration options, the weird idiosyncrasies between most of their offerings, and the absolutely opaque naming applied to most of those offerings… trying to use RDS effectively as a managed database service felt more to me like becoming a full time ops professional.
Idk, I just use the Terraform module and set the Helm values on whatever should consume the db with whatever I get back from the module then call it a day. Then again, I'm not scaling big at all.
Some engineering teams seem to take overcomplication always one step too far. It's very hard to estimate future work and overconfident engineers consistently downplay the costs.
I'm not sure why you're being downvoted. You're right: running databases isn't for every team! Reach out for a managed database if you can.
But tailscale isn't some random group of engs. They've probably got the chops to pull off literally anything they want to. I mean TFA casually mentions online cross-database transfers, multiple zero-downtime schema migrations, inspecting litestream's replication code for feasibility, deftly modifying sqlite WAL checkpoints... all in one breath.
It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations).
> ...rather than improving the product?
Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
----
> if they're this talented, is their time really best spent on...
"People are often surprised and sometimes horrified when they learn that Tailscale maintains its own fork of the Go toolchain. Tailscale is a small startup. Isn't that a horrible distraction, a flagrant burning of innovation tokens?"
"Maybe. But the thing is, you write code with the engineers you have."
"We had a problem: We kept crashing on iOS, and in addition to being awful, it was preventing us from adding features."
"Another team might have decided to cut even more features on iOS to try to achieve stability, or limited in some way the size of the tailnet that iOS could interact with."
"Another team might have radically redesigned the data structures to squeeze every last drop out of them."
"Another team might have rewritten the entire thing in Rust or C."
"Another team might have decided to accept the crashes and attempted to mitigate the pain by making re-establishment of connections faster."
"Another team might have decided to just live with it and put their focus elsewhere."
"The Tailscale team has Go expertise, spanning the standard library to the toolchain to the runtime to the ecosystem. It’s an asset, and it would be foolish not to use it when the occasion arises. And the fun thing about working on low level, performance-sensitive code is that that occasion arises with surprising frequency."
"Blog posts about how people solve their problems are fun and interesting, but they must always be taken with a healthy dose of context. There may be no other startups in existence for which working on the Go linker would be a sensible choice, but it was for us."
> When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
if zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, why are they using s3? migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres
First, S3 and a SQL database aren't comparable. But I think you're bringing up S3 because they're using Litestream to ship WAL frames to S3. Go read the Litestream documentation; Litestream syncs to basically anything. They don't need to "migrate to another S3 compatible backend"; they can migrate to almost anything that can save a file.
It's a super confusing argument regardless, because the industry is lousy with "S3-compatible backends".
> migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres
You may be right. I have no experience migrating litestream but from the docs (https://litestream.io/guides/) it is literally cp'ing files from S3 to wherever and exec'ing one of these one-liners (of course, the devil is in the details):
Litestream author here. Yeah, you're basically right but it's simpler than a DB migration. No need to copy the old data over. You can remove the `-litestream` metadata directory and point it at a new replication destination and it'll automatically re-snapshot the database begin replication.
Some examples: they could get rid of that pointless bootstrap on their website, they are shoving almost 1.5 MBs for a single font alone on their main page and their HTML semantics are nowhere to be found. This will all impact their bounce rate, accessibility and SEO.
I just don't believe the tale of "such skilled engineering teams" which don't show that in their products but blogposts.
What if I told you that the product engineering team is almost never the same team maintaining the website; hell it’s likely the website is contracted out and maintained by the marketing team.
Litestream author here. I'm glad you were able to track down what you were looking for. For posterity, I'll add a response here. Litestream is a one-way, physical replication tool so you have a single primary node that continuously copies changes out to an external destination (S3, SFTP, etc).
If you're looking to have multiple primaries then you can try using the SQLite session extension[1] and copying patch files. That's quite a bit more complicated though.
This is great. We've been using sqlite in production as our exclusive database storage solution for over half a decade. For non-trivial, multi-user systems too.
Replication of data has always been a concern for us, with the current mitigation being periodic VM snapshots. In order to engage larger customers, we would need to tighten this up.
Our next gen data storage technology leverages some dark magic from fintech. Append-only log, single writer, synchronous replication, etc. Most recent benchmarks with replication to 1 sync witness on same lan are in excess of 200k business objects created or updated per second.
The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ourselves into a cloud provider with a managed product like Spanner.
What about managed mysql/posgresql? no lock-in and installing them locally is trivial.
Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.
I'm having a really difficult time migrating my managed postgresql instance in Google Cloud just to get it to a newer version. I'm down to my last option: shut down everything, export, re-import somewhere else again.
I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either.
The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.
they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
"Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.
i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their bespoke solution relies on s3, which is also a vendor-specific technology, so it seems like they haven't avoided vendor lock in? i've seen many cases of developers doing more work to avoid vendor lock-in than it would take to replatform if it ended up being a necessity, and this really feels like that looking at it from the outside. i'd understand this better if mysql or postgres couldn't solve their problem, but that is not the case here, and i can't wrap my head around a company who is ok with their devs reinventing a very good wheel 3 times when the obvious choice would've worked fine the first time. it seems like they are successful in spite of these decisions, not because of these decisions.
You don't know what this database is --- upthread, you said you don't even know what the product is. All you appear to know is that they should be using something like RDS. Isn't that a weird position to take?
it's also kinda weird to write a blog post about a database being used for what is apparently a very specific use case without mentioning what that use case is. the burden of proof is on tailscale to explain why they need to deviate from industry norm here and clearly from the entire comment thread of people wondering the same thing as me they haven't done that. this blog post might actually be valuable if they included more context so that people could learn when something like this might be a good idea, especially since it's not a good idea >99% of the time. as is, no one should be surprised by this response.
This is one of several comments you've written where you've acknowledged you don't know what the use case is. But you've stridently insisted that they should have used Aurora Postgres or Aurora MySQL. You get how strange this take is, right?
One thing that would have helped your writing on this thread: a lot more question marks. It's OK not to understand something! Asking questions helps everybody.
at no point in the blog post or this thread has anyone actually explained what their super special use case is that makes doing what almost everyone else does a bad choice and this replicated sqlite thing a good choice. it's pretty clear you're an investor in this company or have some other vested interest in shilling for them, good luck with that.
The thing about startup decisions is: most of them are "wrong". Or they start right and become wrong later. Successful startups aren't successful in spite of their wrong decisions, they're successful because they can change them very quickly (then write a blog post about it and get more customers).
There's also a strong correlation between people who look at things from weird angles and also build good products. Why are you surprised the people who invented an entirely new way of doing VPNs also don't cargo cult database storages?
Sounds like you need to do a lot more research before commenting on this? It's pretty easy to find in the Litestream docs that it replicates to S3, Google Cloud Storage, Azure, and other options even including SFTP. In fact by off-loading the storage integration details to Litestream, the Tailscale people now get seamless storage vendor independence almost for free.
This is what a really smart and future-proof solution looks like.
Again: this is an infrastructure component, running on many machines internally. It's not a Rails app. We use SQLite in very similar circumstances across hundreds of machines. Having all those machines schlepping all their reads back to RDS would not only be untenably slow, but it would also make the whole system less stable. I don't think you've really thought through the design at all, and you should before making comments like these.
I'd put money on this not being the last database change too! But not for the same reasons you would.
Unless I'm misunderstanding something, it seems to me that Tailscale's controller, discussed in the previous blog post [1], is like a typical CRUD API server, currently only running on one machine (they're not using Litestream's work-in-progress live read replicas), not something widely distributed like Fly's service discovery infrastructure.
I'm not joining the "just use RDS" crowd, but it looks to me like using a managed DB service like RDS would have been a reasonable decision for this application.
Choosing a technology that supports your business needs and growth, and that also lets you easily migrate to something else once it stops being suitable (or your requirements change, or you discover a way in which the tech doesn't work for you, or whatever), doesn't sound like a questionable technology decision, it sounds like a great one.
a little bit oot, but just installed tailscale on my dev stack (ec2, bare metal at home and my macbook air). mindblown by its simplicity and "it just works". Use it for logging purposes (ELK is too heavy for free tier ec2) and CI/CD (again, too heavy for my free tier ec2)
Maybe I missed something. How do you account for the VM downtime? The VM containing the service writing to the sqlite db, it can go down without warning. How do you mitigate that?
It doesn’t. Given that it periodically dumps the WAL to an S3 bucket for actual storage this sounds like async replication. Async replication is prone to lose last bits (last by time) of data when the node goes down. They’re probably more or less fine if the SQLite file is stored on the EBS though, it means they’ll just have to restart that node that failed or reattach EBS volume to a new healthy node. It does mean downtime though.
On another note I’m using Tailscale and it’s a wonderful service and I get it for free.
That was a fun read but I'm missing something here. I've never worked on large scale systems and I don't understand why they didn't choose My or Postgre SQL.
Some comments say that it's because they're complicated but what exactly is complicated about them?
Can someone with more experience explain to me what would have happened if, for example, they used PostgreSQL from the beginning?
P.S Apologies if this is a stupid question.
Edit: Looks like I hurt someone's feelings in this comment.
i don't think it's a stupid question and not sure why you getting downvoted. the earlier blog post from tailscale goes into more detail about how the system in question works: https://tailscale.com/blog/an-unlikely-database-migration/
> Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM.
if they had used postgres/mysql or some other external RDBMS then that would have meant managing a whole other set of dependencies and doing at least an IPC/Pipe roundtrip for every single query instead of being able to use an embedded database like custom-built JSON db or SQLite which is only going to need in-process syscalls for reading/writing to disk/cache.
so, this allows them to scale their queries vertically orders of magnitude more than an external db (SQLite can do an insane amount of simple read queries per second) as well as making their whole dev/test/release cycle simpler and easier to manage.
that's my understanding anyway - hopefully it makes some sense.
It has been almost 15 months since the previous blog post:
> Unfortunately, if we want to move to running more than one CONTROL process simultaneously for high-availability and better release management, that means we no longer have exactly one process managing the data
(Ignoring the fact this may be an April fools joke.)
> We’re going to put everything in a single file on disk again.
For most small scale projects I'm just reading and writing lines to a file on disk. Sure, I could over-engineer some database, or I could get a minimal viable project off of the ground and worry about scaling later. Most projects run into issues long before scalability.
I found myself recently wanting a lightweight key-value store for random structures. For this purpose, I created a toy single-file key-value store [1] in less than 256 lines of C. It's not quite ready for prime time, but it's incredibly easy to use.
Tailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al).
Maybe I'm doing something wrong, but our team has like a few dozens of PostgreSQL RDS running on AWS for the past four years and the ops is... Basically zero?
I don't see how CockroachDB ends up in your list of DBs that lead to vendor lock-in or being unable to run a cluster locally, because it's open source and you can run your own cluster. Is there something about it I haven't spotted that puts it in that list?
CockroachDB is licensed under BSL, so it is not really open source rather a source open or source available project. I checked their LICENSE file [0] and it mentions bunch of other open source licenses with BSL. I randomly checked a code file [1] and it seems the code is licensed under BSL.
Very interesting. I'd never heard of it but apparently MariaDB uses it too.
TL;DR is the source is available but you can't use it in production without paying a fee. However versions older than 4 years become truly open source, so you can use old versions for free.
Interesting compromise. Might be better than "open core".
That’s not quite right. You can very much use the bsl code in production without a license and many, many companies do. The BSL contains a space for additional use grants. In crdb’s license, you can use the product for anything except building a hosted database as a service product. The definition of what that is is a little bit vague but it more or less means that the users of the product you sell shouldn’t be able to cause schema changes. This is all just protection against cloud providers taking the product and reselling it. I believe also the terms of this bsl are that it becomes Apache 3 years after the code has been released.
You absolutely cannot use the BSL code in production without a commercial license. That's literally the first thing it says:
> The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work. The Licensor may make an Additional Use Grant, above, permitting limited production use.
Maybe specific software modifies that as you say, but the base BSL doesn't. I agree it's super vaguely worded ("production" isn't defined at all). Also it turns out MariaDB itself isn't BSL licensed - it's GPL, just some extra projects around it.
> The Licensor may make an Additional Use Grant, above, permitting limited production use.
See the text of the Additional Use Grant for CockroachDB [1]:
> Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for a Database Service.
> A “Database Service” is a commercial offering that allows third parties (other than your employees an contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties.
In all fairness, litestream is no CockroachDB/PlanetScale/YugaByte; but why pay the ops overhead (and your lawyers to make sense of their "OSS" licenses) when litestream would do just as nicely.
TFA goes (emphasis mine):
Litestream..grabs a lock so that no other process can checkpoint. It then watches the WAL file and streams the appended blocks up to S3, periodically checkpointing the database for you when it has the necessary segments uploaded. ___This gives you near real time backups and lets you replay your database from S3 trivially..It's a great hack, and a small enough program that I could read my way through it entirely before committing to it.___
they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
I set up a similar system in the past with redis replicas — it was about 200 MBs of data that I needed to have on quick access to on ~ 300 servers around the world — it was easy to set up and worked really well.
> Footnote: coworkers point out it’s April Fool’s today and request that I clarify this isn’t a joke. Joke’s on them: every day’s April Fool’s in the Tailscale Database Engineering department.
* Instead of an actual database use a JSON file.
* Write a blog post about how that didn't scale.
* Instead of an actual database hand-roll something else.
* Write a blog post about how that didn't scale.
* Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it replicate.
I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place.
Are we supposed to expect another blog post somewhere around a year down the road titled "SQLite didn't scale" or "The new library for replication that was released a few months ago contained a bug that took our prod down"?
Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features?
I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing.
Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about.
If you want content about how to built best-practices-compliant hyperscaler-grade systems, there are places to go to get that kind of content. But you've never had a problem with Tailscale that owed to their JSON file backend not scaling --- how I know that is, you've likely never experienced a stability problem with Tailscale at all, because it just doesn't break. That they got it to work, for a long time, with a JSON file backend should make you think harder about how you build things.
I have nothing to add to this discussion, but I wanted to add that I love this comment. Sometimes we have to take a step back and really think about why we're making all of these choices and picking certain tools for a job.
I love Tailscale and it's true that I am not suffering from stability problems with its database choices.
But, I do have challenges working with their ACL structure -- which is backed by their database -- and specifically would love to be able to update users/groups separately from tag groups and both of those separately from ACL rules and I can't help but wonder if the fact that this feature doesn't exist relates to the fact that the database doesn't normalize these concepts into separate "tables" separated by "foreign keys".
This is great to hear. We've been happy users for a couple of years, and the device tagging support has come a long way (and is a big enabler for sensible ACLs), but trying to use Tailscale's ACLs in general has been our biggest pain point with the product.
> Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features?
IMO both this and "announcements of new, perfectly-formed features" are at the same level (if not lower, because a new feature is still something new). Both are essentially marketing with no technical content that's really worth your time. Except that it's now the third iteration of "we did an obviously questionable thing and replaced it with another questionable solution".
> it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basically a litany of engineering mistakes, and it's one of the things I've most enjoyed writing.
I can agree with that, but I wouldn't bunch your writeup into the same group. That's some pretty decent technical content about the problems that were encountered. Meanwhile in these writeups what were the problems? "JSON file database too slow", "Hand-rolled software has to be maintained in-house"?
> Every serious product gets built the way Tailscale is describing. You start some components with the simplest thing that can reasonably work, and see how far you can take them. The JSON file Tailscale used to use was one of the most interesting and valuable things they wrote about.
Yet they keep changing it, migrating, writing tests for the new code and writing blog posts. The time all these things take even if the size of the new code is less than 100 lines is substantial.
> Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features?
It depends on what that tool is doing and if the thing it's doing is a core competency for your company.
It absolutely makes sense to geek out about stuff like this if it's part of your core product. However, if it isn't, then do yourself and your engineering team a favor and pick boring technologies.
Do you know what DB system tailscale would be using if they choose something boring postgres or Cassandra? Postgres or cassandra.
The cost of not doing that is they are on their 3rd iteration of switching db technologies and the development effort to keep rediscovering the same problems.
The cost of complication is high. They're optimizing for simplicity. Both for low friction development and easy operation.
sqlite is amazingly simple for local dev. Probably even simpler than a json file.
High availability Postgres is complex to operate (you need a whole separate strongly consistent data store to coordinate it). Cassandra is also very complex.
sqlite is, again, simpler. Litestream makes sqlite plausible for production.
You are buying simpler local dev with a more complicated and error prone production.
> High availability Postgres is complex to operate
High available SQLite is complicated. More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system. Instead, you've got to bolt on solutions to fix the fact that you chose SQLite.
Containers make local dev with postgres or other db technologies pretty slick. I have a hard time buying the "oh no, but then we need to run docker" because docker containers have added benefits often simplifying local dev (All your dev tools defined in the docker file. Done, new dev installs docker and moves from there).
> sqlite is, again, simpler. Litestream makes sqlite plausible for production.
SQLite is a fantastic tool, when it's used in the right ___location. A backend database is not the right ___location for SQLite. Litestream may make it "plausible" but it certainly doesn't make it a sane choice.
They don't have a multi-tenant database problem! This is the second weird thing you've said about this design (the other was that they'd need to run Jepson simulations for it). They're not a database hosting provider; they're the single producer and the single consumer of their database. They have a single writing leader, and read-only replicas. That's how Litestream works; it's the only way it works.
> You are buying simpler local dev with a more complicated and error prone production.
How so? Any evidence for that?
> More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system.
Neither was Tailscale, that's why the engineers decided they were a good fit. But I'm sure you know more about it than they do.
> Containers make local dev with postgres or other db technologies pretty slick.
No they don't. Dockerizing everything for local dev bogs you down to Docker's added latency for every operation, this is not even getting into the latencies added by having to periodically `docker pull`, `build` images, provision the disk space for the images, and maintain all that infrastructure over time. Compared to that, an SQLite database is basically nothing.
> All your dev tools defined in the docker file. Done, new dev installs docker and moves from there
Docker is not a good way to do local dev, overall. The entire point of local dev is speed of iteration. If you need a usable test environment, set one up in UAT.
> A backend database is not the right ___location for SQLite.
> SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites)....Systems designers report success using SQLite as a data store on server applications running in the datacenter, or in other words, using SQLite as the underlying storage engine for an application-specific database server.
And they tacked on Litestream, which made for far more interesting failure modes. Hopefully they ran their jepson... oh wait, no they didn't really care about the reasons they ruled out traditional db choices.
Do you understand how Litestream works? What's the Jepson test you'd do here, to make sure your single, statically assigned leader isn't conflicting with... itself? All Litestream does is ship WAL segments. The whole point of this design is not having the distributed consensus problems (and concomitant schema and development inflexibility) that etcd has.
Yes but userland wireguard is unique and useful. This is ignoring textbooks, competitors who write about their exploits warts and all and screams of get a more experienced db team to work with you... shoving it into a json screams of throw it at a disk and let the hardware deal with it
>> shoving it into a json screams of throw it at a disk and let the hardware deal with it
Which...worked. For quite a long time. And required no dev effort. Freeing devs up to focus on other things.
They did it as a POC, and it worked, better even than they had anticipated; why over-engineer it up front? Even when it turns out they are gonna need it (i.e., YAGNI no longer applies), they have been able to push off their "solve problems customers don't care about" to instead favor "problems customers do care about". That...seems like sound engineering to me?
Speaking as someone who has had to fix things like this...
When you reach capacity and it hits a screaming halt as the hardware capacity has been reached and performance doesn't collapse its completely gone.
This is not about best use. Hell json file performance may be well within spec of the system requirements, but jumping up and down without perf numbers or specs is typical of sweep it under the carpet for a generation...
That would be a good critique if it was was happening, but they gracefully transitioned from a JSON file(!) backend to etcd, and gracefully transitioned from etcd to sqlite. I'd say they're going to gracefully transition from sqlite, but I think the think people don't have their heads around is that it's possible they'll never need to do that, because of The Unreasonable Effectiveness of SQLite.
Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property.
We have MySQL and PostgreSQL veterans on the team. We know those options well.
The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself.
Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do).
You might be able to eek out a bit more performance by hand tuning test files... but.. to what end?
I can't speak for Tailscale, but our whole raison d'etre is running containers for people, and almost none of our development environments are containerized either. Containers (and remote dev environments) create friction. They're useful tools when that friction is unavoidable anyways, but when it isn't --- and you can design systems with that being a goal --- it makes sense to take full advantage of that.
I write applications that use Postgres, and don't use Docker for testing. I just create a database based on the name of the test case, delete it, create it, explode the schema in there, and run the tests. The reason is because I don't want to pay Postgres' startup costs every time I run the tests. Tests are run thousands of times a day, while you're in the mindset to rapidly iterate. Any delay is unacceptable.
At some point, migrations were taking too long, so I just started caching the database schema. I have a go generate thingie that creates a fresh database, applies all the migrations, and dumps the schema to a file that gets checked in with the code. (CI checks that the migrations actually result in the database state that's checked in. Yes, there is something that regexes out the hostname from the sql file, since Postgres dumps that in there by default and that results in spurious diffs.)
I have learned that people often put up with really sub-optimal developer workflows. My test is that you should be able to add "print hello world" to the top of your main function, and see that printed on your screen in less than 5 seconds. (Yes, even if your application is complicated. I have a hacked up copy of K8s that starts that quickly, so that you can write an app that deeply integrates with K8s and not have to reuse a cluster between tests, or pay a long cluster creation cost. All obstacles must be removed!) If it takes longer than that, you are just throwing your developer's salaries into the toilet. (If it takes 6 seconds, you open up HN, and there's an hour gone!)
I suggest reading the article. The reason they haven't chosen "boring" tech has nothing to do with the clients running their software and everything to do with their internal development experiences.
> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.
The reason they are ruling out products isn't "this won't work where it runs" its "I don't like this". They talk about worrying about network graphs and proceed to use SQLite and copy garbage around.
If you're dinging them for not using Cockroach, they explained why they're not using Cockroach in the exact paragraph you quoted: if they use Cockroach, they're committing to the scaling and distribution system that Cockroach provides, which might or might not be a good fit for them 2 years from now. If it isn't a good fit, they're stuck with a huge engineering bill to get themselves out of Cockroach.
We're moving towards sqlite for things for the same reason. sqlite is simple to reason about. The distributed state problem not simple, but our distributed state problem is not Tailscale's and theirs isn't yours; every distributed state problem is unhappy in its own way. As they work out the contours of their specific distributed state problem, sqlite isn't the component that's going to break down.
If from your perspective, a super-successful VPN product is 'really struggling', you must have incredibly high standards! We all hope to hear more about your successes, I'm sure.
Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments.
And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?
No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!"
> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.
Seriously.. consider this.. they are worried about the transaction semantics of using db replicas and instead CHOSE SQL LITE!
Sqlite replication via litestream has bascially no overhead. It's a great choice if your database only needs to service one client.
It sounds like you're essentially arguing that people should always use the most robust data store available regardless of cost or requirements. Their needs were served just fine by a json file for quite some time — if that's all you need, then maintaining a full Posgres cluster is the "crazy" solution actually.
Sorry, I probably didn't phrase it very clearly. I don't think there is anything wrong with needing some external help for that, but the combination of their choices is IMO a bit bizarre:
* SQLite which I mostly see as an embeddable database for applications, used on a server.
* Single table with key-value pairs.
* Having the tool for replication be a quite young project that's in active development.
They have a single table with KV pairs right now as an artifact of having used the "database with built-in replication which is tailor built for key-value storage" you complain that they're moving away from. The point of the blog post is that they're doing that in part to move away from the single table of KV pairs.
> SQLite which I mostly see as an embeddable database for applications
This is common but incorrect view of SQLite. It's able to handle many server-side workloads, as long as you're aware of its limitations (and the Tailscale team certainly are).
Those limitations are also the limitations of the backing stores of many distributed state solutions; you just don't see them because they're hidden behind complicated APIs.
There's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.
Threads on HN are very sensitive to initial conditions. A very negative (and I think pretty facile) comment took an early lead here, most likely because of its snark/dunk value. Give the thread time, it'll even out.
Agreed, it's a bit of a shame for HN imo. I see a team who is being brilliant and understanding their ___domain down to the core with as minimal resources as they can get away with.
There is incredible value in incrementally adding complexity only when needed.
tailscale = YAGNI to the MAX! i applaud their willingness to think outside the box 99% of devs seem to be happy to squeeze themselves into just because they read about it in a book by some self-appointed tech god.
Totally off topic.
I am not an UI/UX designer but why can't the page scale to the size of the screen? Sure I could ctrl/cmd++ but is it hard to detect the screen size?
Genuine question, not trying to be condescending.
This should be a new logical fallacy: appeal to current year. Every time I hear someone say "it's $CURRENT_YEAR, we obviously need to implement $PARTISAN_POLICY" I wonder about what fringe politics will be "obviously correct" in 3022.
Nothing in this post claims that the year intrinsically makes this the correct choice. It is the database for 2022 because it is now 2022 and this is the database they are now using.
Seconded. It's a logical fallacy because they are using the year as proof that we should adopt their policy. Except the year only proves what year it is - nothing else.
And, completely apart from the database, it's a really valid point - not a joke at all.
When someone says, "It's 2022. Why are we still arguing about X?" or some such, it means that they haven't been able to persuade the rest of the world about X on the merits of their argument, and so they want to short-circuit the argument by appearing "modern" or "up to date" or "on the right side of history". They want to win by persuading the other side to give up in embarrassment, not by virtue of the quality of their position. (In their own eyes, their position is probably self-evidently the only right and moral one, but that doesn't tell us much.) It's a way of trying to win by default, and it deserves nothing other than being regarded as "you lose by default".
I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? They started using etcd knowing that he would become a bottle neck, that should have been a non-starter, right?
> Using a text file and ...
> It seems like something you'd do as a proof of concept
I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.
> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?
This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions.
The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense.
Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.
Litestream author here. The default configuration is to replicate any new changes every second to S3. This is mainly a cost issue though. S3 doesn't charge for ingress bandwidth but it does charge $0.000005 per PUT request. If you have a non-stop, continuous stream of writes against your database then that's about $12/mo. In practice, most people don't have a continuous stream of writes. Michael Lynch, for example, wrote a post about how he pays $0.03/month for his Litestream replication[1].
I've worked with Raft a lot in my career. I wrote the original etcd Raft implementation (sorry!) [2] and even wrote a visualization explaining Raft[3]. It's a great tool when you need really high durability guarantees but I'm not convinced it gives you better availability guarantees than a single node that can quickly be recovered.
There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues.
A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
the live replication feature litestream is implementing won't be using s3 - it will be direct node to node replication. hasn't been released yet and currently s3 is used for a cold backup that can be quickly restored to a fresh node as far as i know.
https://github.com/benbjohnson/litestream/issues/8
Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision
I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit.
I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years.
It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a ___domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have.
why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?
Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."
I would actually love to work at a company like that. I've been a professional developer for over 20 years now and at least in my experience, too much unneeded complexity has been the root cause of so many defects and product failures. I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages.
If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
> I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by...
For me, it's not so much my fellow developers -- half my age, but quite bright enough to realise the benefits of simplicity -- but that the vendors of the tools our corporation has decided to use have in turn decided to implement them not just as a spaghetti-reinforced mudball, but as a spaghetti-reinforced mudball of spaghetti-reinforced mudballs.
(That these SRMB^2s are then just part of a larger spaghetti-reinforced mudball of SRMB^2s, i.e a spaghetti-reinforced mudball cubed, is of course not the vendors' fault but corporate. But then, aren't corporate systems always SRMBs? So if at least the components were simple, the end result would be only an SRM to the power of one, not three... Sigh.)
> All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages.
i feel your pain dave. resume driven development and FAANG envy seem to be the dominant characteristics of far too many folks in the industry these days.
I'm sympathetic to the idea of avoiding a database because no one on your team has experience with it.
On my old team, our product supported Oracle DB (along with some other databases), but on one on the team really was an oracle expert, and we would frequently run into questions that we didn't really know how to answer. We would have dropped it if we could.
It's easy to say "rtfm" but doing this doesn't make you an expert in the system. And as I bet many NH readers know, there's a big difference between "I read the quickstart guide and got a DB working in an afternoon" and "We've hit an error an error that the manual says nothing about, but our DB expert saw this once, 7 years ago, and remembers how to recover"
> "We've hit an error an error that the manual says nothing about, but our DB expert saw this once, 7 years ago, and remembers how to recover"
Yup, this rings true. Once worked on a large Cassandra deployment – holding people's bank accounts, so, ya know, somewhat critical – and we eventually had to get one of the authors of Cassandra onto our on-call team. Even he was stumped by some of the bugs and failure modes. (And, incidentally, the word 'anticompaction' is still enough to trigger my PTSD.)
The notion that the documentation is an accurate and exhaustive explanation of every single thing about a product is a junior engineer mistake that no one with rough ops experience would say. Nor, for that matter, anyone who's written a database that's in use (as I unfortunately have), or really any other piece of software.
And the same goes for the notion that "everyone should go with MySQL or Postgres for everything because they are Proper Databases that are scalable and won't go down". If I were being charitable, I'd say it's a 'no one got fired for buying IBM' argument; if I were being honest, I'd say it puts me in mind of 'MongoDB is webscale' (https://www.youtube.com/watch?v=b2F-DItXtZs). I don't know why bad engineers are so terrified of thinking for themselves specifically – well, generally, but especially – when it comes to databases. They aren't magical contraptions. They are just portable filesystem wrappers.
I see a dozen comments here, all critiquing their proposal. Surely this is April Fools?
> I'm sorry, but what the fuck? Just writing the blog posts alone might have consumed more work-hours than what it would have taken to set a database up in the first place.
> Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it.
> This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they have different priorities and values. But I'm just confused, why would an engineer want to join a company that is making these decisions?
Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams to pick up and run with: just straight up good design and programming.
This is a thing that happens in other fields, too; some of the hardest dishes to cook seem extremely simple and have few ingredients. They're hard because there's no place to hide: your technique has to be flawless or they don't work. A famous entrance exam for new cooks at fine dining restaurants is "make an omelet". Tailscale is, thankfully (and, to me, irritatingly), more like an omelet than a truffled hash.