It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations).
> ...rather than improving the product?
Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
----
> if they're this talented, is their time really best spent on...
"People are often surprised and sometimes horrified when they learn that Tailscale maintains its own fork of the Go toolchain. Tailscale is a small startup. Isn't that a horrible distraction, a flagrant burning of innovation tokens?"
"Maybe. But the thing is, you write code with the engineers you have."
"We had a problem: We kept crashing on iOS, and in addition to being awful, it was preventing us from adding features."
"Another team might have decided to cut even more features on iOS to try to achieve stability, or limited in some way the size of the tailnet that iOS could interact with."
"Another team might have radically redesigned the data structures to squeeze every last drop out of them."
"Another team might have rewritten the entire thing in Rust or C."
"Another team might have decided to accept the crashes and attempted to mitigate the pain by making re-establishment of connections faster."
"Another team might have decided to just live with it and put their focus elsewhere."
"The Tailscale team has Go expertise, spanning the standard library to the toolchain to the runtime to the ecosystem. It’s an asset, and it would be foolish not to use it when the occasion arises. And the fun thing about working on low level, performance-sensitive code is that that occasion arises with surprising frequency."
"Blog posts about how people solve their problems are fun and interesting, but they must always be taken with a healthy dose of context. There may be no other startups in existence for which working on the Go linker would be a sensible choice, but it was for us."
> When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
if zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, why are they using s3? migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres
First, S3 and a SQL database aren't comparable. But I think you're bringing up S3 because they're using Litestream to ship WAL frames to S3. Go read the Litestream documentation; Litestream syncs to basically anything. They don't need to "migrate to another S3 compatible backend"; they can migrate to almost anything that can save a file.
It's a super confusing argument regardless, because the industry is lousy with "S3-compatible backends".
> migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres
You may be right. I have no experience migrating litestream but from the docs (https://litestream.io/guides/) it is literally cp'ing files from S3 to wherever and exec'ing one of these one-liners (of course, the devil is in the details):
Litestream author here. Yeah, you're basically right but it's simpler than a DB migration. No need to copy the old data over. You can remove the `-litestream` metadata directory and point it at a new replication destination and it'll automatically re-snapshot the database begin replication.
It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations).
> ...rather than improving the product?
Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason.
----
> if they're this talented, is their time really best spent on...
From: https://tailscale.com/blog/go-linker/
"People are often surprised and sometimes horrified when they learn that Tailscale maintains its own fork of the Go toolchain. Tailscale is a small startup. Isn't that a horrible distraction, a flagrant burning of innovation tokens?"
"Maybe. But the thing is, you write code with the engineers you have."
"We had a problem: We kept crashing on iOS, and in addition to being awful, it was preventing us from adding features."
"Another team might have decided to cut even more features on iOS to try to achieve stability, or limited in some way the size of the tailnet that iOS could interact with."
"Another team might have radically redesigned the data structures to squeeze every last drop out of them."
"Another team might have rewritten the entire thing in Rust or C."
"Another team might have decided to accept the crashes and attempted to mitigate the pain by making re-establishment of connections faster."
"Another team might have decided to just live with it and put their focus elsewhere."
"The Tailscale team has Go expertise, spanning the standard library to the toolchain to the runtime to the ecosystem. It’s an asset, and it would be foolish not to use it when the occasion arises. And the fun thing about working on low level, performance-sensitive code is that that occasion arises with surprising frequency."
"Blog posts about how people solve their problems are fun and interesting, but they must always be taken with a healthy dose of context. There may be no other startups in existence for which working on the Go linker would be a sensible choice, but it was for us."