yes, this will definitely vastly increase the Doom fps, haha (I’m the guy that did that project). But I think there’s a lot more to it than that.
tl;dr — Rust would be great for a rewrite, but Go makes way more sense for a port. After the dust settles, I hope people focus on the outcomes, not the language choice.
I was very surprised to see that the TypeScript team didn’t choose Rust, not just because it seemed like an obvious technical choice but because the whole ecosystem is clearly converging on Rust _right now_ and has been for a while. I write Rust for my day job and I absolutely love Rust. TypeScript will always have such a special place in my heart but for years now, when I can use Rust.. I use Rust. But it makes a lot of sense to pick Go.
The key “reading between the lines” from the announcement is that they’re doing a port not a rewrite. That’s a very big difference on a complex project with 100-man-years poured into it.
Places where Go is a better fit than Rust when porting JavaScript:
- Go, like JavaScript and unlike Rust, is garbage collected. The TypeScript compiler relies on garbage collection in multiple places, and there are probably more that do but no one realizes it. It would be dangerous and very risky to attempt to unwind all of that. If it were a Rust rewrite, this problem goes away, but they’re not doing a rewrite.
- Rust is so stupidly hard. I repeat, I love Rust. Love it. But damn. Sometimes it feels like the Rust language actively makes decisions that demolish the DX of the 99.99% use-case if there’s a 0.001% use-case that would be slightly more correct. Go is such a dream compared to Rust in this respect. I know people that more-or-less learned Go in a weekend and are writing it professionally daily. I also know people that have been writing Rust every day professionally for years and say they still feel like noobs. It’s undeniable what a difference this makes on productivity for some teams.
Places where Go is just as good a fit as Rust:
- Go and Rust both have great parallelism/concurrency support. Go supports both shared memory (with explicit synchronization) and message-passing concurrency (via goroutines & channels). In JavaScript, multi-threading requires IPC with WebWorkers, making Go’s concurrency model a smoother fit for porting a JS-heavy codebase that assumes implicit shared state. Rust enforces strict ownership rules that disallows shared state, or we can at least say makes it a lot harder (by design, admittedly).
- Go and Rust both have great tooling. Sure, there are so many Rust JavaScript tools, but esbuild definitively proves that Go tooling can work. Heck, the TypeScript project itself uses esbuild today.
- Go and Rust are both memory safe.
- Go and Rust have lots of “zero (or near zero) cost abstractions” in their language surface. The current TypeScript compiler codebase makes great use of TypeScript enums for bit fiddling and packing boolean flags into a single int32. It sucks to deal with (especially with a Node debugger attached to the TypeScript typechecker). While Go structs are not literally zero cost, they’re going to be SO MUCH nicer than JavaScript objects for a use-case like this that’s so common in the current codebase. I think Rust sorta wins when it comes to plentiful abstractions, but Go has more than enough to make a huge impact.
Places where Rust wins:
- the Rust type system. no contest. In fairness, Go doesn’t try to have a fancy type system. It makes up for a lot of the DX I complained about above. When you get an error that something won’t compile, but only when targeting Windows because Rust understands the difference in file permissions… wow. But clearly, what Go has is good enough.
- so many new tools (basically, all of them that are not also in JS) are being done in Rust now. The alignment on this would have been cool. But hey, maybe this will force the bindings to be high-quality which benefits lots of other languages too (Zig type emitter, anyone?!).
By this time next week when the shock wears off, I just really hope what people focus on is that our TypeScript type checking is about to get 10 times faster. That’s such a big deal. I can’t even put it into words. I hope the TypeScript team is ready to be bombarded by people trying to use this TODAY despite them saying it’s just a preview, because there are some companies that are absolutely desperate to improve their editor perf and un-bottleneck their CI. I hope people recognize what a big move this is by the TypeScript team to set the project up for success for the next dozen years. Fully ejecting from being a self-hosted language is a BIG and unprecedented move!
A tiny thing that's not relevant to this particular piece of work but is worth having in background when thinking about Go is that while Go would like Python typically be described as "memory safe" unlike Java (or more remarkably, Rust) it is very possible for naive programmers to cause undefined behaviour in this language without realising it.
Specifically if you race any non-trivial Go object (say, a hash table, or a string) then that's immediately UB. Internally what's happening is that these objects have internal consistency rules which you can easily break this way and they're not protected against that because the trivial way to do so is expensive. Writing a Go data race isn't as trivial as writing a use-after-free in C++ but it's not actually difficult to do by mistake.
In single threaded software this is no caveat at all, but most large software these days does have some threading involved.
I'm confused what you ascribe to as undefined behaviour. What does that mean in the Go context? There is no mention of what UB is in Go at https://go.dev/ref/spec .
With a race of a non-trivial object in Go you're violating assumptions of Go's language runtime. Physically impossible things can't happen because that's what physically impossible means, but stuff which is merely written down in a document like the one you linked is fair game for such problems.
In terms of concrete examples, this might allow remote code execution, arbitrary reads or writes of memory that you otherwise don't have access to, stuff like that.
Especially given Anders is the one announcing this, given he was the chief architect of C#. But C# AOT is maybe not as mature/lightweight as a Go binary and clearly startup time here is very important. [Edit: the real reason is in the FAQ posted in a bunch of other comments https://github.com/microsoft/typescript-go/discussions/411]
Some big projects have so many people trying to do PRs that it's actually a bit of a hassle to deal with them all. So I don't think maximising the number of contributors should necessarily be one of the top goals for projects that are already big or have guaranteed relevance.
Is learning a language even a thing anymore with $Internal_or_external_LLM_helper plugin available for every IDE? I haven't found syntax lookups to be that much a concern anymore and any boneheaded LLM suggestions are trivial to detect/fix.
"Memory safety" is a term of art meaning susceptibility to memory corruption attacks. They had to come up with some name for it; that's the name they came up with. This is a perennial tangent in conversations among technologists: give something a legible name, and people will try to axiomatically (re)define it.
Rust is memory safe. Go is memory safe. Python is memory safe. Typescript is memory safe. C++ is not memory safe. C is not memory safe.
I mean, no? That's basically a known bug in Rust's compiler, specifically it's a soundness hole in type checking, and you'd basically never write it by accident - go read the guts of it for yourself if you think you might accidentally do this.
At some point a next generation solver will make this not compile, and people will probably invent an even weirder edge case for that solver.
Whereas the Go example is just how Go works, that's not a bug that's by design, don't expect Go to give you thread safety that's not what they promised.
thank you for the clarification. you're right. I guess I was just trying to say that it's a spectrum (even if Rust is very very far along the way towards not having any holes). I can't seem to find it but there's some Tony Hoare or maybe Alan Turing quote or something like that about the only 100% correct computer program to ever exist was the first one.
This is true in that if you pass pointers through go routines, you do not have guarantees about what’s at the end of that pointer. However, this is “by design” in that generally you shouldn’t do that; the overhead the go memory model places on developers is to remember what’s passed as value and what’s passed as a pointer, and act accordingly. The rest it takes care of for you.
The burden placed by rust on the developer is to keep track of all possible mutability and readability states and commit to them upfront during development. (If I may summarize, been a long time since I wrote any Rust). The rest it takes care of for you.
The question of which a developer prefers at a certain skill level, and which a manager of developers at a certain skill level prefers, is going to vary.
Segfaults are very much a memory safety issue. You are correct that concurrency is the cause here, but that doesn't mean it's not a memory safety issue.
That said, most people still call Go memory safe even in spite of this being possible, because, well, https://go.dev/ref/mem
> While programmers should write Go programs without data races, there are limitations to what a Go implementation can do in response to a data race. An implementation may always react to a data race by reporting the race and terminating the program. Otherwise, each read of a single-word-sized or sub-word-sized memory ___location must observe a value actually written to that ___location (perhaps by a concurrent executing goroutine) and not yet overwritten. These implementation constraints make Go more like Java or JavaScript, in that most races have a limited number of outcomes, and less like C and C++, where the meaning of any program with a race is entirely undefined, and the compiler may do anything at all.
I believe that most JVMs implement dynamic dispatch in a similar manner to C++, that is, classes are on the heap, and have a vtable pointer inside of them. Whereas Go's interfaces can work like Rust's trait objects, where they're a pair of (data pointer, vtable pointer). So the behavior we see here with Go is unlikely to be possible in Java, because the tear wouldn't corrupt the vtable pointer, because it's inside what's pointed at by the initial pointer, rather than being right after it in memory.
These bugs do happen, but they have a more limited blast radius than ones in languages that are clearly unsafe, and so it feels wrong to lump Go in with them even though in some strict sense you may want to categorize it the other way.
Sure, that's all true. It does limit Go's memory safety guarantees. However, I still believe that just because Java and other languages can give better guarantees around the blast radius of concurrency bugs does not mean that Go's definition of memory safety is invalid. I believe you can justifiably call Go memory-safe with unsafe concurrency. This may give people the wrong idea about where exactly Go fits in on the spectrum of "safe" coding (since, like you mentioned, some languages have unsafe concurrency that is still safer,) but it's not like it's that far off.
On the other hand, though, in practice, I've wound up using Go in production quite a lot, and these bugs are excessively rare. And I don't mean concurrency bugs: Go's concurrency facilities kind of suck, so those are certainly not excessively rare, even if they're less common than I would have expected. However... not all Go concurrency bugs can possibly segfault. I'd argue most of them can't, at least not on most common platforms.
So how severely you treat this lapse is going to come down to taste. I see the appeal of Rust's iron-clad guarantees around limiting the blast radius, but of course everything comes with limitations. I believe that any discussion about the limitations of guarantees like these should have some emphasis on the real impact. e.g. It's easy enough to see that the issues with memory management in C and C++ are serious based on the security track record of programs written in C and C++, I think we're still yet to fully understand how much of an impact Go's lack of safe concurrency will impact Go software in the long run.
> On the other hand, though, in practice, I've wound up using Go in production quite a lot, and these bugs are excessively rare.
I both want to agree with this, but also point to things like https://www.uber.com/en-CA/blog/data-race-patterns-in-go/, which found a bunch of bugs. They don't really contextualize it in terms of other kinds of bugs, so it's really hard to say from just this how rare they actually are. One of the insidious parts of non-segfaulting data race bugs is that you may not notice them until you do, so they're easy to under-report. Hence the checker used in the above study.
> not all Go concurrency bugs can possibly segfault. I'd argue most of them can't, at least not on most common platforms.
For sure, absolutely. And I do think that's meaningful and important.
> I think we're still yet to fully understand how much of an impact Go's lack of safe concurrency will impact Go software in the long run.
Yep, and I do suspect it'll be closer to Java than to C.
The Uber page does a pretty good job of summing it up. The only thing I'd add is that there has been a little bit of effort to reduce footguns since they've posted this article; as one example, the issue with accidentally capturing range for variables is now fixed in the language[1]. On top of having a built-in (runtime) race detector since 1.1 and runtime concurrent map access detection since 1.6, Go is also adding more tools to make testing concurrent code easier, which should also help ensure potentially racy code is at least tested[2] (ideally, with the race detector on.) Accidentally capturing named return values is now caught by a popular linting tool[3]. There is also gVisor's checklocks analyzer, which, with the help of annotations, can catch many misuses of mutexes and data protected by mutexes[4]. (This would be a lot nicer as a language feature, but oh well.)
I don't know if I'd evangelize for adopting Go on the scale that Uber has: I think Go works best for shared-nothing architectures and gets gradually less compelling as you dig into more complex concurrency. That said, since Uber is an early adopter, there is a decent chance that what they have learned will help future organizations avoid repeating some of the same issues, via improvements to tooling and the language.
Reads of memory locations larger than a single machine word are encouraged but not required to meet the same semantics as word-sized memory locations, observing a single allowed write w. For performance reasons, implementations may instead treat larger operations as a set of individual machine-word-sized operations in an unspecified order. This means that races on multiword data structures can lead to inconsistent values not corresponding to a single write. When the values depend on the consistency of internal (pointer, length) or (pointer, type) pairs, as can be the case for interface values, maps, slices, and strings in most Go implementations, such races can in turn lead to arbitrary memory corruption.
Not all segfaults necessarily point to exploitable bugs, but a segfault is usually very suspicious. On common architectures, you get a segmentation fault when there is a memory access violation. Which usually means you've either read from, written to, or tried to execute code at an address that is not readable, not writeable or not executable in your address space. That is suspicious because unless your program is intentionally doing that (which is relatively rare, and obviously in that case you would want to explicitly catch it with a signal handler) it suggests that some assumption your program is making about memory somewhere is incorrect. Like Go says, arbitrary memory corruption.
Is that exploitable? It depends. It's easier to assume that it is than hope that it isn't.
However, while it is a more serious category of issue, I have two reasons to suggest people don't over-index on it:
- Concurrency bugs that can not lead to segmentation faults are by no means safe, they can still lead to exploits of arbitrary severity. Ones that can are more dangerous since they can violate Go's own safety guarantees, but so can the "unsafe" package, so you need to put it into some perspective.
- Concurrency bugs that can are likely to be less common. In my experience, it is not extremely common to re-assign shared map or interface values in Go. If you are sharing a value of map, slice, string or interface and do plan on re-assigning it (thus causing the hazard in question) you can work around this problem trivially by adding a tiny bit of indirection, using an atomic pointer to the value instead, and re-assigning that pointer instead. Making a new value each time is no big deal since all of the fat pointers in question are still relatively small (just 2-3 machine words) though it incurs more allocations and pointer indirections so YMMV.
And of course I recommend using all applicable linters, the checklocks analyzer from gVisor, and careful encapsulation of shared memory where possible. Even better is to avoid it entirely if you can.
Of course, as much as I love Go, some types of program are going to need lots of hairy shared memory and mutations interweaving. And for that, Rust is the obvious best choice.
Yeah, I was just thinking if an implementation has the propensity to abort or fail early with a segfault, that's better than running with memory corruption and far more difficult to exploit. It's not clear from the upthread example how soon it fails after corruption so there is potentially a narrow window where such a bug could be exploited if found in the wild with the apropos attack surface.
Ah I see what you mean. To be fair, it is still true that not every bug that can lead to a segfault is exploitable, including this one potentially, but on the other hand, I think the point is that Go's memory safety guarantees always prevent segmentation faults: by the time you've hit a segmentation fault, you have definitely broken the type system and nothing is guaranteed anymore W.R.T. memory safety. So any bug that causes a segmentation fault is definitely immediately suspect. I think that's the point they were going for, at least.
tl;dr — Rust would be great for a rewrite, but Go makes way more sense for a port. After the dust settles, I hope people focus on the outcomes, not the language choice.
I was very surprised to see that the TypeScript team didn’t choose Rust, not just because it seemed like an obvious technical choice but because the whole ecosystem is clearly converging on Rust _right now_ and has been for a while. I write Rust for my day job and I absolutely love Rust. TypeScript will always have such a special place in my heart but for years now, when I can use Rust.. I use Rust. But it makes a lot of sense to pick Go.
The key “reading between the lines” from the announcement is that they’re doing a port not a rewrite. That’s a very big difference on a complex project with 100-man-years poured into it.
Places where Go is a better fit than Rust when porting JavaScript:
- Go, like JavaScript and unlike Rust, is garbage collected. The TypeScript compiler relies on garbage collection in multiple places, and there are probably more that do but no one realizes it. It would be dangerous and very risky to attempt to unwind all of that. If it were a Rust rewrite, this problem goes away, but they’re not doing a rewrite.
- Rust is so stupidly hard. I repeat, I love Rust. Love it. But damn. Sometimes it feels like the Rust language actively makes decisions that demolish the DX of the 99.99% use-case if there’s a 0.001% use-case that would be slightly more correct. Go is such a dream compared to Rust in this respect. I know people that more-or-less learned Go in a weekend and are writing it professionally daily. I also know people that have been writing Rust every day professionally for years and say they still feel like noobs. It’s undeniable what a difference this makes on productivity for some teams.
Places where Go is just as good a fit as Rust:
- Go and Rust both have great parallelism/concurrency support. Go supports both shared memory (with explicit synchronization) and message-passing concurrency (via goroutines & channels). In JavaScript, multi-threading requires IPC with WebWorkers, making Go’s concurrency model a smoother fit for porting a JS-heavy codebase that assumes implicit shared state. Rust enforces strict ownership rules that disallows shared state, or we can at least say makes it a lot harder (by design, admittedly).
- Go and Rust both have great tooling. Sure, there are so many Rust JavaScript tools, but esbuild definitively proves that Go tooling can work. Heck, the TypeScript project itself uses esbuild today.
- Go and Rust are both memory safe.
- Go and Rust have lots of “zero (or near zero) cost abstractions” in their language surface. The current TypeScript compiler codebase makes great use of TypeScript enums for bit fiddling and packing boolean flags into a single int32. It sucks to deal with (especially with a Node debugger attached to the TypeScript typechecker). While Go structs are not literally zero cost, they’re going to be SO MUCH nicer than JavaScript objects for a use-case like this that’s so common in the current codebase. I think Rust sorta wins when it comes to plentiful abstractions, but Go has more than enough to make a huge impact.
Places where Rust wins:
- the Rust type system. no contest. In fairness, Go doesn’t try to have a fancy type system. It makes up for a lot of the DX I complained about above. When you get an error that something won’t compile, but only when targeting Windows because Rust understands the difference in file permissions… wow. But clearly, what Go has is good enough.
- so many new tools (basically, all of them that are not also in JS) are being done in Rust now. The alignment on this would have been cool. But hey, maybe this will force the bindings to be high-quality which benefits lots of other languages too (Zig type emitter, anyone?!).
By this time next week when the shock wears off, I just really hope what people focus on is that our TypeScript type checking is about to get 10 times faster. That’s such a big deal. I can’t even put it into words. I hope the TypeScript team is ready to be bombarded by people trying to use this TODAY despite them saying it’s just a preview, because there are some companies that are absolutely desperate to improve their editor perf and un-bottleneck their CI. I hope people recognize what a big move this is by the TypeScript team to set the project up for success for the next dozen years. Fully ejecting from being a self-hosted language is a BIG and unprecedented move!