There is an argument that western civilization went astray when Plato elevated human reason above all other virtues. The obvious problem is that for all of the wonderful things reasoning has given us, it is just as effectively used for deceit both of the self and others. The arguments behind much of today's so called rationalists are laughable -- especially to someone outside of their bubble. It is precisely for these reasons that they have to cloak their ridiculous arguments in arbitrary quantitative analysis and impenetrable jargon. And then there is the irony of their nearly universal militant atheism that comes along with sneering contempt for traditional religion (which they also generally have at best an extremely superficial knowledge of).
I also am particularly amused by the worship of Bayesian statistics without serious reflection on the fact that it is premised on subjective belief in the prior.
When I run a profiler on a compiler I wrote (which parses at somewhere between 500K-1MM lines per second without a separate lexer), parsing barely shows up. I'd be very surprised if the zig compiler is spending more than 5% of the time tokenizing.
I assume there is some other use case that is motivating this work.
No, the problem for a language server is incremental performance, not batch performance. Although there are a lot of bad implementations out there that just reparse the entire buffer on each edit (without the error recovery benefits an incremental parser would give you).
> No, the problem for a language server is incremental performance, not batch performance
"When something is fast enough, people start to use it differently" - Linus Torvalds.
Make your parser able to parse the current file at 30FPS and you do not need incremental parsing anymore nor error recovery. That is probably part of the idea here.
Here that can go both ways - SIMD parsing can allow handling arbitrary changes in reasonable time for files below like maybe 100MB (i.e. everything non-insane), whereas incremental parsing can allow handling small changes in truly-arbitrary-size files in microseconds. A trade-off between better average-case and worst-case time. (of course the ideal thing would be both, but that's even more non-trivial)
It’s not good enough. Incremental parsers can save trees across edits, and you can hang type information off of those trees, so you just aren’t saving parsing time, you are saving type checking time as well. Even if you have a super fast batch parser, you are screwing yourself in other areas that are actually much more expensive.
The runtime cost of type checking is highly dependent on the type system / meta-programming complexity of your language.
For simple languages (Golang?) with a pretty well designed module system: it should be doable to reach ~500KLOC/sec (probably even 1MLOC/s in some case) so more than enough for an interactive usage.
And for complex languages with meta-programming capabilities: they are indeed slow to type check.
But are also a giant pain in the butt to cache without side effects for incremental parsing. It is 2025 and clangd / intellisense still fail to do that reliably for C++ codebases that rely heavily on template usage.
So it does not seem a so-crazy approach to me: It is trading a complexity problem for a performance one.
I am not a fan of PEG. It is straightforward to write a fast parser generator for languages that require just one character of lookahead to disambiguate any alternation in the grammar. This gets you most of the expressivity of PEG and nearly optimal performance (since you only need to look at one character to disambiguate and there is no backtracking). Just as importantly, it avoids the implicit ambiguities that PEG's resolution algorithm can hide from the grammar author that lead to unexpected parse results that can be quite difficult to debug and/or fix in the grammar.
It does require a bit more thought to design an unambiguous language but I think it's worth it. While there is a learning curve for designing such languages, it becomes natural with practice and it becomes hard to go back to ambiguous languages.
For those further interested in PEG vs LL(1) parsers. The first few sections of the Python PEP[1] where they switched from an LL(1) to PEG parser for CPython has a nice short introduction to both and their rationale for switching from LL(1) to PEG.
It still seems to me the PEG revolution hasn't arrived.
PEG has the possibility for composable grammars (why not smack some SQL code in the middle of Python?) but it needs a few more affordances, particularly an easy way to handle operator precedence.
I think current parser generators suck and that more programmers would be using them if anybody cared about making compiler technology easier to use but the problems are: (1) people who understand compiler technology can get things done with the awful tools we have now and (2) mostly those folks think it is performance über alles.
With the right tools the "Lisp is better because it is homoiconic" would finally die. With properly architected compilers adding
unless(X) { .. } -> if(!X) { ... }
to Java would just one grammar production, one transformation operator and maybe a new class in the AST (which might be codegenned), that and something to tell the compiler where to find these things. Less code than the POM file.
I gave up on Restructured text because it didn't support unparsing: I could picture all kinds of scenarios where I'd want to turn something else into RST or take RST and mix it up against other data and turn it back to RST; RST had the potential to work with or without a schema but it never got realized.
- Lisp is better because it manipulates the same data that the program code is represented in (car works on a data list, and it works on a code list as well).
- Lisp is better (at least, Common Lisp) because of image-and-REPL-driven development. Good luck finding exactly that level of flexibility in other REPL-ful languages.
- Lisp is better because of hot code reloading and restarts. Only Elixir/Erlang have a similar mechanism.
- Lisp is better because of structural editing (e.g., paredit). No more character-level editing.
I could go on but just wanted to point out that homoiconicity isn't the entire deal with Lisp.
>- Lisp is better because it manipulates the same data that the program code is represented in (car works on a data list, and it works on a code list as well).
Yeah, but I wanted to emphasize that homoiconicity isn't just some superficial "nice thing" to have, it literally is why we can have powerful macros in Lisp.
I went through quite a few stages of grief reading Graham's On Lisp starting with "this is so awesome" to nitpicking details like "he defined everything else but he didn't define nconc" to "if we was using Clojure he wouldn't be having these problems with nconc" to "funny I can write 80%+ of his examples in Python because most of the magic is in first-class functions and macros are a performance optimization except for that last bit about continuations... and Python has async anyway!"
Notably he doesn't do any interesting tree transformations on the code because tree structures in list are just a collection of twisty nameless tuples that all look alike. If you were trying to do nontrivial transformations on code you'd be better off with an AST in a language like Java or Typescript. In the end the dragon book is On Lisp squared or cubed, that is, games people play with macros are a pale shadow of what you can do if you actually understand how compilers work.
Yes. The best example of something that is popular and homoiconic is Bash. Bash keeps your functions in source code form, though comments may be lost, and they get reformatted. You can see the code when you type "set all", and can copy and paste the definitions back into Bash.
In Common Lisp, the homoiconic feature is the ed function which allows you to edit the source code of a function. Support for ed is implementation-defined!
It may be absent (e.g. in a Common Lisp that compiles everything to machine code). A Common Lisp that compiles all forms and doesn't support the ed function isn't homoiconic.
This is undeniably true. The consequences of a technological collapse at this scale would be far greater than having never had it in the first place. For this reason, the people in power (in both industry and government) have more destructive potential than at any time in human history by far. And they do not act like they have little to no awareness of the enormous responsibility they shoulder.
The hardest part of writing a compiler is designing a language. In my opinion, most compiler courses seem too fixated on the implementation details. The fact that this course targets x86 already misses the mark for me. Computers are very fast, there is no reason to start with a language that translates to asm. This is incidental complexity that distracts from language design.
Hard disagree. There are a lot of implementation "details" that, if you want to do properly, are a lot of hard work and very much nontrivial. For example, do try to write a compiler with efficient incremental compilation, and especially one that does so while also having optimization passes. And that's just one example, most things in compiler implementations actually turn out to be fairly complex. And lots of features that modern languages support e.g. more powerful typesystems, trait/typeclass systems, etc. are also very very tricky.
While designing a language is by no means trivial, it generally really occupies just a very small fraction of the language/compiler developer's time. And, in most cases, the two things (language design + implementation details) have to walk hand-in-hand, since small changes to the language design can vastly improve the implementation end.
> The hardest part of writing a compiler is designing a language.
That's a completely separate field. If you want to learn about language design you should study that, though it helps if you have some background on both to get started with either one.
I disagree. First of all, you can write a compiler for any existing language. But second, inhave experience writing compilers and interpreters. For me the hardest was always to give good information/feedback on errors. Because the compiler cannot go ahead, means it is “confused” eliminates that confusion to help the user is very much non trivial.
Regarding the language design part, many universities I know of offer a {Advanced} Programming Languages class later on after a Compilers 101 class and that is where many of the topics like EBNF, Grammar Design along with different constructs are done.
Besides, for someone learning it for the first time I think designing a new language seems a bit difficult.
The trickster is indeed an ancient archetype that can bring both wisdom and chaos. Historically, however, my understanding is that prior to Plato, essentially all knowledge, including philosophy, was understood to be received from divine sources. It was through the Socratic dialogues that the idea of knowledge as being something gained through human reason gained a foothold.
One could easily argue then that Plato was essentially a prankster and what we know as western civilization is a consequence of his trickery.
In one particular European tradition, maybe? But elsewhere the trickster may themselves be a divine source of insight. Hermes in Greek, the Southwest American Kokopelli, etc.
My point is that the trickster as philosophical root is an idea that has tendrils far beyond a Western viewpoint. I cant find the ref now but IIRC some Native American traditions have the viewpoint that connecting to the divine cannot be made without first laughing, as that opens the mind to the new experience. Reminds me of some Far Eastern traditions where you need a sharp break from your normal world view to achieve an enlightening breakthrough.
Many spiritual and philosophical traditions claim the immortality of the soul. Socrates argues forcefully for it in Phaedo.
Nearly all mystics (and many if not most neuroscientists) also come to the conclusion that our world of the senses is an illusion. This doesn't mean that the illusion doesn't have rigid laws, but it does challenge the materialistic assumption that the soul, or consciousness, becomes nothing at the time of physical/biological death.
If that is too fuzzy and mystical, I'd also suggest reflecting more deeply on the concept of technologically facilitated immortality of physical life on earth. For me, it is clearly a dead end. It can only lead to a complete annihilation of every human value.
> For me, it is clearly a dead end. It can only lead to a complete annihilation of every human value.
could you please elaborate on this? why is it clearly a dead end and why would human values clearly end? any resources you can point to would be great. thank you.
I also am particularly amused by the worship of Bayesian statistics without serious reflection on the fact that it is premised on subjective belief in the prior.
reply