But based on my holistic experience using Emacs—more than a binary "it works"—I'm convinced that the conceptual design carries most of the weight.
Would namespaces, lexical scoping, etc be better? Almost definitely. But it would be a marginal improvement, a difference in degree not kind. Other "best practices" like more aggressive information hiding would go against Emacs's design and make it worse as a whole.
Most "best practices" are neither necessary nor sufficient and, in most cases, not even particularly effective, even when they offer strict improvements over the status quo.
> Would namespaces, lexical scoping, etc be better? Almost definitely. But it would be a marginal improvement, a difference in degree not kind.
That's the thing - it probably would not be.
IMO, the key things that allow Emacs to succeed and last, while being somewhat unique in how it "routinely flouts every single software engineering "best practice"", are not merely conceptual, but cultural or even philosophical. The most fundamental of those, IMO, is that Emacs is ultimately owned by users, who are treated as intelligent partners in the relationship. Everything is designed with extensibility in mind. There's no hard boundary between "core developers", "third-party extension developers" and "end-users" - everyone can be any of them, including all three, at any time.
Maximizing extensibility and interoperability, not demanding "users" to coordinate with "developers" and ask for permission, not chasing user growth (it's all Free Software), expecting that everyone is a responsible adult, encouraging users to make Emacs "their own" - all that leads to an approach that goes the opposite to standard software development practice, because the standard practice is built around entirely opposite assumptions.
I don't see what any of that has to do with namespaces. You can have all of that and not clobber your functions together. It's just a way to delineate groups of functions, not a organizational mandate.
Same with lexical scoping? Not really following you.
Lexical scoping is now available as a toggle and recommended for ELisp code, but Emacs was a mature project and ecosystem when this was introduced, so people know when to use it or not use it. Most languages and environments that start with lexical scoping don't even provide dynamic scoping as an option, failing to recognize when it's useful. And to be fair, it's primarily useful in the thing that Emacs does way more than most software - that is, the ability for some code to override behavior of other code up the call stack, regardless of how that other code got to be called, and without involving intermediate callees in this.
(Or, in short, what you're doing each time you're setting environment variables in a shell call. Environment variables are dynamic scoping, just one level up.)
Namespaces are mostly fine if they're only helping against name clashes, and not trying to do access control. Even then, given the Lisp nature and high extensibility of Emacs (two sides of the same coin, actually), normal namespaces you see in other programming languages are a bad fit, as they don't play well with symbols and various patterns around using symbols as designators resolved at runtime. You'd need a namespacing solution like packages in Common Lisp, which is quite different than what people are used to.
In general, none of that would be a show-stopper - it's more that Emacs always leaned on being easy to extend, including extend interactively (via REPL, or more commonly, just selecting and eval-ing blocks of code), and things like namespaces tend to make it more annoying.
Dynamic scoping is a feature in the Emacs ecosystem. It plays a similar role to dependency injection in other languages. It is what allows you to change global state for some function call. Dynamic scoping together with buffer-local, file-local, directory-local variables is similar to scopes in dependency injection libraries --- a powerful organizational tool.
But based on my holistic experience using Emacs—more than a binary "it works"—I'm convinced that the conceptual design carries most of the weight.
Would namespaces, lexical scoping, etc be better? Almost definitely. But it would be a marginal improvement, a difference in degree not kind. Other "best practices" like more aggressive information hiding would go against Emacs's design and make it worse as a whole.
Most "best practices" are neither necessary nor sufficient and, in most cases, not even particularly effective, even when they offer strict improvements over the status quo.