Complex systems are best built iteratively. You start small, plan, test, make things work well, and then continuously improve, clean, and polish over time, with a focus on maintainability, otherwise known as "this thing is easy to change"
Bigger systems are made out of smaller systems that are, themselves, built this way.
The clean-and-polish bits are critical. The issue with most software is that management at most organizations only focuses on "move fast and break things" and fails to reward -- or actively penalizes -- maintainability.
"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."
Sometimes I think that this arises out of factors usually considered external to the system. As in, a complex system is so embedded in its environment that it can't be said to start or stop at all, which implies that starting a complex system is an oxymoron of sorts, and that is the reason that creating one from scratch can't normally be done.
In turn, this implies that a working (or, to my way of thinking, running) complex system can be created from scratch, but only if you deliberately create the simpler system it bootstraps from. This makes "creating a complex system" look more like biological reproduction.
With that in mind you start to see echoes of this constraint all over the technology ecosystem, like new programming languages piggybacking off of another toolchain until they can self-host.
Just a random example, from this very forum, from Kenton Varda, who at one time maintained Protobuf at Google.
"Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything." [0]
> They just did something that worked well enough, and it worked well enough that it was rapidly adopted by the rest of the company. It was then too late to change anything.
Similar story with the decision to utilise tab characters in makefiles:
"Why the tab in column 1? Yacc was new, Lex was brand new. I hadn't tried either, so I figured this would be a good excuse to learn. After getting myself snarled up with my first stab at Lex, I just did something simple with the pattern newline-tab. It worked, it stayed. And then a few weeks later I had a user population of about a dozen, most of them friends, and I didn't want to screw up my embedded base. The rest, sadly, is history."
That’s not how we build cities; modern cities have urban planning departments, traffic engineers, zoning etc.
Of course, completely rational city planning is a failure and the only ones people like were built before urban planners and zoning were invented. So far this hasn’t stopped them.
> That’s not how we build cities; modern cities have urban planning departments, traffic engineers, zoning etc.
Yeah... Modern, already existing, cities. That's not how cities are "built", in the sense of a city coming to exist where a city hadn't existed before.
Sure it is. When the plans are never completed, change faster than the cities do, and there are more exceptions granted to the plan than compliance, it is hard to say cities are "planned".
By contrast, cities are reasonably good at setting, adopting, enforcing, and evolving standards of various sorts (curbcuts, signage, building codes). The story of the past few decades has been efforts to bridge the gap to slightly larger patterns at the level of streets and neighborhoods rather than leaving that largely to a minmaxing process that litters our cities with endless small and medium tragedies of the commons (eg. giant parking lots surrounding isolated fortress-like stores, office blocks surrounded by blank-walled alleys, or suburban cul-de-sacs forcing long circuitous routes that require a car to get anywhere).
> That’s not how we build cities; modern cities have urban planning departments, traffic engineers, zoning etc.
I don’t know. My father spent his career as an Urban Planner in a Planning Department (in Ontario, Canada), and he never had the power to remake cities that you’re alluding to.
Mostly his job was to do studies of proposed zoning changes of land use and the impact, and make a recommendation. The recommendations were presented to the Planning Board at public meetings.
The board meetings were, of course, part of the larger democratic process, and were just as subject to popular whims as any other parts of government. Landowners and developers had a huge amount of influence, and worked hard to get their projects approved.
I think the “Urban Planners did this” is mostly just a deflection (in Canada and the U.S. at least.) I point to the “smart neighborhood” that Google and the Province of Ontario (on behalf of developers) were pushing hard for in Toronto a few years ago [1] as a more visible example.
patio11 has even written about this process, more from the developers side. [2]
> Mostly his job was to do studies of proposed zoning changes of land use and the impact, and make a recommendation. The recommendations were presented to the Planning Board at public meetings.
But that's the bad part - there shouldn't be zoning or planning boards. People who want to have opinions here should keep them to themselves instead of being able to enforce them; they're the reason for the US housing crisis and nearly everything else. Of course, there's a few more reasons like property tax systems (CA's encourages keeping old single family homes) and paranoid fire codes (which cause giant streets in the US so our giant fire engines will go down them).
If you're reading patio11 you'll probably know that Japan does everything right here by having national zoning where cities aren't allowed to stop you from doing anything.
As a result Tokyo is still livable, cheap compared to US cities, and a house in Japan is affordable because it depreciates like a car rather than being your main retirement asset through land hoarding.
I studied English alongside CS and was really into the creative nonfiction genre in my university days. I remember really enjoying "Close to the Machine" by Ullman, might be worth checking out.
“We build our computer systems like we build our cities - over time, without a plan, on top of ruins.” — Ellen Ullman
…and was prompted to search the web for the author. Wasn't at all disappointed when I read this interview, and decided to share.