This is really inspiring. Reminds me of the quote "A complex system that works is invariably found to have evolved from a simple system that worked."
The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.
On the contrary I don't think you can plan architectural perfection from the beginning. IMO you have to throw it in a prod-like environment early (preferably as an internal app with a minimal SLO) so that you can iterate early and often.
That's what I talked about in the last sentence of my post. It is important to do this, but it is not the whole story. How do you decide in which direction to iterate?
Make it feature-driven. When Facebook came out they didn't know photo sharing would be a definitive feature. When they put it out there they took a slug to the shoulder for not realizing it sooner and then implemented relevant features (like tagging).
Similarly when Redis came out I doubt the creator realized everybody and their grandmother would use lists as a defacto message queue but now look at all the commands list has that facilitate just that.
Key point is to accept and welcome that slug to the shoulder that pushes you in the right direction, to abuse the metaphor.
Similar to my sibling comment I think the value in sw architects isn't the ability to predict the future from experience, rather they should build systems that can accept reasonable change.
Some practical examples would be to implement build systems into their stack that accept database migrations and rollbacks easily. Or realizing they are in a fast-paced team that could benefit from specific frameworks and languages (experience would play a big role here).
All you have to do is listen to your users and do what's necessary if you have scars to remind you that sticking IO here will bring the system to its knees on a real user load, or that this customer is going to ask for LDAP integration next week so you better build your authentication accordingly.
I agree that experience helps, but experience alone is insufficient. Experience should be properly reflected upon, or you can stick to suboptimal practices justifying them by saying "that's how I always do it, worked for me in the past".
Experience can even be detrimental - the well-known second system syndrome comes to mind!
If you are able to rewrite the code base in a short amount of time, the infrastructure problems are easier to handle. That way you can move forward faster and fix problems early on.
Yes, when the code base is small, evolving it is much simpler. One of the challenges is deciding on the exact right moment to introduce big changes - too early and the changes are in the wrong direction or even useless, too late and the amount of work is prohibiting and you are forced to live with suboptimal architecture.
> Yes, when the code base is small, evolving it is much simpler.
There's obviously some truth to that. But I also think it leaves or an important angle: Often larger, more complicated, features are easier to add if you have a lot of infrastructure pieces.
Say, in a database, if you have no durability (journaling), resource management (errors, locks), error checking (deadlocks, days checksumming), it'll be a lot more work to add support for some new query language [feature]. Now databases are my thing, but the same it's true for a lot of areas.
Disregarding that is IMO one of the reasons ground up rewrites fail so often: You'll get bogged down with infrastructure for a long time, without proper non toy users of the facility.
Yeah, I agree. I think the real challange is to keep the code base in such a good shape that a rewrite of the whole thing or parts of is always doable. Someone talked about never writing modules that can't be rewritten in two weeks.
As someone who worked for a company for a few years on a vast Tcl codebase, a company in which almost every engineer lamented the fact that they were working in an unused and unknown language, it warms my heart to see that it was used for this.
I dabbled with it myself, writing a simple TCL interpreter in C, but I always find myself comparing it to FORTH a language I still find it easy to love.
antirez mentions that this forks to save, but I'm not seeing that. The backgroundsave procedure makes a copy of db to dbcopy, but then never saves dbcopy. I wonder how much further the TCL implementation was extended before antirez switched to C.
BTW, I hadn't heard of LLOOGG (the L in LMDB) before. It was this:
"LLOGG was web service I (Salvatore Sanfilippo) and my co-founder Fabio Pitrola ran for seven years for free."
You are right, the intention was that but now I'm reminding that, in the middle of doing it, I thought, ok I reached the limits of the Tcl prototype, let's rewrite in C.
The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.