Here is the story up front.
We once did a hard thing the "right" way: we built custom high-performance S3 read paths for our compute cluster workers, and they performed great. But that design also normalized credential handling in places where it didn't belong. Engineering started laying tracks around the identity problem, and each track made the system harder to install, less secure, and harder to operate. It also ended up being exactly opposite from what mature cloud customers wanted.
Then we had to blast through the mountain:
I’ve been building distributed systems since 1988, and the same pattern keeps showing up: when terrain gets rough, teams lay tracks around the mountain with workarounds, scripts, glue code, and ceremony. Eventually, nobody remembers what the actual system does. The workaround becomes the architecture, and the mountain is forgotten.
Floe rejects that. When something gets in the way, we don’t route around it. We blast a hole straight through the mountain.
Around the office you can hear me tell people at times: "you are laying tracks". This is a gentle clue to people who are not fixing root cause issues, and instead planning or building a workaround.
Laying tracks:
I have found laying tracks doesn't usually happen because of something the engineer is not seeing. Engineers can see the problems, but sometimes things get in the way that they have less control over. It could be schedule pressure, or personal distractions, and yes, we do see the "lazy developer" syndrome contribute. What I've found is that by naming this effect, however, we can gently remind ourselves to lobby for that bit of extra time, to build with pride, to fix something so your teammate or even a whole organization doesn't have to later.
Requirements change. Systems evolve. But change doesn't have to mean adding to your code base. I find evil joy in doing that git rm -r on a whole bunch of code that you maintained for years. It's gone! Life is dreamy....we found a better way.
Doing the right thing in software sometimes requires just pure, human courage. To find the courage to ask for more time, to be unafraid to tell someone their code needs a redesign (with tact of course), to tell the person demanding that your team shift to a new business that the code that was built and maintained for years won't do it, that it's time to cut bait and nuke the whole fucking thing from orbit. Blast a hole through the mountain: it's violent, and sometimes the only way to safely proceed.
It was simple: when we built our first cloud product, our compute cluster workers needed to read from S3 and compatible object stores.
Vendor SDKs were a bad fit for our workers' memory/CPU/storage profile, so we built custom C++ libraries. The vendor libs were sluggish, not custom allocator-friendly, and certainly not compatible with the concurrency model we required. So we built our own. The results were great: throughput improved, and the read path handled amazing levels of concurrency.
But in the same move, we baked access credentials into the core design instead of teaching the system to run under workload identity. We did one really hard thing right, and accidentally set ourselves up to lay tracks everywhere else.
Once credentials sat in the middle of the system, workarounds multiplied:
Every one of those looked "necessary" locally. In aggregate, they were a maze of tracks. And these tracks, man did they derail cars on their path.
The real mountain was a bad assumption: "our workers must handle credentials directly." We reinforced that assumption with an internal myth that workload identity was too hard, even after proving we could solve tougher engineering problems.
The turning point came when one of the engineers carrying most of this pain called out the myth directly. He challenged our mythology, and we finally listened and tried to come up with an answer. There wasn't one. So we removed credential handling from the core product path and implemented workload identity/IRSA. That single architectural move deleted a huge amount of accidental complexity and retired a large chunk of our track network.
Don't be part of the mythology police. You know these guys, the ones who tell one another how difficult a problem is, without doing a bit of work investigating their own claims. Challenge the base assumptions, don't lay tracks around them.
Be brave enough to challenge leadership to stop laying tracks, or fostering losing culture that does. They could be the ones demanding you solve problems the right way, only to take away the time to fix the problems their demands begat. They may already see your easy way around your obstacle. God help you if your CEO is smarter than you are! The blessing will be if he gets that you have to have time to do things right. Pity those of you that work under one of those accursed "being first always wins" leaders, who start laying your tracks before you've started your project if you aren't careful!
This blog series is about fearless engineering — the craft of tearing systems apart, rebuilding them cleaner, learning in motion, and refusing to let complexity and accumulating technical debt win. It’s not theory. It’s not sanitized.
In the context of Floe, and how we are building a database that both embraces the new and pays homage to the lessons we learned in our past, this series will chronicle the honest record of building a service in real time while still actually running the damn service.
Welcome!