The importance of improving software systems incrementally.
I was six months into working at Amazon when my team was gearing up for our big MLP (minimum lovable product instead of MVP) launch of Fulfillment by Amazon (FBA). I was asked to implement the collection of fees that Amazon would charge third-party sellers for using its fulfillment services (namely picking, packing, and shipping inventory on behalf of the seller). I gladly accepted.
One week in, I started to regret my decision. There was a monolithic C++ application that had almost zero documentation. It was unbuildable on my desktop (it maxed out the memory available), and frankly, it was badly in need of some love. The service also had zero unit tests and only a dozen, mostly broken, integration tests. However, there was one reason that kept me on that assignment. I had a fantastic host team that welcomed me with open arms; they were just awesome people. The host team listened, gave me invaluable feedback on my ideas as the away engineer (an Amazon model that I’ll explain in another post), and helped me develop a pragmatic design.
My assignment was to implement new functionality for a non-existent business (launching FBA) in a way that limited impact to any of Amazon's existing businesses.
As engineers, we detest writing and maintaining duplicate functionality/code. All of the design patterns that I learned over the years were about extending capabilities; however, when I presented my design I was almost laughed out of the room. I was not laughing, but I quickly learned that my current design was not how I should proceed.
My design was to refactor all of the spaghetti logic code and enable individual lines of business to decorate/augment their fee structure based on their business requirements. No more switch statements and far less duplicated code. A very senior engineer on the team (whom is still my friend, thankfully ;) ) looked at me after reading the proposal and asked an amazing question, “How do you know you're not going to f*** things up?” I paused and told him that we could run both versions of service (old and refactored) in parallel and we would compare the results. He then said, “Do you think this size and scale of change for a business that we don't even know will succeed is worth investing in building all of this complexity?”
That was my oh s*** moment. He was right. I went back to the whiteboard and hid. Instead of thinking of myself as an architect, I now looked at myself as a surgeon.
What pieces could I replace or augment and effectively reduce the blast radius of change? In the end, I wrote only 1500 lines of code, and over half of those lines were implementing a regression suite that validated that fees were accurate.
We often hire chief architects and engineers for their big data design skills, but should you be hiring surgeons instead? For many of us, our experience tells us to think of ourselves as architects, but are you a pragmatic engineer?Pragmatism is one of the most important skills I teach engineers, and I share this knowledge with senior engineers routinely. Good engineers often add lots of complexity to solve a problem. On the other hand, outstanding engineers simplify things wherever possible.
Reflecting, I’m not sure I was an outstanding engineer, but I was better than good. I reduced the complexity we needed at that time to deliver immediate results. Surgeons do the same thing; they perform an explicit procedure, improving a patient’s health for a specific reason. It would be terrifying (and illegal) having a surgeon swap out your liver when all you needed was your gallbladder removed.
Over the last 20 years, one of the most critical advancements in software has been our ability to incrementally update things like mobile apps and the software in your car. We can incrementally improve and extend software in a matter of weeks. The power of incremental delivery enables us to go faster to market and learn from our customers. So the next time you're faced with the option of building something long term (fixing the whole body), think about how you can break things down and deliver incrementally (fixing one body part) like a software surgeon--only operating on the parts that require immediate attention.