Tech debt seems to be a pretty popular topic. In my experience, there are two types of startups: those who are successful and have tech debt, and those who are not successful. The good news is that if a company is successful, there is the chance to get rid of the old tech debt, and in many of those cases it is a great cause for celebration among the engineers who have had to deal with it.
Recently, we deprecated a service that has been around here at SendGrid for almost 7 years, causing our engineering teams plenty of headache. I figured it might be educational/encouraging to talk about this to help others understand how tech debt can sneak up on you and how we have decided to celebrate when the bill has finally been paid.
It sounded like a good idea at the time
In the olden days, when we were experimenting with things, one idea we had was to create something similar to Return Path’s Mailbox Monitor. Getting to the inbox is what people pay us for after all, so ideally we would be able to see how well we are doing, right? How hard could it be?
It didn’t take long for us to learn just how hard it was. We discovered that if someone wants that capability, they should talk to Return Path for it. But as part of our design, there was some data generated that our Compliance Team was really interested in. So rather than just shutting the service down completely, we disabled most of the toggles in the database and left the service to run and do its thing.
The daemon that time forgot
And so this service continued on its way, despite the fact that it did nothing that it was intended to do. Whenever it would run into trouble, we would drag the author (one of our most senior engineers) off whatever important work he was doing to go and deal with it. But that would be just to get it working. While most other services underwent a lot of upgrades to meet our evolving operational standards, since this wasn’t customer facing, it was pretty much left out of the party.
Official ownership of the service was passed around to various teams, but again since it wasn’t customer facing, it wasn’t given a whole lot of love. Our Operations Team wrote documentation for how to deal with it, which included titles such as “the daemon that time forgot,” which in 2012 stated “even though it is an important part of our infrastructure, it’s basically abandonware.”
There is no such thing as a 0% interest only infinite term loan
By now many people who have been in a similar situation probably know where this is going. While the service itself was very simple, and not customer facing, the path to get to it was convoluted and was in the same flow as customer traffic.
Last fall, we spun up a team with some of our best engineers (which, despite that high bar, they allowed me to join) to look at bottlenecks/performance impacts to our system. Something that was quickly identified was that there was a major bottleneck getting data to this service, and that was in turn having impact on our delivery speeds for our customers. Even though the service itself was not customer facing, the path to get there was, and when it backed up, it was responsible for millions of messages in our queues, which was consuming resources that should be put towards our customers.
At that point, we dug deeper and realized that all of the functionality that this downstream service was being used for could be done, much easier, farther upstream with just a few lines of code. I seem to recall that the entire process of coding/QA/deploy/verify to completely kill this service took less than a week and even included some extra stuff added in around the functionality for our Sales Team, because we love them too.
How to celebrate paying off tech debt
The final death of the service was pretty anti-climactic. After deploying our changes, we watched the queues slowly drain off to 0, turned off the power to the boxes, and all went dark.
That just didn’t seem like a fitting enough end to a service that had caused us all that pain, self-inflicted as it may be. So we decided to take a page from the time-honored tradition of what to do when a very large loan, such as a house, is paid off. When that happens, tradition is to burn the note.
So we printed out the source code to the now obsolete service, brought it with us to our annual kick off trip in Mexico, and f’ing burned it.
People might feel that is a bit extreme, but if you’ve ever lived with tech debt like that, you probably get it. Tech debt is just a fact of life, but that doesn’t mean you have to like it and can’t celebrate when it’s gone.
The big lesson to be learned from this is that it is oftentimes cheaper to look into how to pay off tech debt than how to restructure it. While some people may read this and wonder about our engineering decisions, I’m sure any successful startup will have similar stories. You cannot be successful against someone who will incur tech debt without doing so yourself, but that doesn’t mean you have to keep it forever. No matter how small it seems it will end up costing you if you don’t just deal with it sooner or later.