In 2012 Knight Capital lost $462 million as a result of one badly written application and a lack of deployment procedures.
It was a classic example of a silent killer - technical debt.
From losing millions and destroying your team’s reputation to nearly having your operation shut down as a result of your only server crashing, there are enough technical debt horror stories to keep any engineering team up at night.
Yet the term “technical debt” is still new enough that many don’t know what it actually refers to or why it matters.
Let’s put it simply; if your support queue is out of control, features take too long to develop, bugs are getting ignored, or your engineers are burning out and leaving, you’ve got a technical debt problem.
That’s why we here at Aimably wrote this post - to help you understand, combat, and prevent future technical debt. We’ve split it up into the following handy sections:
- What is technical debt?
- Why you should care about technical debt
- The 13 kinds of technical debt
- 5 signs that you have technical debt
- How to solve existing technical debt
- How to prevent technical debt from forming
Let’s get started.
What is technical debt?
Back in 1992, Ward Cunningham (the inventor of the wiki and co-author of the Agile Manifesto) first used a “debt metaphor” to explain why resources were needed for refactoring to non-technical people.
‘Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. Objects make the cost of this transaction tolerable.’
- Ward Cunningham, The WyCash Portfolio Management System (1992)
Cunningham has since gone on to clarify that shipping code before you understand the full scope of the issue you’re trying to solve is not an excuse for poorly written code.
This “debt” was a result of pushing code with imperfect knowledge and experience, with one outcome being to learn more from the problems you encounter and then to go back and write something better. If code is badly written even considering what you knew at the time, it’s not a result of “debt”, it’s just bad code.
Basically, you do the best you can with what you know to get things rolling, then go back and make it better when you learn more.
This concept was fleshed out and refined over the years to become “technical debt”.
Technical debt’s premise is the same as the original debt metaphor but has been clarified in much greater detail and specificity. Instead of focusing on only why the debt has built up, technical debt has also been defined by the intentions of the person writing the code and, more recently, the nature of the debt itself.
‘Technical debt is a term that has been used to describe the increased cost of changing or maintaining a system due to shortcuts taken during its development.’
- Nicolli S. R. Alves, Leilane F. Ribeiro, Vivyane Caires, Thiago S. Mendes, and Rodrigo O. Spínola, Towards an Ontology of Terms on Technical Debt (2014)
So, rather than being limited to legacy code causing issues, the term “technical debt” refers to the costs resulting from issues including outdated libraries, infrastructure problems, key knowledge holders leaving the company, and so on.
Why you should care about technical debt
So now you know what technical debt is, and it sounds scary, but why should you care? If things grow into problems you can just prioritize fixing them right?
See, here’s the problem.
Once you start to feel the effects of technical debt mounting, your problems can rapidly snowball into a worst-case scenario. Much like financial debt, if you haven’t been keeping up on your payments the interest can leave you unable to pay what you owe.
Here’s a picture of what out-of-control technical debt looks like:
- You’ll lose feature-reliant sales
- The product will have defects
- The product will be less reliable
- You’ll need a larger support team
- There are massive security risks
- Feature delivery time will balloon
- Customers will churn
- Frustration with engineering will rise
- You’ll lose your star engineers
First up, one of the hallmarks of escalating technical debt is your product lacking certain requested features. This can cost you sales, especially if it’s something big like having SSO (which enterprise clients tend to require).
Even if your product has all the features that are deal-breakers for your prospects, if you’ve pushed off dealing with too much technical debt the resulting defects and reliability issues can drive off healthy business.
Nobody will want to use your product if it doesn’t do what you promise it will.
These defects and bugs will also put a massive strain on your support team, as clients will be asking for technical support and finding bugs faster than you can deal with them. It’s a hat trick of bad press, bad UX, and an extra resource drain for your company.
Not to mention the security risks caused by outdated code, vulnerable unnecessary dependencies and bugs, and so on. Even if these don’t drive away customers, you’re putting their data (and your own) at risk every time you fail to close those gaps.
These issues combined will result in spaghetti code, making it far more difficult to develop features than it should be. Even if there are no defects, the practice of picking through your workarounds (and everything you’re ignored doing until now) will cause development to slow to a crawl.
The bottom line is that customers will be frustrated and churn, they and your team will get frustrated that engineering isn’t fixing these issues, and this will eventually lead to engineer burnout.
Unchecked technical debt is, in short, a great way to alienate future and existing customers, grow resentment with and within your team, and lose your star engineers.
The 13 kinds of technical debt
In 2014 Nicolli S. R. Alves, Leilane F. Ribeiro, Vivyane Caires, Thiago S. Mendes, and Rodrigo O. Spínola laid out the 13 kinds of technical debt in Towards an Ontology of Terms on Technical Debt. They separated it into:
- Architecture debt
- Build debt
- Code debt
- Defect debt
- Design debt
- Documentation debt
- Infrastructure debt
- People debt
- Process debt
- Requirement debt
- Service debt
- Test automation debt
- Test debt
1. Architecture debt
Architecture debt covers any issues in the project’s architecture as a result of shortcuts in the original design. Due to these problems being inherent to the design of the product, they tend to require heavier development (refactoring) to fix, and so can be very costly.
For example, you may have originally designed the product with a dependency on an external or third-party library. As development continues these dependencies can pile up, presenting an ever-increasing fault risk.
2. Build debt
Build debts are problems created while building the product which cause future work to be more difficult or cause unnecessary time and/or processing power to be consumed.
For example, an ill-defined dependency could cause the product to run slower than necessary.
3. Code debt
Code debt is what might initially spring to mind in regards to Cunningham’s original debt metaphor - an issue in the code which makes it more difficult to maintain. Assuming that no unnecessary mistakes were made in relation to what you knew when writing the code, this debt generally comes under the “legacy code” umbrella.
For example, while you may have written your code as well as you could, your experience level at the time could have caused you to write code that is difficult to read. Anything that isn’t up to your current coding practices classes as code debt.
4. Defect debt
Defect debt refers to the bugs in your code that you’re aware of but have decided not to solve, usually because other elements are higher priority. These are fully documented (via your own experience, user reports and/or testing) but not solved - the issue is not having the resources to tackle everything while continuing development.
The overlapping issues and vulnerabilities caused by these bugs are present in every team, as no product with ongoing development is defect-free. However, it’s important to tackle as many as you can and to make time for the ones which you’ve been putting off.
An example of defect debt would be two product features interacting incorrectly, but only in a specific fringe case that most users won’t experience. Thus, it is put on the back burner while you handle bigger issues.
5. Design debt
Michał Mazur (Design Team Lead & Partner at EL Passion) describes design debt as “... all the good design concepts or solutions that you skipped in order to reach short-term goals. It’s all the corners you cut during or after the design stage…”.
Honestly, we couldn’t say it better ourselves. The key is that these design shortcuts weren’t made through ignorance, but through a conscious decision to iterate faster than those choices would allow.
In the same article as the quote above, Mazur also gives an example of design debt as your main navigation flow not being tested with users before going live, citing budget restrictions while designing it.
6. Documentation debt
Documentation debt occurs when any of your documentation is not detailed enough, incomplete, or straight-up absent. Relevant parties should be able to understand what everything is designed to do so that nobody is flying blind.
Solving this means that you need to record what everything is designed to do and how it achieves that (and what assets it uses in the process).
One of the more common examples of documentation debt is failing to record and communicate small changes to your code. Alone these might not matter, but cumulatively they can result in much larger knock-on effects if not understood and recorded.
7. Infrastructure debt
Infrastructure debt is exactly what it sounds like - any infrastructure issues which hinder development in some way. This can be anything from needing to update your library and apply a security patch to your current infrastructure being unable to deliver your current demands (thus requiring migration, introducing scaling capacity, and so on).
8. People debt
If there’s an issue with your team’s human element that could affect development, it’s classed as people debt.
For example, delaying on hiring could result in your team not having the availability to carry out everything they need to. Alternatively, there could be one person in your team who deals with a specific side of the product while your other engineers haven’t been trained up to handle the same elements, thus preventing further development.
9. Process debt
Process debt is similar to documentation debt, except it’s concerned with inadequate, incomplete or missing processes. This can lead to your team struggling to deal with things like defects due to their process (the way that they do it) being flawed.
Let’s say you start using Intercom to manage customer support but don’t update your support ticket process to make your CSMs record bugs they encounter in JIRA. This means that your team won’t know what’s wrong to begin with, and thus can’t prioritize what to solve effectively.
10. Requirement debt
Requirement debt covers any compromises the development team made in regards to the requirements they needed to implement or how they implemented them. For example, if your system needs to handle 2000 patient accounts but the result can only deal with 1000, there is a requirement debt because the product does not meet the full requirements.
11. Service debt
Service debt is the name for when you need to change your web service and the problems, time, and effort involved in doing so. This should only be done if it’s certain the debt incurred will be worth the payoff when the new service is fully operational.
12. Test automation debt
The need to automate tests of existing elements and functionality in order to speed up development and scale with your product is known as test automation debt.
For example, in order to scale you can’t keep manually carrying out smoke tests for every product build (especially when the focus is on developing quickly). You need to automate them so that you can focus on other issues, be it automating other tests or developing new features.
13. Test debt
Finally, test debt covers anything that you know is affecting the performance and/or effectiveness of your tests. This includes the maintenance your automatic testing system(s) require, along with any inherent faults of those systems and your manual testing practices, such as your automatic testing having low code coverage.
5 signs that you have a technical debt problem
It’s all well and good knowing what kinds of technical debt you could have, but how can you tell when your technical debt has built up enough to start causing problems? Technical debt is bad, sure, but it’s also inevitable, so where is the breaking point that you need to stay below?
In other words, what are the signs that you have too much technical debt?
The first sign is that you have a large customer support backlog, including it taking a long time to resolve cases. This could be a result of defect debt (user-reported issues going unsolved), code debt (users reporting issues stemming from legacy coding), design debt (a key feature’s design not being user-tested before being released), or so on.
The second is that the portion of your engineering team working on delivering new features is much larger than your other product lines or than that of the same feature teams in similar companies. Documentation, code, build, test, and test automation debt could all be contributors to creating a nightmare of spaghetti code that means you need more engineers to deliver the same amount of features.
The third sign is closely related to that, as taking more time and effort to deliver features will result in a large feature backlog. The more that goes undeveloped, the more opportunities you’ll miss with users who require those features before signing up.
Fourth, you’ll have a large bug backlog which isn’t being tackled. The strain on your engineer’s time and resources will leave lower priority bugs at the wayside, which can be majorly damaging for your UX and usability.
Finally, a buildup of technical debt (particularly test and test automation debt) can show through a large proportion of your unsolved bugs being related to customer support requests.
Think about it - if your team isn’t effectively testing your builds automatically before they go out, far more bugs will be slipping into the public version of your product. Combined with the shortcuts made to your original product vision for the sake of brevity, your customer support will be handling exponentially more bug reports and feature requests than otherwise.
How to solve existing technical debt
Either you know you’ve got too much technical debt or you want to prevent it from getting to that critical mass stage. That’s good! The first step to doing something about your debt is recognizing that it poses an issue and knowing how much of it you have.
The next step is solving your current debt so that your team regains the flexibility to future proof against further debt. This can be split up into leadership solutions and technical solutions.
Leadership Solutions
The primary thing your leadership team needs to do is to create a long-term strategy of how to approach shifting your technical debt and what the chain of command will be for any decision-making which is required.
Basically, everyone needs to know what they’re doing at a high level to tackle the debt, and who they need to report to on their progress or who to ask questions to.
In order to get to this strategy, you’ll need to explain to the executives what the situation is and why it needs dealing with. Don’t take no for an answer, and don’t sugar-coat the outcomes if your debt keeps getting ignored.
You need your execs to buy into the situation, its importance, and the approach you need to take to solve the debt problem. This means you’ll need to be communicating openly and regularly, potentially including board-level reporting on technical debt initiatives, metrics, timelines, and impact.
Once leadership is united in their approach, you need to make sure that your team is able and comfortable enough to be transparent about the problems they face whenever they occur. If something is more complex than initially thought, leadership needs to know about it so that they can factor extra time into their plans.
Speaking of planning, the only way that accurate battle plans against technical debt can be created is if your team accurately communicated how long issues will take to fix.
This doesn’t mean that your team always has to be 100% correct about how long a fix will take, but you should always take the time to accurately assess the issue and be realistic about how long a solution will take to deploy.
All of this will help to foster an environment which leadership should also take advantage of - one that encourages building a solid culture of an engineering team united against the massive challenge ahead.
You want your engineers to be helping each other get things done faster and to a higher quality. They should be safe in the knowledge that they’ve got each others’ backs, and that the rest of the company understands the massive undertaking they’re making.
They should feel supported in what they’re doing and valued for their efforts.
To this end (and to help organize your debt solution) it can even be worth looking to bring in managers who have experience helping teams tackle their technical debt already. It can be much less daunting to tackle the mountain of work ahead if someone can tell you where best to dig.
Finally, your leadership figures need to introduce (or update) your code development best practices and assess whether further training is necessary. This will give your team guidelines on what they should be doing and provide them with the experience to carry them out.
Speaking of coding best practices, let’s jump into the technical solutions…
Technical solutions
It’s going to be brutal, but sometimes you’ve got to amputate in order to save a life. The development equivalent being that every time you see code that doesn’t meet your current standards, you need to refactor it.
Every. Time.
This also means that you’ll need to test, test, and retest your code to make sure that everything works correctly. By testing before and after you’ve written the new code you ensure that what you write brings the intended results, even if those results are just clearing the clutter from your backend.
Some issues will require dedicated teams to tackle, such as full or partial rewrites. These need planning carefully with everyone who will be involved so that, once again, everyone knows what they need to do, why they’re doing it, and how it will fit with the rest of the team’s work.
You also need to plan and implement “known good patterns” that can be leveraged for future work, even if they aren’t directly tied to feature development. This will let your teams continue to expand their work going forwards without having to reinvent the wheel every single time.
For example, if you don’t already have one you should build a framework for dependency injection, as otherwise your team won’t be able to use it in their future work.
How to prevent technical debt from forming
We’re on the home stretch now! You know what technical debt is, why it matters, what kinds of tech debt there are, how to identify it, and how to solve your current debt.
Now let us quickly cover how to prevent further technical debt from forming. Thankfully, this is a pretty simple two-step process.
- Build policies to protect the company
- Get other execs to sign off on those policies
Start by building better policies to help protect your company from technical debt. This includes establishing guidelines for modern code quality and library support that all teams agree with.
To help ensure that your code quality doesn’t break your new policies in the future, try leveraging third-party code analysis tools such as Raxis or DeepSource.
Once you’ve set out your policies you need to get your executives to sign off on them and on the fact that they need to be rigorously adhered to. After all, policies aren’t worth the paper they’re written on if they aren’t upheld.
Getting your other executives to agree on them will play a crucial role in preventing future technical debt. When you’re asked to cut corners (accrue debt) in order to push out items faster, you can refuse on solid ground because it would break the policies that they’ve already agreed to. Everyone should comply with your policies and not only when it suits them to do so.
By communicating your policies with other teams and staying tapped in to when your work needs to expand to accommodate new projects, you then have the flexibility to lay out what work you can handle, how long it will take, and get direct informed permission to complete it.
Things won’t always be smooth sailing, and at times your defenses against technical debt will be tested, but with solid policies, company-wide buy-in to those policies, and proactive communication, you too can shed the weight of unfinished features and endless bugs.
Go ahead. Breathe easy.
