An Open Letter to Fitbit – A Lesson For All

Any IT professional who’s called in when a leading company makes the headlines for a technological disaster may be tempted to tell you that such unfortunate events can be entirely prevented. Notably, DevOps is about coupling the responsibility to use reliable software process improvement methodology [in this case, to protect client information] with the equally relevant reality that data breaches like the one you’ve just experienced are inherently unavoidable.

Still, as inevitable as vulnerabilities are, there are many measures which can be taken to mitigate risk. In 30 years of experience in the field, I’ve found that the weakest link is most often an abundance of human error. People, though well intentioned, are fallible, and as a result, their code is also.

Companies like yours can protect themselves and their infrastructure by automating building the entire system from a credit card and creating backups of all the data. Furthermore, if you automate system alerting, you’ll know the moment that you’re under attack.

Before releases, I might ask the team responsible if:

1. They have baselined the existing code so that we can reliably restore, if necessary.
2. They have baselined the new release so that we can verify that the correct code has been deployed.
3. The new release has been tested and approved for release production.
4. They have notified the stakeholders that a new release is coming so that they are not surprised.
5. Their support team is alert and ready to handle any calls or issues which might arise when the release is deployed.
6. They have tested the system thoroughly enough to have a baseline to which they can compare (i.e. will they be able to identify which bugs are new and which predated the release), and they have established what this new release provides the user (e.g. defects fixed, new features, etc).
7. They have divided the deployment into the smallest increments that can be delivered (if not, they’re increasing risk with a ‘Big Bang Deploy’).
8. There are any other predictable/identifiable risks to other systems when the release is deployed.
9. They have full backups of the current system.
10. They have rehearsed full system disaster recovery.

If 10 items seems overwhelming, then I would encourage you to remember the words of the 16th President of the USA:

“If I only had one hour to chop down a tree, I would spend the first 45 minutes sharpening my ax.”

Abraham Lincoln

The best way to accelerate your delivery pipeline is to understand what would interfere with it and to prepare for any disruptions ahead of time.

If a breach should occur (and from time to time, it will), I might ask the release team:

1. What they are seeing that leads them to believe there is a problem.
2. Who should be notified that there is a problem.
3. If it would be appropriate to take the system offline now.
4. If we can take a snapshot of the existing system (complete with the problem) for later forensic analysis.
5. If we need a full system restore or if restoring an application will suffice.
6. How long the restore can be expected to take and what the necessary steps are.
7. What risks can be associated with the restore and how we can prepare for them.
8. Who performs each task, who gives the approvals, and who notifies others when relevant.

No one likes to have their data compromised, but your public relations people have a much stronger leg to stand on when your IT department can clearly and definitively not just state, but also provide evidence that they are following the industry standard best practices including the new IEEE 2675-2021 industry standard for DevOps.

There are just a few more questions that I would encourage a release team to consider at the tail-end of a breach like this one:

1. What can we learn from the forensic analysis of the corrupted system baseline?
2. What went well? What could be improved?
3. Going forward, what can we do to ensure that we can immediately detect problems and restore application baselines (and full system baselines, when necessary)?

With an appreciation for software security best practices and an eye on improvement, Fitbit, a product already beloved by many, can emerge stronger, better, and more trusted than ever before.

Written by Bob Aiello