Lets play a game. If you go through the correct door you win a million dollaroos.

Alice steps up to play the game and is presented with 3 doors. What should Alice do? There are no discernible differences. Her only choice is to pick a door and hope it is right. Standing there pondering is a waste of time and effort causing undo stress. There is nothing she can do to change her odds of going down a specific path, so she should walk in head held high.

Bob steps up to play the game and is also presented with 3 doors. However, this time Bob has a special Luck O Meter. What should Bob do? Well obviously, Bob should use his Luck O Meter to determine which is the correct door. How good is good enough though? The Luck O Meter does not perfectly answer which door has the million dollaroos, because it extrapolates Bob’s luck for all time if he goes down a specific door. But, Bob can certainly use the Luck O Meter to compare the different doors and choose the one which will give him the most future luck which is hopefully correlated with the million dollaroos.

Does this analogy sound familiar? When you are faced with a decision, how long do you hesitate and ponder the scary unknown? If there is no way to quantify the unknown, then there is minimal use spending time trying to quantify it. Conversely, if you have a way to measure the unknown then you should spend time quantifying it. How much? That’s up to you, and in my experience - it tends to be a learned skill.

Depending on the risk/reward ratio, I tend to follow reasonable statistics through the empirical rule (three sigmas). This states that in a normal distribution, one standard deviation is 68%, two is 95%, and three is 99.7%. If you have worked with me for awhile, you have probably heard that I like 70% complete design docs, and sometimes 90+% design docs. You will also notice my work tends to follow in one of these paths. A design with minimal risk if we get it wrong or it is delayed - 70% is perfect. You did enough to be confident and work through a lot (a standard deviation) of problems. A code implementation which is going to impact the production system, did you test that to 99+% confidence? Does it do what you think it will?

This entire setup is just a framework to help you think about biasing for action. How much is enough? It is certainly a debate. Do we bias for action or dive deep and insist on the highest standards? You will hear a lot of people with different opinions (and rightly so). My answer - just be right, a lot. A lot of people get tripped up on one way and two way door decisions, commonly muddling the waters and going into endless debates. Or thinking something is a 2-way door because “it is reversible with enough effort”. There is nothing in this world that is irreversible, with enough effort.

My general rule of thumb, combining these topics follows:

• If the cost to undo an action is an order of magnitude less than doing an action - do it with no confidence. For example, adding an automated scanner to a code review. Easy enough to undo. (dont think - just do)
• If the cost to undo an action is within an order of magnitude of doing the action - do it with 0 < confidence < 68% . For example, rewriting all tests to use a new framework. (less than one standard deviation)
• If the cost to undo an action is an order of magnitude larger than doing the action - do it with confidence > 68%. For example, a major architectural change. (one standard deviation)
• If the cost to undo an action is an order of magnitude larger than doing the action and there is direct customer impact - do it with high confidence > 95% (two standard deviations)
• I seldomly find myself in a scenario where I need more confidence. But, maybe if you are in charge of peoples lives rather than showing them ads you would want to consider a third standard deviation?

https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

I thought about making one of the doors deadly, but that would be unnecessarily rude to Alice and not further the point. Go watch squid games if you are into that sort of thing.

I choose a million dollaroos because currencies change in value over time. So, this example should hold true for all time because a fictional currency has a fictional exchange rate - you will be rich.

Lets hope with all hope that Bobs Luck O Meter actually points him at the door with the dollaroos for the sake of this example. Although, it would be like winning the lottery which is statistically not a great outcome. Either way, Bob should have one door standout (the two without the dollaroos should be equal) - regardless the choice is in Bobs hands.

If you are a statistician, I am sorry. You are probably thinking - that’s not how the 68% works! Its 34% of either side to the mean, and if the mean is 50% then its actually 85%. Yada, yada, yada. You are not wrong. But, the 70%, 95%, 99% rule holds pretty true in software engineering - and its bizarre how closely it approximates this rule/law.

Well, maybe pandoras box is irreversible. But come on, I am trying to prove a point here.