Image by Mira Cosic from Pixabay
Image by Mira Cosic from Pixabay

As a Texan, I cast my ballot weeks before Election Day because I live in a state that allows in-person early voting. I consider myself lucky to have the privilege of early voting because there are several states that make you wait until Election Day to vote in person.

Election Day has come and gone, and, at the time of this writing, there is still no clear winner of the presidential contest. However, despite an enormous amount of funding, no poll was able to foresee this outcome. The polling was overwhelmingly wrong. Furthermore, money was piled into races to help candidates who seemed to have support based on polling, but, in the end, those candidates lost badly.

Having worked in software development for almost a quarter century, I see a similar level of error when it comes to estimating software projects. These estimates are also overwhelmingly wrong, and the information used to create them often drives what projects get funded.

Polling and software project estimates fail for the same reasons.

Reason #1: The Craving for Certainty

The human experience is bound by time. That means that we experience time in a linear fashion, and we can’t peek into the future to make decisions right now. There are no “respawn” points that we can go back to if things go terribly wrong. So, since we can’t undo the consequences of our decisions, we crave certainty in the outcomes.

For all of human existence, many groups have exploited this craving. Priests, prophets, and pollsters have sold a dream product: an exact forecast of what will happen in the future. The rewards were often great. In exchange for reading the entrails of a sacrificed animal, interpreting the dreams of king, creating a work breakdown structure, putting a needle on the internet, or performing some other obscure ritual, treasures and kingdoms were often given.

However, these predictions and interpretations almost always fail to align with the reality of the future.

Reason #2: A Multitude of Variables

Life is complex, and no forecast is able to capture all of the variables. There are the variables we can see like people, financial resources, calendars, etc. But, there are variables that we can’t see or properly interpret.

For example, no poll of the 2016 election could have foreseen James Comey’s last minute announcement of the re-opening of an investigation into Hillary Clinton literally days before people would go to the polls. Furthermore, national elections play out across vast land areas, populations, and personal experiences. It’s impossible to account for everything.

Reason #3: Self-Deception

As humans, we lie to ourselves. Some of this is protective optimism since constantly reminding ourselves of the dangers of the world probably isn’t mentally healthy. We also try to present the best version of ourselves. So, when a pollster contacts us to get data, we often say what we think we should say instead of sharing our true opinions.

Software estimators also give in to self-deception and protective optimism. We want to believe that our team can tackle that complicated set of features and quickly get it implemented. We don’t want to admit that building software is complicated and even seemingly trivial features can take longer than expected to finish. So, we see the actual time it takes to finish a project run several times longer than the estimate predicted.

Clarity Over Certainty

As someone who regularly has to answer the question, “How long will it take to build this?”, I’ve learned to respond by seeking clarity over offering certainty. That means that I ask clarifying questions about what we’re trying to do. I also invest a lot of time into making sure that everyone involved in the development effort has a strong shared understanding of the project’s scope. Whenever possible, I try to reduce scope to the smallest set of features and fixes that will achieve the desired outcomes of the project. By doing this, I often find that the team makes steady progress with fewer issues cropping up that slow our progress.

I think optimizing for clarity over certainty would also work well in polls. The focus should be on making sure that people understand the issues and where each candidate stands on them. However, most people don’t need to comprehend the entirety of the candidate’s platform. They just need to know the issues that matter to them. If we can provide education instead of prognostication, we may find the voters cast more informed ballots. We can then let go of the myth that we can predict what they do.