Several years ago I read a book about how to make a living as a technology consultant. The main premise was that as a consultant for a business, the most you could ever hope for is to bring about 10% change to that particular business in a given year. At the time I thought this number was so low - I wanted to change tons. I wanted to fix everything that I thought was broken. The book made the case that most businesses really can’t handle more than a 10% change for a variety of reasons, one of which is that upper management does not want to be told that everything they do is wrong. Large changes in the ranks of employees causes instability and change is often equated to risk, and many businesses have a low appetite for risk.
I can’t help but think of this book every time I hear about a new small or medium business that seeks to hire a data scientist to usher them into the world of Big Data. I think the hype around Big Data is so strong that the value proposition isn’t carefully considered. Will a data scientist that makes $200k bring about a game change that makes his company $1 million? To accomplish that, a person would have to be a PhD in statistics, a software developer, a master storyteller, politically savvy, and work for a company that has a risk appetite to even consider whatever data backed initiatives are proposed. The data scientist is being hired to do the impossible: usher greater than 10% change in a business that very likely is not prepared to accept it.
The model of hiring a super smart data scientist or a team of data scientists to change your business is fundamentally broken especially for small and medium sized companies. We need to get to the point where data science is a part of the culture and DNA of a business and inspires many small and palatable changes every day as opposed to huge, risky transformational business shifts. To do this we will need better tools, better education, and more resources. We need to think small.