What the Rising Tide of Machine Learning Means at the Corporate Cubicle

As a bit of a machine learning/intelligence geek, over the weekend I read with interest the news on the MIT Data Science Machine, a set of algorithms that look for interesting patterns in data.  And it brought to mind what I'd observed on the cubicle floor recently (more of that later).

What the Data Science Machine was proving out was how well pattern recognition in data can be cognitively automated, versus typical human cognition augmented with analysis tools. For example, within raw sales data, an interesting simple pattern may be that different types of products sell better at different times of year, depending on location, and perhaps those sales are influenced by drivers not present in the data, such as external factors like weather. The search for trends or patterns that are predictive is the stuff that careers are made of. The question was, who's better? Humans + analysis software, or just software.

"The machine" acquitted itself well, beating 615 out of 906 human teams, and achieving results 94 percent and 96 percent as accurate as the best human counterparts. More interestingly,  "where the teams of humans typically labored over their prediction algorithms for months, the Data Science Machine took somewhere between two and 12 hours to produce each of its entries." I think most of us would take 96% as accurate, for the advantage of that step function increase in speed - and without the cost of labor.

And so, as the proud owner of about 3lbs of wetware that was comfortably attuned to only manufacturing or low level numerical jobs being displaced by automation, it signals that the rising tide of machine intelligence encroaching on the white collar is perhaps nearer than we think. In fact, I'll share a real life experience on what feels like later on.

One Man's Meat is Another Man's Poison

Nick Bostrom, in his book Superintelligence: Paths, Dangers and Strategies, made the prediction of the economic impact of machine intelligence, observing that “what is seen as a boon from one person’s perspective can be seen as a liability from another’s” and portends a profound effect on the economy. The excited advocates of a new technology often have a hard time thinking about what it means for the people on the receiving end of it. And, people’s reactions to change can get really interesting, really quickly.

We’ve seen this script before: at the dawn of the 19th century. The economic shock of the introduction of the power loom and spinning frames slashed the salaries and work prospects of the hand weavers and artisans that preceded them. They (understandably) took their anger out on the machines that were disrupting their livelihoods, signing their acts of sabotage with “Ned Ludd did it.” - spawning the (inevitably short lived) Luddite movement.

The Rational Luddite

Prior to the power loom, a handloom weaver could produce 20 yards of cloth in an hour. After the initial introduction, a power loom could produce 40 yards in the same amount of time. The result was immediate: those that stuck to the old methods of productivity effectively had their pay slashed in half.

Contrary to popular thought, the Luddites behaved not irrationally, but rationally in the face of an unfavorable economic shift with a collective (and rather destructive):


And almost precisely 200 years later, we’re beginning to see the same human response in the face of machine learning – and soon, more disruptively, deep learning. First shock and disbelief, perhaps a little embarrassment, and sometimes even anger. 

The acceleration in machine learning and intelligence promises greater productivity improvements than perhaps those even experienced at the beginning of the industrial era. Algorithms can increasingly create more accurate inferences,  predictions, and recommendations, often based on vastly more datasets that our software assisted wetware can sift through. Comparable or better results, with less labor, and a step function increase in speed.

The Rising Tide on the Cubicle Floor

So what does it mean for traditional corporate roles for example – perhaps in the Halls of Analysis, and how will people react in the face of it? Let’s get back to that real-life example which I observed recently.

Typically, medium to large organizations have an investment in teams building forecasts. Accuracy is prized and increased accuracy farther out into the future, is prized even more.

The tools du jour are mostly spreadsheets, or statistical tools like SAS and R, or data mining tools to build predictive models depending on the organizations level of sophistication. In this one example, the company had a team using in house proprietary models and analysis tools to build their forecast. Management was naturally looking to increase their forecasting accuracy, so wanted to try out advanced machine-learning techniques. They wanted to see if an automated predictive model assisted using big data could could unlock some competitive advantage.

But it turned out the new machine-learning based model really sucked. In fact, it was worse than the in-house models that the organization was already using.

Case closed. These guys already had a solid lock on their forecast. Nothing to see here.

Wrong. The in-house forecasting team had deliberately provided the erroneous data to throw the machine-learning algorithm off. After correcting the issue, it turns out the new approach was logging a substantial improvement in accuracy, and doing it near instantaneously. 

Need less to say, as this was going on, the large in-house team was frantically looking for other roles in the organization, seeing the writing on the wall of their models (and corporate role) being replaced by a more accurate technology that required fewer people and took less time – all while frantically trying to conceal the embarrassment of how much their old models were off.

My prediction: We’re going to be seeing a lot more of that.

It’s worth turning back to the implications from the Luddite Movement. Some people either struck out in anger or blocked change (understandable, but not a successful or sustainable strategy), others readjusted to the rising tide -- working with the machines rather than competing with them. However, the amount of change so quickly had profound societal impacts, and as the pendulum swung further into the employer having leverage, it sowed the seeds for the labor movement.

Only those that swam to higher ground were able to economically survive.