The Last Mile of Big Data: Big People

There's this fantasy we all have at work that someday we're going to have instant access to perfect information. Now more than ever, it seems within reach, since we know that there are huge amounts of data being produced every second of every day. If you believe the hype, the next generation of Big Data tools will crunch billions and billions of cells of unstructured data and tell you exactly how to run your business.

Sadly, that fantasy is far from reality. Yes, we have more data and better tools today than we did 10 years ago, which helps us make better decisions. But Big Data only takes us so far. Just like the last mile – the final leg of getting connectivity or goods to the end consumer - is the most complex challenge in telecommunications and logistics, getting from a warehouse (or lake) of data to actionable insights is the most complex challenge in the corporate world.

Delivering on the fantasy

To deliver on the Big Data fantasy, we need Big People. To make people big, we need another kind of tool: software that enables people to collaborate across time and space on a massive scale.

Why? Because there's a huge distance between data and insights that only humans can bridge. Big Data is just the base layer. It only enables better decisions when the people who know what to look for have the ability to explore it together and make the connections that lead to insights. Big People is about capturing people's knowledge and ideas, memorializing collaborative problem-solving processes, insights and decisions, and making it all accessible to everyone forever.

It’s a new twist on a problem as old as humanity itself. Humans generally want to collaborate and seek wisdom from others. Yet we often find it difficult to do, since we're genetically programmed to protect our egos and to distrust people outside our own tribe.

Talking to Neal

Consider this situation: You are struggling with a marketing analytics problem. You could wade through a data lake, or you could walk down the hall and talk to your colleague Neal, because he’s been with the company forever and knows not only the answers to a lot of your questions, but the history and context behind the current situation. And, if he doesn’t know, there’s a good chance he can tell you where to look or who to talk to next.

But you might hesitate to talk to Neal, for any number of reasons. Maybe you don't know him well, or you don't trust him. Maybe you’re afraid that your question is stupid, or you don't even know what questions to ask. Or maybe Neal works halfway around the world and is hard to reach.

Those are just the obstacles to collaboration on a person-to-person level among colleagues. Enabling collaboration with Neal on a massive scale is an even more difficult problem to solve. Inside companies, especially large ones, marketing, supply chain, finance, IT, R&D, and sales are in such different spheres it can seem like they're from different tribes. They all hold valuable knowledge about the company, they're all working on the same problem from different angles, and they might as well be strangers when it comes to getting them to collaborate.

This problem has existed for decades, but it's becoming more critical to solve it as the world becomes more complex, and businesses require cross-functional, creative answers to notoriously difficult problems. We need tools that break down barriers to communication and trust, unlock knowledge, facilitate difficult conversations and capture all sides of the story. Email is simply not sufficient for the modern enterprise.

There's so much 'soft' information that people share every day or hold inside their heads, on paper, or at the bottom of their email inboxes, that is lost to the organization unless there’s a way to share it on a large scale. It's a huge cost that there’s no way to quantify, so we don't think about it too much. But it's very real.

Wanted: Big People

For example, in a former life as a management consultant, I worked on a project with a public-sector client embarking on a multi-year, multi-billion dollar capital expenditure project. My team's mandate was to identify areas for savings and process improvements.

The project started with the assumption that our client had spend and vendor selection data from all the sourcing events that they'd run in the past decade. Our assumption was technically correct, but the data was spread across disparate systems and tough to organize. To make matters worse, a lot of it was on paper, including printed emails. Unfortunately, the organizational knowledge that would have helped us make sense of all this data had never been captured.

For example, we could see which suppliers were awarded how much business – but we never figured out why they were chosen, and most of the people involved in those decisions had either left the organization, or didn't remember all the details.

This project should have been a slam-dunk, but instead we had to abandon it, not because the client hadn't captured enough data, but because they hadn't captured the wisdom and thought processes of the people that walked through their doors every day. We really needed Big People.

The technology opportunity

This is the opportunity for collaborative technologies: to store not just information but know-how, interaction and insights of the experts in your organization, and make it easily accessible so you never have to say, "It's been 10 years and we have all this data. Let’s do something with it." It should all be available in real-time and it should be actionable.

I’m not suggesting we don't need data. We need both Big Data and Big People. Big Data has come a long way, but that last mile is still very soft. It's not very tangible, it's not the same across companies, and it's different based on the person you're talking to and the nuances of a given situation. It's time to turn our attention to the solving the people side of the problem, because no matter how much data you have or how powerful the tools become, the last mile of data will always be Big People.

Sunny Manivannan is a senior product manager at Coupa focused on collaborative features and products. He is passionate about helping people get more done in less time with less work. When he is not working, he likes to play tennis and poker.