Machine Learning: Understanding Its Potential and Limits

Machine learning is being talked about an awful lot in the business world these days, and rightfully so, since we’ve made big strides with it in just the past few years. We are finally starting to see some exciting applications in a wide variety of industries. However, the way ML is being talked about now makes it sound like it’s magic. Suddenly, it’s the solution to every problem. People who haven’t experienced it imagine that whatever it is they want, they can just press a button and the machine will pop out the answer.

Although there are some amazing things being done with machine learning, the machines are still learning. In most cases, machine learning today is a combination of old and new technology and human intervention. And in some cases, it’s what is known as a Mechanical Turk—an elaborate bit of automation concealing a massive human effort underneath.

While we will never get to a point where there’s button press and we’re happy with it, the promise of machine learning is that as humans teach and train the machine, it gradually and continually becomes smarter and faster.

For example, there’s been loads of machine learning going on in mobile technology to make sure that all your calls of are of high quality. You don't get as much call loss as you used to, even though the load on the network is growing, because there’s heavy duty ML going on underneath the covers.

Failure to Learn

The danger is that it doesn’t get better—that the machine fails to learn, and what’s been offered as a machine learning solution continues to rely mostly on human effort. Then every time you come to do something, it's Groundhog Day. You keep going around the same hamster wheel and it never gets easier. 

You don’t see improvement over time. The calls still get dropped. The suggestions on the website don’t get better. The helpdesk does not become smarter. They end up being the same if not worse, because if they lose the people doing it, you get different results from a different set of people.

I suspect we’ll see that happen with some applications that are today being labeled as ML solutions. At the moment it’s a bit hard to tell which applications are really using machine learning to arrive at a result, because most applications today still legitimately need a lot of human assistance from subject matter experts to help the machines learn, and quality assurance people on the back end, correcting errors in the output and feeding the right information back in.

Putting it in Perspective

This kind of effort is what has gotten us to where we are today. For example, incredible advances in machine learning in medicine, such as the ability to diagnose diseases, are only now possible because of all the data from MRI scans that doctors have been through and said, "That one’s a tumor, that one’s a fracture, that one’s a blood disorder."

In what we’re doing in spend analytics at Coupa, the machine has learned from a whole lot of manual classification that we did, and it continues to do so. To put where we are in perspective, the machine will accurately classify roughly 80 percent of spending data, meaning it will understand 80 percent of what you are purchasing and put it into the correct commodity bucket. But you still need to employ people to do QA and keep training the machine, because the amount of variation in language to describe things across the world is almost infinite.

Massive Acceleration

We’ll never get to a point where the machine will be able to do everything automatically. It’s not magic. But, if we’re doing it right, what we should be able to do is accelerate massively. It took us seven years to be able to train our machine, six years to be able to get enough data together, but now we have a machine that’s able to classify spending data automatically.  

We’ve got enough knowledge in the system so people can do things that would previously have taken months in just a few days. It's only the final bit that we have to ask the QA team to verify. So, we carry on training the machine, building up more and more evidence of what a thing is or what it refers to, filling in gaps in its knowledge. Those gaps get narrower and narrower, but it's like chasing the horizon. You'll never get there. That's what’s happening all over in the world of machine learning. 

Kicking the Tires

As we move forward with machine learning, it’s important to understand those limitations. Right now, there's loads of brilliant stuff happening, but you've got to kick the tires a bit to make sure make sure the answers are mostly coming from machine learning and not from a bunch of humans.

To know if the machine is learning and getting better, you have to look at key performance indicators over time, such as how many calls have been dropped, or how many help desk answers were correct within two minutes. You might start at 50 percent correct answers, and then 75 percent and then you’re up to 85 percent as the machine learns and gets better, quicker, more accessible.

The way we describe it in our space is by talking about how much we've improved our service level agreement. We're excited when we can go from 95 percent of spend data classified to 97 percent. It's a bit boring, but it’s a big deal for the customer, because that can represent a few hundred million dollars more of classified spending.

If it’s truly machine learning, even if you've got a poor learning set, if you've kept the feedback loop going, a year later you should be looking at dramatic improvement.

So, if someone says, "Look, we've got a machine learning application, but it's not very clever yet," that's nothing wrong with that, because that’s the state of a lot of these applications right now. What you should be wary of is if they say, "We've got a machine that's brilliant," when actually under the covers they're doing it all manually, and it will never learn. It’s as brilliant now as it will ever be.

Paddy Lawton is General Manager, Coupa Spend Analytics. He is the founder of London-based Spend360, which was acquired by Coupa in January of 2017. Prior to that, he was the CEO of  Digital Union. He holds a BSc in Computing and Software Engineering from Oxford Brookes University.