The Butterfly Effect of Artificial Intelligence

Data is one of the most important survival factors for companies in the age of digitalisation. But no one knows how to determine the value of data. There are neither byte rating agents nor data stock markets or highly-professional experts who could measure data in carats or ounces. To be absolutely sure, most people rely on mass. The “big” in “big data” means: “As big as it gets”. True to the motto: the bigger the better.

Data minimalism instead of “big data”

At OTTO we would prefer to become data minimalists. Less data means faster scaling, more sustainability, fewer costs and more responsibility for the private sphere of our clients. However, before we can become data minimalists we needs to know: how much do we get from different data?

Data is only as valuable as what we make of it. The most prominent data driven process is the topic on everyone’s lips: artificial intelligence (AI). And in this process, it’s no longer mass which is decisive: a few months of data material from a platform like otto.de, from which the algorithm can learn, is in fact an AI gold mine. But where exactly in the masses of unused data sludge and sand the gold nuggets which best advance the AI are hiding - that’s unclear. Some experts claim that it’s impossible to measure it in such detail.

In an experiment, we proved the opposite: as a guinea pig (bearing in mind that animal testing is also opposed in the field of Artificial Intelligence) we used a recommendation algorithm, which recommends suitable alternative products to our customers if items are either sold out or out of stock. For example, it recommends products which other customers frequently viewed before or after viewing the unavailable item. Which of the alternatives are the most suitable is calculated by a neural network (with an algorithm developed by Google) using many, many user sessions. Almost 100 million, or 6 months of click data.

Our question: can a single visit by a user, a.k.a. one session, change something in the recommendations for all other users?

One dataset can change a huge amount

To preempt the answer: one session can change a huge amount - up to 40 per cent of the alternatives in our tests - even though less than 0.001 per cent of the data changes! A real “butterfly effect”. That such small changes have such a large effect was surprising even to us.

During the tests our AI algorithm proved itself to be quite a moody and sensitive soul, and it didn’t divulge these findings easily: even though we trained it in the same way a second time, a quarter of the recommendations differed. Poor conditions. Ultimately we wanted to measure the data influence and not the instability of the algorithm.

The problem: the AI has to cope with the enormous amounts of data – “big data” being the key term. And in doing so it sometimes decides randomly between alternatives that it considers to be equivalent. We had to eliminate this level of randomness from the experiment. That is only possible with a greatly reduced database, in this case: 1.3 million sessions, or around 2 days.

From this mini version of the recommender we built 500 versions, and in each one we omitted a different dataset. We then compared the results and looked at how far they differed due to the single altered dataset. The result: long visits to the website alter more in the overall results, but not always. The AI prefers to learn from sessions with a lot of different product groups or uncommon products. In addition, a lot changed through sessions with spontaneous purchases (e.g. using vouchers).

Since this way of determining the value of a data point is so new, we have written an academic article on the subject and we presented it at the international Workshop on Explainable AI.

Next we want to show what the data butterfly effect means in terms of Euros. To do this we need to test whether the results only change with regard to content or whether there are also qualitative changes – there is still a lot to research. And we will surely be able to wrest a few more secrets from our moody AI...