This article is based on a talk I gave at the DecompileD conference in Dresden, on 6 February 2025.
Software is eating the world
Who’s heard of this quote before? Let’s do some audience participation, please raise your hand if you have.
Marc Andreessen, co-founder of one of the most successful software venture capital firms in the world, a16z, penned them almost 14 years ago. Think of any successful software company - they’ve invested in them. His point is that virtually every industry has been revolutionised by software, and every product nowadays is touched by it in some way.
Machine learning is eating the world
Now how about this quote? Again a show of hands if you’ve heard it before.
Trick question! Nobody really said this. Yet. I would argue that it’s safe to now update “software” to “machine learning”. With the rise of genAI and ChatGPT, and even before that, machine learning and artificial intelligence are taking over every industry one by one. So many products nowadays have some form of machine learning, or at the very least AI, in them. Recommendations like Spotify’s Discover Weekly. Drug development. Fraud detection. Ad targeting. It goes on.
Today I want to talk about two things. First, to share my experience in how the machine learning part of a product is typically developed. And second, how genAI offers an alternative approach - to the degree that in my most recent role, it replaced traditional machine learning entirely.
So here’s my more clickbaity title for the talk: Machine learning is dead, long live genAI! Today we’ll discuss to what extent that is true.
Why listen to me?
So what are my qualifications to talk about this topic? In short, I’ve been building machine learning products for the last ten years.
I first did a PhD in machine learning at the University of Cambridge, before transitioning into product management at Yelp to build a machine learning-powered reporting metric for large advertisers. I also worked on ad targeting and location intelligence.
I then joined a startup in Berlin called Apheris, building a SaaS product for federated (or distributed) machine learning, like AlphaFold across multiple private datasets. And now I’m working at Blinkist, which was acquired by a B2B learning company called Go1. First I worked on the Blinkist app’s recommendation systems, and then after transitioning to the B2B side I worked on the core search experience and content metadata for finding the right learning materials out of over 100,000 training courses.
During my PhD and at Yelp, everything was about predictive models, like matrix factorisation, random forests, and support vector machines. Deep learning was almost exclusively used for computer vision. After that, deep learning models were becoming more and more popular, but still required a ton of data, so they often weren’t an option. At Apheris everything was about deep learning - and at this time ChatGPT launched.
It’s in my current role that we started working exclusively with generative AI - to improve search performance, and to automatically improve the signals and metadata that we have about our learning content.
What is machine learning?
Let’s dive into those types of machine learning a bit more. The way I like to think about AI, ML, and genAI, is as follows:
The most general category is artificial intelligence - it’s any system that exhibits human-like behaviour. This could be machine learning, but also simple heuristics based on data.
A subset of AI is machine learning, which involves the training of a model with lots of data, typically with labels showing what output is expected - e.g. here’s a cell that has cancer, here’s one that doesn’t. The trained model can then make predictions for future cell samples.
Finally, generative AI is a group of pre-trained machine learning models that are used to generate… things. GPT is the most famous, and it’s for text generation. These were shown 300 billion words from internet sites and resources to train them, and can now generate realistic text. There are other types of models for other types of outputs, such as images (e.g. Midjourney, DALL-E) and videos (e.g. Synthesia, Sora).
Building products with machine learning
First, so we know what we’re comparing genAI with, here’s how products with machine learning are typically built. Let’s take as an example the Yelp Store Visits (YSV) product I worked on. The value of the product was to provide advertisers an estimate of how many store visits their ads drove. For the ML part of this product, we wanted to predict which store a user visited based on their location data, which we had for a small subset of users.
Here’s the six stages of building an ML product:
Define what we are predicting. In this case, we want to make sure we accurately capture as many store visits as possible based on GPS data, to the right store, without missing many visits.
Agree on a success metric. How are we measuring the quality of the predictions? In this case we care both about recall (did we identify all visits?) and precision (did we get the right store?), and decided to combine them into one success metric (F1 score).
Gather training data. The more training data a model has, the better the performance of the model. This data needs to be clean and unbiased, so putting this together typically takes a lot of time (e.g. 1-2 months). For YSV, we could use high-confidence signals like reviews and check-ins, but needed to clean the data a fair bit.
Build an MVP model. Now the temptation is to immediately build a complex ML model, which could take us 1-3 months. We’ll get to why that is a bad idea in a second. But in general it’s way better to start with what I call an MVP model - something extremely simple you can put together in a day. For example, we could simply pick the closest store. Notice the parallel with traditional product management: starting with an MVP product.
Measure the baseline & validate the product. Here’s why we want a quick MVP model, rather than spending weeks or months building a complex model. Firstly, and most importantly, we can plug the MVP model into the larger product, and we now have a working product MVP that we can test with customers. Odds are, based on this feedback we will learn something that makes us change what we should be predicting or how to measure success. Secondly, it allows us to set up a baseline, so we know how much better any more complex approach is (if at all). More on this shortly.
Build a proper ML model & iterate. Once we have determined that the product really solves a customer problem, we can iterate on the ML model, and test how much better we can make it. A key question to ask: how much customer and business value is there in improving the model further? Because we have the baseline, we may be able to establish that more easily than without. The answer may be: there isn’t, the model is already good enough.
Coming back to measuring a baseline: for the YSV product, we could estimate how many more correct store visits we were detecting, and from that inferred how much more reliable the reporting became. We had quantified the effectiveness and upsell potential of the reporting, so we could tie a revenue estimate to model improvements. This is very powerful for evangelising the value of your product. I call this an impact model, and I've written more about it here.
Common pitfalls
Here’s four common pitfalls I see and have experienced with building ML products:
Not solving an actual customer and business problem. Leadership may pressure you to do ML because it’s the trendy thing to do, but make sure you always focus on what customer problem you’re solving, and how it will drive business impact.
Solving the wrong problem. This is a key reason why the “MVP model” is so important. When I worked at Yelp on another ML product, we worked with another team to identify users to show specific high-value ads to (e.g. car insurance). For this kind of problem, you can either improve the audience size (how many people we can show the ads to) or relevance (how likely they are to actually want car insurance, measured by the click-through-rate). The team that we worked with believed we needed to focus on improving relevance, and wanted an ML model from the start. We pushed back and started with a simpler heuristics-based approach, which we could deliver in weeks rather than months. From that “model MVP”, we quickly learned that the problem was actually audience size, not relevance, and we could quickly adjust our approach. If instead we’d spent a quarter building the model, we would have taken 5-10x as long to deliver value, and the project could have been killed.
Incorrect success expectations. Because ML products often involve a lot of research and testing ideas out, there’s a good chance initiatives don’t work out, or if they do: the return is hard to predict. In fact, oftentimes the return is either zero, or an absolute game changer. Help your stakeholders understand that ML product management is less certain than traditional product management.
Incorrect timeline expectations. Timeline estimates in software are notoriously hard. For ML products they’re even harder, because sometimes ideas don’t work or need further investigation. What I found helpful was to always timebox explorations and initiatives, and share a progress update with stakeholders at the end of those timeboxes.
Building products with genAI
Now let’s talk about how things are different with genAI. With ChatGPT we simply define a prompt, and can use that framework to make predictions.
Let’s look at an example. At Go1 we have a catalogue of 100,000 learning courses - for compliance, leadership, Excel, etcetera. Our users and managers search through our library to find the right learning materials, relying on our metadata to choose the right ones - such as the description, format, learning outcomes, level. Unfortunately, some of this metadata is either missing or unreliable.
Let’s look at the “Level” metadata - this is either “Advanced”, “Intermediate”, “Beginner”, or “Suitable for everyone”. In other words, a nice predictive classification task.
With ChatGPT we can create an MVP model in minutes. The prompt may look something like this:
You’re a learning specialist, assessing training materials for corporate employees. Your goal is to classify each course’s level, falling in one of four buckets: “Advanced”, “Intermediate”, “Beginner”, or “Suitable for everyone”.
Here is the definition of a course’s level: [definition]
Here are examples of levels for existing courses: [spreadsheet with examples]
Please generate the course level predictions for these courses: [spreadsheet]
We can write this prompt in literally five minutes! Compared to weeks and months of making a proper machine learning model. We can also easily feed in any prior knowledge or descriptions, especially if they are in text form, without any extra complexity - we simply add it to the prompt.
Furthermore, we may not have clean, reliable training data - in the Levels example, we get the existing metadata from our content partners, and we don't always trust its accuracy. That makes training a proper ML model on this data difficult (garbage in, garbage out). However, we may have a small set of high quality data, perhaps manually verified, that we can feed into the LLM. This is called few-shot learning, and it performs surprisingly well. (I would always start here, over jumping to RAGs or custom-trained LLMs).
Given how fast and flexible this approach is, we should ask ourselves…
Is ML dead? Long live genAI?
As the Germans would say: jein. Yes, and no.
The key takeaway from this talk is that with genAI you can get things started faster and more easily, compared with traditional machine learning. We can get to that stage of an “MVP model” very quickly, and incorporate prior knowledge more flexibly.
And that’s really important for us working in product: instead of spending a month gathering training data and developing a first model, we can now prototype something much faster with ChatGPT, giving a baseline performance and something we can validate with users.
However, there are cases where I believe traditional machine learning will still be better than genAI, so let’s dive in deeper.
genAI vs ML
In my experience, you should use genAI for:
Quick prototyping: you want to quickly prototype a model or system in days
Little training data: if there is little data, a general model like GPT4o will probably do as well as a proper trained ML model
Subjectivity: if predictions are fairly subjective, such as the level of a course, then a more specialised model probably won’t do better than genAI, because the task is ambiguous
Text-based data: LLMs are really good at processing text-based inputs and generating text as output
Prior knowledge: if you have text-based prior knowledge (e.g. description of what each level means, or defines a good learning outcome), it’s really easy and flexible to add. For ML models, you have to figure out the right way to encode and incorporate those features, and train a new model.
In contrast, you should use traditional ML when:
Lots of structured data & clear success metric: the more data you have, and the less subjective the output, the better the trained ML model will be. If the data is structured and tabular, traditional machine learning is probably a good fit.
Costs & latency: generating Learning Outcomes for 100,000 courses, using a lot of other metadata as input, cost us a 5-figure sum. If you expect to generate lots of predictions constantly, maybe don’t use OpenAI’s APIs. Similarly, if you need to create predictions on the fly, and low latency is key, ChatGPT may not be the right fit.
Performance matters: if improving the performance to the maximum possible matters a lot, such as with ad targeting, and you expect to iterate regularly, then traditional machine learning will be better.
To zoom in on that last point: here’s a quick analysis of how well different predictive approaches perform. As you can see, genAI (in yellow) performs pretty well, and when there is very little training data it’s almost identical. But traditional models are better the more data there is.
Because we all love flowcharts, here’s a visual way to go through this.
Ask yourself: are you building a first prototype, like we did with the Yelp ad targeting use case? If the answer is yes, then genAI (or a heuristic) is a great choice.
Next, determine if the output is subjective. If it is, like with the Level metadata, a machine learning model probably won’t do super well anyway - use genAI.
Then, check how much high-quality training data you can get. In my opinion, if it’s less than 1,000 examples, genAI will probably do as well as traditional ML.
Finally, is there customer and business value in optimising for performance, and getting the last 20% out? We saw this with the Yelp Store Visits product, where we had a clear impact model. There was definitely value in improving it further, so we kept working on more advanced machine learning models.
Wrapping up
Here’s what you should take away from today. No, machine learning isn’t dead. Yes, genAI can sometimes do it better and faster.
If you’re working on ML or data products, and you or your data science team hasn’t used genAI yet, I would strongly encourage exploring this.
In my opinion, using genAI, you can get 80% of the performance in 20% of the time (or even less - maybe 5%). The question for you is: do you need that remaining 20%? If not, then indeed:
Machine learning is dead - long live genAI.