// HACKER NEWS — CYBERSECURITY
Don't classify, hallucinate!
Using LLMs to classify products, search queries, etc is by now boring. Yet it can still be difficult to constrains the LLM’s output to the legal vocabulary of brands, colors, categories, etc your system allows.
In the Wayfair WANDS e-commerce dataset, for example, you want to classify a query like “wood coffee table” into its most appropriate category. Of which there are hundreds:
The classic way to implement this would be with structured outputs. You tell your provide it must constrain its outputs to a list of legal values. In Pydantic, you create a giant literal of legal output values:
This works. But there’s a way to do this a lot cheaper with small / dumb models at scale. Not to mention, there’s an upper limit you can send
Luckily, there’s an easy pattern that makes LLM classification pretty seamless.
Just ask a dumb LLM to invent plausible, fake classifications for your query:
Now we’re not sending the list of legal classifications. We’re instead, asking the LLM to make stuff up:
It’ll then make up some BS that doesn’t actually exist in your real taxonomy like:
Actually it’s extremely helpful. You can now resolve that into the real vocabulary.
It’s very cheap to build an in-memory set of embeddings of the REAL classifications. As I’ve done in this notebook and this utility.