About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.
SVMs in big data were already well liked back in 2010 as they could be scaled via map-reduce.
An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.
So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.
LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.
I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.
Yeah, its pretty wild! I think one of the benefits, but also potential risks, with LLMs, is you can just point them at anything without a lot of expertise.
Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.
I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.
Will be interesting to see how much or little this changes with LLM costs.
True I guess. I think most companies would at least want sign off from someone who's heard of SVMs that it isn't doing something mad.
Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.
I think this kind of change will pop up more often.
In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....
I don't know where the focus on building a single model that can do everything comes from.
I was thinking about this recently as well. Smaller, specialized models that can run on device just make so much more sense to me than these huge generalized models that need huge infrastructure run.
Does it though? Let's say you use an LLM to write code, do you think it benefits from the fact that the same model is capable of translating corporate messaging into internet slang for marketing?
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements.
This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.
Interesting! I had the idea to compute embeddings of log lines to look for “black swans” in my logs. A small model running locally would do. Then PCA, SVM - any old school method should work, I guess. Haven’t seen it done but it must be tried by someone somewhere.
This is just the AI grift and uneducated hypebros. There is zero reason to use an LLM as a classifier. But AIbros with zero technical knowledge will push for it anyways even if it is 1000x slower and 1000x more expensive to run.
You can can't build an SVM-classifier for a ticketing system without domain data. You can ask an LLM to do it, and it will do it very well.
With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.
It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.
I mean just have a coding agent do the whole process from start to finish. Tell it you want to bootstrap an SVM from LLM runs on ticket data. I can bet that it'll do it all for you without much input from you at all.
There are absolutely good reasons to use smarter things for classification. Svm with tdf idf for example is super fast but will make very bad classifications at times in ways entirely unfixable with that system (thinking back to bats being an acronym for a theology degree classification or similar)
I can spin up a classification solution with an LLM in ca. 5 minutes that is likely to do an okay job. To build a custom classifier, I need lots of training data first.
That is the reason.
Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
> but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
Absolutely!
But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!
This is a bit too radical and misses some nuance. In order to start building classifiers you at least need some ML domain knowledge, like what to google at least. Meanwhile, point anyone at the jev/open-jev-clones API and have them start classifying in <10 minutes. There's some value there. (same used to be valid for LLMs before jev became a thing. You can use llms as zero-shot classifiers with the right API wrapper, w/ no domain data).
At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
In some big companied many core corporate teams which hate that level of transparency and build their own opaque systems. I've seen this for departments like IT, HR, legal, etc.
I'd rather have tickets for everything and full transparency except for confidential info.
If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded.
If your Boss belongs to that club oh man god say the division.
Well if they use VPN point to point connections managed by AWS they'll get an email per week until they die for each VPN instance from AWS that they need to put in trash so I understand them.
Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.
This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?
Not sure if it would even need to be a small Chinese model. Amazon would surely have the necessary hardware and infrastructure to run a bigger Chinese model with a low per-token cost too. They probably get their Sonnet at a discounted rate too since they do the inference "locally", but I imagine the savings for not having to pay Anthropic their share would still add up.
You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
I don't see how "bonsai + cheap model" is any better than the solution they used which was just "cheap open-source decision model that can be run internally at Amazon".
50x cheaper, damn. I recall Chinese LLMs being similarly cheaper (and quite suitable for such simple tasks).
I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?
I don't mean send the data to china obviously — there must be an American provider that offers it?
Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.
What I don't understand is why are they not offering the decision model via Bedrock yet. Sagemaker / Lambda are quite expensive if you're not using the model continuously.
This pattern is playing out a lot. Running LLM inference at scale on apps typically lacks ROI. But I increasingly see folks use LLMs to help improve classical techniques.
So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.
Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.
If you were actually involved in that work pre-LLM, you know it didn't go in a circle. Most projects died at training data and problem definition. That isn't the case with LLM applications which can ship with zero data collection and labelling effort.
Now you can iterate on your 20 ticket categories in days against real workloads and not commit to training an efficient model until you've proven the prototype in production.
> One thing that surprised us was that JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19.
Meanwhile they could have just switched to Haiku 5.5 to get same savings. No need for complex orchestration involving GPU instances and would have just been a one line change in the code.
[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.
So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.
LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.
I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.
Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.
I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.
Will be interesting to see how much or little this changes with LLM costs.
So between llm and pseudo experts, at this stage, I prefer to go with a LLM.
Well not anymore... You can just get AI to build it.
Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.
Your model only took minutes to train if you don't count the data collection and labelling work
In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....
I don't know where the focus on building a single model that can do everything comes from.
Does it? All the latest frontier models that has made technical details publicly available is using MoE; the "microservices" of LLMs.
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.
It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.
That is the reason.
Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
Absolutely! But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
I'd rather have tickets for everything and full transparency except for confidential info.
heck, there's even a ticket if the office toilet is clogged
If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
Isn't it detrimental?
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
It’s shocking how effective this is.
I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?
I don't mean send the data to china obviously — there must be an American provider that offers it?
Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.
So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.
Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.
— "Let's build efficient classifiers"
— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"
— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"
Now you can iterate on your 20 ticket categories in days against real workloads and not commit to training an efficient model until you've proven the prototype in production.
That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...
Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?
But just think of the street cred!