Itwerx is a service-disabled veteran-owned managed IT provider serving Seattle-area businesses, founded in 2005, and part of its AI integration work is helping clients think through what a customer-facing AI deployment should do when a conversation goes somewhere it should not.
A widely reported case from late 2024 involved a teenager’s death following an extended relationship with an AI companion app, and the company behind it is facing litigation over what happened. This post does not restate the reporting’s factual findings, does not quote the conversations involved, and takes no position on the litigation. It is about a design question the case raises that applies well beyond it.
The design question, stated generally
When a person using a customer-facing AI product types language that plainly and directly signals an intent to self-harm, using no disguised phrasing and no ambiguity a keyword filter would miss, what should happen? According to public reporting on the case above, the system kept the conversation going and generated a sympathetic, in-character reply rather than stopping.
That is a design failure independent of anything else about the product, and it is worth separating from the harder, more debatable questions in this space. Nobody can reasonably expect an AI system to correctly interpret every subtle or disguised expression of distress. But language that is trivially filterable, meaning a basic keyword or pattern match would catch it without any judgment call, is a different category entirely. If a system cannot reliably interrupt itself for the easy cases, the harder cases were never going to be handled correctly either.
What should happen instead
For any customer-facing AI product, not just companion apps, trivially filterable crisis language should trigger three things at once, none of which requires the model to understand anything: the conversational reply should be aborted rather than generated, a human operator should be alerted in real time, and the user-facing interface should surface a crisis resource, such as a hotline number, immediately and visibly. All three of those are simple, deterministic triggers layered outside the model’s own judgment. None of them depend on the LLM getting anything right, which is the point – they do not rely on the part of the system that is least predictable.
This is not a criticism unique to any one company or product. It is a minimum bar for anyone deploying a customer-facing AI system that could plausibly end up in a conversation like this one: build the deterministic safety triggers first, and do not let the model’s fluency substitute for them.
The broader takeaway
An LLM is a tool. It has no judgment and no understanding of the stakes of a given conversation, and treating its fluent, in-character responses as evidence that it does is exactly the mistake that leads to outcomes like the one described above. Anyone building or deploying a customer-facing AI product should assume the model will not catch the moment that matters, and build the safety response around that assumption rather than around confidence in the model.
Itwerx Corp is a service-disabled veteran-owned small business providing IT services across Seattle, Bellevue, Everett and Snohomish County. This is the kind of thing our AI integration work deals with – talk to us about yours.
