Jev Message Intent Detection: A Complete Playbook for Chat Classification
·3 min read
Jev Message Intent Detection: A Complete Playbook for Chat Classification
Message intent detection is one of those problems humans solve instantly and rules never get right. A colleague writes: "I took a look at that proposal — it's got some ideas. Though knowing your team's style, I'd expect a more mature approach. Thanks anyway~". That is not a question. It is not praise. It is passive-aggressive, and a keyword router will never know.
This guide walks through a complete Jev message intent detection setup: state design, question wording, probability thresholds, and how to ship it to production without an LLM in the hot path.
TL;DR: Give Jev the raw chat text as
state, ask one choice question with three exclusive options (genuine question / passive-aggressive / casual chat), and branch onconfidence. Below 0.55–0.7, fall back to a human or a slower model. Typical latency: ~140ms.
Why message intent is a System One problem
Kahneman's Thinking, Fast and Slow describes two modes of judgment. System One is fast, intuitive, and pattern-matching — exactly how humans read tone in chat. System Two is slow reasoning. Most product features that need "is this urgent / is this sarcastic / is this spam" are System One jobs.
LLMs can do this, but they were trained for System Two-style generation: long answers, explanations, tool calls. For high-volume chat intent classification, you pay for tokens you do not need and then parse prose you did not want.
Jev is built for the opposite: typed probabilistic decisions on a fixed label set.
The three-way intent taxonomy
Keep options mutually exclusive and human-sounding:
| Option | What it covers | Example signal |
|---|---|---|
| Genuine question | Sincere ask for information or help | Question mark, clear request |
| Passive-aggressive | Indirect criticism, polite hostility | Backhanded compliments, "anyway~", ellipsis |
| Casual chat | Low-stakes social noise | Emojis, greetings, off-topic banter |
Why not add “Other”?
“Other” becomes a magnet for uncertainty. You lose visibility into which real label the model is unsure about. Prefer a flat distribution across real options, then use a confidence threshold for the unknown mass.
Step 1 — Design the state (paste raw text)
{
"state": "Colleague: I took a look at that proposal — it's got some ideas. Though knowing your team's style, I'd expect a more mature approach. Thanks anyway~"
}
Rules for good states in message intent detection:
- Keep tone markers. Punctuation, fillers, hedges, emoji — they are the signal.
- Include speaker labels if you have them ("Colleague:", "Customer:").
- Do not pre-clean into summaries. "Colleague is criticizing us" removes the evidence Jev needs.
- Keep PII out if your compliance requires it — order IDs help, payment details do not.
Step 2 — Define the question
{
"type": "choice",
"text": "What's the real intent behind this message?",
"options": ["Genuine question", "Passive-aggressive", "Casual chat"]
}
Option wording checklist
- Use words annotators would pick without a style guide (“passive-aggressive” yes; “negative social signal with indirect framing” no).
- Prefer 3–5 options for choice questions.
- One decision per question. Reply urgency is a separate question (score or choice), not a second option on this one.
A second question for urgency often looks like:
{
"type": "score",
"text": "How soon should I reply?",
"options": ["Today", "Within 3 days", "Not urgent"]
}
Step 3 — Call Jev (via OpenRouter)
Model ID: typesafe/jev-1.13
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "typesafe/jev-1.13",
"messages": [{
"role": "user",
"content": {
"state": "…raw chat…",
"questions": [{
"type": "choice",
"text": "What is the real intent behind this message?",
"options": ["Genuine question", "Passive-aggressive", "Casual chat"]
}]
}
}]
}'
Example response
{
"answers": [{
"type": "choice",
"text": "What is the real intent behind this message?",
"selected": "Passive-aggressive",
"confidence": 0.71,
"distribution": {
"Passive-aggressive": 0.71,
"Casual chat": 0.15,
"Genuine question": 0.14
}
}]
}
confidence is the top option's probability. Always log the full distribution — entropy is your early-warning signal for drift.
Threshold tuning for chat intent classification
On a labeled sample of 50–100 messages, try:
| Top probability | Action |
|---|---|
| ≥ 0.85 | Auto-route / auto-prioritize |
| 0.55 – 0.85 | Soft action (suggested label in UI) |
| < 0.55 | Human review or slower LLM |
Intent classification is high-volume and low-stakes — until it is not. A customer-support tone detector that mislabels sarcasm is annoying; a compliance triage that mislabels threats is not.
Production checklist for message intent detection
- Store
selected,confidence,distribution, latency, model id - Alert when average entropy rises week-over-week (drift before accuracy drop)
- Re-check thresholds after launching new product areas (vocabulary shifts)
- Keep a golden set of 30 hard cases for regression tests
- Fail open to human review on timeouts — do not block the user
When to use an LLM instead
Use an LLM when you need a rewritten reply, multi-step reasoning, or open-ended labels. Use Jev when the label set is fixed and you need a value to switch on. Many stacks do both: LLM drafts, Jev decides.
FAQ
Can Jev detect sarcasm vs passive-aggression?
It separates them if your options do. If you need both, add “Sarcastic” as a fourth option and re-tune thresholds.
Does language matter?
Jev is used across languages; keep the state in the original language and write options in the language of your label set.
How is this different from keyword sentiment analysis?
Sentiment is polarity. Intent is what the speaker is doing. “Thanks anyway~” is positive sentiment words with negative intent.