TryJev
← Back to Blog
question designbest practicechoicescorenoulpatterns

How to Design Effective Questions for Jev (Options, Granularity, Thresholds)

·2 min read

How to Design Effective Questions for Jev

The quality of a decision model is mostly the quality of the questions you ask it. This guide covers question design for Jev: wording, types, granularity, and the threshold logic that turns probabilities into product behavior.

TL;DR: One decision per question. 3–5 exclusive options. No “Other”. Write the question as the if you will compile.

Start from the branch, not the data

Bad:

“Analyze this message and describe the customer's emotional state.”

Good:

“Which team should handle this?”

The second question maps to code:

if (selected === 'billing') return enqueueBilling();

If you cannot name the branch, the question is not ready.

Three types, three jobs

Type Use when Returns
Choice Pick one of N queues / intents selected + per-option probability
Score Place on a scale (urgency, fit) weighted position + per-level probability
Noul Yes/no with confidence probability 0–1

Choice

{
  "type": "choice",
  "text": "What is the real intent behind this message?",
  "options": ["Genuine question", "Passive-aggressive", "Casual chat"]
}

Score

{
  "type": "score",
  "text": "How soon should I reply?",
  "options": ["Today", "Within 3 days", "Not urgent"]
}

Noul

{
  "type": "noul",
  "text": "Does this message contain a refund request?"
}

Option wording that works

Do

  • Use everyday labels annotators already know (“passive-aggressive”, “billing”).
  • Keep options parallel in grammar and length.
  • Make them mutually exclusive (no overlapping “angry” and “frustrated” unless that is the point).

Don't

  • Invent jargon (“negative affect signal”).
  • Mix abstraction levels (“billing”, “tech”, “existential dread”).
  • Put two decisions in one option (“billing_or_sales”).

Why “Other” is usually a bad default

  1. It absorbs all uncertainty.
  2. You cannot see which real label failed.
  3. Models learn to use it as a shrug.

Better pattern: allow a flat distribution, then threshold:

if (confidence < 0.55) return humanReview();

Granularity rules of thumb

Type Sweet spot Notes
Choice 3–5 options 7+ smears probability mass
Score 3–4 levels “Low / medium / high” beats 1–10
Noul binary If you need three outcomes, use choice

Need 15 support queues? Cluster into 4–6 macro-queues, then a second call for sub-queues.

Separate separate decisions

Intent and urgency are two judgments. Ask two questions in one API call — do not fuse them:

"questions": [
  { "type": "choice", "text": "…intent…", "options": ["…"] },
  { "type": "score", "text": "…urgency…", "options": ["…"] }
]

Fused questions force the model to average two judgments; both get softer.

State is half the question

A great question on a thin state underperforms. For classification:

  • Keep tone, punctuation, fillers.
  • Include speaker / channel when relevant.
  • Avoid summarizing away the evidence.

Thresholds as product policy

Treat confidence as a policy input, not just a model score:

Confidence Policy example
≥ 0.85 Full automation
0.55–0.85 Suggest + confirm
< 0.55 Human queue

Tune on 50–100 labeled samples. Re-tune when vocabulary drifts (new features, new markets).

Anti-patterns

  1. Kitchen-sink questions — “Classify, summarize, and suggest a reply.”
  2. Taxonomy without a branch — labels no code uses.
  3. 11-point scales nobody can defend.
  4. Hidden “Other” disguised as “General”.
  5. Prompts that ask for essays on a decision model.

Checklist before you ship

  • Each question maps to a real switch / if
  • Options exclusive and human-worded
  • 3–5 choice options (or 3–4 score levels)
  • Separate questions per decision
  • Threshold + human fallback defined
  • Logging includes distribution + latency

FAQ

Should options be English?
Match your label set and UI. Keep state in the user's language.

How do I test question quality?
Write 20 adversarial cases; a good question separates them without extra rules.

Can I use few-shot examples?
Put representative text in state patterns and keep questions clean; prefer examples in docs, not in every prompt.

Related reading