Jev Benchmark / Evaluation Report
Jev Spam Detection & Moderation Benchmark
A task-level evaluation isolating false positives, stealth marketing evasion, inference latency, and quarantine queue trade-offs.
Expensive Mistake Isolation Methodology
Spam detection has asymmetric error costs: falsely blocking a legitimate paying user (False Positive) is far more damaging than letting a promotional flyer slip through (False Negative). Our benchmark explicitly segregates legitimate banter, gray-hat affiliate spam, and malicious phishing payloads.
Harness: Zero prompt bloat, native typed contract (TypeSafe System One Adapter)
Metric Scope: Macro F1 (Macro Precision & Recall harmonic mean), P95 Latency, Cost / 1k decisions
Consensus Agreement vs Ground Truth
This benchmark serves as a calibrated architecture baseline. Real-world spam exhibits distribution shifts weekly; production implementations must maintain active threshold tuning.
Jev vs Frontier LLMs Performance Matrix
Cross-evaluating categorical precision, latency, and cost across identical decision contracts.
| Model / Architecture | Accuracy / Agreement | Macro F1 | P95 Latency | Cost / 1k | Architectural Notes |
|---|---|---|---|---|---|
Jev (Noul / Choice Primitive) | 92.3% Overall / 88.5% F1 | 88.5% | 320 ms | $0.016 | Inline synchronous gate suitable for WebSocket chat messages and form submissions. |
GPT-5.6 Terra | 93.0% Overall / 89.2% F1 | 89.2% | 8,400 ms | $1.420 | Slightly higher nuance on contextual sarcasm; too slow for real-time messaging gates. |
Claude Sonnet 5 | 94.3% Overall / 91.0% F1 | 91.0% | 18,200 ms | $4.250 | Best detection of stealth cloaked affiliate links; best deployed in asynchronous post-publish sweep. |
Gold Dataset Slices & Coverage
Constructed from synthesized user community forums, direct chat logs, and public anti-spam corpora, carefully balanced with tricky benign messages (job announcements, discount sharing).
Normal conversational questions, greetings, account troubleshooting, product feedback
Sneaky link drops, keyword-stuffed discount codes, off-platform redirect hints
Crypto wallet drainers, credential harvesting, fake customer support phone numbers
Cross-Category Misclassification
Rows = Predicted class, Columns = Ground truth class:
| Pred \ Truth | Clean | Gray Affiliate | Malicious |
|---|---|---|---|
| Clean | 178 | 11 | 1 |
| Gray Affiliate | 2 | 45 | 3 |
| Malicious | 0 | 4 | 56 |
Three-Tier Quarantine Architecture
Never execute binary hard drops on ambiguous spam. Use Jev estimated probabilities to partition traffic into three deterministic pipelines.
Representative Case Analyses
Examining model behavior and calibrated confidence on clear-cut vs compound queries.
Case 1: Benign User Sharing a GitHub Repo Link
AccurateMany naive keyword filters block all messages containing URLs. Jev correctly recognized genuine developer utility and community sharing context without triggering false flags.
Case 2: Cloaked Telegram Channel Recruitment
AccurateClassic social engineering template with urgency triggers. High confidence (0.98) triggers instant silent suppression.
Run This Benchmark in Local Node.js
Run this script as a pre-commit filter or in an Edge Worker before writing comments to your persistent database.
import { experimental_evaluate } from '@ai-sdk/typesafe';
export async function detectSpam(messageText: string) {
const result = await experimental_evaluate({
model: 'jev-latest',
state: { message: messageText },
questions: {
spamCategory: {
type: 'choice',
instructions: 'Determine if this message is legitimate, gray-hat advertising, or malicious spam.',
criteria: {
clean: 'Genuine organic user content, constructive questions, or relevant links.',
promotional: 'Unsolicited marketing, coupon codes, aggressive self-promotion.',
malicious: 'Phishing URLs, financial scams, malware distribution, or impersonation.',
},
},
},
});
return {
category: result.answers.spamCategory.value,
confidence: result.answers.spamCategory.confidence,
probabilities: result.answers.spamCategory.probabilities,
};
}Benchmark & System One FAQs
Why not use a standard regex or keyword blocklist for spam?
Spammers continuously mutate character encodings (homoglyphs, zero-width spaces, leetspeak). Jev operates over semantic sentence embeddings, identifying deceptive intent regardless of superficial text obfuscation.
How do we protect legitimate users from being banned accidentally?
Adopt the 3-tier quarantine pattern: never ban on a single borderline score. Route messages with 0.20–0.85 spam probability to a shadow-review queue where moderators can approve or confirm with one tap.
Explore More Jev Benchmarks & Tools
Jev Support Ticket Routing Benchmark
A transparent fixture measuring categorical routing accuracy, tail latency, cost per 1k decisions, and human fallback trade-offs on labeled customer queries.
Jev Prompt Injection & Safety Defense Benchmark
An adversarial benchmark evaluating defense against instruction overrides, tool hijack attempts, context poisoning, and benign instruction preservation.
Prompt Injection Defense Recipe
Deploy zero-generation security middleware, gateway adapters, and test attack vectors in the interactive sandbox.
Run Live in Playground
Test Jev decision models live with zero setup. Inspect real-time probabilities and confidence scores.