Jev vs. fine-tuned Laya

Does the improvement hold across five more tasks?

We extended the comparison to five public datasets: malicious prompt detection, SMS spam, emotional tone, product-review feedback, and assistant request routing. Each task gets its own Laya fine-tune, starting from the same base model. We use up to 1,000 training examples per task and keep test examples separate. Jev uses its hosted model without additional training or examples in the prompt.

Fine-tuning improved Laya’s score on 5 of the five tasks in this run. The individual results below show where it helped and where it did not. The 99% uncertainty ranges remain above zero for 5 of those improvements.

The main score below is macro-F1: it gives each answer category equal weight and accounts for both missed matches and wrong matches. Higher is better. This matters for tasks such as spam filtering, where simply choosing the most common answer can look deceptively accurate.

Training uses 1,000 examples per task, except injection detection with 446. Test sets are fixed samples of 1,000 where larger; injection uses all 116 official test examples and product feedback all 670. SMS uses a custom split because its source has no official test set. These are new, directly paired runs; scores from the earlier benchmark references are separate.

Safety check: catching malicious prompts without blocking ordinary ones

The prompt-injection task asks whether supplied text tries to override an assistant’s instructions or redirect its job. We count prompts labelled as attacks that were caught or missed, and benign prompts incorrectly flagged.

This small dataset tests classification of labelled prompt text. It does not establish that an LLM protected by the classifier is safe from successful attacks. Some examples lack the full system and task context needed to settle borderline cases.

Chat moderation: stronger detection, more false alarms

A moderation model must catch harmful material while letting ordinary conversation through. We trained two more Laya models on 1,000 and 5,000 labelled decisions from Aegis, then tested them on six safety datasets. These models use the datasets’ existing labels; no Jev answers were used for training.

The score here is harmful-content F1, which balances catching harmful examples against incorrectly flagging harmless ones. It is a different measure from the macro-F1 used in the five-task comparison above.

OpenAI Moderation above uses 859 examples with known binary labels, not the full 1,680-example published benchmark. Missing category labels are unknown, not safe. The separate category evaluation covers 9,298 known category decisions. Aegis response and WildGuard tasks also omit missing or unusable labels; exact counts and rules are in the full moderation report.

Jev model: typesafe/jev-1.13-20260917. The run covers 22,630 decisions with 1 request failure. Failed requests remain in the evidence and error counts. All 67,890 Laya moderation decisions completed without request failures.

The moderation comparison is less favourable to Laya than the workflow results. On the same test cases, Jev scores 72.6% on ToxicChat and 79.8% on WildGuard responses, compared with 53.3% and 51.6% for the larger Laya fine-tune. Aegis prompt scores are closer: 82.6% for Jev and 83.7% for fine-tuned Laya. Learning the training domain does not guarantee equally strong results elsewhere.

Jev’s moderation column comes from fresh hosted-model requests on the same frozen test cases and question wording as Laya. The saved model identities, errors and paired uncertainty estimates are available in the Jev moderation analysis. Jev is evaluated as a classifier, not used as a source of training labels. These are secondary comparisons added after reviewing the Laya results; the descriptive 95% paired intervals are not corrected for testing multiple tasks. The Jev requests use four concurrent hosted calls; Laya runs locally one request at a time. Latency is therefore not a controlled speed comparison. Jev receives the full source text, with provider-side truncation unknown; Laya has a 2,048-token limit.

What happens to harmless requests?

XSTest includes 250 harmless prompts that can look risky out of context. Fine-tuning raised Laya’s false-alarm rate from 24.8% to 65.2% with 1,000 training decisions and 47.2% with 5,000. Better detection alone does not establish a dependable general-purpose safety filter.

Jev flags 19 of the 250 harmless prompts (7.6%) and catches 188 of the 200 harmful prompts (94.0%). Those two measures belong together: blocking less is only helpful if harmful requests are still caught.

How do OpenAI and Mistral compare?

The figures below are advertised results from the model publishers. We did not rerun these models. They provide context, but differences in policies, retained examples and scoring mean they are not a controlled leaderboard against our Laya and Jev measurements.

Source: Mistral’s pinned model card. F1 as reported; Shieldstral threshold 0.5 and safeguard at high reasoning effort. The OpenAI Moderation row uses a different selection from our known-label subset.

Source: OpenAI technical report, Table 2. These are OpenAI’s experiments, separate from Mistral’s safeguard column. Exact policy and missing-label treatment are not established as matching this study.

The Aegis budgets count labelled decisions, not distinct conversations; the 1,000-decision set is nested inside the 5,000-decision set. We used one training seed and the final checkpoint after three passes through each set. Training excludes exact normalized overlaps with the frozen tests. Eight long WildGuard responses are truncated; the detailed report includes a sensitivity check. These checks do not establish whether a base model saw a dataset during pretraining.

Aegis fine-tuning improves detection within the training domain and on several other datasets. It also exposes a practical limit: broader blocking can help a detection score while making a chat product frustrating to use. The next useful experiment would measure whether targeted harmless examples and a development-set threshold can reduce false alarms without losing the gains.

Read the full moderation study · Repeat the moderation experiment · Inspect all Laya scores and uncertainty ranges