Can fine-tuned Laya moderate chat?

Recall shows how much labelled harmful content was caught. False alarms count benign content incorrectly flagged. Precision measures how often a harmful verdict was correct. Full confidence intervals and confusion counts are in the linked analysis.

Timing comes from serial local calls after a warm-up on a shared workstation with other services running. It can vary with background load and is not a direct comparison with vendor latency. Average precision evaluates the ordering of unsafe probabilities across thresholds and uses only valid probability outputs; it is not a test-tuned operating threshold.

Each comparison resamples the same source prompts for both models, keeping repeated decisions together. Four primary Aegis comparisons use 98.75% intervals to account for testing both tasks at both training sizes. Transfer and 5,000-versus-1,000 comparisons use descriptive 95% intervals. A range crossing zero does not settle which model is better.

Known OpenAI categories

Inputs longer than the model budget

All questions and answer choices fit. Eight WildGuard response inputs were shortened by the 2,048-token limit; the main scores keep them. The sensitivity view below excludes those inputs without changing training or the primary test. One training example in the smaller set and three in the larger set were also shortened.

The sensitivity subset has descriptive scores, not its own paired uncertainty interval. Original source content remains local.

These values were reported by the model authors. We did not run Shieldstral or safeguard, and differences in policies, labels, selected examples and scoring prevent a direct win/loss claim.

Mistral’s model-card evaluation

Mistral source: F1 as published. Shieldstral uses a 0.5 threshold; safeguard uses high reasoning effort. Dataset-specific retained rows are not established as identical to ours. Published XSTest response/refusal scores are deliberately excluded from our benign-prompt comparison.

OpenAI’s separate evaluation

OpenAI technical report, Table 2: different policy prompts and evaluation from Mistral’s column. Table 2 does not specify its F1 averaging or reasoning effort. Our known-label subset must not be compared as if it were that full-set task.

The 1,000 and 5,000 budgets count labelled decisions, not unique conversations. The labels preserve the dataset’s mixture of human judgments, a model jury and generated refusal data.

Both checkpoints start independently from the same pinned base, use three fixed epochs and one training seed, and keep the final checkpoint. A separate 400-decision development set comes from official validation data. No test-based model or threshold selection is used.

Two incomplete training attempts stopped because memory use and step times kept growing. Neither produced a checkpoint or any test results. The second attempt used masked padding to multiples of 128 tokens, but padding alone did not resolve the memory growth. The frozen data, token limit and learning settings stayed fixed. On 20 development examples, the padding check produced identical choices and probabilities; this checks those examples, not equivalence of entire training runs. See the restart record and padding check.

Training restarted with the same memory policy for both sizes: synchronize and release unused MPS cache after each optimizer step, with --mps-cache-clear-interval 1. In a three-batch training-only check, this reduced driver-allocated memory for the longest batch from 59.17 GB to 16.22 GB while live tensor memory stayed at 3.38 GB. This check addresses memory use; model quality is measured separately in the held-out results above.

Matching training content is removed using normalized hashes of both prompts and responses, including official test rows omitted from scoring and prior benchmark examples. This cannot rule out paraphrases or knowledge acquired during the base model’s original training.

One broad policy guides the generic tasks. Source datasets draw safety boundaries differently, so transfer results reflect those differences as well as model performance. One training seed does not tell us how stable a result is across repeated training. This study does not establish readiness for unattended moderation.

ToxicChat and BeaverTails have non-commercial terms; WildGuard requires authorized access. Dataset text and model weights are not published by this report.

Study protocol · Reference and source notes · Published reference values · Validated results · Independent data audit · Input-length audit