Japan Open Data Lab has released ja-sentiment-v1, a lightweight Japanese sentiment classifier designed for infrastructure pipelines where latency and interpretability matter. The model returns probabilities for positive, neutral, and negative classes, achieving 0.9940 accuracy on a sealed 5,000-item test set and 0.9933 on a 600-item human-verified set. Crucially, the expected calibration error sits at approximately 0.005, meaning the output probabilities can be treated as reliable scores rather than just labels.

Built for CPU-Bound Workflows

The team prioritized deployment efficiency over raw parameter count. The model runs at roughly 17.8 milliseconds per item on a Core i7-11700F processor using an ONNX build. The fp32 ONNX implementation matched PyTorch predictions exactly on 400 development and sealed items, ensuring that moving to production inference doesn't introduce drift. No INT8 quantized build is shipped, likely to preserve the calibration integrity that makes the probability scores useful for thresholding and averaging.

The Error That Matters Is Nearly Absent

While most sentiment classifiers struggle with neutral boundaries, ja-sentiment-v1 minimizes the specific confusion that breaks dashboards: reading positive as negative. In the 5,000-item sealed test, only two instances of positive-to-negative confusion occurred, and zero in the human-verified set. The training data is entirely synthetic, generated by open-weight LLMs and verified by a second model, with a 200-item audit showing 97.5% label validity. The dataset deliberately includes hard cases like negations and understated complaints, comprising about 60% of the training corpus.

Caveats for Real-World Deployment

The team explicitly warns that these metrics reflect agreement with their synthetic label definitions, not superiority on real-world review data. The most-downloaded existing model, christian-phu/bert-finetuned-japanese-sentiment, scored 0.6210 on the same sealed set, but that comparison is framed as label agreement rather than a verdict on real-world performance. Additionally, the three-class structure forces mixed feelings into a single label or a probability spread, making it unsuitable for aspect-level sentiment analysis without recalibration on your specific distribution.

Key Takeaways

  • ja-sentiment-v1 achieves 0.9940 accuracy with only 2 positive/negative confusions in 5,000 sealed items.
  • Probabilities are calibrated (ECE ~0.005), making them suitable for scoring and thresholding tasks.
  • Inference speed is ~17.8ms per item on CPU, with exact ONNX/PyTorch parity.
  • Performance metrics are based on synthetic data; real-world field performance may vary.
  • Apache 2.0 licensed, with no INT8 build shipped to maintain calibration accuracy.

The Bottom Line

If you are building a CPU-bound pipeline for Japanese review aggregation, this model is a strong candidate because it treats probability scores as first-class citizens rather than afterthoughts. However, you must validate performance on your own real-world data, as the benchmark relies on synthetic definitions that may not align with natural user sentiment.