All videos
0:00 / 0:00
ai

I Tested Jev vs 12 Local Decision Models. Here's What I’d Use...

The AI Automators3 October 2026Watch on YouTube

Part of series

Ep. 4 · Jev AI + Claude: Live Demos

Praktische demonstraties van Jev AI gecombineerd met Claude voor realtime visuele en zakelijke automatisering.

View the series

What you'll learn

  • Thirteen decision models, including Jev and twelve local open models, were tested on over 7,000 decisions each.
  • Jev performs best overall, but the gap with the local models is smaller than expected.
  • The question type makes a difference, and a tiny model topped one of the tests.
  • Local models are faster on short inputs, but that speed advantage changes once the input gets longer.
  • Some models could not accept the questions at all, which affects which model you should choose.

Frequently asked questions

Which models were compared in the Jev arena?
Jev was compared with twelve local open decision models, including Winnow 12B, Decider 4B, Laya, SemIf 4B (OpenJev), Nimble 9B, CLM 8B, Plumb 4B, Qwen 3.5 4B and ModernBERT NLI.
How many decisions were tested per model?
Each of the thirteen models was tested on over 7,000 decisions, run on the creator's own workstation.
Is Jev always the best choice?
Jev came out on top overall, but the gap with the local models was smaller than expected. A tiny model managed to top one of the tests.
When are local decision models faster than Jev?
Local models offer a speed advantage on short inputs. Once the input gets longer, that advantage changes completely.

Topics

Read next

Description from the channel

👉 Access our AI Builders course & join our community: https://www.theaiautomators.com/?utm_source=youtube&utm_medium=video&utm_campaign=tutorial&utm_content=jev-arena 🧪 Jev Arena: the app, the results from this video and the code to run the tests on your own hardware https://github.com/theaiautomators/jev-arena 🔗 Jev / TypeSafe TypeSafe: https://typesafe.ai/ Jev model docs: https://docs.typesafe.ai/models 🔗 Models in the arena Winnow 12B: https://huggingface.co/EldanRing/Winnow-12B Decider 4B: https://huggingface.co/Mapika/decider-4b Laya: https://huggingface.co/convaiinnovations/laya SemIf 4B (OpenJev): https://github.com/TheoLeeCJ/SemIf-OpenJev Nimble 9B: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B CLM 8B: https://huggingface.co/Contrastive-LM/CLM-v0.1-8B Plumb 4B: https://huggingface.co/crh225/plumb-4b Qwen 3.5 4B: https://huggingface.co/Qwen/Qwen3.5-4B ModernBERT NLI: https://huggingface.co/MoritzLaurer/ModernBERT-large-zeroshot-v2.0 🔗 Benchmarks mentioned JevBench: https://github.com/fstandhartinger/jevbench DecisionBench: https://huggingface.co/spaces/Hanno-Labs/decision-bench-leaderboard Jev is the decision model everyone is talking about, but a wave of small open models like Winnow, Decider and Laya now promise the same kind of AI decisions on your own hardware, with inference kept inside your own infrastructure. So how close are they really, and which ones should you actually use? To find out, I built a Jev arena and ran 13 decision models through over 7,000 decisions each on my own workstation. I measured how often they got the decision right, how quickly they responded, whether the question type made a difference and what it takes to run them locally. Jev came out on top overall, but the gap was smaller than I expected. A tiny model topped one of the tests, some models couldn't accept the questions at all, and the speed advantage of running locally changed completely once the input got longer. If you're choosing between hosted Jev and a self-hosted decision model, this should tell you which ones are worth investigating and what to check before you commit. #AI #Jev #TypeSafe #DecisionModels #LocalAI #SelfHostedAI #OpenSourceAI #Winnow #Laya #Benchmarks #AIAutomation #AIAgents #AIBuilders