Login
You're viewing the mastodonapp.uk public feed.
  • Chi Kimchikim@mastodon.social
    Oct 16, 2026, 10:24 AM

    😲 OMG, #Qwen3.8-Flash-Next-oQ4e-mtp on oMLX is the first local model I could run on my M3 Max 64 GB that could tackle my benchmark for #ComputerUse with custom GUI and custom harness. It scored 23/25! It's very slow on my machine, with prompt processing at 201.3 tok/s and token generation at 22.8 tok/s, because I could only offload 50% of the experts. Nonetheless, I'm very impressed because all the local models I tried previously weren't even able to complete the first level.
    #AI #LLM @vick21

    💬 1🔄 0⭐ 0

Replies

  • 💬 2🔄 0⭐ 0
  • Chi Kimchikim@mastodon.social
    Oct 16, 2026, 1:05 PM

    @vick21 The fan is actually pretty quiet. The model has 125B total parameters with 6B active, 51B n-gram embedding, and 4B MTP, soI have to offload 50% of the experts. The GPU isn’t running at full speed because SSD streaming is the bottleneck. This would be an amazing model to run on macs with 128GB. :)

    💬 1🔄 0⭐ 0
  • victor tsaranvick21@mastodon.social
    Oct 17, 2026, 4:31 AM

    @chikim That's a really good news! I had to lower my Qwen3.8-27b to oQ4e eventually and that really helped while not sacrificing any quality. Will give the flash a try over the weekend.

    💬 1🔄 0⭐ 0
  • Chi Kimchikim@mastodon.social
    Oct 17, 2026, 10:45 AM

    @vick21 Hope it works for you. I’m curious to hear what you think of it. The benchmarks on the model card show that it’s roughly comparable to Opus-4.6-Max, but more importantly, the general vibe on the internet seems to check out as well. The complain is it just takes a lot longer and more tokens. Even just the fact that it's the first local model I could use with VOCR for computer use is very exciting! It's great mystery to me why Gemini-3.8-flash still fails clicking the right coordinates!

    💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • Chi Kimchikim@mastodon.social
    Oct 17, 2026, 4:27 PM

    @vick21 Yes, it's 125B parameters model. That's why I can only offload 55% resident experts in memory, and stream the rest from SSD.

    💬 0🔄 0⭐ 0
  • Chi Kimchikim@mastodon.social
    Oct 16, 2026, 1:20 PM

    @vick21 Also, I've tried some coding task on Pi, and it's not bad. I'm definitely considering retiring qwen-3.8-27b! When I tried a few weeks ago, oMLX kept crashing with this model, but it got pretty stable. I've been running it for about a week now with 0 crashes!

    💬 0🔄 0⭐ 0