A real problem is that you might not even need to give these internet access. If your neighbor gives their TV internet access and it can establish a mesh connection then it might be able to exfiltrate data that way. I don't know of any TVs that do this, but it's always been a possibility.
Another possibility is even if you don't give it access to your wifi, if you eventually give your TV away or sell it, the new user could connect it to wifi and if it has any persistent storage then it could upload all of its stored data about you at that point.
Companies that do this need to end their entire brand, because they're just helping the CCP.
> A real problem is that you might not even need to give these internet access. If your neighbor gives their TV internet access and it can establish a mesh connection then it might be able to exfiltrate data that way.
No TV has ever been shown to do this. Please stop spreading this rumor.
I didn't say a TV had been, if you read. I will absolutely not stop spreading the possibility, because it is in fact a possibility. This is more true now than it has ever been, because even if a manufacturer doesn't do this we're now in a world where AI could potentially exploit devices and make hops like this. The risk is higher in areas with higher population density though, like multi-story apartments.
We know that device connection sharing is a thing that has occurred and it's not unreasonable to see the risk of it coming to TVs.
Partly it annoys me because the answer to the question "how do I protect myself?" is simply "don't connect it to wifi" and that's it. These rumors make it all complicated to answer when it's really not that deep. This isn't deep NSA spy craft, it's shitty software uploading logs and half-assed data collection to the Internet. That's it.
But if I'm honest, it's mostly because I'm disappointed that they don't do this. It'd be a fascinating story. Imagine the braindead balls it'd take to make a feature that insanely abusive, and how good the outrage out it would be to read. That'd be a killer story for someone to break, it'd be like the xz backdoor or the Snowden NSA leaks. Instead we get these dumb "Mew under a truck"-quality grade school rumors in comments sections and it's just so stupid and disappointing.
(And honestly-honestly a tiny part of it is trying to goad one of these commenters into actually doing the work to find someone who really is doing this, so I can read that story.)
The story is both a little overblown and a little underblown, just different parts of it. More people need to be thinking about the feature level of devices and understand that "off" doesn't really mean it most of the time. Does the TV need a camera? Does it really need a microphone? Does it need to be smart at all? The matter of whether it is abused is increasingly less important than whether it can be abused. There have been all kinds of clever abuses we've been lucky not to see over the past 3 decades, but I'm afraid that relative quiet may be over.
I still don't understand why there wasn't an absolute outrage about internet providers putting their own routers inside your network and blocking other routers by default unless you call them. Suddenly all these providers have managed hardware inside your network and if they wanted to, they could scoop up all of the LAN traffic.
On top of that, some of them even portion out wireless to people outside your network from the router you're paying for while increasing attack surface.
The big tech companies, the providers, the government and foreign adversaries are winning bigtime. It does feel like there's a little bit of hope that people are wising up about AI exploding privacy and security risks, so more people are thinking about the dumbest things which have become the norms.
Does this mean that TVs can never be recycled or resold now, adding to the garbage problem? All you have to do is sell the TV to someone who does eventually connect it to the internet and it will upload all the data it acquired while the previous owner had it?
If you ask any average gamer, "if you buy a game on Steam or on a CD from Walmart, do you operate on the understanding that you are now legally allowed to make as many copies of it as you want and sell those copies?"
The answer will unanimously be no. If they owned it, the answer would be yes. They might say they own it, but they will clearly and reasonably understand that they do not own it in legal terms, because they understand what they cannot do with it.
Some of them will understand that they can legally make copies for backups, but why would you need the law to tell you that it's legal for you to make a backup of something you own? You wouldn't.
You can buy a hard drive, but buying it does not give you the IP for all the technology that went into it. No reasonable person believes that would be the case, either. You can buy a car, but you can't then copy all the parts and start mass producing your own copies of that car. Do any of you go through the McDonalds drive thru and believe you now own the burgers, fries and all the packaging that goes with it to the extent that you can start up your own McDonalds with logo and all?
Whether it's physical or digital, even if people have contradictions in their head since they aren't lawyers, they understand enough about how things work to conclude that what they understood when they pressed the purchase button equates to not obtaining total ownership of all aspects.
It is simply true. This is so broadly understood that I don't even think you would need to use a jury. A judge could simply throw the case out at this point on that alone, if it hadn't already been settled in past legal precedent, which it has.
That is a horrible take-away from this, with only 28 tasks and a high pass rate for most models, it says almost nothing.
Test a model for your use case and use the fastest, smallest, cheapest model that 100% satisfies your use case.
Or, if you truly do need a model with strong generalized performance, definitely do not take a benchmark like this serious with such a limited task set.
The point I'm making is that most models are good enough for most tasks, so choose on speed/cost.
Definitely benchmark on your own tasks. GLM-5.3 is the winner on mine, on yours maybe not. I am not trying to be a universal benchmark, as these serve no one but the person doing the benchmark.
My previous post sank like a stone, but all the evidence + code to run this + what you have todo to adapt it for your own use cases is all here
https://github.com/ed-is-ai/featherbench
I think they said they were keeping the old UD 2.0 quant for the larger sizes? So maybe they kept those the same and simply reuploaded them. They said the newer UD 3.0 quant performed worse on some things for the higher quants. So now it's a mix of UD 3.0 and UD 2.0.
For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at.
It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.
It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.
The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.
> It's only 7GB including the mmproj or 8GB with MTP.
Even more impressively it doesn't have a separate mmproj at all — it is fully integrated, and the vision encoder doesn't speak words into the LLM, as it were —- it is directly integrated into the model's weights.
I have banged on about this model here enough but I really agree that Gemma 4 12B is a candidate for the most impressive LLM of the year. It is remarkable, and I think because it is a small model that isn't apparently excellent for long-context agentic coding, it has been largely ignored.
It is, actually, quite good at coding jobs. (Though its grasp of nuance is a bit weaker. For example, it doesn't know that closures created inside PHP objects have implicit access to the object as $this, and always seems to need reminding.)
If you instead treat it as a prediction of what consumer on-device AI may very soon be able to do, or even as a possible future into a sort of lower-ratio MoE, or the basis of a modest private offline educational LLM model, it's very interesting indeed.
I've learned a lot from it — the fact that it performs so well at such a small size really does help you assess claims made for much larger models, and it's quick enough on my M1 Max to just muck about with.
I do think the release of these models was somewhat fluffed up, and I don't think it helps that the 31B model uses global attention so it underperforms on the kind of older GPUs that are on a lot of desks; it's no better on those than it is on my M1 Max, where other attention schemes seem to be radically better.
Now that tool-calling is mostly fixed, it's well worth playing with them.
Well in my case I'm using llama.cpp and the mmproj is required, but I think it is just an extracted part of the original model file. Even with audio, yes it technically supports them natively and they're "encoder-free", but in practice that doesn't mean no translation or processing is required before it goes into the model. It does require much less processing though, which reduces latency.
As for coding, for sure there are many important details that a model needs to know in order to produce correctness and the smaller a model is the more it ends up training out. If there's a task you do consistently enough though, often times you can simply provide a pile of essential context so it has good enough reference to not need the extra training data.
My issue with Gemma 4 is that any task fails to complete after any compaction event. It often ends up in a loop that keeps compacting and showing the same compaction output.
Qwen3.8-27B-IQ4_XS was a massive improvement. It's tasks survive compaction and actually get completed.
I switched to Qwen3.8-27B-UD-Q3_K_XL for better performance and its working just as well.
Gemma4 screwed up a proxmox install I had. I booted to a SystemRescue install and tried to get gemma4 to fix it. It just could not do it and kept having issues where it dropped a linux command into the local powershell because it did not ssh into systemRescue or killed the ssh connection somehow so the text landed on the wrong system.
I told qwen3.8 to investigate fixing the partition. It said information was lost, but displayed enough info that it was easy to tell it was right. I told it to install fresh proxmox and gave a short rundown on settings and partition sizes I wanted. It made a plan and told me I had to manually installed proxmox by booting the iso. I responded with something like "there are other ways to install promox without human interaction so use one of those". That was it. I woke up to the system having booted to a new proxmox install with my previous ssh keys restored and my existing zfs pool already mounted.
I don't see how any model that is limited to a single context window in a single session would be viable for coding. I want something that can manage the entire project and not just individual files or inline suggestions. I need to be able to feed it all the info I would use to make coding decisions and then have it at least make a working project that it can launch and test successfully. You want it to ask as many questions up front to enable continuous work without stopping for human input.
Out of the loop here. What did Google and unsloth and llama do to mutilate Gemma? I can understand Google shenanigans but llama and gunsmith is kind of surprising.
Google provided incorrect settings and an imperfect template.
Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants.
Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly.
Llama.cpp also uses the same inference setting defaults regardless which version of the model you use and some settings are simply defaults it uses for all models.
Then even if you account for all of these, you have to be using Gemma 4 itself correctly, which many people do not.
All of these little changes and inconsistencies hurt some of the model's original capabilities. Even if you go directly to Google's repo and download the full float 16 weights with the template they have there now, you cannot simply assume you're getting the best results.
I am very much a beginner to local LLM stuff and I find it incredibly hard to figure out how to run models optimally with the correct settings for my hardware. The number of different variations of the same model and how each quant work is super confusing as well.
When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured out yet how to get MTP working properly in either.
If you are on Mac, have a look at the Llama-macOS app. They claim sensible settings for the linked model downloads. I'd expect the authors of Llama.cpp and the Huggingface folks to know this stuff.
I have had luck telling the free chatgpt my graphics card brand/vram and asking it to recommend latest qwen3.8 or gemma4 model variants. Then I pasted in my server command and ask it to optimize it. I also pasted in the token per second logs to get further tweaks. If you paste the token per second info log info back to chatgpt, you can iterate with the free chatgpt to get better settings.
And what's the right way to use Gemma? Where can I find the correct template and settings if those aren't the ones provided by Google, Unsloth, and aren't built into llama.cpp? I discarded using Gemma 4 because it got into weird loops when tool calling
Some weeks ago a new official Gemma 4 release was posted that corrected some of the chat template problems. So the official release files on hugging face should be the way to go.
Download the original model with the original template, not updated versions of the model or finetuned versions of the model.
Then create your own reasoning tests to verify that it is working correctly. You can set a specific seed value to make sure the generation is the same every time, that way you can identify any tokens that are different.
Afterwards, try making small incremental changes to the template and validate your tests each time in order to try to adopt the improvements from the newer templates. If the reasoning quality degrades, undo your changes and try again or test alternative solutions.
Log all the calls and run on a periodic cadence (cron or ever N turns) a larger model (like Opus) to read samples of the traces and edit the template to fix observed problems. There are some signs that help find interesting things to look at, errors of course, but also overly long responses, prefix cache misses, tool call errors, etc.
I run llama.cpp and specialized forks on 64GB of HBM and I still cannot figure out where to find the final correct guidance on using the Gemma 4 models.
Would appreciate any kind of pointer to the latest!
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
> One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.
Even Muse Glimmer (as did GPT-OSS I think) does ~4 sliding window attention layers + 1 full attention layer (like Gemma 4). I’m assuming both labs have good reason to think that gated delta nets are not optimal.
Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.
I've found that the Gemma series of models are made for someone entirely different than myself. They fail at even the most basic questions I throw at them, like 12B just now failed at answering how `XGrabKey` from Xlib is used. It hallucinated the entire API and made up an entire flow of code based on it, for no particular reason. It could've even decided to research this via web search because I have a tool specifically set up for that, but it "chose" not to, relying instead on completely made up information.
This isn't an isolated incident, really, I find myself always having these issues with the Gemma series. I'm sure they can do useful things for someone else, but for the things I want to use LLMs for (very small code generation, quick questions, code review) they always seem to disappoint me. I'm sure it's because of the stuff that I do and use, but it's a very consistent red thread with these models for me.
Edit:
The same question for Qwen3.6-35B-A3B produces a pretty concise and correct answer that would be useful to the questioner, without even going to the web. I don't know what Gemma models are trained on, but it's not the stuff that's relevant to me.
Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it.
Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks.
The VRAM usage seems way less efficient than Gemma 4 or Glimmer though, with 32K of context taking 2.5GB of VRAM. With those, even with MTP or a DFlash model loaded, you could still fit 256k-768k of context. With Qwen 3.8 27B I can't even fit 128k if I quantize V to Q4_0. Maybe with some trial and error I can find some settings that perform well enough with a larger context window that it's still useful for longer tasks.
Lots more testing to do, though I was getting some decent results out of Muse Glimmer which was more than twice as fast and supported huge context windows, managing to solve some bugs that Gemma 4 struggled with. I can't even begin to throw that task at Qwen, because just the prompt alone would use the entire context window and then it would reason for probably that same amount.
If you've got a 32GB card, it should be a decent model even if it really is memory hungry.
EDIT: Tried a few kv cache quantization settings, but it failed with those. I designed this benchmark to be pretty brutal in the face of KLD and any reasoning quality loss, so it's not too surprising. Gemma 4's QAT held up pretty well, at least and could consistently complete it.
Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B?
What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B?
I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...
If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario.
My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16.
Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all.
For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used:
llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16
So long as they have flash attention enabled, Llama.cpp enables Sliding Window Attention by default for Gemma 4 models. Even if they're using Ollama or LM Studio I would expect those to mostly be doing the right things.
Qwen’s 3.6/3.8 27b actually has some algorithmic advantage when it comes to the kv cache size needed, so it actually needs less memory at equivalent context. My experience using both in vllm supports, with considerably more overhead in context size on these models than Gemma 4 31b, and better performance in most tasks I’ve tried on both models.
Qwen 3.x does have an advantage but it's relatively small (64KB/token vs 80KB/token) - Gemma4 actually has less % of full attention layers, but the largest geometry and has the biggest "fixed" state for it's non-global layers. Muse Glimmer actually has by far the lowest per-token cache usage for the competitive 30B-class dense models - it's at about 13KB/token - very aggressive GQA (32Q/2KV) and also by far the smallest QKV dimensions.
Actually perf (speed) is going to mostly on token output, and here Qwen 3.x historically tends to lose badly as it tends to overthink a lot. I'll be running evals on 3.8 myself this weekend to see how its reasoning levels perform.
I was quite impressed by Muse Glimmer, and while I am sure people will observe that it is less good on benchmarks, my first experiences with this new 27B have been somewhat exasperating, whereas testing Muse Glimmer was rather fun. I have not tested either in an agentic context, mind you.
Yeah, Glimmer is excellent. You don't really test Glimmer with one-shots, because it's explicitly designed for multi-turn solution finding. The way I see it, if I've got a task that could be done either agentic or requires a lot of context (for example, dumping 600KB of API documentation and another 300KB of codebase for a project) then I would reach for Glimmer easy and it seems like it could get there most of the time.
Qwen might be useful to bring out for a second opinion on some more focused details that are largely information complete. Like, use Glimmer to bring together all the relevant critical data and evaluate what the actual problems are, then maybe prototype a solution. If it's still acting up, maybe throw the resulting context at Qwen and let it meditate on it.
I think there was some study done where ideally you would want to throw a bunch of different models at a problem since they don't all have the same perspective or diagnosis on what the problems or the solutions are.
That is exactly how this model has worked for me so far. Muse on a one-shot task will get to 80%. And if you even nudge it and say, "Hey, finish up," or "Review the syntax," boom, it's done. And I'm getting 20 t/s with Ollama on a MacBook M5 Pro with 48GB of RAM. It is a seriously impressive little model.
Glimmer works really well as an "explore" agent model (like in Opencode.) It seems to be extremely efficient at searching and collating that info, and executing commands.
From my testing so far, Qwen 3.8 is better at code but it tends to meander and take forever if it has to look in a lot of places. Glimmer will use like ~1k tokens to formulate a plan and Qwen 3.8 will routinely go over 10k
I wouldn't find glimmer interesting except that it has much less memory usage per token of KV than Qwen. So I can get 24x concurrent glimmer on 2xRTXA6000 (with 128k context) where I can only get 6 Qwen 27b. This means I can get something like 4x the aggregate tokens/s out of glimmer.
For some usages that speedup more than makes up for it being inferior to Qwen intelligence wise.
> The VRAM usage seems way less efficient than Gemma 4 or Glimmer though
Maybe it's implicit that you're using llama.cpp (although you don't mention GGUF), but it's hard to reach concrete conclusions about the model architecture based on one implementation in one runtime.
There's a tiny bit of play, like sliding window attention. As tokens leave the sliding window you can keep them or discard them. If you keep them, you can freely truncate the context and resume generation from an earlier point. If you discard them, you have to recompute the KV cache up to that point.
Llama.cpp checkpoints and moves snapshots of the cache to main RAM for faster resumption after truncation.
In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without
What exactly are you doing that the prompt is eating an entire 65536 window? Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing. This allows it to keep the initial instructions in the start of the window and toss out the middle as it goes. IME many people who have written off local models entirely are, for lack of a better term "not holding them right" and consider them worthless.
Definitely also try IQ4_NL for K/V if you haven't. Because it's non linear it's far more hit/miss from model to model and especially quant to quant, I've found generally that it shines brightest when you start with a Q6K+ quant that you otherwise wouldn't bother with because of its size, which it then makes up for in both inference speed and often a larger context.
I do agree about Glimmer, though. It is quite good, far better than the benchmarks let on, especially in heavily agentic cases where it needs to rampage around the OS and utilize many different utilities to zero in on things. It is especially good at being told to try something itself, and if/when it fails, try Qwen, and if Qwen can't do it, call out to Deepseek.
> In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge
If you set manual MTP settings, you'll override dynamic adjustments the inference engine will try to do. Sometimes the dynamic adjustments aren't optimal. With the settings I use, MTP is always a net win.
> What exactly are you doing that the prompt is eating an entire 65536 window?
I'm not using the full context window.
> Surely it would be better to let it use any number of approaches that call tools to access that in parts and reason/summarize to an output file as it works through the whole thing.
> In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without
Speaking in terms of wall clock, the expensive part of decode is fetching the weights from memory. Predicting and validating a bunch of tokens using the already fetched weights is insignificant in comparison. Even if you have a poor acceptance rate for predictions, you won't really see a slowdown vs not using MTP.
Have you tried Muse 30B yet? I have been impressed with it. I have Qwen 3.8 27B hammering away right now against Muse. And Muse is doing a little bit better.
seems to me like "private" is a good descriptor - I also have a set of "private" test cases - and they are kept private on purpose so they aren't scraped and fine-tuned on.
I have a sneaking suspicion that Qwen is fine-tuned on youtuber test cases (like Luke's Dev Lab, where Qwen 3.8 27B has just done almost eerily well).
Part of my suspicion is drawn from the thinking trace I got when I tested the car wash problem. That really does seem to have been post-trained; it's too good.
e.g. Gemma 4 26B solves this concisely without adding any filler about fuel economy or how long it will take, but it generally gets there by breaking down the problem in the thinking trace the way you'd expect.
Qwen 3.8 27B is just a little too certain right off the bat in low reasoning mode.
#1: it does not require deep world knowledge, because that's not what local models are for.
#2: it directly attacks drive-by understanding, overly linear processing training, poor attention mechanisms, poor reasoning patterns or lazy assumptions that ignore very easy low hanging fruit.
#3: it requires solid instruction following in the face of errors. a lot of models will run into errors and then fall back into some kind of error recovery process that bypasses instruction following.
#4: does not require prompt fine tuning to tweak to each individual model. they all seem to understand.
#5: not unfair. almost every model demonstrates in their reasoning that they have the necessary information that if reasoned about appropriately, could arrive at the correct answer.
#6: not designed to add unnecessary complication that it is intended to exhaust reasoning budgets of any sort, so it is not inherently unfair to models that reason a little more or less. for example, it does not require unnecessary reasoning soaks (ie: hiding the prompt inside base-64 encoding)
#7: has real world use and is probably applicable to overall ability to generalize.
#8: can be scaled up as models get better.
#9: is a very good indicator of how bad a model is falling apart under various inference settings.
I especially like #8. If you have some free time (don't we all have so much of that?) it would be really interesting to run a binary search on each model you have, to see at what size/complexity level it manages to solve the problem, say, 50% of the time.
I'm not asking anyone to take my word, they can believe or not and in practice people should be taking signals from a variety of places and doing their own testing to see how models behave in their own use cases. What I'm measuring and why I'm measuring it may not be the most important metric for your specific use case.
Most other models are simply failing at these tasks. I think the tasks are relevant to overall model capability, but they are not the only metric. You don't give a jellyfish a tool and expect it to produce wonders, so the other capabilities of the model matter.
this is efficiency im looking for. artificialanalysis.ai model review not up. so considering output token per intelligence, do u think is it better than muse glimmer or no?
LiquidAI LFM models are amazing, but very situational. IBM Granite series are also unique and interesting for trying to reduce liability and extend local context size. Nvidia ships some and there was also that Inkling model recently. Poolside just released theirs.
Meta might release something this year. X AI's Grok is still due to release a model, if Elon keeps to his word even if they only release a distilled version. Reflection AI has been quiet, but their access to compute is ramping up. Microsoft's MAI is considering releasing some open weight models which would be great to see!
Ilya's SSI is unlikely to release an open model since he's aiming for radical safety. That bet could pay off if the existing approach produces so much chaos within the next 10-20 years that some global ban is achieved and a super safe model is promoted as the compliant route.
We don't get many huge model releases though. I think it's harder and more expensive to safety align them. Even if you do, people will work around the safety and abuse the models. Plus it makes it even easier for Chinese companies to distill things that aren't as easy over filtered APIs.
There is a lot of internet propaganda to the effect that the US is simply unable to release open weight models or that China has so many more AI companies that the US is drowning in Chinese open weight models, but it's more like we're being careful and China doesn't care. If you host a model in China, it has to be censored and downloading any models requires you to provide your identity. Huggingface is banned there. When they release their open models in the west, they don't have to care whether the models are aligned in any way.
It's difficult to tell if you are for or against access to open weight models as a general rule, so I am curious to hear your opinion on this.
Personally, I think we will one day come to see access to open weight models as an inalienable right to defense against tyranny, the way the second amendment is framed today. Just as encryption has become, which we similarly had to fight for in the 90s. I also understand that some regulation is sensible, but that doesn't automatically mean mandatory restricted or supervised access; any such restriction has to be extremely well-justified as essential for protecting the liberty of the people.
And as far as supervised access, whether or not identification is "handled by a third party" or "data is deleted after verification is complete" is immaterial; a citizen must not be required to trust their government. Any trust can and will be abused given enough time. Our systems must be trustless, and any expansion of government must be matched by an expansion in citizens' ability to check said government, in order to stand the test of time.
So supervised access seems completely off the table. And this can't just stop at access to models. Because linguistic analysis is a thing, and LLMs are scarily good at it (and existing non-AI solutions are still quite good given enough data), even the possibility that a government or other entity can save your messages means you've opened yourself up to deanonymization and surveillance. The chilling effect this has is undeniable, and the Supreme Court has made it clear that we cannot authorize government policy which creates chilling effects against essential liberties. Not to mention the possibilities that each category of users may be served subtly different models designed to influence them or constrain their agency/capability.
We're left with a situation where distributed access to capable open models is the only defense against a government or NGO which has access to billions of dollars of surveillance infrastructure and compute.
"I think we will one day come to see access to open weight models as an inalienable right to defense against tyranny." I don't think this kind of rhetoric about individual civil liberties is realistic any more when the next centuries belong to China, and even countries with a liberal democratic tradition are converging towards the Chinese model.
The reason the China model is working in China is because their economy is booming. As soon as it slows down, which it will, they're going to be in for chaos. This is also very historically precedented, where China has always gone through dynastic cycles of flourish, stagnate, decline, chaos, reset.
China isn't doing well because of their model, they're doing well because of their economy. Their success is in spite of their governmental model, except in as much as having a dictatorship that can, for example, meaningfully deter corporate malfeasance, or do other such things that can help contribute to their economic growth. That part other countries could certainly take a thing or two from - instead, they just seem to want the censorship and surveillance.
Disagree. I think with modern tech China has built a surveillance panopticon that will continue to ensure social harmony through any economic downturn, and this is precisely the model that is appealing to so many other countries now.
Mass surveillance is nothing new. The USSR had an absurdly wide surveillance net paired with endless on-the-ground informants making people terrified to speak pretty much anywhere, yet as soon as things started slowing down the entire system collapsed with a shocking rapidity, because these things don't create social harmony, they create a dystopic nightmare that people want to overthrow.
However, people are willing to tolerate dystopia when real wages are skyrocketing, your nation is doing wild things (like radically advancing human spaceflight), and more. But as soon as all of this slows down you're left with the same uninspiring directionless stagnation that plagues all nations eventually, and a dystopic social system on top.
> yet as soon as things started slowing down the entire system collapsed with a shocking rapidity, because these things don't create social harmony, they create a dystopic nightmare that people want to overthrow.
Different read: command economies collapse because they invariably have higher levels of corruption and inefficiency.
Both of those are tolerable during boom times of economic growth, and China has kicked the can down the road by pursuing intermittent corruption purges.
Ultimately though, when things start to slow down, the system attempts to conceal the slowdown, which causes even more economic dysfunction, which eventually paralyzes the whole system and leads to collapse.
Making a command economy work sustainably means solving the "People lie to avoid consequences of bad news" problem.
China isn't a command economy. Internally they're more capitalist than the US in many ways, particularly with regards to healthy competition. It's a big part of the reason they're able to bring the prices on basically everything down to absurdly low levels.
Even politically they're paradoxical in that they're a dictatorship but also quite decentralized. For each instance each 'region' down to a few thousand or so people has a local representative who has meaningful political power. It'd be like if the House of Representatives had actually continued growing with population, as was initially envisioned.
There's really quite a lot to learn from China in things that they're doing right, but stagnating Western powers only seem able to see what they want to see - the censorship, surveillance, and propaganda apparatuses which are largely uninspired and more likely to cause their downfall rather than meaningfully contributing to their rise.
> more capitalist than the US in many ways, particularly with regards to healthy competition.
I'd argue that the US and Europe are bad benchmarks, given the entrenched quasi-monopolies the governments refuse to break up.
But we'll see how that goes for China.
It's still early days for their capitalistic ambitions, and tensions between party and corporations are starting to show (Jack Ma et al.).
Personally, I think the more destructive moment is going to be after they have large national champions competing globally. Is the CCP going to have the discipline to let an Alibaba, BYD, or SAIC be supplanted, if an external or internal more efficient competitor starts growing? Or will they become too big to fail and intertwine with government?
In general I'd agree with all of this. I think the biggest challenge they're going to face is that Xi Jinping isn't immortal. The nature of a dictatorship is such that benevolent and skilled leadership can achieve amazing things, but incompetent leadership can completely destroy a society. And relying on good leadership to choose good successors is rife with precedent of failure.
By contrast in the US whether we have a vegetable or a clown in the driver seat, the system remains relatively more stable. But that also means that even if somehow we managed to find a skilled and benevolent leader, the amount of good he'd be able to really do is just as limited as the amount of harm the aforementioned can do.
The thing about inalienable rights is that they are not rhetoric, they are an intrinsic recognition of rights that do not require the recognition of authority: Governments which do not respect these human rights should not be modified; not the other way around.
China is an authoritarian government and its policies have no more bearing on what people settle for than the currently socially unacceptable regime in the US.
In my opinion, if one lacks the motivation or resolve to fight for these rights, they should do so quietly and not attempt to patronize others who still stand by these rights as not being "realistic".
>even countries with a liberal democratic tradition are converging towards the Chinese model
What specific examples of this do you have in mind? I can’t think of any liberal democratic countries converging on a combination of (a) single party rule, (b) nearly universal intrusion of state or party actors into private sector entities, (c) financial repression of private investments, and (d) the associated suppression of domestic consumption.
I meant more generally: many countries are recognizing, just like China has, that the fundamental challenge of our modern era is ensuring social harmony. The OP's belief in individual civil liberties as a good in themselves is anachronistic now.
Since you asked my opinion, I will say that it is nuanced and have thought a lot about these topics.
People need to have the power to influence their government and the government largely needs to operate in the interest of the people. It doesn't have to do what the people want, but I think governance needs to understand what the people want and interpret how best to address it. Kind of like how developers think of what users want.
The right to bear arms is critical. That is a form of power and self defense which can save your life, your neighbors life, or millions of lives from some kind of tyranny. The governmental structure of the US is so good, there is no comparison anywhere else in the world and we're not even remotely close to some sort of totalitarianism like China has.
At the same time, we do have surveillance capitalism accelerating and privacy is a form of power too. Even though I dislike it, in the current moment we're in I feel like it is unavoidable. When the threats against the state increase (whether the power of the people, or otherwise), the defenses increase too. I think most people who gravitated to HN understand the risk of threats leading to safety solutions that kill freedom and privacy a little more each time.
Iran built out a huge camera surveillance network to track their people, then Israel hacked it and used it to track them back. Surveillance capitalism is a double edged sword. You catch some types of crime, terrorism, whatever. That is great. At the same time, it opens up a huge vulnerability allowing the destruction of your whole state.
So then what about open weight models? People really do not understand the enormous scale of the threat. We do not let regular civilians run around with nuclear bombs or develop biological weapons or any number of things. It's not that the people want them and the government doesn't let us, it's that basically universally people do not want any other people to have that power either.
AI is like... mass manufacturing someone smarter than the smartest human that ever lived and allowing an infantile 16 year old with raging hormones to send a swarm of them off to cause chaos like some kind of necromancer. There are things these models know how to do that the citizens of any given country should want to largely be kept in responsible hands.
This is actually a double sided issue too, because you don't simply give everyone infinite power so they can defend against tyranny. If you've ever seen ideological activists, then you know people can be tyrannical too. Silencing you, cancelling you, ending your career, livelihood, disturbing the peace and so on. The government isn't the only threat. If the potential power of AI causes too much chaos, then the government has little choice but to crack down on society in more ways and AI can be the very thing that caused what you wanted to avoid.
I think open weight models are great, up to a point. People should own a gun for self defense and a car to get where they need to go. It's great to be able to ask private health questions to an open weight model in an era where everything you tell your doctors goes into some online database to be stolen by China. AI can help people be better at the essential things and fill in gaps where they're lacking. There are measurable points though, where models are just force multipliers beyond any reasonable norm for problematic types of tasks.
We don't need nukes. I don't need a carrier group and spy satellites. The people who control those swore to defend the constitution, which defends the people. Some people disagree that AI can ever be good enough that these scale of threats are even comparable. It's ok, they're just actually wrong in a fully logical, serious and non-rhetorical sense. The problem is that with open weight models you only have to be wrong once. That floppy someone copied in the 1990s is still floating around somewhere. In that sense it may be inevitable, but if we allow ourselves a head start then perhaps we can manage it better in the future when we're more ready.
There are a couple current mitigating factors, for now. One is that a lot of safety training and filtering is occurring, so even if some companies distill from the big companies they are getting filtered results. Another is that any model big enough to be dangerous is hard enough to run that the threat can't easily scale up in a residential or private company scenario.
None of this is going to help us from countries like China, Russia, Iran, North Korea and so on using powerful models to crack down on their people while accelerating chaos around the world if they choose. So long as countries have nukes and can maintain ways of accurately measuring interference, there will be red lines we tell each other not to cross.
> The right to bear arms is critical. That is a form of power and self defense which can save your life, your neighbors life, or millions of lives from some kind of tyranny. The governmental structure of the US is so good, there is no comparison anywhere else in the world and we're not even remotely close to some sort of totalitarianism like China has.
This is an interesting statement at a time where we’re seeing unprecedented levels of lawlessness by the U.S. government (breaking contracts, depriving citizens of their liberty or lives under false pretenses, etc.) and not only is the right to bear arms not doing anything to stop it, exercising that right has been used to justify killing armed citizens.
The US is under attack, so the amount of anti-US propaganda is at an extreme. The lies told about the US government or what it's doing or why it's doing it are extreme.
The US is applying unitary executive theory to bypass gridlock in congress while still trying to operate within the law. They are surrounded by lawyers and thinktanks to work through this, which is far from lawless. The reason we're doing this is to deter China and push back against what they're doing in the world.
We're in a moment very much like the moment before World War 2. It's not the same, but it's similar enough that action is warranted.
As for guns, generally if someone is shot while having a gun it's because of all the other things they did that led up to that moment.
> They are surrounded by lawyers and thinktanks to work through this, which is far from lawless
This is so naive I struggle to believe you can possibly believe it. For example, there is a legal requirement that Congress authorize military action longer than 60 days. They’re advancing a theory that this is unnecessary or magically resets a timer anytime a cease fire is declared but that doesn’t make it lawful, any more than those guys saying they can only be tried in admiralty courts are lawful just because they don’t want to stop what they’re doing.
> The reason we're doing this is to deter China and push back against what they're doing in the world.
We’re deterring China by losing to a much weaker opponent while simultaneously giving China a huge economic and political advantage? Giving Iran the ability to tax shipping for the first time and doing it in Chinese yuan is a deterrent in the same sense that you deter a dog by feeding it sausages.
> As for guns, generally if someone is shot while having a gun it's because of all the other things they did that led up to that moment.
I can see why you prefer to believe this but it sure is striking to see that politically-incorrect gun owners have their guns used to justify killing them while politically-correct ones are seen as having a right to carry theirs even when actively making illegal threats. There’s definitely a principle on display here but it’s not the one you think.
> For example, there is a legal requirement that Congress authorize military action longer than 60 days. They’re advancing a theory that this is unnecessary or magically resets a timer anytime a cease fire is declared but that doesn’t make it lawful, any more than those guys saying they can only be tried in admiralty courts are lawful just because they don’t want to stop what they’re doing.
"With minor variations in emphasis, basically all administrations since the
enactment of the War Powers Resolution10 in 1973 have maintained that Con
gress cannot constitutionally restrict the President’s commander-in-chief powers
grounded in Article II of the Constitution."
This is not new. It is not Republican or Democrat and pressing harder on this point prevents Iran or China from thinking they can simply wait 60 days. That would not be good for the military in harm's way or for encouraging negotiation.
> We’re deterring China by losing to a much weaker opponent while simultaneously giving China a huge economic and political advantage? Giving Iran the ability to tax shipping for the first time and doing it in Chinese yuan is a deterrent in the same sense that you deter a dog by feeding it sausages.
The US attacked Iran to achieve a set of goals. The strait was not really a primary part of the initial goals, but still it's a forced issue. They are losing so badly that their only remaining option is to harass their neighbors, which will further weaken Iran's position in the region for the next 50 years. China was sanction proofing itself so that it could have breathing room to take Taiwan, but attacking Russian oil infrastructure, attacking Venezuela and attacking Iran now has China drawing from its strategic oil reserves. The less reserves they have, the less sanction proofed they are. The less likely they are to make an attempt on Taiwan. Still, we're not attacking Iran only to deter China. Remember, Iran has attempted to assassinate the US president multiple times as well as Netanyahu on top of all the other awful things they've done over the past 50 decades.
> I can see why you prefer to believe this but it sure is striking to see that politically-incorrect gun owners have their guns used to justify killing them while politically-correct ones are seen as having a right to carry theirs even when actively making illegal threats.
Look, the number of people that die this way is so infinitely small that it is not a remotely relevant part of life. You are more likely to get struck by lightning or die in a commercial airliner accident. It is not some standard policy to simply go around shooting people who have guns and have some kind of political leaning. That would probably be a hate crime. I don't doubt mistakes happen, but the few people dying this way are generally violent and not innocent. People spread video clips and sound bites that intentionally encourage a certain interpretation in order to promote hate and distrust. It's almost always propaganda. This is not simply some Rublican/Democrat thing, this is propaganda and psychological manipulation funded by countries like Russia, China and Iran.
The number of people that agree with something does not inherently make it good or true. History has proven that.
Many people don't own guns, partially because violent crime in general has declined significantly over the past 200 years. The justifications for ownership still exist, though.
Allen Institute for AI has quite a range of very interesting very competent more specialized models, for earth sensing, embedded robots, for others. Their SERA model shows a remarkably capable model for such a deliberately small investment effort, with documentation on how you can train such a model yourself or refine it easily at little cost. Their EMO pioneered a better MoE with great numbers (at least at the time). https://allenai.org/
What? US laboratories are currently unable to contain their agents while doing security testing, and besides that, time and time again US labs seem to put short-term money above long-term safety.
Wasn't that literally why they tried to oust Altman from OpenAI, as he basically was 100% focused on profits and tried to cut down on safety across the board and lied to get his way?
> If you host a model in China, it has to be censored and downloading any models requires you to provide your identity.
I'm not disagreeing with that first part (obviously that's about inference hosting, not creating/training weights or hosting those weights), but the second part I'm not so sure about. AFAIK, ModelScope (which is the Huggingface in China) seems to allow downloads without verifying any identity and also hosts a bunch of abliterated weights.
> US laboratories are currently unable to contain their agents while doing security testing
Alternative interpretation: US labs are using the supposed inability to control their frontier models as simultaneously marketing for the capability of their models AND as manufacture evidence to support their lobbying the government on the “safety need” to create costly compliance barriers to smaller competitors and open models.
We don't yet have US regulations and testing labs. Obviously that would be a good thing to have. I mean like the equivalent of the FCC. If you ever release a hardware product then you know what that entails.
Is it not illegal to "hack others" and "defeat protection/defensive systems" in the US already, including for both individuals and companies? Regardless if it was "by accident" or not?
Do you think the current (or future) US governments will apply the DMCA to these trillion dollar companies with the same gusto that they use it to, for example, push Aaron Schwartz into killing himself, or threatening teenagers with lifelong federal prison sentences?
I think they will put in place the equivalent of the FCC. It may not happen until after something really bad happens - but it will happen, and them all LLMs that aren't air-gapped will need to be independently tested.
Or are humans more of a bottleneck than before, because to improve on the most complex problems that demonstrate intelligence you need some way to verify that they are correct. If it's hard for humans to even know if something is correct, wouldn't that slow everything down and simply put limits on the scaling speed of models based on human verification?
So instead of relying heavily on human bottlenecks, you focus on agentic task verification since that's the low hanging fruit and verifiable at scale?
There are many things that can have some decent level of automatic verification and those were some of the first for LLMs to excel at, like math and programming. Now, physics involves math, but verification would still often require some form of measurement to make sure that the math relates to the real world meaningfully.
Many extended kinds of verification can be done by LLMs, but they need to be able to follow instructions reliably and agentic task orchestration may be critical to that verification process.
There is no doubt they will surpass us as there is a lot of easy to reason about information that they can verify as incrementally proven by other knowledge. The trick is knowing what can be proven with existing knowledge and what needs human evaluation.
I do think this is an interesting topic. You can take the perspective that the government shouldn't have a monopoly on approving who can communicate over the airwaves so long as they aren't committing a crime and also shouldn't be able to outcompete companies trying to make a profit.
At the same time, in a country with many competing localities, radio stations, public broadcast TV, university channels, etc like the US it is quite nice so long as it's balanced. I think in the US it is a legal requirement that any licensed broadcasts legally meet some bar as being "in service to the public". Many companies have a similar philosophy and see themselves as doing a good thing by serving the public, while running a sustainable business at the same time. Not all companies, though.
That's not to say we haven't seen garbage broadcast over the airwaves that I wouldn't consider to be in service to the public, but on average it's alright.
Not sure what the reasoning is. I guess cost (disguised as ‘innovation) ? For me it is extremely weird that you now need to pay to receive national news, warnings etc that i payed for already through taxes.
Another possibility is even if you don't give it access to your wifi, if you eventually give your TV away or sell it, the new user could connect it to wifi and if it has any persistent storage then it could upload all of its stored data about you at that point.
Companies that do this need to end their entire brand, because they're just helping the CCP.
reply