I am not sure where this comment is from (possibly without looking at this project?). This project is running quasi-frontier model at reasonable tps (~30) with reasonable prefill performance (~500tps) with a high-end laptop. People simply project what they see from this project to what you optimistically can expect.
You can argue whether the projection is too optimistic or not, but this project definitely made me a little bit optimistic on that end.
I am actually getting interested in QAT these days, especially for LSQ+ type, but it doesn't seem like people have done that enough in open-source world at least, for 2-bit / 3-bit OPD with LSQ+ basically.
the industry has largely moved away from QAT because the hardware required for running a quantized model are an order of magnitude less than training/QATing the fp model.
That's why things like Autoround, GPTQ, AWQ have been so popular, you don't even need enough hardware to run the original model on gpu, just cpu is enough due to the data efficiency
Thanks. I think it is a good explanation, but also suggests a gap. QAT to me, if done right, is the only way to recover performance for extreme quantization regime. The only thing matters of course, if whether it can work. My confidence in QAT comes from the LoRA can recover most quality misses in quantization, and that is still different from QAT for extreme quantization, so it could be very wrong. I need to try it anyway.
DeepSeek v4 Flash is still over 100GB at Q4 IIRC, and Q4 has generally been the sweet spot. Although it's an MoE so it might run a lot faster that this dense Mistral model if you have the RAM.
"Q4 has generally been the sweet spot" for self-hosting, yes. For any real meaningful work it's dumb AF. The only way to get reasonable intelligence from mid-size Gemma or Qwen is to run full precision BF16. Anything else is just an emulation of AI.
I would disagree. I have 8 GB of VRAM and 32 GB of RAM. I can either run a 4B BF16 dense model fully on GPU at around 30 t/s or Qwen3.6 35B A3B Q5_K_M at 20 t/s with GPU offload. Which one would I choose?
> but transformers are not AGI, and they will never be AGI
Like the claim "transformers are AGI", this needs proof, otherwise should be prefixed "I think". And honestly, positive proof is easier than negative proof (you just need to make one transformer model that is a AGI, whereas the never claim requires you to enumerated all possibilities).
That's like saying we should wait for positive proof of AGI from combustion engines. That'll never happen, no matter how much you tweak the engine. It's just not possible.
The negative proof is there in the definition itself. Transformers are not AGI, they're frozen human intelligence of the autocomplete variety. That can never be AGI and anyone who says otherwise doesn't understand transformers or AGI.
This kind of proof isn't really as water tight as you claim. It's a lot like saying state machines are limited to processing regular expressions, and then completely ignoring how easy it is to add a stack or linear memory to a state machine to make it a PDA or Turing machine.
So yes, the LLMs can be trivialized as just randomized autocomplete, but if you add a database or memory to the side very basic MLPs can become a Turing machine. It's going to take a lot more proof to say a Turing machine could never be intelligent. And you can do more than just give the LLM side memory - you can have them invoked recursively, use message passing as coroutines, and so on...
You might be technically correct if you ignore anything other than the very restrictive definitions you're using, but even there I'm not certain. If you had a LLM with a trillion token window, is that good enough to act as a memory? Human brains aren't infinite either.
Agreed. It is nonsensical to argue that a 3B transformer that hard-capped to decode 100 tokens is "intelligent". Of course when we are evaluating whether "transformers" is intelligent or not, we are talking about taking transformers as a core part of the system in some ways and enhance it with some other means (as you said, it is pretty trivial to making transformers a Turing machine, hence can carry out any compute, including intelligence (if you are in the camp that intelligence is computable, I don't think it makes sense to argue with anyone who otherwise believes intelligence is not computable)).
Lol, I totally agree about anyone using the non-computable angle.
However, I've got a 20GB GGUF file on my disk that can write code better than 99% of the people I ever worked with in the last 25 years, and ravens seem pretty clever with about 2 billion neurons... I have no idea what the lower bound is.
I honestly don't know. tim@apple.com is unavailable for quite some time now (since I tried a few years ago), while lisasu@amd.com still works around that time frame.
Yes! Can confirm. I emailed him in March 2020 after my 16-day old MacBook Pro had a logic board failure resulting in endless kernel panics. It was just past the return date so I couldn’t just return it and get a new one, so my local Apple Store had sent it in for repair. Then covid hit and everything shut down, so they couldn’t get it fixed and sent back either.
I had emailed with an explanation of what had occurred, and asked if I could get a refund so that I could just purchase a replacement. Within two hours of sending my email, an assistant from his office called me to arrange sending me a replacement. I was really impressed. I honestly figured I would just have to wait until the repair depot opened again, because I didn’t think I would hear back about my email.
Then a month or so later I got a call from the repair depot asking what address I’d like my repaired laptop sent to, since it was supposed to be sent back to the store for pickup (but stores were closed.) So I guess the right hand knoweth not what the left hand doeth in that case, because the person on the phone from repairs was pretty confused when I said no thanks.
I would have assumed that some assistant goes through the inbox and only a (random or filtered) subsample of those mails actually gets read by Tim Cook.
He also donated to Kamala Harris campaign. He would also donate to the next Democratic president for their inauguration if they still choose to do this corruptive thing. And your point is?
Trump is the president. People voted him into the Office. Tim Cook didn't give him the golden statue before he is in the Office.
Everyone in the United States is complicit to the horrible things done by the Trump administration by your logic. I partially agree, but I also think burning Apple to the ground will not be Tim Cook's legacy and he is in no place to go against the executive branch.
It is not about Trump, it is about the corrupted executive branch. Tim didn't do any crime against humanity in his act.
No, before Trump 2 nobody would’ve taken bribes and gifts so openly like this. It’s not even in the same league and it’s some really self-serving argumentation to pretend otherwise.
Every complicity is another nail in the coffin of our democracy.
If a cop says your problems go away for $100, you pay it, because the downside is huge by comparison. The problem is the cop getting away with it, not that you paid the bribe.
I hope you’re not comparing a gold trophy to a straight up bribe. It’s like giving Trump your Noble peace prize.
Having the prize doesn’t make you the winner. But it feeds Trumps ego sooooooo muuuuuch, it’s probably the “best” thing you can do to get on his good side without actually giving him anything.
That's a lawful FBI. This is a lawless executive branch. As we all know by now, executive branch has a lot a power that cannot be limited by Congress nor the Courts and erasing a few zeros from 4T market valuation is a piece of cake (as we witnessed daily how they moved billions around the market to their favorite inside traders).
> Everyone in the United States is complicit to the horrible things done by the Trump administration by your logic.
This is a ridiculous strawman. I’ll give you the benefit of the doubt and assume ignorance instead of malice.
I wrote that going above and beyond to curry favor with an autocrat in order to protect your profits is collaboration.
And you read, what? Existing under a government means you necessarily support it because there was an election? You do understand an election means some people voted the other way, right?
He's not an autocrat precisely because there was an election. He won the election because he got the most votes. He has since failed to do most things he campaigned on because his power is very limited by virtue of our government's structure.
Sorry for dropping the implied “wannabe” in autocrat, I figured HN commenters would be smart enough to infer that based on context. He is pushing and breaking boundaries on every front. No, he never accomplished any of the outlandish promises he made about the economy because he was lying and his team is incompetent, same reason the Iran war is a disaster. Project 2025 has been going pretty damn well though.
reply