"Surgical "is the kind of wordage that LLMs seem to love to output. I have had to put in my .md file the explicit statement that the word "surgical" should only be used when referring to an actual operation at the block...
you're right, they are tools. that's kind of the point. PAL is a subprocess that runs a python expression. Z3 is a constraint solver. regex is regex. calling them "surgical" is just about when they fire, not what they are. the model generates correctly 90%+ of the time. the guardrails only trigger on the 7 specific patterns we found in the tape. to be clear, the ~8.0 score is the raw model with zero augmentation. no tools, no tricks. just the naive wrapper. the guardrail projections are documented separately. all the code is in the article for anyone who wants to review it.
In this day and age, without serious evidence that the software presented has seen some real usage, or at least has a good reviewable regression test suite, sadly the assumption may be that this is a slopcoded brainwave. The ascii-diagram doesn't help. Also maybe explain the design more.
Fair. "Does consolidation actually improve recall quality on a running system?" is exactly the benchmark I haven't published, and it's the one that would settle the question.
What I do have right now:
1178 core unit tests including CRDT convergence property tests via proptest (for any sequence of ops, final state is order-independent)
Chaos test harness: Docker'd 3-node cluster with leader-kill / network-partition / kill-9 scenarios (tests/chaos/ in the repo)
cargo-fuzz targets against the wire protocol and oplog deserializer
Live usage: running on my 3-node homelab cluster with two real tenants (small — a TV-writing agent and another experiment) for the past few weeks. Caught a real production self-deadlock during this period (v0.5.8), which is what triggered the 42-task hardening sprint.
What I don't have and should: a recall-quality-over-time benchmark. Something like: seed 5,000 memories with known redundancy and contradictions, measure recall precision@10 before and after think(), and publish the curve. That's the evidence you're asking for, and you're right it's missing. I'll run that and post the numbers in a follow-up.
The ASCII diagram fair point too — website has proper rendering (yantrikdb.com) but the README should have an SVG.
Appreciate the pushback — this is more useful than encouragement.
I kind of agree with the comment here that a lot of stuff happening around comes out from an idea without proof that the project has a meaningful result. A compacting memory bench is not something difficult to put off but I'm also having difficulties understanding what would be the outcome on a running system
I have been using the memory while building it. I have a central server and all my workspaces are connected to it via the MCP server. This changed everything for me. But that's me. Now I don't have to repeat things, the agent knows my preferences, can connect different projects I am working on without me asking and it knows my infra so can plan the test deployments and stuff on its own. That is somewhat I was aiming for.
Is there already a name for that effect where grandiose plans somehow appear to be more feasible than the simple mundane step-by-step issues resolution that even though they clearly stand in the way of said grand plans, are not worth investing thought and effort in?
When it comes to software projects my pet-name for it is the "big-bang theory", but in the article's domain that's kind of already taken.
It appeals to me because if you've ever taken a flight you can see how the details get progressively erased as you lift. Details that matter for a lot of reasons even if you can't see them.
It's also called "vision". It's what provides and powers directions on large and long term scales. Those "simple" and "mundane step-by-step issues" are just chores by themselves, yet at the same time may become stepping stones in the context of a well thought vision that people buy into and rally behind.
Like another commenter wrote... anyone can cast a fireball. Sam has been called a sociopath by many who know him personally. So it seems more likely than it might be otherwise.
1) Still unreliable at logic and general inference: try and try again seems to be SoTA...
2) Comically bad at pro-activity and taking the right initiative: eg. "You're right to be upset."
3) Most likely already reaching the end of the line in terms of available good training data: looking at the posted article here, I would tend to agree...
The problem is that LeCun was obviously wrong on LLMs before. You have to take what he says with the caveat that he probably talks about these in a purist (academic) way. Most of the "downsides" and "failures" are not really happening in the real world, or if they happen, they're eventually fixed / improved.
~2 years ago he made 3 statements that he considered failures at the time, and he was quite adamant that they were real problems:
1. LLMs can't do math
2. LLMs can't plan
3. (autoregressive) LLMs can't maintain a long session because errors compound as you generate more tokens.
ALL of these were obviously overcome by the industry, and today we have experts in their field using them for heavy, hard math (Tao, Knuth, etc), anyone who's used a coding agent can tell you that they can indeed plan and follow that plan, edit the plan and generally complete the plan, and the long session stuff is again obvious (agentic systems often remain useful at >100k ctx length).
So yeah, I really hope one of Yann, Ilya or Fei-Fei can come up with something better than transformers, but take anything they say with a grain of salt until they do. They often speak on more abstract, academic downsides, not necessarily what we see in practice. And don't dismiss the amout of money and brainpower going into making LLMs useful, even if from an academic pov it seems like we're bashing a square peg into a round hole. If it fits, it fits...
reply