Would be great if there were before and after comparisons of some outputs in the README, makes it more easier to understand how much value this will be adding If I want to use it
Now imagine there are 2 futures from here on, either joe becomes so much mature and internalizes so much context around the product and your team that your boss realizes it's faster and much cheaper to remove you as the middle man and promote joe to your level and your boss will directly work with joe or either joe starts asking for more money per PR because joe realizes how valuable he is to you and your company
This actually seems awesome, just going through the video on their site made me burst out with 100 different ways in which I would tinker with it, spend time in learning different gaits, some RL or some tricks to have it jump or do a back flip
I can see some parallel between how arduino gave hobbyist a mini computer they can use in any way imaginable, I can see this kind of thing being used to solve unimaginable problems by hobbyist or RL computer scientist or heck any UG student with claude subscription
> Thinking, tool calls, code, etc. remain completely unconstrained.
I wonder how much of these tricks actually work? As a side effect does it make the model produce thinking tokens to remember not to do that, “ohhh wait the user instructions says I should not use PG style in my thinking switching back to my …”
Slightly unrelated but I have no idea how wind speed predictions are made up high in the altitudes, is it just interpolation of data that are get from various radiosonoids across various timestamps and finding a regular or seasonal pattern?
Generally at altitude the wind goes from west to east. During the Cold War the US would launch spy balloons to overfly Russia whilst the wind was blowing against the Russians reciprocating. This is generally true in the current Ukraine war too.
As I understand it, it's a combination of things: aircraft reports, atmospheric motion vectors (e.g. a cloud doesn't have any propulsion, so if a cloud moves 30 km in an hour, you have learned something about the wind), Doppler wind lidar, and satellite measurements.
The numerical predictions of weather models often have many vertical components as well, so solving it for ground level also requires extending the forecast to the air, depending on the model.
To predict anything on the ground, you have to predict air temperature, humidity, wind speed and direction etc. for the entire troposphere. The output of the models in altitude are as relevant to the meteorologists as the output on the ground for manual expert analysis.
There are of course several different global weather models that predict the behavior of the atmosphere, and it is a lot more complicated than just interpolating past data. But they all do rely on radiosonde data for daily calibration of actual atmospheric conditions.
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state
# (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the
# level that contains that step (level 0) our counters start exactly one action behind.
# That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces
# the framework's lagged state exactly.
# CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET
# level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
The comment more likely means the harness source was read, not memory from a previous run and the first few turns of bp35 appear to be a cold start.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
Run policy search long enough with enough exploration and you can solve any of these games. But solving 110 games in 9 hours with a single RTX Pro 6000 doesn't seem likely. And if you did, you could keep the solution secret in exchange for the mountain of VC you would get to productize the approach. Not expecting it.
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
You are not given the rules or the winning conditions. You are only given a potentially windowed and/or degenerate visualizer of the underlying game state along with the UI and told to just figure it out. And you as a human will, in a couple moves. An LLM? Not so much. But they do eventually solve them. And given enough moves, victory is inevitable, but you are penalized for taking more moves than a human, yet also slightly punished if you find a better solution by capping your reward to 115%.
I am very much skeptical on the joint optimization of the harness and the model weights, I mean we can argue that claude code itself was a joint optimization of the claude, they released the first claude model and then the claude code harness then improved claudes performance on the harness then improved the harness after the model improved, but this was a process managed by humans all the time, it would be interesting to see claude improving it's own weights or harness in the future (or it might already be happening at anthropic right now)
I just didn't find the SIA paper to be testing this out rigorously or either provide sufficient evidence that it works
reply