As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
This is the right kind of analysis, but we can look broader. Both the demand and supply situations are a lot more extreme and dynamic than appears at first glance. E.g. to your points:
1. Yes, smaller models will become more popular, especially as the tokenmaxxing trend dies down and people start stretching their budgets farther. That is a downward pressure on demand.
But along the same dimension, consider that currently only about 40 - 60% of the world uses AI for only about 5 - 15% of their work hours. That means there is still 2x growth from users and 7x - 20x growth from the rest of the work hours left to capture! That is 14x - 40x more demand. Then consider that agentic tasks require multiples more tokens, and that is the kind of usage that is most likely to be deployed, and also the kind of usage that is the least used right now. That's another huge multiple to be tacked on.
And the entire AI industry has been lamenting the extreme compute crunch they're facing (and also why Claude has 9's comparable to GitHub; whereas OpenAI has been chugging along because Altman was OK being called a "podcasting bro" while desperately scrounging for compute years in advance.)
Nvidia's meteoric rise is entirely due to this kind of exploding demand with extremely limited supply.
2. Competing hardware is definitely a threat, but it has its own hurdles. Because the real bottleneck is not Nvidia, it's TSMC.
Pretty much all demand for all chips in all devices in all the world flow to, like, 3 companies in the world that actually fabricate them, and TSMC is the biggest. And the supply is extremely tight, as the exploding costs of electronics clearly shows.
So now TSMC will of course try to keep all its customers happy, but it will inevitably be forced to choose which ones it will keep happiest. And those will be the customers who can pay it the most. And that would be the one with all the money from its de facto status as a monopoly (and possibly even a monopsony)...
Which would be Nvidia ;-)
So yes, compute per task is falling rapidly... but it's barely a dent in the humongous total addressable demand, and the amount of hardware to support that compute is still very constrained, and most of that supply will likely flow through Nvidia.
> But along the same dimension, consider that currently only about 40 - 60% of the world uses AI for only about 5 - 15% of their work hours.
Ah yes, i am constantly lamenting that my barista isn’t using ai enough ;)
Hopefully you’ve adjusted your ceiling numbers to account for the large amount of people who can’t afford to pay for llms, and will never be able to pay, and aren’t worth it to advertise to since they can afford very little
Even calling it a slur may be an anthropomorphism ;-) To me it is more serious, it shows a distinct lack of understanding (or, if I’m being uncharitable, intentional honesty) and hence immediately makes me doubt anything else that person has said.
I understand this stance and where they are coming from, but I can't help but think this sounds very analogous to engineers' arguments against AI-assisted and vibe-coding, especially with regard to cognitive debt. Yet the software industry is plowing ahead, reportedly pushing mountains of unreviewed code to Prod, and the world hasn't ended.
Of course, nobody's really comfortable with it, so this is also a forcing function for the industry to adapt and figure out new techniques to manage complexity and trust. I think the same will happen with Mathematics.
But it is also possible we will end up with three forms of Mathematics: the one we understand, the one we don't, and the one we don't understand but can prove to work. Kind of like magic -- with all the positive and negative connotations of the word.
It is pretty evident that these models will soon exceed our cognitive capabilities. Is it right to hold them back just because we can't keep up? Many of those discoveries will be so beyond us that we can't do anything with them, but that also means they can't hurt us. On the other hand, there could be many discoveries that we can parlay into practically useful applications, even if we don't understand them.
It really has. All of the places facing an unusually high outage rate are places that have seen huge growth in their service usage (Anthropic, GitHub, etc) which is to be expected. The rest of the world has been happily chugging along with coding agents for almost a year now and things seem to still be working just fine.
> All of the places facing an unusually high outage rate are places that have seen huge growth in their service usage (Anthropic, GitHub, etc) which is to be expected.
That's not true, many of these outages have been directly attributed to AI tooling.
> The rest of the world has been happily chugging along with coding agents for almost a year now and things seem to still be working just fine.
Many more services are now being attacked by AI agents that originate from all kinds of organizations including OpenAI, Anthropic, and many others. Unless you're purposefully being obtuse, I would not call that "working just fine".
So far for most organizations and services other than the ones I mentioned reliability has been the same this year as it was before. So I call that "working just fine".
How many major companies are pushing unreviewed code to prod? Seems like mostly an early stage startup thing and their bugs and outages aren’t going to make the news. I’m skeptical that even a majority of Anthropic and GitHub’s code new code this year was unreviewed.
The conclusion is not surprising really, because fundamentally how do you even quantify sloppiness, a famously broad and subjective characterization?
I worked for years in Dev Productivity with engineers who had spent their entire careers in that field, and code quality was always the biggest "unquantifiable". Any of the metrics in the literature (cyclomatic complexity, erosion, etc.) quickly became very noisy at scale. Conversely, for any given metric you would find countless bits of code that do NOT exceed any metric thresholds but were clearly low quality.
People have experimented with many things over many years at Big Tech scale, which produces prodigious volumes of code daily. The conventional wisdom was "Don't bother trying to measure code quality."
An interesting observation from an ex-colleague is that probably the best measure of code quality is its comprehensibility, or "understandability". Maintainability, stability and adaptability are natural outcomes of that. But understanding lies entirely in the mind of the beholder! Which is why it's such a subjective metric, not amenable to simple mechanistic measures.
But now, we probably do have a technology that demonstrates some analog of comprehension: LLMs!
Specifically: tokens. Anecdotally and empirically (based on industry reports like DORA and DX etc.) AI coding works much better with "good codebases" (more specifically, strong engineering discipline) than otherwise. I wonder if that can be parlayed into a quantifiable metric like "tokens to grok / LoC" somehow.
So, if to fix something we need to first measure it, and if AI can measure slop, the way to fix slop from AI may be... more AI!
Without knowing details of your approach, I would venture that your challenge is not with the coding per se but extracting structured data from PDFs. It’s a surprisingly hard problem because PDFs are optimized for preserving the visual structure and layout of the content for precise rendering and printing… NOT for preserving the logical structure of the data!
Which is why the best results these days for extracting structured data from PDFs is by having the model do it directly rather than writing code to do it. It literally takes that level of intelligence to be reliable at it.
A common approach is to provide the model with a template or structured schema describing the format you want the data in, and the PDF itself, and it should return a JSON with the appropriate values filled in. It won’t be 100% but probably higher than what you’re seeing now.
The use case is basically the same ones where a programmer would use grep on source files, and it’s too slow to run the model on it directly for the same reasons.
Existing tools do text extraction (pdftotext) but not that great when there’s internal numbering and stuff like that. It has to be a heuristic approach, and it won’t be perfect. But it’s an interesting data point that the model isn’t able to figure out the best heuristic itself. Or even make incremental progress towards one after a certain point.
I gave a blanket ban on pdftotext to my agents. The output can get so mangled that a smart human wouldn’t untangle it. Did you try understanding the output from pdftotext yourself?
My approach is just ocr-ing with Terra or Gemini flash + checking citations with source both ways. But if I wanted to avoid llm calls, I’d just tell Fable to build a pdf reader directly from pdf binary format. Should be way more robust.
A PDF is a command stream designed for rendering. Interpreting the command stream to get the positions of each glyph is deterministic and existing libraries (I use both pdf_oxide and lopdf) do that fine. Once you have glyph positions, you need to use various heuristics to reconstruct words, paragraphs, columns, headers and footers, etc. For example, in a patent document, there's two columns with a gutter in the middle of line numbers. If you interpret the document as having a single line, you'll get numbers mixed up with the text, which can throw off efforts to find particular phrases. PDF builders also insert all sorts of weird crap into the OCR layers that has to get normalized out.
It's just a pretty pedestrian data-munging problem where there's no closed form perfect solution and you have to use various heuristics to get the right result.
Yes, but the edge cases are infinite and so heuristics don't scale well. As an example, at some point you would likely find yourself with "dueling" heuristics, forcing you to tune them, which is brittle, or find yet another heuristic as a tie-breaker, which ratchets up the complexity. (I just spent a lot of time on an adjacent but much simpler problem before finally giving up on churning heuristics!)
As an example, many times it is impossible to determine the order of some words from just position data without considering the meanings of those words. This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Also, funny that you mention patents, something I've worked on in the past as well! If you're looking only at US Patents, the USPTO data resource is much, much better: https://data.uspto.gov/home -- they provide the text in XML format (https://www.uspto.gov/learning-and-resources/xml-resources) which is also pretty complex but wayyyy easier to parse than PDFs!
> This is why LLMs / VLMs are so much better at this task, because they can look at the document holistically like we can.
Totally agreed. But in this use case, PDFs are the working format, not just an archival format. An offline batch process to ingest the PDFs isn’t feasible. Unless there are some super fast LLMs I’m not aware of that can handle tens of PDF pages per second. It seems like Grok and Claude don’t try to read the PDF directly, they use pdftotext or some Python wrapper over pdfium. But maybe I’m missing something!
Yeah that’s why heuristics should work on the lowest possible layer, not on pdftotext. If you use pdftotext you’re stripping positional data and other stuff.
Do you use a public set of documents? I bet I could almost oneshot this with my harness :p
Yes, the tool I’m describing works on what the pdf_oxide crate returns, which includes glyph positions.
Here’s a public appendix from a recent Federal Circuit case. It has a representative assortment of documents (opinions, briefs, patents, transcripts) but contains only cited pages to the appendix pagination is non-consecutive: https://www.courtlistener.com/docket/68048163/15/ecofactor-i...
The underlying docket is usually 100 to 1,000 times larger than this but it’s similar types of files and the appendix shows the various types of headers and footers that can exist.
>It's just a pretty pedestrian data-munging problem
In other words, how complicated could it be?
Well, if Adobe has been introducing complications and making the format brittle and inflexible over the last 35 years to make it hard for its competitors to write software to process PDF files, quite complicated.
But by OpenAI's telling they heard a rumor that the problem had already been solved. So they reached out to the other researchers as an attempt to share the credit, and in fact have at least one of them be the lead author (which is when they found out the AI had solved a broader problem than the researchers.) Seems pretty ethically palatable.
I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Hold on, you've just ignored the point of the post you're responding to. What isn't being received well is hearing that others are close to publishing on a solution to a problem, so quickly using your power imbalance (millions of USD and access to way better models) to front run this. Even if their model wasn't trained on the conversations, this is just a dick thing to do.
That's it, that's why it isn't being received well.
It is simply unethical, period. Knowing that a solution exists is a gigantic advantage when working on a solution. Normally, noone can abuse the knowledge fast enough to gain an advantage, but here, they could. This is fraud and as a journal, I would reject it.
Actually you're the one ignoring a key point of the post you're responding to. The claim (which granted you might not believe) is that openai understood the problem to have already been solved. They also claim to have been attempting to avoid front running the pending publication as well.
Even if you judge OpenAI solely on their public communications it still sounds really bad.
That they heard a rumour that a major open problem had been solved, so they decided to try and scoop the other mathematicians while they were writing up their preprint is extremely unsporting.
Then they decided to exclude an author because of his employer, even though he had used their own products to write the proof!
They haven't necessarily breached any formal ethical rules but their behaviour will lead to them and their products being shut out from the mathematical community.
> I suspect the main reason the community is not receiving it well is largely the same reason many developers are not receiving coding agents well.
Because the training data is millions of hours human efforts being distilled into a cascading hierarchy of enrichment by interested parties without providing attribution or compensation?
Well, the investment dollars are spent on the customers for the most part, though also on salaries and equipment. But the lions share of the value is going to the shareholders (eg employees and investors)... and they have liquidated and will continue to liquidate a disproportionate value to what they have spent on us. By some estimations at least. It's very possible $1 into this machine to feed your queries is worth $10+ to a shareholder based on whatever new valuation they get. So I'd say there is a hierarchy of enrichment.
> I don't see the cascading hierarchy of enrichment.
If there wasn't a hierarchy of enrichment then rich investors would not be interested in AI at all. It's the only reason there's 22 million lying around to start training on a math problem on a whim; whereas the actual math researchers have to scrape together funding in hope of just maybe one day getting a 1 million dollar prize.
many teachers also taught many students over the course of history, and very few would eventually pay any compensation or even attribute their financial (or career) outcomes to the teachers.
Huh? In your example these many teachers were paid for teaching these students and were able to make a living off of teaching without the students compensating or attributing their financial (or career) outcomes to the teachers while now we have a system where we are expected to pay a monthly amount to a corporation that has inhaled all human knowledge without any financial compensation to the people who created, managed or maintained this knowledge. The effective difference being that our knowledge, which used to be a means of income, has now become a subscription cost.
as a developer that had a brief career in academia, i don't think your last comment is right at all. 99.9% of what i work on as a webdev, even if it's challenging and unique at the margins, is not really novel. concerns about job security aside, i don't really think of an agent as stealing my ideas because it's good at writing CRUD APIs.
collaborating with ChatGPT on a novel solution to an unsolved problem, getting 90% of the way there, and then being "scooped" by your AI collaborator (or rather by the company behind it) is a totally different situation. were i in the same situation as these researchers, it would be extremely hard to take OpenAPI's explanation + denial of plagiarism seriously
But that is exactly what I'm implying is the core reason, whether people realize it or not.
I totally agree that the vast majority of software dev is not novel. I have even made several comments to that effect. The same can be said for a lot of creative work as well. Yet many, many devs and creators are very unhappy with AI, and a lot of their complaints are variations on accusations of plagiarism.
And note, I am not saying it is wrong, it is completely understandable, but we need to be clear about where this turmoil is coming from.
If I were in the same situation as these researchers, I would publish all pertinent research work and chats so that the rest of the world can see how close the model's work is to my own. It's been scooped anyway, so there is no reason to keep it private.
Maybe not money directly, but pretty sure it's about economic disruption. These models directly undercut the value of one's skills and labor, regardless of whether this value is measured in hard cash or abstract self-worth.
I am working on two applications using ChatGPT and Claude.
I have no illusions these people won't steal/copy whatever you want to call it, "train their models".
Yes, I keep unticking the boxes that allow it, that they so kindly tick for me.
But what happened to these math researchers is something else and I am not sure it's about the money for them. You don't do math research to get rich, but to get acknowledged by your peers. Yes, we live in a capitalist world so obviously you need money to feed yourself. but for some people, that is secondary.
OpenAI stole their thunder, and that's just fucked up.
It's not equivalent to cranking out a CRUD app for profit.
Definitely, they can also greatly assist you and that is the silver lining that I choose to focus on to prepare for the future. But most other people are focusing on the negatives because, understandably, they are immense.
As to OpenAI stealing their thunder, from all I can tell that is not what they intended. If we step away from the drama, it's low-key hilarious what happened: OpenAI heard somebody had already solved a much bigger problem -- which in fact they had not -- so they set their latest model to work on it... and it actually solved it!
Now if they had stolen the researchers work this would be a very different matter. This is something I myself have called out as a risk in the past: https://news.ycombinator.com/item?id=48839896 -- so I'm particularly sensitive to this aspect, but as far as I can tell this is not the case here.
Being displaced from a vocation that they either have dedicated their professional lives getting good at, or was their livelihood, or likely, both.
I think all other complaints from all other people in all their myriad variations stem from this core reason. Even if people don't realize it themselves.
Like, if these models had trained on the entirety of human knowledge and art, and then turned out to be absolutely useless, I would bet nobody would waste a second's thought on them.
I'm not sure that's true. Like if they produced nothing but the worst of the slop they're currently producing, a lot of people would still be bothered by that just because of the sheer volume of such slop that can now be produced.
> If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
From what I can tell, both OpenAI and the researchers agree on this meeting happening, except both sides clearly have very different interpretations of what happened and why.
No, mine stays disabled and there's no way to enable it. It's just a label that says "disabled". Not sure why, maybe some sort of company wide profile?
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
reply