I think mathematics is feeling that concern with training acutely. The average math department pays its way by teaching in all forms and all levels: they don’t bring in the magnitude of grant money that concretely impacts a university’s budget. But they are already dealing with the consequences of that - that’s why universities that are cutting due to “low student demand” are going after math programs.
That’s a good reminder of a company that might have a very familiar ethos: Pacific Gas & Electric. Criminally convicted of 64 counts of involuntary manslaughter after towns were destroyed by wildfire. But oh well, what are we gonna do with a limited liability enterprise? At this point their liability insurance covers all the financial penalties they’ll need to spend.
> We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews
And, soon, it looks like we’ll be training on the reasoning traces of failed airlines and startups, which seems to open up similar hazards. I wonder if we’d be training on the next Lehman Brothers too?
It's funny, but to relax I was reading Tim Berners-Lee's FAQ for kids. He has a section about "I'm interested in Math -- what exciting stuff is there we don't do at school?". He quickly sketches out what ideas might get us to Maxwell's Equation, ironically enough. But, what a joy it is to see a person who believes in the public domain, and who we have everything to thank for right now, saying "The main thing is, to have fun."
As an aside, open core research reactors that glow are a very cool marketing tool for STEM. I went to a college with an open-core TRIGA run by undergraduates that loved to pulse it. It never failed to awe high school students.
> To me it doesn't seem like what AI has destroyed is the ability for mathematicians to develop understanding and share it with each other, but rather it's destroyed the yardstick (solving open problems) that has traditionally been used to measure how much they have contributed to that understanding.
I think you're missing key details in the statement. You can hate problems and sprinting towards solutions, and still have a deep impact in mathematics – Grothendieck once famously said you can often "soak the walnut" to crack open an issue. Rather, it's how frontier labs are inherently commercial incentivized to act like this, because it is in their best fiduciary interest. That is the misalignment.
I'm extremely sad that, in all of this, it is hard to celebrate the use of formal verification as a tool for genuine progress in the mathematical frontier... the LLMs that enable it are too entangled with commercialization.
Some smoking guns were agents calling themself "oai..." and making explicit comments with "evil"... depressingly enough, I doubt the next models will be less idiotic about this. Welcome to AGI...
reply