As much as I'd love to believe it, it is now a conservative take. Sure, having a solid architecture in mind still matters right now, but manually writing code is completely unnecessary and, before long, even designing the architecture is going to be completely automated.
Here I am starting to think a central part of knowledge work is the knowledge gained.
Maybe you are right, maybe not, let's see. Unfortunately I kinda agree, because humans are really good at being lazy and going in the path of least resistance (including me). It's genuinely difficult to not use AI even if it makes my work worse, as long as it's easier and faster.
Does writing table schemas count as writing code? Is that architecting? Both?
I personally don’t trust coding agents to have enough context to write domain-specific table schemas, and I don’t have the patience to transcribe all of the context into a natural language prompt. If I ask it to, it’ll write something for sure, and maybe that can be a jumping point for me, but at some point I have to physically write what the columns will be.
You should definitely try it if you haven't already. Most of the models are deeply familiar with domain specific areas in ways that most of us aren't. In fact, I would say the problem is generally the opposite. If model size and effort is is high, it will over architect what an app needs. I find myself reigning in a giant email notification system with subscriptions and "channels" and other such nonsense when sometimes you just want a simple one off.
Yes, the LLM knows the definitions of the words in my domain, that’s clear and obvious.
No, the LLM doesn’t know our product strategy and why certain things matter and certain things don’t. That’s very much what I get paid to do. There’s not even agreement within our team about what path we should take through the domain-product space, there’s no chance in hell that the LLM will choose a profitable random walk through that domain-product space.
If it were able to do that, then AGI would have already been achieved and we’re only compute power away from OpenAI or Anthropic making the marginal utility of any piece of code $0.
Yeah, it's not just over-engineering, it is just that generic systems that look like everything else usually don't solve a business problem, but I think I would be less dismissive of its world model in general.
LLMs are not a "random walk", they take in information and they explore the space according to the way they've been instructed.
I’m taking this view now as well. If you’re reading code, you’re probably doing it wrong. You should absolutely be setting criteria that can be objectively measured and rejecting code that doesn’t meet those criteria or perform as specified. We are all senior software engineering managers now, with a fleet of cheap and ambitious young engineers doing all the authoring.
But reading code? What does that accomplish, other than to slow your dev process down enormously? Serious question.
Man, I think that code is still the artifact that we produce as developers. Code is the truth. I don't find it difficult or super time consuming to just... read the code, either. I've highlighted quite a few issues with LLM/Agent output from just glancing at the code.
I'll let you know how it goes... My new VP of engineering is a 'no looking at code' type of guy and is ripping 10K LOC PRs / Docs / plans against our 25 year old codebase and I would not say that they're 'good' PRs.
Maybe I'm completely wrong, but I think reading the code is more valuable than ever when working in a full-stack / small company role. I can tell you exactly what the business logic or functionality is for a certain piece of our system, in truth, without having to step through and make sense of ambiguous docs (that were also AI generated).
(I have a sneaking suspicion that in two years or less, my small team is going to significantly compromise the integrity of this codebase. Maybe by then we can refactor with GPT 12.)
> You should absolutely be setting criteria that can be objectively measured and rejecting code that doesn’t meet those criteria or perform as specified
This is the classic "make no mistakes".
On a serious note, I might set as criteria "avoid code duplication". Does that mean that the model/agent will actually follow it?
> What does that accomplish, other than to slow your dev process down enormously?
I am an OSS developer and I often see PRs (i.e. from the general public) that look correct, pass all CI checks, are heavily documented and they are still wrong.
Most of the times either they duplicate code that already exists somewhere else, or they implement a "feature" by opening a can of worms for subsequent "features" in the same area.
> even designing the architecture is going to be completely automated
I mean, people have been dreaming about this for decades. The whole reason UML was so overdesigned in the first place was in hopes that people could code by drawing boxes and lines. The full COBRA spec had software autonomously buying components in digital marketplaces and installing without human supervision.
It is funny to see engineers insisting that there's no way a machine could do this better. If anything, the surprise is that it took good old human language -- that second L in LLM -- to get the computer to sling code. The assumption was that a computer would just "speak code" like some kind of native tongue, but instead it just understands human language and associates that with code, and relies upon things like compilers and tests to see if it's right. Just like humans.
Predictably, now that it's actually happening, engineers are worried. As they should be, but I think it's a short-term worry. The job is changing, productivity is leaping, but its still a world where computers don't need to do stuff, humans do, so humans will be making it happen one way or another.
Seems like content curation is becoming an increasingly important problems in online communities. Eventually a new equilibrium will be found, I'm wondering what it will look like.
Agreed, what I understand from RSI would be models creating new models, or at least upgrading their own weights/architecture. It does not seem to be the case here.
This technique is very useful to gain intuition for a given sample size. Just run a few simulations with uncorrelated data and then you can get a sense of how extreme the estimators can be.
Yes. However, consciousness is in the eye of the beholder (we have no good definition or test), and by training models to act as if they are conscious, combined with our general tendency for anthropomorphizing, we may enter a world where people start acting as if models are conscious, regardless of the fact of the matter. (Which, again, cannot really be established.) The SF Bay Area already has a historical tendency to attempt to project its own radical belief systems over the rest of the world. So I think it's rational to be concerned that one day soon "woke" ideals may include punishment for putting human concerns above machine welfare.
It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs.
Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
That's the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There's a statement that "consciousness is very likely biological"... and then that's it. No argument for it, no citation.
If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."
I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.
About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance https://www.youtube.com/watch?v=j2zv4jlo2Nw ).
If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors?
There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.
I am not saying that everything that a LLM say is true (the human without pain receptors can also say "I have pain" without feeling any pain).
Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.
There are chemical reactions that LLMs lack because they do NOT have a biology. They are just mathematical weights. Now, let’s say we can truly make sentient AI in the future but it does not have the biology that we do. Then it would just be a totally literal, emotion lacking, sentient machine.
But then what is sentience? Dogs are sentient to a certain extent and so are dolphins…
We don't know anything about how consciousness works? Why couldn't there be a material fact about the biological substrate that is necessary for qualia? I don't see how that possibility makes you a crypto-dualist. If anything, the idea that the abstract computations are the most important thing is maybe more crypto-dualist.
This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
Cephalopod and decapod brains aren't much like human brains either, yet it's still reasonable to regard them as being sentient and worthy of protection from unnecessary suffering.
I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is.
So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
My response was to a comment appealing to brain reconstruction. My point is that a brain may be reconstructed, but LLMs do nothing of the sort. Substrate was never the point.
Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.
Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
>
> Short story
This looks more like a dating app simulation than a chatbot
reply