>Notably, the neural network component behaves like a compressed index of the training data.
The "Compression" part is indeed about the model finding order in the training data. But this is not about applying some predefined compression algorithm on it, but by actually learning an algorithmic representation of the data.
This has nothing to do with "consulting the training data", because the training data cannot be reconstructed based on this information.
Roughly speaking, there are two types of information being stored in the NN: Memorization and generalization, or, shannon entropy and kolmogorov complexity.
great distinction. can we use it to freely produce and share mp3/h264 copies of any media? because it is provably impossible to reconstruct original based on that information.
But keep in mind: So far there is no way to train the models while completely avoiding memorization and only including generalization. That would be a great way to avoid any copyright issues, but all attempts I have seen so far were fairly limited.
And I'm not sure video, images and motion are actually 3 different things. Images are just still motion and videos capture motion, so it's really just "video and audio" of which one is visual, the other is not, thus my confused/surprised comment.
>When Intel finally shipped it at the 45-nanometer node in 2007, Gordon Moore called it the biggest change in transistor technology since the late 1960s. The breakthrough was not the material. It was learning how to process the material at scale.
Its curious that they picked this example. The challenge with HKMG was not the material itself, but how to integrate into into the transistor stack.
There were two completely different approaches: Gate first and replacement gate. Gate first is what the industry was already using for silicon oxide so everybody tried to go with as little change as possible. Only intel decided for replacement gate, which worked much better and reaped some other benefits on the way.
This was a watershed moment in the industry and ultimately led to some of the players dropping out of the cmos race.
But is this really a "scale-up" problem? It required development of novel manufacturing processes (atomic layer deposition), but was still mainly a process integration and device engineering topic.
The part of the thesis I have to agree with is that there is a data problem. The development above relies on executing lots of time consuming and tedious split experiments that often cannot be parallelized. The outcome of this relies heavily on the experience and diligence of the experimenters.
It's probably well suited for an "autoresearch" approach, bridging to the phyiscal world and dealing with the timescale is the challenge.
I had Opus 4.5 design an LLM inference engine in verilog, including firmware and automated verification a while ago: https://github.com/cpldcpu/smollm.c
It's of course far from optical. But lowering the implementation through the abstraction levels turned out to be extremely powerful.
Can you suggest some tutorials for Verilog and FPGAs in general?
I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.
hm.. has been quite a while for me. The good thing about the Tang Nano is that it is supported by the Yosys open source toolchain. There are quite a few resources on the web when you search for the combination.
Wow! And it also implements a very interesting variant of SUBLEQ that is turing complete.
>This VM implements an OISC - a One Instruction Set Computer. That instruction takes three signed 32-bit operands, a, b and c, and runs a program from memory m[] as follows:
1 PC (program counter) starts at 0
2 Fetch the next instruction (32-bit signed operands a, b and c)
3 If the low bit on any operand is set, remove it, and replace that operand with m[operand] i.e., a dereference of that address
4 Set m[b] = m[b] - m[a]
5 If m[b] is 0 or negative, set the PC to c, otherwise increment PC by 3 words
There's always someone somewhere who, with hindsight, did something that could be retconned into being similar to something important we've got today, von Däniken being an extreme example. Not putting down Losev's work, but accidentally stumbling on an interesting physical effect that you treat as a curiosity and engaging in targeted research to turn in into a product is a very different thing. For example the FET was envisaged multiple times in the same time frame as Losev's work, but wasn't rigorously pursued until Bardeen et al came along.
> He used these junctions to build solid-state versions of amplifiers, oscillators, and TRF and regenerative radio receivers, at frequencies up to 5 MHz, 25 years before the transistor. He even built a superheterodyne receiver.
That one calls them "negative resistance diodes" but I don't see how you can make a functional solid state amplifier and the like without it being a transistor.
The USSR famously invented everything the west did but years or even decades earlier, only for some reason never commercialised any of it, to the point where it became a bit of a running joke like the Su-24 "validating" the design of the F-111 which preceded it by some years. So I'd take any claims like this with a bit of a grain of salt.
Yeah, that pattern can be seen everywhere in semiconductors. E.g. the transistor invention vs. Lilienfeld, Heil, Matare etc. So the scope is more narrow than "Inventend Semiconductors".
Generally, there seems to be a tendency to disregard discoveries from outside the US. I think this pattern can still be observed today...
Other examples: Invention of light bulb, telephone.
What do you mean with "open-source"? Of course, the inference code for all the open weight models is publically available - see llama.cpp or hf transformers.
There are, however, very few models where also the full training pipeline is available. Olmo by AI2 comes to mind.
Ahah I was just thinking about that tiny web server the other day and even submitted it here, but it didn't get any traction. Back then (and even now) I thought it was very impressive!
The "Compression" part is indeed about the model finding order in the training data. But this is not about applying some predefined compression algorithm on it, but by actually learning an algorithmic representation of the data.
This has nothing to do with "consulting the training data", because the training data cannot be reconstructed based on this information.
Roughly speaking, there are two types of information being stored in the NN: Memorization and generalization, or, shannon entropy and kolmogorov complexity.
This paper is highly underrated: https://arxiv.org/pdf/2505.24832
reply