Hello fellow regards, I know from the title that your probably thinking this is the most contrarian take you have ever heard. Just chill and hear me out for a second.
The basis for training LLMs currently uses a framework called back propagation. I'm not going to attempt to explain how that works to you regards, but essentially it's incredibly energy inefficient. There has to be a "forward pass" and a "backward pass" for every single training cycle. These neural networks also have to have central synchronization.
I know what you're thinking, there has to be a better way. And in fact, there might be. Predictive coding is the new frontier of AI research. Each neuron in these networks has a single local prediction error which makes the network much less computationally expensive to train. There is also adaptive Inference which massively decreases the energy consumption of each individual query.
Let's walk through the life cycle of a neural network and make actual quantitative comparisons between the two training frameworks. According to research papers I've read (no I won't cite them, this is reddit), in the deployment lifetime of an LLM, a PC network will use 10-15% of the energy of a BP based network. With the vast majority of savings coming in inference and training capabilities.
Now let's talk about investments in the area. The former databricks AI head has entered the field to try and win the race towards PC integration in neural networks. Intel is currently rolling out models and hardware infrastructure so these models can be adopted ASAP. There are also numerous startups in the field funded by the likes of sam altman, google, and nvidia that are making leaps and bounds of progress.
So why haven't we switched? That's a phenominal question, lets dig into it. On current AI chips PC is 3-4 times more expensive than BP networks. Most of the energy savings I talked about is theoretical using neuromorphic chips which are necessary to optimize the energy savings from the PC algorithms. Also currently PC networks struggle with breadth destroying accuracy. So why am I still bullish on this technology? Well, because there is actually precedence that it is possible to deploy this technology, and its actually inside of your right now (your brain). Your brain uses a PC framework to run incredibly powerful neural networks on just 20W which is incredible.
Also I know this was super high level and somebody who knows the topic well will be pissed about the depth I went into the problem. However, I think I still get the point across pretty well.
LMK what you think