Neural nets, including LLMs, have almost nothing to do with statistics. There are many different methods in Machine Learning. Many of them are applied statistics, but neural nets are not. If you have any ideas about how statistics are at the bottom of LLMs, you are probably thinking about some other ML technique. One that has nothing to do with LLMs.
That’s where the almost comes in. Unfortunately, there are many traps for the unwary stochastic parrot.
Training a neural net can be seen as a generalized regression analysis. But that’s not where it comes from. Inspiration comes mainly from biology, and also from physics. It’s not a result of developing better statistics. Training algorithms, like Backprop, were developed for the purpose. It’s not something that the pioneers could look up in a stats textbook. This is why the terminology is different. Where the same terms are used, they don’t mean quite the same thing, unfortunately.
Many developments crucial for LLMs have no counterpart in statistics, like fine-tuning, RLHF, or self-attention. Conversely, what you typically want from a regression - such as neatly interpretable parameters with error bars - is conspicuously absent in ANNs.
Any ideas you have formed about LLMs, based on the understanding that they are just statistics, are very likely wrong.
If parameters aren’t neatly interpretable then it’s bad statistics. You’ve learned nothing about the general structure of the data.
Linear regression models are often great tools for explaining the structure of the data. You can directly see which parts of the input are more important for determining the output. You have very little of that when using neural networks with more than 1 hidden layer.
This article use different wording than me, but in essence: Statistics is mostly about using a known model to explain the data. Machine Learning is mostly about finding any model that predicts the data well. Different purposes with some overlap. Some statistical methods are used in Machine Learning, but that doesn’t necessarily mean all of Machine Learning is statistics.
The boundary between statistical inference and ML is subject to debate—some methods fall squarely into one or the other domain, but many are used in both. […] Statistics requires us to choose a model that incorporates our knowledge of the system, and ML requires us to choose a predictive algorithm by relying on its empirical capabilities.
Another (potentially lower quality) article that is not from Nature, but discusses the meme in particular:
Because of the large number of variables in machine learning datasets, the models developed from them can be simultaneously extremely accurate and almost impossible to understand. Statistical models, on the other hand are typically easier to understand because they are based on fewer variables, and the accuracy of relationships is supported by tests of statistical significance.
Seeing your comment I wondered how someone publishing in Nature could have possibly left out the use of statistics for prediction. That would be a wild oversight that only someone with little knowledge of the topic would make, and surely not something that the editors of Nature would miss. Upon clicking the link I see that they mentioned it in the very first sentence and apparently ignore it if someone happens to call the prediction model a machine learning model. Using statistical models for prediction has been used since the start of the field, and renaming things that have been used for decades as “machine learning” doesn’t suddenly make them not statistics.
Artificial neural networks are statistical models, with numerous statistical approaches associated with their use, development and interpretation.
That book probably doesn’t go much further than neural networks with 1 hidden layer. Maybe 2 hidden layers at most.
IMO, statistics is about explaining data. Regression is useful to explain how parameters relate to each others. Statistics that don’t help us understand data isn’t useful statistics.
Modern machine learning has strayed far away from data explanation. Now it’s common to deal with more than a dozen hidden layers. It might have roots in statistics, but mostly it’s about brute forcing any curve to the data. It doesn’t help us understanding the data better, but at least we have approximated some function.
Software developer here, the more I learn about neural networks, the more they seem like very convoluted statistics. They also just a simplified form of neurons, and thus I advise against overhumanization, even if they’re called “neurons” and/or Alex.
Neural nets, including LLMs, have almost nothing to do with statistics. There are many different methods in Machine Learning. Many of them are applied statistics, but neural nets are not. If you have any ideas about how statistics are at the bottom of LLMs, you are probably thinking about some other ML technique. One that has nothing to do with LLMs.
Hahaha. People are great.
That’s where the almost comes in. Unfortunately, there are many traps for the unwary stochastic parrot.
Training a neural net can be seen as a generalized regression analysis. But that’s not where it comes from. Inspiration comes mainly from biology, and also from physics. It’s not a result of developing better statistics. Training algorithms, like Backprop, were developed for the purpose. It’s not something that the pioneers could look up in a stats textbook. This is why the terminology is different. Where the same terms are used, they don’t mean quite the same thing, unfortunately.
Many developments crucial for LLMs have no counterpart in statistics, like fine-tuning, RLHF, or self-attention. Conversely, what you typically want from a regression - such as neatly interpretable parameters with error bars - is conspicuously absent in ANNs.
Any ideas you have formed about LLMs, based on the understanding that they are just statistics, are very likely wrong.
“such as neatly interpretable parameters”
Hahaha, hahahahahaha.
Hahahahaha.
If parameters aren’t neatly interpretable then it’s bad statistics. You’ve learned nothing about the general structure of the data.
Linear regression models are often great tools for explaining the structure of the data. You can directly see which parts of the input are more important for determining the output. You have very little of that when using neural networks with more than 1 hidden layer.
“If parameters aren’t neatly interpretable then it’s bad statistics.”
Haha, keep going guys. You obviously know a lot about statistics.
https://www.nature.com/articles/nmeth.4642
This article use different wording than me, but in essence: Statistics is mostly about using a known model to explain the data. Machine Learning is mostly about finding any model that predicts the data well. Different purposes with some overlap. Some statistical methods are used in Machine Learning, but that doesn’t necessarily mean all of Machine Learning is statistics.
Another (potentially lower quality) article that is not from Nature, but discusses the meme in particular:
https://www.datarobot.com/blog/statistics-and-machine-learning-whats-the-difference/
Seeing your comment I wondered how someone publishing in Nature could have possibly left out the use of statistics for prediction. That would be a wild oversight that only someone with little knowledge of the topic would make, and surely not something that the editors of Nature would miss. Upon clicking the link I see that they mentioned it in the very first sentence and apparently ignore it if someone happens to call the prediction model a machine learning model. Using statistical models for prediction has been used since the start of the field, and renaming things that have been used for decades as “machine learning” doesn’t suddenly make them not statistics.
Artificial neural networks are statistical models, with numerous statistical approaches associated with their use, development and interpretation.
That book probably doesn’t go much further than neural networks with 1 hidden layer. Maybe 2 hidden layers at most.
IMO, statistics is about explaining data. Regression is useful to explain how parameters relate to each others. Statistics that don’t help us understand data isn’t useful statistics.
Modern machine learning has strayed far away from data explanation. Now it’s common to deal with more than a dozen hidden layers. It might have roots in statistics, but mostly it’s about brute forcing any curve to the data. It doesn’t help us understanding the data better, but at least we have approximated some function.
Care to reiterate?
Just because wheels have roots in horse wagons doesn’t mean cars are horse wagons
Wheels have roots in carriages
Whether you are a badass with a horse carriage or a cuck with a horseless carriage doesn’t really drive home your point
You know what I’m meaning
Software developer here, the more I learn about neural networks, the more they seem like very convoluted statistics. They also just a simplified form of neurons, and thus I advise against overhumanization, even if they’re called “neurons” and/or Alex.
How so?