Now take a set of words. They have "sizeness", "weightness", "colorness" and "densityness"...and "pythonness" and "haskellness" and even "adjectiveness" and "verbness" and so on and so forth...You can readily see that this can encode arbitrary patters (like language grammer or program syntax)
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...
my take from all these is that is perhaps possible to teach similarity search to 5th graders. not an elementary teacher myself, but many math concepts are much easier to explain with allegories (visual and other) than with formalism. a good example perhaps is what Mamikon does in his visual explanations of calculus (https://calteches.library.caltech.edu/4007/1/Calculus.pdf)
formalisms come only after there is understanding of the phenomena and a specific need to communicate concepts across members of community - in a concise and commonly comprehensive way. my understanding is that education often fails to train understanding and jumps directly to formalisms. and this is right also for STEM schools, where it is presumed everyone has some innate inclination to the matter. but this presumption, in my experience as university teacher, is not entirely correct, and students are not universally prepared to express lack of understanding, and particularly when it is expected from them to rather indicate understanding.
it seems very weird, the concept, to reward the correct answer, while suppressing and discouraging the expression of misunderstanding or confusion. why is weird? because learning comes trough trial and error, and the error part is half of it. so we need to encourage people to express their errors or lack of understanding, so that we can then fill in the gaps and eventually stand on a common ground. sadly, many forums, HN included, does not cherish misunderstanding. our posts here mostly get upvoted whenever they cross a sort of threshold about the amount of truth in our writing, and not because of daring expressions of confusion.
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...