Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.
Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
> The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.
Also:
>Some of my customers today: A PM is now the full R&D team. He reloads $1,000 a day in tokens and merges code faster than any team he's managed.
> Tokenmaxx, not peoplemaxx. Come tokenmaxx with me.
The Units seem to be independent i.e. could be followed in any order. Can someone knowledgeable confirm this? I know Set theory etc is the basis of many things usually in a formal mathematical setting, hence asking.
The society of mind is an interesting reference. I remember browsing through it around the late 90's when it came out. It seemed to provide some theory for the basis of some of our cognitive functions in terms of a collection of cooperating agents. But then, I guess, what the agents themselves are made of was not clear/understood? Are today's LLM models capable of taking the form of those agents, and can we take inspiration from SoM to see how they can evolve together towards a more powerful (real/AG?) intelligence?
Seems like the fact of a large India-Egypt trade link via the red sea was known atleast a year back, and specifically this evidence from Berenike. This [0] link describes the author William Dalrymple talking about it and also about his book [1] which is already out, which presumably covers this in more detail. A lot of Indian scholars are (re)discovering Indic history and we can expect much more of ancient India specific history to come out, which was unknown or has been forgotten over the ages, given the ancient nature of the Indian civilization.
Slightly OT, but if you are interested in this sort of thing, William Dalrymple and Anita Anand co-host the Empire podcast, which has many episodes and guests and recommended reading covering lots of ancient history.
> A lot of Indian scholars are (re)discovering Indic history and we can expect much more of ancient India specific history to come out, which was unknown or has been forgotten over the ages, given the ancient nature of the Indian civilization.
This, there are also very real links connecting famous civilizations of the Ancient Near East such as the Sumerians with the Dravidians of South India.
Tbf we have no clue if the Harappa valley civilization was dravidian. I think current consensus edges towards a lost austronesian language rather than a dravidian one (albeit certainly coexisting with dravidian cultures), but we'll likely not have good answers without archaeological evidence of cultural comparison (like a rosetta stone)
I don’t know about the Dravidians. But in English (UK/US) “chai tea” does not just mean any tea. It is commonly used to refer to to black tea spiced with specific mix of spices.
It is true that “chai” means tea in many languages, but the meaning in English is more specific. (At least in the usage i have encountered.)
It’s not that straightforward because every Indian is a healthy mix of ANI+ASI+. In fact there’s no ASI ancient DNA sample available annd it’s a proposed phenotype. ANI/ASI also goes thousands of years before these empires arose so again back to the original claim that none of these empires called them Dravidian.
Another month, another article falsely claiming that these trade routes haven't been known for years. It's also not limited to India-Egypt. The Greeks traded with Ancient Ethiopia. As did the Romans, who also traded with India and even as far as China. That sea route through the Arabian Gulf has been well established for millennia.
This article [0] proves that the Pāṇinian Shiva Sutras are optimal in some mathematical sense. I'm just learning to get into this, learning Sanskrit grammar, etc. and get a sense of the Asthadyaayi [1] so that I can approach it later in more detail. There are various ways to approach it and some online sites make an attempt, but as per an acquaintance who has been taught under a teacher, it [1] is almost impossible to understand without a teacher. Still, this document [2], seems to give a good understandable overview with pointers for further study.
Thanks for developing, and now open sourcing, Harmonic. It has been my favorite HN client for a few years now, something to it that make you not want to leave!
Could you explain in simple terms (if possible) what is the similarity? For context, I worked for a brief time with ART before Y2K in BU CNS, and took a few courses there, but had to leave it for 'reasons'.
