It's considerate communication. Lurching into the next lane .08 seconds after the blinker first flashes says things like "Your life isn't worth the basic consideration and respect of communicating my intentions" and scales up to "I'll communicate, but you're not worth any sort of common courtesy" - that can be upsetting to people.
It doesn't even have to be real. There's huge room for miscommunication. Unpredictable movements and perceived aggression, or unwillingness to be considerate to other drivers on the road, there's a whole wealth of information being processed, regardless of how little is actually real.
Now add the total lack of accountability for the driver's emotional state (don't you love yelling at other drivers, completely free of judgement?), and you can see how things spiral into road rage so relatively easily, even if everyone involved is normally a pretty chill, rational person.
If you're tailgating or brake-checking, or being inattentive and sloppy, you're basically threatening people's lives with a few tons of high speed metal, even if you don't intend that at all.
Ideally, the rules of the road are meant to reinforce a mutual understanding of the game being played. Behavior occurring when expected, proper signaling, observing limits, and making the effort to communicate where possible is a signal that you and the other driver are both operating by the same set of rules, giving you both confidence that neither of you are going to be a danger.
I've seen little "cute" exceptions where locals develop a subculture of dangerous assumptions and then get aggravated when someone from out of town doesn't immediately get it. There are other areas where aggression and what amounts to flagrant disrespect are the norm, so you've always gotta be adaptive, but ideally you get people conspicuously following the same set of rules as a sort of game theoretic optimal strategy for driving.
Some of these companies have (local) law enforcement subscriptions, and default opt-in disclaimers throughout their ToS to make it all tidy and legal.
None of them have contracts with, nor can they sell to, federal agencies. Agencies have to provide a warrant, and the processes are verified through each of the companies' respective legal teams.
Their recordings data is not generally available for sale; that's a legal minefield, but there are official channels to go through. Geofence warrants and things like that aren't conducive to real-time surveillance, and the practice of using those types of reverse-search , differential analysis uses of sensitive data is under review by the Supreme Court; it's thought that they're going to weigh in on the side of the 4th amendment and prohibit overbroad fishing expeditions, even if there's snazzy math behind it.
TLDR; They need to pay the company, either via subscription or direct charge for T&M, require warrants, and the use is limited in scope. It's burdensome and expensive enough that they're not going to be using it for arbitrary random "let's scan everyone's doorbell cams in case there's an illegal immigrant!" situations, but if there's a drug dealer, violent offender, or some specific high value target, they're going to use the broad surveillance tools wherever they can.
Differential analysis is amazingly powerful. If you're in the US - 30 bits is all you need. And not all bits are equal - some come with implicit anchors, allowing you to segment and search efficiently.
If you know the state, the median number of bits needed is 23. If you know the city, around 10 bits is all you need to identify you as a unique individual.
A drunk raccoon with one eye and a missing paw can sieve out 10 bits of information about a particular person.
You can do probabilistic assumptions and segment the population by fuzzy characteristics you get, like stylometry, assumptions about native language, interests, etc. For a giant database like the spies and agencies have, they can do probablistic ID with extreme accuracy based on a tiny number of leaked bits.
If you snag a giant pile of readily available website data, then tag the person of interest based on that data, then any time you process new data, you can get a probability of that new data being associated with an already known person. Set a five nines threshold, or higher, and then assume those matches are legitimate, and you can chip away at all sorts of identity handles. From there, you can start doing contrastive searches, sieving out known quantities, improving the statistical accuracy of those fuzzy parameters.
Deanonymization and such is borderline trivial, consumer compute is about 5 generations past the threshold where a global database would be considered particularly difficult or challenging.
Fingerprinting is very easy, but obfuscating it is incredibly challenging, with all of the implicit, deliberately leaky data transactions that are imposed on us.
These technologies are easily and readily available for whoever wants to pay for it. I personally believe the scope of the privacy nightmare will result in a glut of faux “privacy” oriented services that just serve the monster more.
I can tell you know what you are talking about here, but communicating this to the masses is difficult to why this matters or how bad it is. Unfortunately there’s a strong industry incentive to keep the status quo.
I take the more cynical view privacy is impossible to participate online, let alone anonymity, and would love a disruptor in the space, but at this point, I’ve become so cynical that participating as minimally as possible in it seems the only real solution. But I am not fatalistic, and cannot expect people to consume things the way I do, so I keep trying. You can’t really fight it in any real way, so my approach has been monitoring, observing, informing, and learning.
They're a nonsense company, and trusting them with any information is foolish.
They'll store everything and anything, because data is valuable, and won't delete anything unless legally compelled to and held accountable by third party independent verification. This is the default.
The purpose of things is what they do. They're an adtech user data collection company, they're not a user information securing company.
"US Senator says AT&T, Verizon blocking release of Salt Typhoon security assessment reports"
A US senator is using it for political grandstanding. She is an ineffective twit with no power and no principles, no right under law to receive what she demanded, and she made sure to run to the press with it "see! look, I'm a principled, powerful senator holding those evil corporations feet to the fire!"
The problem is that the vulnerability exploited by salt typhoon is a systemic flaw implemented at the demand of Cantwell and other of our legislative morons.
You cannot have an "only the good guys" backdoor. That doesn't work. People are bad, and stupid, and fallible. You can't make policy or exceptions that depend on people being good, and smart, and infallible.
She's using the inevitable consequence of a system she helped create for her own political benefit. She voted for the backdoor back in 94 against the strenuous and principled objections by people who actually know what they're talking about.
Bobblehead talking points should not serve as the basis for technical policy and governance, but here we are.
> The problem is that the vulnerability exploited by salt typhoon is a systemic flaw implemented at the demand of Cantwell and other of our legislative morons.
Assuming you're talking about CALEA, I find it hard to blame Cantwell personally given that she first joined the House in 1993, and CALEA was passed in 1994. She wasn't in much of a position to "demand" anything against the headwinds of a bipartisan bill passed in both chambers by a voice vote.
The point remains that she's pretending the problem is AT&T, when really it is the US government's demand for a backdoor.
This should be trumpeted as an example of why we cannot mandate encryption backdoors in chat, unless we want everybody to have access to every encrypted message we send.
You can tell this whole thing will be a nothingburger on the government side because the only thing she can actually do is pull in some CEOs to (not) answer questions and receive a congressional tsk tsk.
It's not even a strongly worded letter, lol. Senators and congress people should have to wear shock collars, and on majority polling get hourly "feedback" from their constituency, and for senators, weekly national feedback.
The convention of states project seems like it might be the only way out - there's a shot at implementing term limits, clearing up some of the money in politics issues, no risk of a runaway convention, etc, and we can bypass the people deliberately fouling up the system.
The country is such a dumpster fire. Fucking congressional hearings. The best case scenario is a little video clip that legislators can use to campaign with.
Each election period they have to take a break from eroding citizens' rights catering to lobbyists. The video clips help them pretend they were doing something other than insider trading while in the seat.
>You cannot have an "only the good guys" backdoor.
So what? If I store a document in a private Google doc. I know that technically a Google employee could read it if they really wanted to, but the policies, security, and culture in place make it have a 0% of happening. It's possible to design proper access systems where random people are not able to come in and utilize that access.
So you think there's no Google employees with privileged access gooning on private images, stalking, selling access, disrupting individuals, etc?
Schmidt notoriously had a backdoor, and I'd be far more shocked if executives did not have backdoor access and know all the workarounds and conditions in which they have unaccountable, admin visibility into any data they might want to access.
These are human beings, not diligent, intrepid champions of moral clarity with pristine principles.
>I know that technically a Google employee could read it if they really wanted to, but the policies, security, and culture in place make it have a 0% of happening.
We know it's non-zero as they have already had occasions when it has happened that Google employees used their access to stalk teenagers.
This is such a backwards take. You are ignoring that the system you cite as evidence that secure systems with backdoors can be designed and protected from random access has not been perfectly protected.
And you say it's stronger now.
Ok, so which country or neighbor is going to be the one to hack our national encryption system with a back door the first time? The second time? The third time? Before we manage to get it right (which we never will), what damage will be done by the backdoor? Probably something like Salt Typhoon, which you also conveniently ignore as a counterfactual to your claim.
It not being perfectly protected is by design. Security comes with trade offs.
>Before we manage to get it right (which we never will)
Keep in mind that modern encryption isn't perfect either. You can just guess the key and then decrypt a message. In practice if you make the walls high enough (requiring a ton of guesses) than it can be good enough to keep things secure.
>And such access kicked off an internal investigation and got him fired. Privacy is taken seriously.
The complaints of the victim's parents kicked off an internal investigation, months later. It's not like google found this and took care of it on their own. Also, it has happened before too.
I found my big summer hike. It's the farthest point that can be seen from the highest point near where I live. I can make the hike and then get some pictures of that highest point, from the farthest point away it has a line of sight.
It 100% needs to be online. Imagine you're trying to think about a new tabletop puzzle, and every time a puzzle piece leaves your direct field of view, you no longer know about that puzzle piece.
You can try to keep all of the puzzle pieces within your direct field of view, but that divides your focus. You can hack that and make your field of view incredibly large, but that can potentially distort your sense of the relationships between things, their physical and cognitive magnitude. Bigger context isn't the answer, there's a missing fundamental structure and function to the overall architecture.
What you need is memory, that works when you process and consume information, at the moment of consumption. If you meet a new person, you immediately memorize their face. If you enter a room, it's instantly learned and mapped in your mind. Without that, every time you blinked after meeting someone new, it'd be a total surprise to see what they looked like. You might never learn to recognize and remember faces at all. Or puzzle pieces. Or whatever the lack of online learning kept you from recognizing the value of persistent, instant integration into an existing world model.
You can identify problems like this for any modality, including text, audio, tactile feedback, and so on. You absolutely, 100% need online, continuous learning in order to effectively deal with information at a human level for all the domains of competence that extend to generalizing out of distribution.
It's probably not the last problem that needs solving before AGI, but it is definitely one of them, and there might only be a handful left.
Mammals instantly, upon perceiving a novel environment, map it, without even having to consciously make the effort. Our brains operate in a continuous, plastic mode, for certain things. Not only that, it can be adapted to abstractions, and many of those automatic, reflexive functions evolved to handle navigation and such allow us to simulate the future and predict risk and reward over multiple arbitrary degrees of abstraction, sometimes in real time.
That's not how training works - adjusting model weights to memorize a single data item is not going to fly.
Model weights store abilities, not facts - generally.
Unless the fact is very widely used and widely known, with a ton of context around it.
The model can learn the day JFK died because there are millions of sparse examples of how that information exists in the world, but when you're working on a problem, you might have 1 concern to 'memorize'.
That's going to be something different than adjusting model weights as we understand them today.
LLMs are not mammals either, it's helpful analogy in terms of 'what a human might find useful' but not necessary in the context of actual llm architecture.
The fact is - we don't have memory sorted out architecturally - it's either 'context or weights' and that's that.
Also critically: Humans do not remember the details of the face. Not remotely. They're able to associate it with a person and name 'if they see it again' - but that's different than some kind of excellent recall. Ask them to describe features in detail and maybe we can't do it.
You can see in this instance, this may be related to kind of 'soft lookup' aka associating an input with other bits of information which 'rise to the fore' as possibly useful.
But overall, yes, it's fair to take the position that we'll have to 'learn from context in some way'.
Also, with regards to faces, that's kind of what I'm getting at - we don't have grid cells for faces, there seem to be discrete, functional, evolutionary structures and capabilities that combine in ways we're not consciously aware of to provide abilities. We're reflexively able to memorize faces, but to bring that to consciousness isn't automatic. There've been amnesia and lesion and other injury studies where people with face blindness get stress or anxiety, or relief, when recognizing a face, but they aren't consciously aware. A doctor, or person they didn't like, showing up caused stress spikes, but they couldn't tell you who they were or their name, and the same with family members- they get a physiological, hormonal response as if they recognized a friend or foe, but it never rises to the level of conscious recognition.
There do seem to be complex cells that allow association with a recognizable face, person, icon, object, or distinctive thing. Face cells apply equally to abstractions like logos or UI elements in an app as they do to people, famous animals, unique audio stings, etc. Split brain patients also demonstrate amazing strangeness with memory and subconscious responses.
There are all sorts of layers to human memory, beyond just short term, long term, REM, memory palaces, and so forth, and so there's no simple singular function of "memory" in biological brains, but a suite of different strategies and a pipeline that roughly slots into the fuzzy bucket words we use for them today.
It's not just faces. When recognizing objects in the environment, we normally filter out a great number of details going through the visual cortex - by the time information from our eyes hits the level of conscious awareness, it's more of a scene graph.
Table; chair behind and little to the left of the chair; plant on table
Most people won't really have conscious access to all the details that we use in recognizing objects - but that is a skill that can be consciously developed, as artists and painters do. A non-artist would be able to identify most of the details, but not all (I would be really bad compared to an actual artist with colors and spatial relationships), and I wouldn't be able to enumerate the important details in a way that makes any kind of sense for forming a recognizable scene.
So it follows from that that our ability to recognize faces is not purely - or even primarily - an attribute of what we would normally call "memory", certainly in the sense of conscious memory where we can recall details on demand. Like you alluded to re: mammals and spaces, we're really good at identifying, categorizing, and recognizing new forms of structure.
I suspect we're going to need hypernetworks of some sort - dynamically generated weights, with the hypernet weights getting the dream-like reconsolidation and mapping into the model at large, and layers or entire experts generated from the hypernets on the fly, a degree removed from the direct-from-weights inference being done now. I've been following some of the token-free latent reasoning and other discussions around CoT, other reasoning scaffolding, and so forth, and you just can't overcome the missing puzzle piece problem elegantly unless you have online memory. In the context of millions of concurrent users, that also becomes a nightmare. Having a pipeline, with a sort of intermediate memory, constructive and dynamic to allow resolution of problems requiring integration into memorized concepts and functions, but held out for curation and stability.
It's an absolutely enormous problem, and I'm excited that it seems to be one of the primary research efforts kicking off this year. It could be a very huge capabilities step change.
Yes, so I think that's a fine thought, I don't think it fits into LLM architecture.
Also, weirdly, even Lecun etc. are barely talking about this, they're thinking about 'world models etc'.
I think what you're talking about is maybe 'the most important thing' right now, and frankly, it's almost like an issue of 'Engineering'.
Like - its when you work very intently with the models so this 'issue' become much more prominent.
Your 'instinct' for this problem is probably an expression of 'very nuanced use' I'm going to guess!
So in a way, it's as much Engineering as it is theoretical?
Anyhow - so yes - but - probably not LLM weights. Probably.
I'll add a small thing: the way that Claude Code keeps the LLM 'on track' is by reminding it! Literally, it injects little 'TODO reminders' with some prompts, which is kind of ... simple!
I worked a bit with 'steering probes' ... and there's a related opportunity there - to 'inject' memory and control operations along those lines. Just as a starting point for a least one architectural motivation.
> That's not how training works - adjusting model weights to memorize a single data item is not going to fly.
Apologies; I think I got us all kind of off-track in this comment thread by stretching the definition of the term "fine-tuning" in my ancestor comment above.
Actual fine-tuning of the base model's weights (as one would do to customize a base model into a domain-specific model) works the way you're talking about, yes. The backprop from an individual training document would be a drop in the ocean; a "memory" so weak that, unless it touched some bizarre part of the latent vector-space that no other training document has so far affected (and so is until then all-zero), would be extremely unlikely to affect output, let alone create specific recall of the input.
And a shared, global incremental fine-tune of the model to "add memories" would be a hare-brained idea, anyway. Not even just that it wouldn't work, but that if it did work, it would be a security catastrophe, because now the model would be able to recall all this information gleaned from random tenant users' private chat transcripts, with nothing to differentiate that info from any other info to enable the model (or its inference framework) to compartmentalize it / prevent cross-tenant info leaks.
But let me rephrase what I was saying before:
> there's a way to take many transcripts of inference over a period, and convert/distil them together into an incremental-update training dataset (for memory, not for RLHF), that a model can be fine-tuned on as an offline batch process every day/week, such that a new version of the model can come out daily/weekly that hard-remembers everything you told it
As:
> for a given tenant user, there's a way to take all of their inference transcripts over a given period, and convert/distil them together into an incremental-update training dataset (for memory, not for RLHF), that a LoRA can be rebuilt (or itself fine-tuned) on. And that the work of all of these per-tenant LoRA rebuilds can occur asynchronously / "offline", on a batch-processing training cluster, gradually over the course of the day/week; such that at least once per day/week (presuming the tenant-user has any updated data to ingest), each tenant-user will get the effect of their own memory-LoRA being swapped out for a newer one.
---
Note how this is essentially what Apple claimed they would be doing with Apple Intelligence, re: "personal context."
The idea (that I don't think has ever come to fruition as stated—correct me if I'm wrong?) is that Apple would:
1. have your macOS and iOS devices spend some of their idle-on-charge CPU power to extract and normalize training fulltexts from whatever would be considered the user's "documents" — notes, emails, photos, maybe random text files on disk, etc.; and shove these fulltexts into some kind of iCloud-persisted database, where the fulltexts are PKI-encrypted such that only Apple's Private Compute Cloud (PCC) can decode them;
2. have the PCC produce a new/updated memory LoRA (or rather, six of them, because they need to separately imbue each of their domain-specific model "adapter" LoRAs with your personal-context memories);
3. and, once ready, have all your iCloud-account-synced devices to download the new versions of these memory-imbued adapter LoRAs.
---
And this is actually unnecessarily complex/circuitous for a cloud-hosted chat model. The ChatGPT/Claude/etc version of this architecture could be far simpler.
For a cloud-hosted chat model, you don't need a local agent to extract context from your devices; the context is just "past cloud-persisted chat transcripts." (But if you want "personal context" in the model, you could still get it, via an OpenClaw-style "personal agent"; such agents already essentially eat your files and spit them out external memories/RAGs/etc; the only change would be spitting them out into plain-old hidden-session chat transcripts instead, so as to influence the memories of the model they're running on.)
And you don't need a special securely-oblivious cluster to process that data, since unlike "Apple looking at the data on your computer" (which would upset literally everybody), nobody has any kind of expectation that e.g. OpenAI staff can't look at your ChatGPT conversation transcripts.
And cloud-hosted chat models don't really "do" domain-specific adapters (thus the whole "GPT" thing); so you only need to train one memory-LoRA per model. (Though I suppose that might still lead to training several LoRAs per user, if you're relying on smart routing to different models within a model family to save costs.)
And you don't need to distribute the memory-LoRAs back to client devices; as they can just live in an object store and get just-in-time loaded by the inference framework on a given node at the moment it begins an inference token-emission loop for a specific user. (Which might thus cause the inference cluster's routing to benefit from sticky sessions in a way it didn't before—but you don't need it; the LoRAs would likely be small enough to fetch and load within the ~second of delay it takes these cloud-hosted models to allocate you a node.)
Sentiment is crucial - if you know sentiment is incorrectly oriented, you can capitalize on it. If you know it's correct, you can identify mispricing, and strategize accordingly.
It doesn't even have to be real. There's huge room for miscommunication. Unpredictable movements and perceived aggression, or unwillingness to be considerate to other drivers on the road, there's a whole wealth of information being processed, regardless of how little is actually real.
Now add the total lack of accountability for the driver's emotional state (don't you love yelling at other drivers, completely free of judgement?), and you can see how things spiral into road rage so relatively easily, even if everyone involved is normally a pretty chill, rational person.
If you're tailgating or brake-checking, or being inattentive and sloppy, you're basically threatening people's lives with a few tons of high speed metal, even if you don't intend that at all.
Ideally, the rules of the road are meant to reinforce a mutual understanding of the game being played. Behavior occurring when expected, proper signaling, observing limits, and making the effort to communicate where possible is a signal that you and the other driver are both operating by the same set of rules, giving you both confidence that neither of you are going to be a danger.
I've seen little "cute" exceptions where locals develop a subculture of dangerous assumptions and then get aggravated when someone from out of town doesn't immediately get it. There are other areas where aggression and what amounts to flagrant disrespect are the norm, so you've always gotta be adaptive, but ideally you get people conspicuously following the same set of rules as a sort of game theoretic optimal strategy for driving.
reply