iframe sandboxing is wildly underleveraged. I think it's because it doesn't work well with "modern" app development - you need the ability to slice bits and pieces out yourself.
I've been just using plain typescript/html and it's so easy to say "yeah all of that rendered content goes into an iframe", I've got all of d3 entirely sandboxed away with a strict CSP and no origin.
I do hope that iframe sandboxing grows some new primitives. It's still quite hacky - null origins suck and I want a virtual/sandbox origin primitive as well as better messaging primitives.
I think the reason it's under leveraged is that there's so little useful documentation about it - particularly about its support in different browsers.
For something like this that's security critical I'd really like to see each of the browser vendors publishing detailed, trustworthy documentation about their implementations.
The technology itself is very widely deployed due to banner ads, so it's at least thoroughly exercised.
It's multi-faceted. Docs is part of it, but also, no one cares about security and they won't do literally anything to improve security if there's a papercut.
There's just a lot more work to do before I expect to see this. It would solve so many problems though. I personally put d3, markdown rendering, etc, all into iframe sandboxes, which means the entire library could be malicious and it won't matter. But it requires way more effort than I'd like.
Typically you do things like this to either work in restricted envs (distroless) or to evade detection logic. It's not about bypassing a boundary, it's about getting things done in the env you have available.
I've had to remove any of the "knowledge" about me from any agent I use. "As a security engineer, blah blah blah" or "as a rust developer blah blah blah" even though my questions has nothing to do with those topics and they're a huge distraction.
Yeah, I've disabled memory in everything I use. It's super distracting to have it infer connections between conversations where there is none. It's also kind of sleazy feeling. Like, manipulative in the sense that it thinks it knows what I'm into so it's going to weave that into the conversation.
If we didn't have evidence that these things cause something like psychosis in some people, it'd seem innocent. But, since the sycophancy combines with the long-term relationships some people think they're having with matrix math to trigger serious mental health problems, it feels more sinister.
Anyway, having a long-term memory makes them dumber and more easily confused. I don't have any use for a dumb agent.
This assumes that the tokens it outputs are a good description of the tool's behavior. That's not necessarily true though. For example, the LLM may be trained such that a lot of its input data is "LLMs often hallucinate", so the LLM may be biased to say "I hallucinated that" even if there's some more structural issue.
I think there's something here to consider, but it's sort of like assuming that the LLM has reasons for doing things when it only has weights for which tokens are produced - thats the sum of its reasoning.
Maybe it's the case that LLM tokens to correlate to truth values or that this approach actually provides value but there's probably good reason to be skeptical, given that we'd need to posit some sort of causative function of "token outputs" to reasoning about prior behaviors.
There's one extra process that takes up a tiny bit of CPU and memory. For that, you get an immutable host, simple configuration, a minimal SBOM, a distributable set of your dependencies, x-platform for dev, etc.
That’s fair, NixOS avoids the direct stuff from Docker itself but if you’re basing on an Alpine image or something that would probably be more minimal / smaller
Not by default but tools like agent-sandbox.nix (bwrap, seccomp) or other nixpak (just bwrap but more popular) can provide those capabilities if you want in a fairly simple interface
This is why there's an endless cycle of shitty SaaS with slow APIs and high downtime. People keep thinking that scale is something you can just add later.
Let's say you're a team of 1-3 technical people building something as an MVP, but don't necessarily want to throw everything away and rewrite or re-architect if it gets traction.
What are your day 1 decisions that let you scale later without over-engineering early?
I'm not disagreeing with you btw. I genuinely don't know a "right" answer here.
I don't think there's a right answer, you need to sit down and try to think about these problems upfront. What will scaling look like? What decisions will you regret? Make the guesses you can, but don't ignore scale or performance.
I'd argue on the contrary that it's the last decades' over-engineering bender that's coming home to roost. Now too many things have too many moving parts to keep stable.
It's sort of hilarious to compare "talking to people" with analytics. I'm not defending Github here, but you can't possibly think that "talking to 1M customers" is viable.
Not really. (a) People hate responding to surveys and hate emails, you're more likely to lose users than to get data (b) there's no way you're surveying people's in a way that gets you information like "time spent on a page" or "time between commits" or whatever.
This is just nonsense tbh. Surveys and customer outreach solve completely different problems from analytics.
I agree you can't practically get the same information as you could with telemetry.
Survey data is still real data that can be used for "analytics".
Some people also hate telemetry. It feels invasive. I have a guess about what direction the percentage of consumers who hate telemetry is moving toward.
I'm not taking a side on whether a product should add telemetry. I'm rejecting the absurd notion that these suggestions are at all giving the same information.
No one claimed that they give the same information, only that it's viable to produce a good product that solves your user's needs without using telemetry. The whole point is that you don't get the same information, e.g. no private data that the users haven't provided informed consent for upload to your servers.
Kagi has a user forum (as well as listening to comments on other sites like Hacker News) and does not (at least supsosedly) collect telemetry. They seem to be doing fine when it comes to feedback.
So, my perspective on this is that the blog post doesn't motivate me very much.
First of all, the constant framing around Mythos as being capable of tackling "hardened" targets is invalid to me. Or it heavily depends on what "hardened" means. Firefox is an old codebase that has rapidly adopted new features over decades, much of its security has followed after implementation (JIT -> JIT hardening, single process -> multiprocess, etc), it's written in C++, it's extremely oriented towards performance and benchmarking, it has massive attack surface, etc. They have an incredible team working on it - genuinely best in class people are working to make Firefox safer. But does that mean Firefox is a hard target? That's not really my view of it, it's just more expensive than, say, phishing. I would maybe contrast this against Firecracker, which I think is a hard target despite having a tiny fraction of the investment of Firefox.
Similarly, Linux is a codebase with plenty of security investment - lots of research goes into Linux security. It is also an extraordinarily soft target in my opinion. Firefox is far better than Linux, I think, but the point is that Mythos has picked a lot of very specific targets here, called them "hardened", and I think it's misleading from Anthropic's marketing - they frame things like "heap spray" as advanced techniques and that's just nonsense.
Second, this article seems to be trying to convey a few things and then it has a really intense conclusion. I want to separate these out.
The article wants to convey that one of the advantages that attackers have is that the attacker's human-based attack exploration has been more effective than the difficult to scale automation-based attack exploration employed by defenders. The argument is that new AI capabilities change this because automation-based attack exploration via AI will close that gap.
One conclusion is then that because this gap will close that defenders can "win", which presumably means that attacks become too expensive (and therefor attackers move to other paths).
The final conclusion is seemingly that Firefox will have 0 exploitable vulnerabilities.
I want to separate that final conclusion out and focus on the first part.
1. The argument seems to rely on the advantage attackers have being singular. Attackers have a lot of advantages, so I'm not sure that removing this one will be sufficient.
2. It seems incorrect to me that AI will be so radically effective at finding vulnerabilities that attackers won't be able to find enough to build a chain. In fact, I think this is almost certainly false for a codebase like Firefox. It would be one thing if Firefox froze it's codebase today and spend years on hardening what's there, maybe I'd buy that... but no, I don't buy it at all for Firefox as it exists in reality.
3. This all relies on Firefox reducing the bug density such that viable chains can not be built. There is, I think, some threshold in which a codebase's bug density is so low that full attack chains are not viable. I do not think Firefox will ever reach that low of a threshold. The claim here seems to be that bugs can be found so quickly that the threshold can be met, or it rejects my threshold idea entirely.
I don't think any of this makes sense to me, personally. I don't think we've ever seen a moving codebase made significantly safer via any "bug squashing" technique. The value of squashing bugs is to track where bugs crop up so that you are informed about what mitigations should be built; you see a lot of vulnerabilities that leverage JIT RWX? Time to harden how JIT pages are emitted. Any bug isn't the interesting part, it's the mitigations and layered defenses that help. Fixing 100 XSS vulns on a website will never be as good as deploying a CSP, or updating your CSP, etc. This has always been the case, I don't think AI is so radically different that it will change this.
So anyway, I sort of don't buy anything the blog post says so far. And then it ends on this note:
> The defects are finite, and we are entering a world where we can finally find them all.
As far as I'm concerned, the defects in Firefox should be considered ~roughly infinite. But even if we say "no, there's some finite number of them", the idea that we'll drive to zero is just... not something I'm going to take seriously.
Earlier in the article "Nevertheless, we’ve all long quietly acknowledged that bringing exploits to zero was an unrealistic goal." is stated so it does seem to me that whoever wrote this actually believes that a zero vulnerability firefox is achievable. I think that it's not even achievable to reach a low enough threshold to break entire chains of exploits, so obviously I think it's not correct to say zero is achievable.
I think I can probably justify this to some degree.
1. We've never seen any bug squashing technology, including insanely highly leveraged ones like fuzzing, meaningfully reduce exploitability in similar projects. I doubt anyone thinks that 0 days are rare ITW because of fuzzing when it's very obviously because of sandboxing and mitigations. Is that contentious? Feel free to push back.
2. Rice's Theorem makes it seem highly implausible that we can reach a true "zero point" computationally through formal means, which to me implies that AI would have to be so effective at exploring insanely massive state spaces for a moving target for so many different properties, that it just doesn't sound realistic to me.
I am extremely skeptical of a lot of the statements made in this post. I do not think Mythos will help defenders by squashing bugs at all in a codebase like Firefox, I do not think Firefox is safer just because they patched 500 vulnerabilities, I do not think Firefox will meaningfully reduce vulnerability counts long term with Mythos, and I do not think that this should change their strategy of using fundamentally safer technologies / implementing mitigation techniques.
I will weakly predict that AI usage for bug squashing will look quite a lot like fuzzing. A win in the short term, just another automated tool in the long term, and the highest impact will be watching it for trends. I think AI will likely fare worse than fuzzing though.
Unfortunately it's unclear whether it was Mythos, an earlier model, or even an eagle-eyed employee.
I tend to agree that bug squashing your way to perfectly secure software is unlikely, but there are plenty of projects that managed to fuzz/test/audit their way to making it much harder to find serious vulnerabilities. If we can do the same again with LLMs in a way that leaves the remaining vulnerabilities out of reach of anyone except extremely skilled humans (perhaps with LLM assistance) then that's still an OK outcome that buys us time to build stronger foundations.
> but there are plenty of projects that managed to fuzz/test/audit their way to making it much harder to find serious vulnerabilities
Agreed! But I think those projects have certain things in common, like being tightly scoped, slowly developed, and built with safety in mind from day 1.
I don't think that any of the projects that have managed to meaningfully improve safety through fuzzing have the same qualities as projects like Firefox, Linux, etc.
I've been just using plain typescript/html and it's so easy to say "yeah all of that rendered content goes into an iframe", I've got all of d3 entirely sandboxed away with a strict CSP and no origin.
I do hope that iframe sandboxing grows some new primitives. It's still quite hacky - null origins suck and I want a virtual/sandbox origin primitive as well as better messaging primitives.
reply