>"Someone else is always auditing the code and will save me from anything bad in a macro before it would ever run on my machines." (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
He got some hate for that in the replies but I think it is a great point. Pardon my snark, but every "security expert" that voiced their opinion over the last couple of days, yet failed to recognize the situation for almost a month, should reassess their reaction once the anger subsides.
The current macro situation is dangerous (and wasteful), hopefully some real progress can be made.
> (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
> Pardon my snark, but every "security expert" that voiced their opinion over the last couple of days, yet failed to recognize the situation for almost a month, should reassess their reaction once the anger subsides.
I'm not a "security expert", but I wondered why I hadn't noticed this situation, since I have the habit of always diffing between the previous and the current version of the crates in ~/.cargo/registry/src whenever cargo tells me it downloaded a new version of a dependency, and a new binary file would stick out like a sore thumb. But real life has intruded for me in the last few weeks, and the last serde_derive on my ~/.cargo is still 1.0.166, so it seems I luckily avoided all this mess.
(Yes, I know that reviewing the code after it has already compiled is suboptimal, and obfuscated malicious code or even the "new dependency" trick could easily pass through my cursory inspection; but at least it can be reviewed by comparing it with the previous version, unlike a precompiled binary blob where even small changes can lead to a huge difference between one binary and the other, not to mention that comparing objdump disassembly output is painful.)
This isn't a serious comment. It really does not matter whether "almost anyone" became aware, it matters whether the sorts of people who pay attention to this became aware. OS packagers became aware of this super quickly and were working quietly to get this fixed. My understanding is that some security teams at large companies quickly flagged this as well and likely reached out via private channels.
The public outrage happened weeks later but it's not as if no one was paying attention.
> reassess their reaction once the anger subsides.
You have completely missed the point. It is because things go unnoticed that security minded folks are upset. We don't get off from finding security problems, we get off on being safe to begin with.
But the whole reaction here was 'this makes it more difficult to find security problems' (running a binary is not intrinsicly a security problem, if instead you are just going to compile it and then run it, it's just harder to audit a binary) and yet no-one was even doing the easy bit (auditing the source code). If someone had injected malicious code into the build.rs file it would have exactly the same effect.
> yet no-one was even doing the easy bit (auditing the source code).
Someone did: the Fedora maintainer who raised the issue. One of the reasons I avoid installing from wild-west package managers like Cargo/NPM/..., is specifically because Debian/Fedora/... maintainers performs some basic checks like this so I don't have to.
Respectfully, I think we agree? The current proc macro situation allows security vulnerabilities and a potentially dangerous change went unnoticed for weeks. This RFC aims to address that, at least in part. If the comments I read reflected your view, I wouldn't have made my snarky remark. Unfortunately, I read many comments full of fury aimed at dtolnay while ignoring the system that enabled it.
Isn't this exactly the kind of adversarial behavior that got the University of Minnesota banned from contributing to the Linux kernel? Made an intentionally bad-faith commit, then released it in order to prove a point about how insecure the software ecosystem is?
Great, he made his point that we're all dummies and we'll all just blindly run any ol' code he sends us. In the process he did very serious damage to his reputation and any trust relationship he had with the broader community. Other projects have banned people for this kind of behavior.
Not exactly, UofM was accused of deliberately contributing security flaws to the Linux kernel. dtolnay proved a point, but it's not like his binary included a malicious payload.
If dtolnay planned ahead of time and indeed expected everything, kudos to him, because he had the guts to trade his credence with public good (I don't). That said, I would appreciate a confirmation that this is a very exceptional stunt that won't be repeated again.
I feel for him/her. When you provide a usefull service in OSS, this is the result of the slightest transgression, real or imagined.
I also think this displays 'emperors clothes' behaviour. In a perfect world, many eyeballs make all bugs shallow, and long term, it actually mostly works. But short term, we collectively severely fall short of our ideals. I think part of the harshness an OSS contributor receives in this kind of situation comes from everyone realizing we've been caught with our pants down.
It's human to shoot the messenger, but let's not forget this only happens because someone worked hard enough to make the whole world trust him or her.
Thanks to dtolnay for boldly doing what it took to make some real progress on the problem of proc macro compile times. It's unfortunate that we reacted the way we did, driving him to abort the experiment. I'm sorry I contributed to the pile-on with my first comment on the infamous GitHub issue, the one I directed specifically at him. I'll be slower to react next time.
I disagree that what was done with good but agree with matklad [0] about basic expectations of transparency and communication when you maintain de facto standards. This is too big of an "experiment" to run in production code on the ecosystem without first seeking input of drawing attention to it. If any of this was done to "push the conversation forward", it is done in an unhealthy way that burns a lot of bridges.
As for making progress on the idea of improving build times, I agree with kayabaNerve from the Pre-RFC [1]
> It's effectively impossible to fairly review this on its merit now, nor to say it isn't being reviewed on an accelerated time span than it would otherwise have been.
Due to the way that the derive feature was causing the dependency chain in Cargo to be longer than necessary, thereby making it impossible for cargo to compile multiple crates in parallel.
The dependency chain is what led to the longer compile times and why the prebuilt binary existed. It was to shave time off how long it would take to get serde with the "derive" feature enabled compiled.
I’ve seen a lot of complaints about this. I would love some insights into the author’s intentions, though. Why did they choose binary distribution? They must have had a reason.
It makes a lot of users's crates faster to compile from scratch, and compile time is a common complaint in Rust. It could be celebrated as a big total ecosystem win for compilation speed, but other issues overshadowed the whole thing.
I think also one of the issues was that it wasn't opt-in, it wasn't deterministic from day one (I truly believe it should have), and it broke building when using git references as the detection was just triple based.
Nice to see the maintainer stop being so stubborn.
I wish I could say this left a good taste in my mouth as a strong proponent of Rust.
Especially after dtolnay banned me on the repo after I commented on the PR. Can't even give thumbs up on someone else's comment let alone open an issue or a PR.
Interesting to see how glorified janitorial roles make people think they have some real power.
> Interesting to see how glorified janitorial roles make people think they have some real power.
Not hard to imagine why he would have blocked you or the possibility you're not telling the whole story. This combination of entitlement and disregard is thoroughly toxic.
Some of his crates are a bit opinionated, and I don't agree with all the opinions, but the amount of effort and ingenuity that has gone into them is breath-taking, and we get the results of it without paying a dime. Maybe that deserves some gratitude and grace...
even if it's a 'glorified janitorial role', you ain't paying him for it, so I don't really understand your objection to him not wanting to deal with your feedback on it (especially if it's just participating in a dogpile and not actually a useful contribution).
Authors comment:
>"Someone else is always auditing the code and will save me from anything bad in a macro before it would ever run on my machines." (At one point serde_derive ran an untrusted binary for over 4 weeks across 12 releases before almost anyone became aware. This was plain-as-day code in the crate root; I am confident that professionally obfuscated malicious code would be undetected for years.)
He got some hate for that in the replies but I think it is a great point. Pardon my snark, but every "security expert" that voiced their opinion over the last couple of days, yet failed to recognize the situation for almost a month, should reassess their reaction once the anger subsides.
The current macro situation is dangerous (and wasteful), hopefully some real progress can be made.