It's been my experience that there are 2 types of security people.
1. Are the security people who got into a security because it was one of the only places that let them work with every part of the stack, and exposure to dozens of different domains on the regular, and the idea of spending hours understanding and then figuring out ways around whitelist validations are appealing
2. Those that don't have much technical chops, but can get by with a surface level understanding of several areas and then perform "security shamanism" to intimidate others and pull out lots of jargon. They sound authoritative because information security is a fairly esoteric concept and because you can't argue against security like you can't argue against health and safety, the only response is "so you don't care about security?!"
It is my experience that the first are likely to work with you to help figure out how to get your application past the hurdles and challenges you face viewing it as an exciting problem. The second view their job as to "protect the organization" not deliver value. They love playing dressup in security theater and their depth of their understanding doesn't even pose a drowning risk to infants, which they make up for with esoterica, and jargon. They are also unfortunately the one's cooking up "standards" and "security policies" because it allows them to feel like they are doing real work, without the burden of actually knowing what they are doing, and talented people are actually doing something.
Here's a good litmus test to distinguish them, ask their opinion on the CISSP. If it's positive they probably don't know what the heck they are talking about.
Source: A long career operating in multiple domains, quite a few of which have been in security having interacted with both types (and hoping I fall into the first camp rather than the latter)
It's a good test, however, I wouldn't ask it in a public setting lol, you have to ask them in a more private chat - at least for me, I'm not gonna talk bad about a massive org (ISC2) knowing that tons of managers and execs swear by them, but if you ask for my personal opinion in a more relaxed setting (and I do trust you to some extent), then you'll get a more nuanced and different answer.
Same test works for CEH. If they felt insulted and angry, they get an A+ (joking...?).
> If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
Seems to me like if this is true I'm screwed no matter if I want to "embrace" the "AI revolution" or not. No way my manager's going to approve me to blow $1000 a day on tokens, they budgeted $40,000 for our team to explore AI for the entire year.
Let alone from a personal perspective I'm screwed because I don't have $1000 a month in the budget to blow on tokens because of pesky things that also demand financial resources like a mortgage and food.
At this point it seems like damned if I do, damned if I don't. Feels bad man.
My friend works at Shopify and they are 100% all in on AI coding. They let devs spend as much as they want on whatever tool they want. If someone ends up spending a lot of money, they ask them what is going well and please share with others. If you’re not spending they have a different talk with you.
As for me, we get Cursor seats at work, and at home I have a GPU, a cheap Chinese coding plan, and a dream.
> If someone ends up spending a lot of money, they ask them what is going well and please share with others. If you’re not spending they have a different talk with you.
Make a "systemctl start tokenspender.service" and share it with the team?
This is the part that feels right to me because agents are idiots.
I built a tool that writes (non shit) reports from unstructured data to be used internally by analysts at a trading firm.
It cost between $500 to $5000 per day per seat to run.
It could have cost a lot more but latency matters in market reports in a way it doesn't for software. I imagine they are burning $1000 per day per seat because they can't afford more.
They are idiots, but getting better. Ex: wrote an agent skill to do some read only stuff on a container filesystem. Stupid I know, it’s like a maintainer script that can make recommendations, whatever.
Another skill called skill-improver, which tries to reduce skill token usage by finding deterministic patterns in another skill that can be scripted, and writes and packages the script.
Putting them together, the container-maintenance thingy improves itself every iteration, validated with automatic testing. It works perfectly about 3/4 of the time, another half of the time it kinda works, and fails spectacularly the rest.
It’s only going to get better, and this fit within my Max plan usage while coding other stuff.
LLMs are idiots and they will never get better because they have quadratic attention and a limited context window.
If the tokens that need to attend to each other are on opposite ends of the code base the only way to do that is by reading in the whole code base and hoping for the best.
If you're very lucky you can chunk the code base in such a way that the chunks pairwise fit in your context window and you can extract the relevant tokens hierarchically.
If you're not. Well get reading monkey.
Agents, md files, etc. are bandaids to hide this fact. They work great until they don't.
I wonder if this is just a byproduct of factories being very early and very inefficient. Yegge and Huntley both acknowledge that their experiments in autonomous factories are extremely expensive and wasteful!
I would expect cost to come down over time, using approaches pioneered in the field of manufacturing.
Have these people done the math on how many engineers they can hire in other countries for USD$200k/yr? If you choose the timezone properly, they will even work overnight (your time) and have things ready in the morning for you.
Same. Feels like it goes against the entire “hacker” ethos that brought me here in the first place. That sentence made me actually feel physically sick on initial read as well. Everyday now feels like a day where I have exponentially less & less interest in tech. If all of this AI that’s burning the planet is so incredible, where are the real world tangible improvements? I look around right now and everything in tech, software, internet, etc. has never looked so similar to a dumpster fire of trash.
Yes, exactly this. My biggest issue is how uncurious the approach seems. Setting a "no-look" policy seems cutting edge for two seconds, but prevents any actual learning about how and why things fail when you have all the details. They are just hamstringing their learning.
We still need to specify precisely what we want to have built. All we know from this post is what they aren't doing and that they are pissing money on LLMs. I want to know how they maintain control and specificity, share control and state between employees, handle conflicts and errors, manage design and architectural choices, etc.
All of this seems fun when hacking out a demo but how in the world does this make sense when there are any outside influences or requirements or context that needs to be considered or workflows that need to be integrated or scaling that needs to occur in a certain way or any of the number of actual concerns that software has when it isn't built in a bubble?
Isn’t that the whole point of this approach? Everything is specified just in terms of how the end user will actually use the software, at a high level. Then the LLMs basically iterate relentlessly until the software matches what the end user wants to do.
The biggest rewards for human developers came from building addictive eyeball-getters for adverts so I don’t see how we can expect a very high bar for the results of their replacement AI factories. Real-world and tangible just seem completely out of the picture.
I read that as combined, up to this point in time. You have 20 engineers? If you haven't spent at least $20k up to this point, you've not explored or experienced enough of the ins and outs to know how best to optimize the use of these tools.
I didn't read that as you need to be spending $1k/day per engineer. That is an insane number.
EDIT: re-reading... it's ambiguous to me. But perhaps they mean per day, every day. This will only hasten the elimination of human developers, which I presume is the point.
I think corporate incentives vs personal incentives are slightly different here. As a company trying to experiment in this moment, you should be betting on token cost not being the bottleneck. If the tooling proves valuable, $1k/day per engineer is actually pretty cheap.
At home on my personal setup, I haven't even had to move past the cheapest codex/claude code subscription because it fulfills my needs ¯\_(ツ)_/¯. You can also get a lot of mileage out of the higher tiers of these subscriptions before you need to start paying the APIs directly.
In big companies there is always waste, it's just not possible to be super efficient when you have tens of thousands of people. It's one thing in a steady state, low-competition business where you can refine and optimize processes so everyone knows exactly what their job is, but that is generally not the environment that software companies operate in. They need to be able innovate and stay competitive, never moreso than today.
The thing with AI is that it ranges from net-negative to easily brute forcing tedious things that we never have considered wasting human time on. We can't figure out where the leverage is unless all the subject matter experts in their various organizational niches really check their assumptions and get creative about experimenting and just trying different things that may never have crossed their mind before. Obviously over time best practices will emerge and get socialized, but with the rate that AI has been improving lately, it makes a lot of sense to just give employees carte blanche to explore. Soon enough there will be more scrutiny and optimization, but that doesn't really make sense without a better understanding of what is possible.
Assuming no overtime, one day translates into 3x 8 hour shifts, or 3x engineers. Suddenly, $260k a year buys 3x engineers.
Now, assuming that the dark factory stuff can actually work as conjectured, it will work 24x7, 365 days a year, it does not require annual leave, sick leave, observance of public holidays etc. So $365k (adjusted for 24x7, 365) works out to be a cheap deal.
I do not really agree with the below, but the logic is probably:
1) Engineering investment at companies generally pays off in multiples of what is spent on engineering time. Say you pay 10 engineers $200k / year each and the features those 10 engineers build grow yearly revenue by $10M. That’s a 4x ROI and clearly a good deal. (Of course, this only applies up to some ceiling; not every company has enough TAM to grow as big as Amazon).
2) Giving engineers near-unlimited access to token usage means they can create even more features, in a way that still produces positive ROI per token. This is the part I disagree with most. It’s complicated. You cannot just ship infinite slop and make money. It glosses over massive complexity in how software is delivered and used.
3) Therefore (so the argument goes) you should not cap tokens and should encourage engineers to use as many as possible.
Like I said, I don’t agree with this argument. But the key thing here is step 1. Engineering time is an investment to grow revenue. If you really could get positive ROI per token in revenue growth, you should buy infinite tokens until you hit the ceiling of your business.
Of course, the real world does not work like this.
Is the time it takes for an engineer to implement PRs the bottleneck in generating revenue for a software product?
In my experience it takes humans to know what to build to generate revenue, and most of the time building that product is not spent coding at all. Coding is like the last step. Spending $1k/day in tokens only makes sense if you know exactly what to build already to generate this revenue. Otherwise you are building what exactly? Is the LLM also doing the job of the business side of the house to decide what to build?
Right, I understand of course that AI usage and token costs are an investment (probably even a very good one!).
But my point is moreso that saying 1k a day is cheap is ridiculous. Even for a company that expects an ROI on that investment. There’s risks involved and as you said, diminishing returns on software output.
I find AI bros view of the economics of AI usage strange. It’s reasonable to me to say you think its a good investment, but to say it’s cheap is a whole different thing.
Oh sure. We agree on all you said. I wouldn’t call it cheap either. :)
The best you can say is “high cost but positive ROI investment.” Although I don’t think that’s true beyond a certain point either, certainly not outside special cases like small startups with a lot of funding trying to build a product quickly. You can’t just spew tokens about and expect revenue to increase.
That said, I do reserve some special scorn for companies that penny-pinch on AI tooling. Any CTO or CEO who thinks a $200/month Claude Max subscription (or equivalent) for each developer is too much money to spent really needs to rethink their whole model of software ROI and costs. You’re often paying your devs >$100k yr and you won’t pay $2k / yr to make them more productive? I understand there are budget and planning cycle constraints blah blah, but… really?!
I assumed that they are saying that you spend $1k per day and that makes the developer as productive as some multiple of the number of people you could hire for that $1k.
I didn't always agree with Scott Adams on everything he did and said, but "The Dilbert Principle" taught me more about living in a corporation and management than any other book on business and his dilbert comics were a source of endless wisdom and amusement, which I use often today.
I don't know what the alternative is, but I don't think I've ever found a situation yet where the solution has been His Majesty's Government being able to exercise more control over what people can see and hear.
Okay, everyone here is talking about dick pics but let's be clear here the goal is
>A major expansion of the UK’s Online Safety Act (OSA) has taken effect, legally obliging digital platforms to deploy surveillance-style systems that scan, detect, and block user content before it can be seen.
Do we really believe that no government forever is not going to use this to prevent certain "misinformation" from circulating?
And by misinformation we mean things like MPs breaking COVID lock down rules or "problematic" information about the PM being involved in a scandal, or the list is endless.
Let's be clear this isn't at all and never has been about dick pics this is 100% about being able to control what you can see and share.
I don't understand the downvotes that you are getting.
There is a clear intent to muzzle the population that is going on in Europe with this new legislation and then with Chat control. Those who can't see that need to remove the blinders they have on.
First, it's the nudes and then it's something else. Once there is a capability to filter what can be shared between two adults in private message, then can anyone say that any government is not going to come back for more and ask more and more things to be removed or censored?
2. Those that don't have much technical chops, but can get by with a surface level understanding of several areas and then perform "security shamanism" to intimidate others and pull out lots of jargon. They sound authoritative because information security is a fairly esoteric concept and because you can't argue against security like you can't argue against health and safety, the only response is "so you don't care about security?!"
It is my experience that the first are likely to work with you to help figure out how to get your application past the hurdles and challenges you face viewing it as an exciting problem. The second view their job as to "protect the organization" not deliver value. They love playing dressup in security theater and their depth of their understanding doesn't even pose a drowning risk to infants, which they make up for with esoterica, and jargon. They are also unfortunately the one's cooking up "standards" and "security policies" because it allows them to feel like they are doing real work, without the burden of actually knowing what they are doing, and talented people are actually doing something.
Here's a good litmus test to distinguish them, ask their opinion on the CISSP. If it's positive they probably don't know what the heck they are talking about.
Source: A long career operating in multiple domains, quite a few of which have been in security having interacted with both types (and hoping I fall into the first camp rather than the latter)