We have recently taken a huge step toward being truly lean at Earbits, and it's not easy. Particularly when you've got as much product as we do, and have goals with deadlines, it's hard to say, let's stop for a minute and position ourselves to start doing this smarter.
I think it's easiest to get it wrong if you are more focused on mimicking examples from the book instead of applying the underlying principle, which to me is: Validate or invalidate your assumptions with the smallest application of resources possible, and double down on things that move the dial.
In our business, that can be pretty tough to do. There are a lot of moving parts to our product and it's not exactly a funnel. But you have to break each experiment into parts. For example, we have an assumption that people want more things to do on our site, and that they want more control over what to listen to. But before we go building a bunch of complex things for them to do, we decided to see if they would click on a series of tabs that seemed to "do something cool". And they did. So now we know that they're looking for more to do. Next, we fake a few different things that they could do, with real features when easy to build, and smokescreen tests when not. Only if a big percentage of users try to use a feature do we truly build it, and perhaps then we'll try skinning it different ways, since that's easy. If the final feature increases our core metrics (a whole other discussion), we keep it. If not, we don't, because maintaining a feature that doesn't improve our company health is a waste of resources.
Admittedly, you still have to take leaps of faith. The point is to make them as short a leap as possible. But I'll tell you that while it can be a bit painful to work in this manner, I feel huge validation killing features, running A/B tests and making decisions based on data. We may have come a long way flying blind, but I have a feeling 2012 is going to yield unreal results for us thanks to this disciplined approach.
We have recently taken a huge step toward being truly lean at Earbits, and it's not easy. Particularly when you've got as much product as we do, and have goals with deadlines, it's hard to say, let's stop for a minute and position ourselves to start doing this smarter.
I think it's easiest to get it wrong if you are more focused on mimicking examples from the book instead of applying the underlying principle, which to me is: Validate or invalidate your assumptions with the smallest application of resources possible, and double down on things that move the dial.
In our business, that can be pretty tough to do. There are a lot of moving parts to our product and it's not exactly a funnel. But you have to break each experiment into parts. For example, we have an assumption that people want more things to do on our site, and that they want more control over what to listen to. But before we go building a bunch of complex things for them to do, we decided to see if they would click on a series of tabs that seemed to "do something cool". And they did. So now we know that they're looking for more to do. Next, we fake a few different things that they could do, with real features when easy to build, and smokescreen tests when not. Only if a big percentage of users try to use a feature do we truly build it, and perhaps then we'll try skinning it different ways, since that's easy. If the final feature increases our core metrics (a whole other discussion), we keep it. If not, we don't, because maintaining a feature that doesn't improve our company health is a waste of resources.
Admittedly, you still have to take leaps of faith. The point is to make them as short a leap as possible. But I'll tell you that while it can be a bit painful to work in this manner, I feel huge validation killing features, running A/B tests and making decisions based on data. We may have come a long way flying blind, but I have a feeling 2012 is going to yield unreal results for us thanks to this disciplined approach.