I think this is the first YC company with french founders so congratulations to the team for that, and most importantly congrats for your launch I look forward to test it! Dotcloud seems awesome!
I want it to be known that Solomon and Sebastien are awesome. They set us up with hosting for our project when we applied to YC several months ago and Solomon took several hours of his own time to really sit down with me to walk us through getting setup. Their system is really kick-ass and amazing. We were working with Django for our framework, and configuration for a project is really straightforward. It was awesome not to have to worry about server configurations at all and just focus on getting work done.
Every service has it's own local mail server (postfix). It can be preconfigured to act as a relay for other mail services like sendgrid or google mail.
As a developer struggling to also be a sysadmin I find this very interesting. We were just about to start looking for a sysadmin to manage what we're doing, so it will be interesting to see how DotCloud works. We're Python focused, so this looks to be right up our street. Curious to know what others do in terms of managing the servers etc - do you think that it could be dangerous to rely on a service like this? Do you think its better to understand from end-to-end what you're using and why?
I HIGHLY recommend Dotcloud. We have been using them for a couple of months and its insane how easy it is to get setup. The Dotcloud guys have done a spectacular job in making it painless to use their system. Ours was a Python/Django app and just took a few lines of configuration to get up and running.
Is there any reason you're running an outdated version of Nginx (0.7.65 vs 0.7.68 vs 0.8.54)? It's one major revision behind current and a few minor behind stable. Which brings me to a question I had lingering in my mind, what is your "software" upgrade process since users have no control over it?
Another question, do you cache non-static objects (cookie based caching when proxy_pass'ing)? If so, how are you handling the different cookies for each particular app? Can this be turned off?
- 0.7.65 is the version currently used in ubuntu lucid. We use it only as a proxy.
- On services like python-wsgi we currently use 0.8.52 and we will soon upgrade to 0.8.54
- On the ruby-passenger service we follow the nginx pulled by passenger
So it's always dependent of the context. The idea is to keep the stacks stable and secure at all times.
Once we have tested and approved an upgrade we create a new revision of the corresponding service and deploy it across the platform. If the upgrade requires downtime we schedule a rolling maintenance window and notify users.
On caching:
We don't cache by default (but this could change). Users have the option to add caching services like varnish.
Cassandra and Hadoop alone need a good chunk of domain knowledge to keep them running smoothly. Seeing them listed casually next to so many other deployment stacks makes me feel slightly dizzy.
I'm definitely looking forward to see if you'll be able to pull this off.
You're correct, there's a beefy learning curve to properly configure, fine-tune and scale each of these components. Our job is to tackle that learning curve so you don't have to.
As a developer, you get a little more value out of that deal every time you want to play with something new.
It works for us, because after a while you start seeing patterns in proper automation. There are only so many ways to store, modify and move bits around.
You're preaching to the choir, most of my day- and nightjob involves just that. ;-)
That's where a bit of skepticism stems from, as I've dealt with both of these databases first hand, also various other stacks ranging from java, python, ruby, even lua.
The not-so-stellar uptime record of Heroku shows that making one size fit all is quite hard, even when you're doing it only for a relatively small set of components.
Doing it well for nearly all of them is nothing short of the holy grail in systems management.
I'm glad there is going to be more competition in this space.
Currently, I don't really have the need or desire to manage my own Hadoop cluster, but I haven't been completely satisfied with Amazon's Elastic Map Reduce offering either.
Definitely interested in any beta tests and/or feedback for that!
Congrats to the dotcloud team! I already had the luck to work with seb and sam from the dotcloud team and I can testify that they are extremely skilled and reliable guys. They won't fail you nor your servers :)
My 2 cents
I like the courage trying to manage this. Two thumbs up.
To be honest, I am skeptical, that a newcomer startup can do the heavy lifting of supporting such a big stack. Each of these has a lot of specialities that need to be known and to fight.
Background: Our stack uses appjet, nginx, varnish and couchdb. Each of these has different challenges. Think of optimization, scaling, resource limiting, leveling, monitoring, statistics and enforcing governor limits/notifying customers. We needed over a year to establish this. I don't want to sound negative, just think about all this, what we needed learn.
If you think about it, isn't it incredible that you had to invest so much time and money in building out your infrastructure? How many web businesses had to re-invent the wheel to get to the same result?
Yes, it's hard work for us. But 90% of the work is the same across all deployments. We take advantage of that fact to offer massive savings in engineering and sysadmin time.
Fantastic idea; wish I had thought of it. I'm very curious to see how efficient your "glue" code is between the components. It's one thing to have a modular hands-off system, it's often another to have that system perform reasonably well. What level of system administration do you expose/hide to the developer? How do you plan to make money?
Our focus is on giving control a developer needs: how do I install language-specific packages? How do run my tests? How do I pass settings to my app? How do I run that custom build command?
We don't offer root access. But we're working on customization tools that will make you forget you ever had to ssh into a server directly.
As for pricing: we're already charging a few test customers, and will expand paid plans to all beta users very soon.
If I understand correctly, your basic assumption is that a developer only needs access to the configuration options. Essentially, a developer isn't going to hack the code, they just modify the package configuration as needed. I think this would be a fair assumption and targets a huge number of users looking to leverage the cloud.
As for pricing, looking back, I realize that my question may have been a bit blunt and offensive. What I was really curious to know was whether you were going to be charging "per instance", "per hour", "per instance-hour", "per package", etc. For example, is Apache+MySQL on a single machine (if that's possible) less expensive than Apache on one machine and MySQL on another? What about compared to Nginx+Cassandra? (FWIW: You don't have to answer this directly if you want to keep the cards close while in testing...)
None the less, I wish you the best. (signed up for a beta.)
Heroku has a fixed stack with tiers that they could tune, tweak and horizontally scale. Is that what dotcloud provides for each of the listed technologies ?
very interesting. I am interested in seeing how it actually works though. Its still behind the doors. Hope to get invite. (btw, heroku is not just deployment platform, it helps u scale as well. Interested in seeing if dot cloud helps in that fashion)
Yes, if the host is good. I love .Net but I'm continually jealous of things like Heroku which really support a fast and nimble development approach favouring smart developers rather than heavyweight process. Azure isn't quite there.
I like the strategy of using other YC companies as alpha customers. If any of the companies on the platform get traction it looks pretty good for DotCloud as well.
We're taking one step at a time. First we'll give you metrics to help you decide when to scale up or down. Eventually we'll offer to take that decision for you, with a configurable ceiling.
In our experience, for most customers 10-second manual scaling is just as good as auto-scaling.
Our problem with Heroku was that we didn't know how many units to allocate, and ended up way over-spending. But if we had tweaked it to be just right, we would have been screwed by an unexpected surge.
It means that you type 'dotcloud scale' and wait 10 seconds :)
We offer resource usage data, so you can make an informed decision when scaling. You are correct that a very sudden surge can still screw you - but we are preparing an alert feature to help you mitigate this.
I don't mean to be negative, but I like your product a lot less without autoscaling. If it's hard to do, then it'll be nearly impossible for me; if its not hard, then I'll be annoyed that I'm writing code that you guys could have trivially written.
Database migration is a complex subject:-) You have full control over you database (with sql prompt and everything). That means that you can use any migration tool you like.
If we start seeing a recurrent pattern we'll try to automate.
Cocorico :)