"If you disable any direct shell access to the server and force all configuration changes to be applied by running the recipe from version control, you have an excellent audit mechanism that ensures every change to the environment is logged."
Um. Yeah. You're also creating a situation where, should anything possibly go wrong (now, how could that happen), you cannot hop on the server, diagnose, and possibly do some quick-try fixes. I can see limiting shell to a small trusted subset of colleagues (sorry, devs), but not eliminating it altogether.
Automated management, "system-as-code", devops, and all that has its place. The problem with virtually all existing systems for automated configuration management (cfengine, puppet, chef, Oopsware), is that they are highly non-transitive, and effectively bin years or decades of administration experience. They're fine once you've worked out the kinks, but as anyone who's worked with these tools can tell you, the one thing they're best at is screwing up all of your systems simultaneously.
"Etch" (http://sourceforge.net/projects/etch/) is one tool I've looked at briefly that appears to take a different tack, and is amenable to taking on-host changes and incorporating them into the configuration management system itself. One of the criticisms I've seen of it is that it's rather Linux and/or specifically Debian-centric, which may well be as Debian offers some very strong tools for managing, assessing, and maintaining system state (policy, APT, debconf). While dependencies can be painful when they keep you from doing what you want to do, as with most good safety systems, it's generally because you really don't want to go there (and if you do, there are means, within the framework provided by Debian, to get you there).
One way to avoid snowflake systems is to use tools that manage dependencies, stick within them to the greatest extent possible, and where that's not an option, to put your own modificiations within that same framework.
I just start writing code to provision/configure the machine for the role in question and don't stop until I have something that can reliably take a blank-slate server and have it rolling by the end of the function.
Easiest way to do devops I've seen yet. I didn't care for chef/puppet/cfengine.
Particularly since I can cherry-pick servers for testing pretty easily with fabric before I run the code against the rest of the machines.
I did the same thing. What's nice about the Phoenix approach is that you start with a known state (base Ubuntu install or whatever) and end with a known state (running your setup) and completely ignore all other possible states. A tool like chef or puppet has to have a lot of complexity around examine a node, determine what state it's in and figure out how to get it to the desired state.
I also use boto in my fabfile, so that I can run a single command to launch an EC2 instance, install & configure software on it, save an AMI and terminate the instance. It's a like having a Makefile that compiles machine images:
> fab build:<role> <region>
I wouldn't go as far as Fowler's "disable the shell" idea, but it doesn't take much discipline to do all significant changes by building new AMIs.
If you prefer some other IaaS, fine. The point is that fabric can drive both the virtualization API and the actual set up of servers over SSH. I have a fake region called "dev" that runs the build against a vmware image on my local machine. It could work with Softlayer or Joyent or Rackspace or CloudFoundry or whatever.
Fabric and an IaaS provider is a really nice way to implement the PheonixServer pattern.
I decided to use Fabric as well, because then I could just use apt to install packages, which strikes me as a far superior way than building everything from source.
However, it took me days to complete the script and make it moderately reusable (and completely idempotent and able to handle different configurations), and it still doesn't do as much as I want it to. For example, I just noticed that Ansible[1] can tell whether a change needs to restart a service (e.g. if a config file changed) and only then do it.
That's pretty useful, but my script can't really handle it.
Do you have a large number of servers or just smallish deployments? Fabric is cool, but at some point the pull model starts to be better than push. You probably don't want to actually wait at the console and try to spot errors when deploying 100+ nodes - and this is where chef/puppet is actually nice.
The primary weakness of my approach is that for convenience's sake, I rely heavily on apt-get. So hypothetically I'm tied to the debian family. (we use Ubuntu LTS of a couple different versions)
Strictly speaking this disadvantage is unnecessary, you could do all from-source builds using fabric if you wanted. Didn't seem like a constructive use of my time though.
While that page has a long list of things they do, the important bits relevant to your comment are
1. a tighter focus on idempotence than Fabric
2. an easy-ish way to integrate package management so you could potentially use the same script to kick off either yum or apt depending on the box
Oh God, why do they put hosts in /etc/ansible? Why would I corrupt my system installation with project-specific files? What I want to do is make a "<project>-deployment" folder and have everything related to deployment there.
Why the hell would I store anything about deploying one of my projects in /etc/?
The original reason I abandoned chef and puppet is that I didn't like something getting between me and the shell.
Why would I make the same mistake twice? My only real dissatisfaction with Flask is that it cannot handle dispatching its work in parallel. Not a big deal though.
I had a modest epiphany some weeks back thinking about server configuration management: if you want to manage configurations on a given node (to use Chef lingo), it's best to work within a system that facilitates and/or encourages this.
For a number of reasons, Debian (or faithful derivatives) comes closest to this of any platform I've encountered.
My startup has been in production on EC2 for 6 months, and I think we've never had a server up for more than 3 days, and never booted a production box from an AMI more than 2 weeks old.
Once you have more than a few servers you will go crazy if you don't have a good configuration management setup. The real advantage of using a tool like puppet over something home grown is that you can hire someone to come in and manage puppet who will be able to understand how your automation works without having to have your system admin sit down and explain the 1000 little arcane perl scripts that make everything work.
We use client-server puppet, but require all the updates to be manually run, which makes rollouts a little bit more deterministic and avoids the "Hey we just changed puppet and everything broke" effect.
This idea - combined with the Phoenix Server concept - seems very powerful to me.
In an odd way, it feels like this is main advantage that PaaS offerings like Heroku have: they force you into the mindset of not relying on hand tinkering with the server config.
Um. Yeah. You're also creating a situation where, should anything possibly go wrong (now, how could that happen), you cannot hop on the server, diagnose, and possibly do some quick-try fixes. I can see limiting shell to a small trusted subset of colleagues (sorry, devs), but not eliminating it altogether.
Automated management, "system-as-code", devops, and all that has its place. The problem with virtually all existing systems for automated configuration management (cfengine, puppet, chef, Oopsware), is that they are highly non-transitive, and effectively bin years or decades of administration experience. They're fine once you've worked out the kinks, but as anyone who's worked with these tools can tell you, the one thing they're best at is screwing up all of your systems simultaneously.
"Etch" (http://sourceforge.net/projects/etch/) is one tool I've looked at briefly that appears to take a different tack, and is amenable to taking on-host changes and incorporating them into the configuration management system itself. One of the criticisms I've seen of it is that it's rather Linux and/or specifically Debian-centric, which may well be as Debian offers some very strong tools for managing, assessing, and maintaining system state (policy, APT, debconf). While dependencies can be painful when they keep you from doing what you want to do, as with most good safety systems, it's generally because you really don't want to go there (and if you do, there are means, within the framework provided by Debian, to get you there).
One way to avoid snowflake systems is to use tools that manage dependencies, stick within them to the greatest extent possible, and where that's not an option, to put your own modificiations within that same framework.