Using Chef to Automate Google Compute Engine
Monday, April 14, 2014
Our friends at Google recently published a comprehensive overview of how to manage Google Compute Engine infrastructure via the various automation platforms available. The Compute Engine team invited us to add our perspective on this topic and what follows here is a look at why we love Compute Engine, how our customers are succeeding with Chef+Compute Engine, and technical details on automating Compute Engine resources with Chef.
Chef is betting on Compute Engine
You’ve often heard us reference the ‘coded business’. In short, we propose technology has become the primary touch point for customers. Demand is relentless. And the only way to win the race to market is by automating delivery of IT infrastructure and software.
This macro shift began in part because of Google’s success in leveraging large-scale compute to rapidly deliver goods and services to market. And when we say ‘large-scale’, there aren’t many, if any, businesses with more compute resources, expertise, and experience than Google.
So it makes a ton of sense that Google would pivot their massive compute infrastructure into an ultra-scalable cloud service. Obviously they know what they’re doing and now everyone from startups to enterprises can tap into Google’s compute mastery for themselves.
Working with the Compute Engine team fits perfectly into not only our view of how the IT industry, and business itself, is changing, but also what our customers want. Choice. Speed (lots and lots of speed). Scale. Flexibility. Reliability.
Why customers love using Chef and Google Compute Engine
Cloud-based delivery
Like the Google Cloud Platform, Chef offers customers all the benefits of cloud-based delivery. New users can get instant access to a powerful Enterprise Chef server hosted on the cloud, no credit card is required, and you can manage up to five instances for free.
When you want to use Chef to manage larger numbers of nodes, you add this capability on a simple, pay-as-you-go basis. Customers can get started using Chef to configure Compute Engine in minutes, start to finish. Ian Meyer the Technical Ops Manager at AdMeld (now part of Google) praises the SaaS delivery model of Hosted Chef:
“Prior to deploying Hosted Chef,” said Meyer, “we did everything manually. It generally took me a couple of weeks to get access to the servers I needed and at least a day to add a new developer. With Chef, I can now add a couple of developers within 20 minutes. Additionally, when we set up a new ad serving system with data bags, the set-up time goes from two to three days to an hour. This is simply one of those tools that you need regardless of what your environment is.”
Speed & Scale
Just as customers are choosing Compute Engine for its speed, our customers appreciate how Chef’s execution model pushes the heavy lifting to the Chef client(s) rather than compiling configuration instructions on the server. Chef stands well above the field with a single Chef server handling 10,000 nodes at the default 30-minute update interval.
Flexibility
Our customers tell us that Chef is more flexible than any other offering. When the situation calls for it, Chef allows advanced users to work directly with infrastructure primitives and a full-fledged modern Ruby-based programming language.
Community
Chef customers can tap into the shared knowledge, expertise, and helping hands of tens of thousands of Chef Community members, not to mention over 1000 Chef Cookbooks. The Chef Community provides a vibrant, welcoming resource for learning best practices. In recent years, high profile vendors have contributed and built on top of Chef including Google, Rackspace, Dell, HP, Facebook, VMware, AWS, Rackspace and IBM.
Google will be a featured partner at this year’s ChefConf. Join Google’s Eric Johnson as he shares technical details about Chef’s integration and future roadmap with Compute Engine.
Chef and Compute Engine: Under the Hood
Chef makes it easy to get started with Compute Engine. Once you’ve obtained a Compute Engine account and configured your Chef workstation, you can extend Chef’s knife command-line tool with the knife-google plugin:
That last command will walk you through a one-time configuration of your knife workstation with Compute Engine credentials.
Now you can use
This command takes the following actions:
And now you have a Jenkins master. This and similar knife commands may be integrated into automation that can also spin up Jenkins tester systems for a complete continuous integration pipeline backed by Compute Engine.
You can then use Chef Server features like search to manage the pipeline as long as you need it. But since Chef makes deployment so simple, and Compute Engine makes it so fast, you can just destroy part or all of it when it’s no longer needed...
… and recreate nodes ‘just-in-time style’ when demand picks back up again.
The quick turnaround on deployment and convergent configuration updates via Chef + Compute Engine allows teams to experiment with developer automation at very low cost.
To get a deeper sense of how you can exploit the capabilities of Compute Engine, please visit our GCE page outlining details around Chef’s knife-google plugin and explore the community library of coded infrastructure.
-Contributed by Adam Edwards, Platform Engineering at Chef
Chef is betting on Compute Engine
You’ve often heard us reference the ‘coded business’. In short, we propose technology has become the primary touch point for customers. Demand is relentless. And the only way to win the race to market is by automating delivery of IT infrastructure and software.
This macro shift began in part because of Google’s success in leveraging large-scale compute to rapidly deliver goods and services to market. And when we say ‘large-scale’, there aren’t many, if any, businesses with more compute resources, expertise, and experience than Google.
So it makes a ton of sense that Google would pivot their massive compute infrastructure into an ultra-scalable cloud service. Obviously they know what they’re doing and now everyone from startups to enterprises can tap into Google’s compute mastery for themselves.
Working with the Compute Engine team fits perfectly into not only our view of how the IT industry, and business itself, is changing, but also what our customers want. Choice. Speed (lots and lots of speed). Scale. Flexibility. Reliability.
Why customers love using Chef and Google Compute Engine
Cloud-based delivery
Like the Google Cloud Platform, Chef offers customers all the benefits of cloud-based delivery. New users can get instant access to a powerful Enterprise Chef server hosted on the cloud, no credit card is required, and you can manage up to five instances for free.
When you want to use Chef to manage larger numbers of nodes, you add this capability on a simple, pay-as-you-go basis. Customers can get started using Chef to configure Compute Engine in minutes, start to finish. Ian Meyer the Technical Ops Manager at AdMeld (now part of Google) praises the SaaS delivery model of Hosted Chef:
“Prior to deploying Hosted Chef,” said Meyer, “we did everything manually. It generally took me a couple of weeks to get access to the servers I needed and at least a day to add a new developer. With Chef, I can now add a couple of developers within 20 minutes. Additionally, when we set up a new ad serving system with data bags, the set-up time goes from two to three days to an hour. This is simply one of those tools that you need regardless of what your environment is.”
Speed & Scale
Just as customers are choosing Compute Engine for its speed, our customers appreciate how Chef’s execution model pushes the heavy lifting to the Chef client(s) rather than compiling configuration instructions on the server. Chef stands well above the field with a single Chef server handling 10,000 nodes at the default 30-minute update interval.
Flexibility
Our customers tell us that Chef is more flexible than any other offering. When the situation calls for it, Chef allows advanced users to work directly with infrastructure primitives and a full-fledged modern Ruby-based programming language.
Community
Chef customers can tap into the shared knowledge, expertise, and helping hands of tens of thousands of Chef Community members, not to mention over 1000 Chef Cookbooks. The Chef Community provides a vibrant, welcoming resource for learning best practices. In recent years, high profile vendors have contributed and built on top of Chef including Google, Rackspace, Dell, HP, Facebook, VMware, AWS, Rackspace and IBM.
Google will be a featured partner at this year’s ChefConf. Join Google’s Eric Johnson as he shares technical details about Chef’s integration and future roadmap with Compute Engine.
Chef and Compute Engine: Under the Hood
Chef makes it easy to get started with Compute Engine. Once you’ve obtained a Compute Engine account and configured your Chef workstation, you can extend Chef’s knife command-line tool with the knife-google plugin:
gem install knife-google knife google setup
That last command will walk you through a one-time configuration of your knife workstation with Compute Engine credentials.
Now you can use
knife with the cookbooks on your Chef server to deploy infrastructure from Chef recipes to Compute Engine instances. Here’s an example where we use Chef to create a Jenkins master node hosted in Compute Engine. Note that this command assumes your local user has previously used 'gcutil' (bundled with Cloud SDK) resulting in a valid SSH Key that has been registered with the Compute Engine Metadata service:
knife google server create jenkins1 -Z us-central1-a -m n1-highcpu-2 -I centos-6-v20140415 -r 'java,jenkins::master' -x $USER -i $HOME/.ssh/google_compute_engine
This command takes the following actions:
- Creates a CentOS VM instance in Compute Engine's us-central1-a zone with machine type n1-highcpu-2
- Registers it as a node named ‘jenkins1’ with the Chef Server
- Configures the node’s run_list attribute as ‘java,jenkins::master’
- Uses the ssh protocol to run chef-client with that ‘master’ recipe from the Jenkins community cookbook on the new system.
Chef Client finished, 19/21 resources updated in 40.207903203 seconds
And now you have a Jenkins master. This and similar knife commands may be integrated into automation that can also spin up Jenkins tester systems for a complete continuous integration pipeline backed by Compute Engine.
You can then use Chef Server features like search to manage the pipeline as long as you need it. But since Chef makes deployment so simple, and Compute Engine makes it so fast, you can just destroy part or all of it when it’s no longer needed...
# Commands like this destroy unneeded nodes knife google server delete tester1 -y --purge
… and recreate nodes ‘just-in-time style’ when demand picks back up again.
The quick turnaround on deployment and convergent configuration updates via Chef + Compute Engine allows teams to experiment with developer automation at very low cost.
To get a deeper sense of how you can exploit the capabilities of Compute Engine, please visit our GCE page outlining details around Chef’s knife-google plugin and explore the community library of coded infrastructure.
-Contributed by Adam Edwards, Platform Engineering at Chef
Follow