Jeremy's Thoughts

Thoughts on DevOpsDays NYC

Sat, 19 Jan 2013 17:01:52 GMT

I’m currently on the train on my way back from DevOpsDays in Brooklyn. The conference was great — lots of smart people facing a lot of similar problems and trying to see what we could learn from each other. The scale was small, with only like 100-ish people present and not a ton of huge, in your face sponsorship. And the venue was a college campus. And so I kept making these comparisons in my head to LUG meetings, installfests and small scale Linux conferences.

Obviously the subject matter was a bit different — talking about and thinking about running large scale production infrastructures is a little bit different than the next cool Linux distribution. This tended, I think, to more discussion around patterns and best practices than about the specifics of “you should do X to get Y to work”. So a higher level and more abstract discussion.

The composition of the audience and attendees was a pretty similar make-up. Linux events always had a strong majority of the attendees who self-identified as sysadmins and then there tended to be a smaller number of developers. And many of the latter group had ended up in that camp due to necessity. The breakdown for DevOpsDays felt pretty similar with an interesting twist where there were speakers who said they were (paraphrasing) “developers first and fell into operations because they needed to”.

One thing that felt more evolutionary than anything else was that the side channel discussion for the event took place on Twitter rather than on IRC. I have (fond) memories of many conferences where attendees sat in an IRC channel and then basically continued to interact on IRC long after the conference had ended. In fact, I made many friends in this fashion. Similarly there was an ongoing discussion on Twitter using the #devopsdays hash tag and I have followed (and am being followed by) a number of the other attendees and hope to keep in touch and call them friends in the future.

And maybe the thing that struck me the most strongly was where people were “from”. Not in the sense of where they lived but rather where they worked. The attendees were almost all from startups. We were in Brooklyn and not the heart of downtown Manhattan, but NYC is probably home to more financial services companies than anywhere else in the world. And all of those companies have *many* people working in software dev and operations-y roles. But they weren’t there.

So it feels like “the DevOps movement” is going through a similar growth and evangelism pattern as open source and Linux did years ago. Maybe that’s why it feels so comfortable to me.

Originally published at Jeremy's Thoughts. Please leave any comments there.

Can’t code with AWS outages so blogged instead

Mon, 22 Oct 2012 23:07:56 GMT

Although I haven’t really talked about it here, I joined a new startup a couple of months ago called Stackdriver where we’re working on building a hosted solution to make infrastructure monitoring and management suck less for users of the public cloud. After a having to duct tape the various pieces together a couple of times now, it’s super clear that the need is there so it’s exciting to be working on solving it. More on the side of being at a very early startup to come in the future.

Today I had planned to do some work around some of our provisioning and deployment code and Amazon had another EBS outage making the AWS API pretty unavailable for much of the afternoon. So after doing some other things, I took a look at what fails along with EBS to help us remember what fails along with EBS and thought it was interesting enough to share.

Originally published at Jeremy's Thoughts. Please leave any comments there.

A repo for the chef-omnibus packages

Wed, 30 May 2012 00:28:25 GMT

I finally got around to trying the Chef omnibus installer and it’s a step up from what I was doing previously but still not great. Grabbing a shell script with curl or wget and piping it to your shell is an anti-pattern which I wish had never taken off. Luckily, in this case, the shell scripts is just pulling down an rpm and installing it. One step nicer would be if there were just a repo that you could use via yum and have things a yum install chef-full away. And as I thought that this afternoon, I remembered the baseurl support in createrepo. Thus, without further ado, I’ve thrown together a quick set of repos that just point to the files in the opscode s3 bucket and minimizes the amount of storage I have to do If you want to use them, just drop a file into /etc/yum.repos.d named something obvious like chef.repo

[chef]
name=Chef Omnibus Packages
baseurl=http://katzj.fedorapeople.org/chef-omnibus/el$releasever/$basearch
enabled=1
gpgcheck=0
#gpgkey=

I’ve only tested the EL6 x86_64 package but I went ahead and created the repos for EL5 and EL6, both i686 and x64_64. Yes, the packages aren’t signed right now. Hopefully that’s something that can be remedied relatively easily. And even better would be if Opscode would just integrate the simple call to createrepo into their build process for the omnibus installer.

Originally published at Jeremy's Thoughts. You can comment here or there.

A Puppet User Trying Chef

Thu, 19 Apr 2012 15:50:18 GMT

I have a decent amount of experience at this point with puppet both from experience using it to manage the infrastructure running Fedora as well as setting it up at a pretty large scale at HubSpot. But in a new gig, I decided it was worth rounding myself out a bit and giving chef a try. Not out of any deep seated dislike of puppet but there are a few pieces that I’ve continued to run up against which are a little grating and so I figured it was worth broadening my horizons. The nice thing is that both are fairly successful open source communities and realistically, as long as you’re using a system, you probably can’t go that wrong or switch in the future.

Side-note: I’ve also been playing with Michael Dehaan’s new project, ansible which is also interesting. But I don’t think it’s mature enough to use for a production environment yet and I also was mostly interested in it as a better remote execution layer as opposed to another full fledged config management tool. But yeah. It’s there. It’s interesting. I’ll probably write more about it later.

With a little bit of chef time under my belt, I have to say that I’m not struck by drastic differences. The terminologies are different, the DSL used on the config side is a bit different but they act pretty similarly and you can get either of them to do what you want. That said, there are a few things (good and bad) that I’ve noticed about chef and figured I’d share for others who are looking at deciding for themselves. Note that a few of the things in the dislikes section may well just be me missing something and being a n00b… suggestions welcome!

Things I’ve Liked

Hosted Chef is a very very nice option to have. Props to the Opscode team for building an infrastructure to run the server side for youand especially for making the barrier to entry nearly zero by letting you manage up to five hosts for free. Given some of my headaches around running a puppetmaster previously, I’m glad not to be having to pull together everything to run a chef server
Knife is actually pretty cool. I was skeptical before using it but it does a pretty nice job of encapsulating a lot of common tasks for you
Knife gets really cool with the addition of the ec2 plugin. Launch servers, register them with hosted chef and have them ready to go. I’ve built all of the surrounding bits and as the environment I’m dealing with grows, I think I’ll grow out of being able to use knife ec2 effectively, but it’s great for an easy starting point
Chef solo seems to work okay and have a few niceties over a master-less puppet setup but I didn’t spend much time with masterless puppet, so it’s probably just that I didn’t find the related nice pieces

Things I’ve Disliked / Been Annoyed By

The package support in the Fedora/CentOS/RHEL universe is pretty poor. I realize that all the cool kids use Ubuntu these days but tons of server infrastructures are not. Todd does a great job with the puppet (+ ecosystem) packages for Fedora and EPEL. Would love to see someone do similar for all of the Chef stuff
A lot of the cookbooks that are out there and published are Ubuntu specific. Even the ones which strive to work across distros often end up coercing the Fedora universe to look more like Debian. Which isn’t necessarily a path I want to go down

Probably just a side effect of this but a lot of cookbooks using things which aren’t the standard init system (eg, depending on runit)

knife-ec2 makes you think you can get away with using it but I keep tripping across things it doesn’t support and making me consider abandoning it
Trying out cookbooks from others drives me crazy. I’m pretty sure I’m missing the good workflow here but polluting my checkout by adding vendor branches and auto-committing things. There’s gotta be something I’m missing here

So am I now a rabid chef fan? Nope. But it’s a nice system with some definite advantages for certain use cases. I suspect I’ll find more of them as I use it more.

Originally published at Jeremy's Thoughts. Please leave any comments there.

Monitoring Driven DevOps

Tue, 28 Jun 2011 00:23:45 GMT

Wrote up a nice post that maps pretty well to the Ignite talk I gave at Velocity about using using monitoring to help drive your infrastructure development.

Go check it out over on the HubSpot dev blog

Jeremy's Thoughts

Thoughts on DevOpsDays NYC

Can’t code with AWS outages so blogged instead

A repo for the chef-omnibus packages

A Puppet User Trying Chef

Monitoring Driven DevOps

Velocity 2011

My new role

2010, The Quick Look Back

I’ve moved my bike blogging

Stop Using the Word “Cloud“

Time to get back into a training routine

EC2 and Fedora: Still stuck at Fedora 8

Why do all deployment systems suck?

10 Year NCSSM Reunion: Has it really been that long?

Commuting Pedal Failures

Review: Amazon Kindle 2

Garmin 705 Tip: Sending Routes Between Devices

First Week at HubSpot

The New Chapter Begins

Beginning A New Chapter

2009 Seacoast Safari Recap

Maui Trip: All the Rest

Jeremy vs the Volcano (Haleakala)

Sooner or later, it was bound to happen

Repeating the cycle, time to kill rhpl