Jeremy's Thoughts
Cycling, Fedora, MIT SDM and Miscellany

Jeremy Katz
Date: 2013-09-10 10:42
Subject: Build systems are the new black
Security: Public
Tags:fedora

So I got into a little bit of a twitter argument last night about distro packaging. Which actually wasn’t where I was trying to go (this time ;) ) One of the problems with twitter is that 140 characters can be hard sometimes. So let’s see if more characters help.

There is a big push afoot from a lot of people towards omnibus packaging. It seems especially prevalent in the world of things written in Ruby I suspect because most even “current” Linux distros are or were shipping some Ruby 1.8.x build up until very recently. And people want to take advantage of the newer language features.

I have a lot I could write on this topic. But I’m not going to today.

The main point I was trying to make is that in going down this path, people are putting together increasingly large build chains that have complex dependencies and take a long time. And so they’re starting to do things like caching of sources, looking at short-circuited builds and a whole lot of other related things. Which, incidentally, are all things that all of the Linux distributions ran into, fought with and figured out (admittedly, each in their own way) years ago. So rather than just rediscover these things and do it yet another way, there’s a ton of knowledge that could be gained from people that have built distribution build systems to make things better for the omnibus world.

So reach out to people that have worked on things like the Debian build system, the OpenSuSE build system, plague, koji and a slew of others. There are some tricks and a lot of tools which are just as valuable as when they started being used. Things like caching build roots and how to fingerprint them for changes, what ccache can and can’t be good for, how to store things reasonably in a lookaside to not depend on upstream repositories being down while you build, incremental builds vs not, ccache, …

Just because you think that building system RPM or DEB packages doesn’t meet the needs of your users doesn’t mean that you have to throw away all of the work and experience that has gone into the toolchains and build systems present there.

Originally published at Jeremy's Thoughts. Please leave any comments there.

Add to Memories | Share | Link



Jeremy Katz
Date: 2013-01-19 12:01
Subject: Thoughts on DevOpsDays NYC
Security: Public
Tags:devops, fedora, work

I’m currently on the train on my way back from DevOpsDays in Brooklyn. The conference was great — lots of smart people facing a lot of similar problems and trying to see what we could learn from each other. The scale was small, with only like 100-ish people present and not a ton of huge, in your face sponsorship. And the venue was a college campus. And so I kept making these comparisons in my head to LUG meetings, installfests and small scale Linux conferences.

Obviously the subject matter was a bit different — talking about and thinking about running large scale production infrastructures is a little bit different than the next cool Linux distribution. This tended, I think, to more discussion around patterns and best practices than about the specifics of “you should do X to get Y to work”. So a higher level and more abstract discussion.

The composition of the audience and attendees was a pretty similar make-up. Linux events always had a strong majority of the attendees who self-identified as sysadmins and then there tended to be a smaller number of developers. And many of the latter group had ended up in that camp due to necessity. The breakdown for DevOpsDays felt pretty similar with an interesting twist where there were speakers who said they were (paraphrasing) “developers first and fell into operations because they needed to”.

One thing that felt more evolutionary than anything else was that the side channel discussion for the event took place on Twitter rather than on IRC. I have (fond) memories of many conferences where attendees sat in an IRC channel and then basically continued to interact on IRC long after the conference had ended. In fact, I made many friends in this fashion. Similarly there was an ongoing discussion on Twitter using the #devopsdays hash tag and I have followed (and am being followed by) a number of the other attendees and hope to keep in touch and call them friends in the future.

And maybe the thing that struck me the most strongly was where people were “from”. Not in the sense of where they lived but rather where they worked. The attendees were almost all from startups. We were in Brooklyn and not the heart of downtown Manhattan, but NYC is probably home to more financial services companies than anywhere else in the world. And all of those companies have *many* people working in software dev and operations-y roles. But they weren’t there.

So it feels like “the DevOps movement” is going through a similar growth and evangelism pattern as open source and Linux did years ago. Maybe that’s why it feels so comfortable to me.

Originally published at Jeremy's Thoughts. Please leave any comments there.

Add to Memories | Share | Link



Jeremy Katz
Date: 2009-10-16 15:46
Subject: EC2 and Fedora: Still stuck at Fedora 8
Security: Public
Tags:amazon, ec2, fedora

Amazon’s EC2 service is great for being able to roll out new servers quickly and easily. It’s also really nice because we don’t ever have to worry about physical hardware and can just spin up more instances as we need them for experimenting or whatever.

Unfortunately, they’re still stuck in the dark ages with the newest AMIs available for Fedora being Fedora 8 based. With Fedora 12 around the corner, that’s two years old — something of an eternity in the pace of distribution development. I’d love to help out and build newer images, but while anyone can publish an AMI and make it public, you can’t publish newer kernel images, which really would be needed to use the newer system.

So, if you’re reading this at Amazon or know of someone I can talk with to try to move this forward, please let me know (katzj AT fedoraproject DOT org). I’d really strongly prefer to continue with Fedora and RHEL based images for our systems as opposed to starting to spin up Ubuntu images for the obvious reasons of familiarity.

Comments
Add to Memories | Share | Link



Jeremy Katz
Date: 2009-01-19 17:09
Subject: Rumours of my death are (largely) unfounded
Security: Public
Tags:fedora, fudcon, life, sdm
The rumours of my death are largely unfounded.  I've just been either busy working or trying to relax while not on a computer since this is as much of a "break" as I get.

I have, though, done various updates to twitter and identi.ca if you have some obsessive need to know what I've been doing.  It hasn't been that exciting, though.  Basically boils down to the following relatively short list
  • Went cross-country skiing a couple of times.  With the very wintry weather we've had thus far this winter, it's been something good to be able to get outside and do as it hasn't exactly been ideal biking weather
  • FUDCon F11 was held in Cambridge at MIT.  Since it was in E51 and I knew where things were, I spent a fair bit of time running around.  I had some good conversations, but didn't give any presentations and didn't really get any hacking done with the hackfest
  • The SDM 09s have started and I helped some with their first design challenge.  Was fun to watch and they seem a good bunch
  • Have been trying to read a fair bit and so made good progress on my book backlog.  Still hoping to finish that before classes start back up
  • Some poking and prodding in the hopes of getting Fedora 11 alpha out the door in a semi-decent shape
  • More work on the new initramfs tooling, although it's making slower progress than I'd really like
  • Getting extra sleep
 
1 Comment | Post A Comment | Add to Memories | Share | Link



Jeremy Katz
Date: 2008-12-17 14:24
Subject: dracut — Cross distribution initramfs infrastructure
Security: Public
Tags:dracut, fedora

As davej was alluding to, we've started trying to make progress on a new initramfs infrastructure that can be used across distributions instead of just being done for a single distribution. Thus far, the code is fairly early and proof-of-concept-y, but I can successfully boot root partitions, root on LVM or root on encrypted LVM. The next step is to clean up the build side so that I can switch to using it with my laptop. I wanted to go ahead and get the code out first, though.

The majority of the ideas behind the code are that we want to use the same tools and utilities that exist on the booted system (so regular bash, regular udev, no nash, no statically linked nonsense, no klibc) and also that we want to not have the initramfs as a slow-down. The latter means that we run pretty much everything off of udev rules right now so that as soon as the devices are available, we mount and switch into the rootfs.

So go ahead 'git clone git://fedorapeople.org/~katzj/dracut.git' and read the README, HACKING and TODO files. Also sent to lkml to get the mailing list started. More hands more than welcome…

Comments
Add to Memories | Share | Link



Jeremy Katz
Date: 2008-12-17 14:24
Subject: dracut -- Cross distribution initramfs infrastructure
Security: Public
Tags:fedora
As davej was alluding to, we've started trying to make progress on a new initramfs infrastructure that can be used across distributions instead of just being done for a single distribution. Thus far, the code is fairly early and proof-of-concept-y, but I can successfully boot root partitions, root on LVM or root on encrypted LVM. The next step is to clean up the build side so that I can switch to using it with my laptop. I wanted to go ahead and get the code out first, though.

The majority of the ideas behind the code are that we want to use the same tools and utilities that exist on the booted system (so regular bash, regular udev, no nash, no statically linked nonsense, no klibc) and also that we want to not have the initramfs as a slow-down. The latter means that we run pretty much everything off of udev rules right now so that as soon as the devices are available, we mount and switch into the rootfs.

So go ahead 'git clone git://fedorapeople.org/~katzj/dracut.git' and read the README, HACKING and TODO files. Also sent to lkml to get the mailing list started. More hands more than welcome...
Post A Comment | Add to Memories | Share | Link



Jeremy Katz
Date: 2008-12-07 21:08
Subject: Semester nearly over
Security: Public
Tags:fedora, olpc, sdm
At this point, the semester is almost entirely over. While I still have one day of each class left, pretty much every assignment is done and turned in. Only have to finish up the principles assignment for System Architecture and that's mostly a matter of sitting down and throwing some together from the notes that I've got from the semester.

Overall, it's been a good semester. The workload ended up being a bit higher than I expected, but it was probably what I should have thought. I knew that System Architecture was going to be time-consuming, but it was still more so than I thought. Similarly, Project Management ended up requiring more time both for the homeworks and the project than I really expected from the outset.

Those along with working to get all of the Fedora 10 bits working on the OLPC meant that it was just a very busy semester. But, now it's actually the time when I get nearly two monhs to relax and "just" work. Well, and I also am hoping to try to get some progress underway for my thesis so that I don't have to do it all while juggling classes also. It should still be a good sort of break, though. And then, it's on to the spring semester. Which I still need to figure out what I'm going to take -- suggestions welcome :-)
1 Comment | Post A Comment | Add to Memories | Share | Link



Jeremy Katz
Date: 2008-11-10 23:06
Subject: The end of the semester comes early it seems
Security: Public
Tags:fedora, life, sdm
For some reason, the end of this semester seems to be coming early this time around. Although the end of classes isn't until the middle of December, it seems that pretty much everything is due before Thanksgiving. Which, coupled with trying to get Fedora 10 out the door is going to make the next few weeks a pile of pain.

So if the updates from me seem sparse, that's why. And then of course, to make things even more fun, I came down with something the end of last week which I haven't quite managed to completely shake. Although at this point, it might just be my usual congestion for this time of year.
1 Comment | Post A Comment | Add to Memories | Share | Link



Jeremy Katz
Date: 2008-11-04 13:45
Subject: Party like it's 1999
Security: Public
Tags:fedora, gnome

I'm somewhat excited about the new gnome-shell stuff that has been blogged about a few times by various people since the GNOME summit a few weeks ago. And so when Owen posted that there was some initial code available, I had to check it out, even if it was pretty retro GNOME 0.9 at the moment. There was, though, one important piece missing...














Wanda!
Wanda!
(Click the link for the movie)

Not what I intended to work on today. But it was fun and gave me a chance to try out writing a tiny bit of code using both Javascript and Clutter, neither of which I'd used before. The takeaway is that it's really simple to do simple things with what is being worked on. Still not sure about anything more complicated at this point

Post A Comment | Add to Memories | Share | Link



Jeremy Katz
Date: 2008-10-31 15:33
Subject: Working around graphical glitches on the XO
Security: Public
Tags:fedora, olpc

Thanks to a pointer from warren, I've just pushed a workaround to the spins-kickstart repo to help some of the graphical glitches that have been seen on on the OLPC. This includes the blacked out buttons and also the cut-off text. Unfortunately, this probably won't make the Fedora 10 Preview release next week, although it will be in the final release. If you'd like to test it yourself, you can edit /etc/X11/xorg.conf and add a line

Option "MigrationHeuristic" "greedy"
below the EXA line.


1 Comment | Post A Comment | Add to Memories | Share | Link



browse
my journal
September 2013