Teaching at the Ruby on Rails Workshop for Women

Today I taught the beginner class at the Ruby on Rails Workshop for Women. First of all, I want to thank Mary Tolbert for getting me involved. Also, a big thanks to Sarah Allen and Liana Leahy putting things together and giving me a chance to say that "I taught a class at Harvard."

I was a bit intimidated in teaching the class at first. I was upgraded from TA status to teacher mid-week and was a bit worried that I wouldn't know what to say. Maybe I was right to be worried. Also, many of the TA's were rockstar rubyists and I have to admit that I felt a little silly speaking authoritatively in front of them.

Once I started getting into the material I started to feel pretty comfortable. Watching people have a-ha moments is really rewarding. Trying to explain programming with out falling back on computer science terms is kinda tricky on the fly. Maybe if I do something like this again, I can speak on things a little more smoothly.

It's pretty incredible how much the students were able to pick up in a day. I know that we had some people in the class that had never written code in their life and it was really cool to see them hacking in just a few hours.

If I were to do it over again, I would suggest just a plain ruby class for beginners. Or maybe a two day class where Rails is covered in the second day. When I thought that the students were getting comfortable with Ruby, we only had one session left. I think that everyone was with me when we were modifying the default index page in Rails, but I wanted to make sure that the students saw the ruby bits too. So I rushed through controllers and views in 15 minutes and I would be surprised if anyone got anything out of it. It probably would have been best to punt on the controllers and just explain a little about HTML and CSS.

I noticed that for a lot of beginners, application switching was killer. Switching between the terminal, browser and text editor is second nature for me, but not so much for those just getting started. I wish I had a ton of screen real estate to keep all three visible at once, but I don't know if that would actually help.

We didn't get to really cover git or Heroku in the class. I was able to tell the students enough to use them, but not understand them. The localhost vs. Heroku server was definitely a stumbling block for some. For beginners, I might punt on Heroku and just work from localhost without any source control just to get started.

I got a ton of positive feedback on Twitter and in person. This was super encouraging. I've always been considering teaching at some level later in my career, so it seems like I am on the right track there.

Overall, it was an awesome experience. One thing I was surprised to hear is how intimidating it is for women to participate in local open source meet ups. I suppose that being a 6' 2" person who has played contact sports leaves me in a position where I am generally not to scared of software developers. But getting up in front of the class today with all of the TA's looking at me, I think I'm starting to get it. Hopefully, events like these can start to even out the gender balance so that more women feel comfortable participating in the Ruby and FOSS community.

Building Tokyo Cabinet for use with Java on OS X

I've been really interested in playing with Tokyo Cabinet lately. I thought that it would be fun to take a hack at the GitHub Contest using Scala and Tokyo Cabinet. I then set out to build Tokyo Cabinet and its Java bindings (since I can call those easily from Scala). The Java bindings for Tokyo Cabinet are not pure Java, they use JNI, so you need to compile some C as well as Java. Everything looked fine and dandy until I tried to run some code. I then ran into this stack trace from Scala:



To translate, what is going on here is that by default Tokyo Cabinet will build a 32 bit binaries. Java 1.6 on OS X is 64 bit and will look for a 64 bit version of the library. Here is what I did to make things happy.

When running the configure script for Tokyo Cabinet itself, I added a flag:



I tried the same trick when configuring the Java bindings, but it didn't seem to end up in the resulting Makefile. So I edited the Makefile by hand. In the end, my CFLAGS line looks like this:



After that, I was able to get a small Scala script to create a Hash database.

As an aside, the Scala IDE for Eclipse seems really nice. I had tried it out a few months ago, and it has clearly made a lot of progress since then.

Glassfish is looking speedy

Let me start by saying that most benchmarks should be taken with a grain of salt, so please do the same here.

I work on Project Laika, and for our deployments we are looking to switch to JRuby. We already have java code hanging around, and it looks like we will need some more soon (I'm thinking it will be easier to deal with SOAP web services in java and calling the classes from JRuby). I wanted to run some numbers to make sure that our performance wouldn't fall through the floor.

The Setup

I decided to pit Mongrel 1.1.5 running on MRI 1.8.6 against Glassfish v3 Prelude 1 and JRuby 1.1.5 on Java 5. I'm running Rails 2.0.2 in both setups (I know we are behind the times). I ran them both on OS X 10.5.5 and had the Rails apps hit the same MySQL database.

I used ab to grab the numbers. I had it hit the site 1000 times with 10 concurrent requests. I hacked the Laika app slightly so that you didn't have to log in.

The Test

I wanted to get a feel for how the app would perform, so I did two simple tests: dynamic content from a typical page and static content. The dynamic content was the patient template library in Laika which contains code that you'd expect in a Rails app: AR pulling info from the DB and putting it into ERB templates. I also pulled down the Rails 404 page to get a feel for serving static content. This is probably less meaningful, as you'll probably have Apache or Nginx serve up your static stuff.

The Results

Glassfish, hands down. The results are the average number of milliseconds it took to serve each request.


It beat Mongrel in both static and dynamic content easily. Glassfish v3 also makes it ridiculously easy to deploy Rails apps. You can use the Glassfish gem, and serve up your app with a single command. I installed the Glassfish server, so I could run JEE apps alongside my Rails stuff, there is a single command where you point the app server to the root directory of your app and you're done.

With Rails 2.2 now out and offering thread safety, and JRuby being the only interperter that can take advantage of it... Glassfish and JRuby are really worth checking out.

SezHoo and WikiSym 2008

I'm proud to announce the release of SezHoo! It's a tool we created at MITRE to help establish reputations for authors in wiki sites... specifically those run on MediaWiki (the same software that runs Wikipedia).

The way that SezHoo works is that it goes through a wiki article's history and tracks the authorship of each word in the article. We use a pretty popular technique in text analysis called shingling to help us out.

For each revision of an article, we break it up into shingles. We're using shingles that are 6 words long, so each word will have 6 shingles associated with it. We credit authorship of the word to the author of the earliest shingle.

This allows us to add some information to a MediaWiki installation:

Pages
We show the authors of a page, and the percentages at which they have contributed. We also offer this information at the section level.

The information we compute is better than doing diffs with a standard MediaWiki install. Diffs are line ending based, so changing a word in a paragraph that has a single line ending can make it difficult to determine who wrote the paragraph. In addition, since we are looking for authorship of a word across all revisions at once (as opposed to a revision vs. revision comparison with a diff), this technique is immune to copying and pasting of text and reversion of vandalism.

Authors
We create a reputation for authors with 3 dimensions: quantity, quality and value. For all metrics, we assign a 5 star rating. The way this is done is by computing a raw score for all authors and then ranking them. Authors in the top 20% have a 5-star rating in that dimension.

For quantity, we simply calculate the number of word contributed by each author. This includes words that may no longer be in the current revision of the article. Value takes the percentage authorship of a page, and multiplies that by the number of page views (it's valuable if it's being read).

Quality is a lot more tricky. We assume that wiki articles are a Darwinian environment. That is, quality text will "survive" in that as the article is edited, other authors will leave the text alone. Poor quality text will be "killed" or deleted by other authors. We determine quality by looking at the percentage of text that is expected to be alive for an author after 8 revisions (you can change 8 to whatever you like, but it works well for our internal wiki). To calculate the percentage alive after 8 revisions, we used a Kaplan-Meier estimate. This was necessary due to the fact that authors will have phrases which are "alive" (they are in the current version of an article). So if a word has survived 3 edits and is in the current version of an article, when calculating survival probabilities you don't want to say that it "died" at 3 revisions, but you don't want to lose the data either. Kaplan-Meier handles that for us.

In the end, you get a 5-star rating for each author in the dimensions of quantity, quality and value.



I'll be presenting this work at WikiSym 2008. We're doing some other interesting stuff which we haven't rolled into SezHoo yet, so it's worth coming to my talk even if you've read this blog post.

Using Hadoop

Since I've gotten back from OSCON, I've had a chance to use Hadoop at work. For those who aren't familiar with it, Hadoop is an open source framework for implementing map reduce jobs.

There are plenty of tutorials on Hadoop around the web, so I won't do any of the basic intro stuff. I wanted to write about some of the stuff I didn't find all that easily.

Most of the Hadoop documentation talks about dealing with input where your records are on a single line (like an Apache access log). From using the Google/documentation/experience, we have found that Hadoop works just fine with multi-line records. We're using Hadoop to process XML, specifically a dump from Wikipedia. The dump is a single 33 GB file, where there is a root tag, and then several million child tags (representing Wikipedia pages). Using this code I found on the Hadoop core user mailing list, we can have it so that the mapper gets the XML for one child node (or one Wikipedia page). This is nice, because the XML for a single page is relatively small. We then use JDOM to deal with the contents of the individual pages.

We are using HDFS to store our input and output. By default, it will chop files into 64MB chunks, which get shipped to the mappers so that the file can be processed in parallel. One thing that I was concerned about was how records would be handled that spanned the splits. So far, we haven't seen any issues, and this answer in the Hadoop FAQ seems to indicate that the records spanning the split will be handled. It may be possible that those records get dropped, it would be hard for us to tell at this point... but the good news for us is that it won't affect the data we're trying to collect much.

As for our setup and usage. We have 5 machines in our cluster, most of which are run of the mill dual core Intel machines running Linux. The jobs we're running on the 33GB XML file are taking around 45 minutes, which seems pretty fast to me.

OSCON 2008: Wrap Up

So I've returned from Portland after all of the OSCON activities. The conference was good, but I definitely didn't feel like it was a good as in years past. The keynotes were OK, but there weren't any that were spectacular. I hit a few sessions that were bombs, but I didn't get to one that rocked. Many were good, but nothing off the charts.

Hadoop was big on Thursday. Derek Gottfrid of the New York Times talked about how they used Hadoop and Amazon EC2 to process tons of data. Derek's presentation style is great, which mate the talk entertaining. Some folks from Yahoo were also getting into the nitty gritty details of how the whole thing works too.

The MySQL Proxy talk was good. It seems like a pretty handy tool for performance tuning and all sorts of SQL trickery.

The last talk that stood out to me was Selectricity. The project is a both a site to run elections as well as the software you can use to run elections where ever you want. One point that Benjamin Mako Hill made that I thought was interesting is that most election research goes into government elections... and these are the least likely to change. By building a tool to allow folks to conduct elections for simple things (what movie to see, who will lead the coffee club, etc.) using methods different from plurality, it's a good way to sneak in alternative voting methods to the masses. That way if people get familar with Condercet when voting for the next American Idol, they may be more likely to push for election reform in government elections.

I'm not sure if I'll hit OSCON again next year. I like going because it's nice to get an overview of a lot of different techologies, as opposed to something like Rails Conf. But things did feel pretty shallow this year.

OSCON 2008: Day 3

Today, I gave my first OSCON talk on Laika. I think that the talk went pretty well. I had plenty of good questions from the audience, and I think I may have been able to snag a few people who were interested in contributing.

The speaking experience was pretty cool. I was in a fairly small room, and probably had about 20 to 30 people in the audience, which was pretty non-threatening. I would have been more nervous in one of the more cavernous rooms with 200 people.

As for the rest of the conference today...

XMPP has a lot of buzz for communicating in the cloud. There were a few talks on that today.

There is a lot of Ruby stuff going on outside of web apps. RAD seems to have a lot of buzz for using Ruby to work with Arduino. Ruby's also behind Adhearsion, a tool for building IVR's.

I missed the talk on CouchDB, but some of the folks I'm out here with said it was great.

On a conference logistics note... I was kinda bummed that some of the talks had filled, so I couldn't get in. I wound up missing the talk on Hypertable as well as one on XMPP in the cloud.