Rooting your Droid Incredible

I have an old Droid Incredible that I'm hanging onto until the iPhone 5 comes out. In the meantime, I've been fighting with the "Phone storage space is getting low" issue. Like many people seeing this problem, I have plenty of free space available. I was able to fix it by rooting my phone and running a custom ROM to repartition the phone's storage. Here is how I did it.

Disclaimer

Rooting your phone may void your warranty. Back up your data before doing this. Also know that I haven't been terribly rigorous in understanding everything that is happening here. You could say that I am cargo cult-ing some of the instructions. For me, since I am due for a phone upgrade, the worst case scenario is that if I brick my phone, I'll head to the Apple store and pick up a 4S.

Overview

What this will do is give you root access to the phone. This will allow you to run custom ROMs. Once you have this, you can replace everything on the phone with something like CyanogenMod. For the time being, I have chosen to stick with the OS shipped on the phone but booted into a custom ROM to fix the issues I am seeing.

The "Phone storage space is getting low" message is caused because application files and some data is stored on a partition that is 150MB. It's mounted to /data/data. Once your apps or that data hist 140MB, you'll get the error and your phone will shut down sync services, meaning you won't see new emails. I found a ROM that repartitions this space to 750MB and haven't had a problem since (roughly a month).

Gaining Root

I use a Mac at home and used Unrevoked 3 to gain root access. The latest version of the software (3.32) did not work in attempting to gain root. I got an error message stating that the firmware was too new. I was able to pull down version 3.22 by changing the download URL. That worked fine. You'll need root access to boot the phone into custom ROMs. You'll need custom ROMs to do the repartitioning.

Repartitioning

I bought a copy of ROM Manager from the Play Store. This will let you boot custom ROMs. I used it to boot the Convert2Ext4_no_data_limit_normal_dalvik ROM. You can find a description of what the ROM does and other variants of it in this XDA Developers forum post. You will need to move the ROM image to your phone to boot it. I just downloaded it to my Mac, mounted the phone's SD card as a disk and copied it over.

Success

After you run the custom ROM, your phone should now be repartitioned and the low storage space error message should go away. With root, you should be able to do other nice things like remove Verizon bloatware. I haven't tried that yet.

Text Messaging

After rooting and repartitioning the phone, I noticed that I could send text messages, but I couldn't receive them. This seems to be a relatively common problem among people who do this sort of thing. After some Googling, I came across this post in the Verizon community forums. Basically, if you go to http://dl3.htc.com/misc/inc8049.apk on your phone, it fixes the issue and you can get text messages again. I have no idea what the file does, so download at your own risk. It goes to HTC, so I figured it was relatively safe.

Debugging MapReduce in MongoDB

On a project that I am working on, we are doing some pretty intense MapReduce work inside of MongoDB. One of the things we've run up against is the lack of solid debugging tools. Some Googling basically tells you that print() is all you've got.

We've decided to take a different approach and debug our MapReduce code in the browser. Since the code is JavaScript and modern browsers have really excellent support for debugging (breakpoints, variable inspection, etc.) it's pretty easy to do.

All you need is a web app (or even static HTML file) that will:

1. Load up one of your documents that you would like to map in the browser. Since the documents are JSON, this is easy. In our project, we have JSON fixture files and a small web app that allows you to choose which fixture to use for testing.
2. Mock the emit() method. You can just have it write to a Hash that you can inspect later.
3. Load up the Map and Reduce functions. If you keep these in separate .js files, you can pull them in with a simple script tag.
4. Bind the map function to the document so that it has the correct context. In a MongoDB mapper function "this" is set to the document that you are mapping. You can easily do this with the bind() function in Underscore.js. I'm sure that other JavaScript frameworks provide a similar function.
5. Put a link on the page that will let you run the bound function.

This will emulate the MongoDB MapReduce environment, but you can now use the browser's debugging tools.

Using Underscore.js with MongoDB

I've been using MongoDB for a while now and have been really happy with it. I wanted to share something we are doing on of the projects I work on that makes working with Mongo even better.

MongoDB allows for the use of JavaScript to do lots of work on the server side. This includes running MapReduce jobs on collections, but also can be used in where clauses and for doing grouping. Being able to use JavaScript for these things is handy, but using just the core JavaScript language can be less than ideal. That's why we prime our MongoDB environment with Underscore.js.

On the Underscore website, it claims to be a JavaScript utility belt. I've found that to be the case. It has functions like any or include that save you the trouble of having to write for loops to iterate over arrays. While the MongoDB documentation describes how you can store individual functions for server side use, it didn't really touch on how you could load an entire library like Underscore.

It turns out you can load up libraries like this pretty easily using db.eval(). I recall reading (but can't currently find the docs to prove this) that every MongoDB connection has a JavaScript context associated with it. If you create functions in this context, they will exist as long as the connection is around. So if you just eval the Underscore.js library before you do any work with your connection, you will have access to all of its functions to do your work.

Here is an example of how to use Underscore.js with the Ruby driver. In this example, I'll set up the MongoDB connection with Underscore.js, create a sample dataset of cars, then use Underscore to group them by make without repeating model.

The only downside to this approach is that db.eval does not seem to work with sharding. That is OK for me right now, but YMMV. Also note that I am using the awesome_print gem to pretty print the results.

Git rm may cause insanity

Ran across this today, and wanted to help others avoid the same fate. If you use git rm to remove the last file in a directory, it will remove the directory as well. If you are in that directory, odd things can happen that will potentially drive you insane.

Let's create a git repository with a folder that has a single file in it:

$ cd /tmp
$ mkdir foo
$ cd foo/
$ git init
Initialized empty Git repository in /private/tmp/foo/.git/
$ mkdir bar
$ cd bar/
$ echo 'hello' > splat.txt
$ git add splat.txt
$ git ci -m 'adding a text file'
[master (root-commit) 069f11b] adding a text file
1 files changed, 1 insertions(+), 0 deletions(-)
create mode 100644 bar/splat.txt

So I now have a git repository and placed the file bar/splat.txt under revision control. Now if I do:

$ git rm splat.txt
rm 'bar/splat.txt'

This will not only remove splat.txt, but it will remove the whole bar directory. I say this will drive you insane, because if you try to move or copy a file into your current directory, you'll get an error that will probably catch you off guard. Like:

$ cp ~/.gitignore .
cp: ./.gitignore: No such file or directory

There is a file called .gitignore in my home directory, it's just that my current directory no longer exists. It took me about five minutes to realize what was going on... and I was starting to wonder if I knew how to use the cp command.

The reason I ran into this is that I was rearranging my .vim folder to use pathogen. I keep all of my dot files under source control and stumbled upon this while clearing out my vim autoload folder.

Gwibber on Ubuntu 10.04 issues with FiOS

I just upgraded one of my machines to the latest and greatest Ubuntu because I plan on taking it on the road later this week. After I got everything set up, I fired up Gwibber, my favorite Twitter client on Linux. Immediately, I started running into problems. I couldn't get Gwibber to load any new tweets. There seem to be several people who are experiencing this issue with Gwibber, but their troubles are related to the language settings. That was not the case for me.

I did some digging by firing up Gwibber in a terminal:

$> gwibber-service -o -d

This is what I got:

Gwibber wasn't refreshing because it was timing out on the DNS lookup. I have Verizon FiOS as an ISP. FiOS having terrible DNS seems to be a problem. I switched over to Google DNS and every thing is snappy and working properly. If you're using Gwibber on FiOS and having issues, try this out. It may save you an hour or three.

Gwibber, or whatever library it is using for network communication, picked a pretty short timeout, but this is pretty lame. Verizon really needs to step it up here. People will perceive FiOS as slow because it takes forever to look up an IP, even though the network is pretty quick in my experience.

Using xargs with git

Sometimes, when I'm working on a project, I'll create a bunch of new files and realize that I have a ton of untracked stuff that I need to add to my git repository. Since generally only use git on the command line, it would be painful to copy and paste all of the untracked file names from the output of git status into separate git add commands.

The two commands I have found handy for dealing with this situation are git ls-files and xargs.

If you run the command:

git ls-files -o

It will show you all of the untracked files in your working directory, one file per line. A problem that you will run into here is that it also shows files in your .gitignore. To get around this issue, you just need another argument to specify your .gitignore:

git ls-files -o --exclude-per-directory=.gitignore

Now that you have all of the files you want to add, you just need to run git add on all of them. This is where xargs comes in handy. It will read from standard in, break it up on line endings and then feed each line as an argument into another command. Putting it all together, you get:

git ls-files -o --exclude-per-directory=.gitignore | xargs git add

That last command will add any untracked files to your git repository. The beautiful thing here is that we can also leverage some UNIX-y goodness if we want to as well. Let's say we're working on a project and we only want to commit some XSLT we have been working on. You can do this by throwing grep into the command chain:

git ls-files -o --exclude-per-directory=.gitignore | grep xslt | xargs git add

This will only add files that contain "xslt" in their names. This same approach comes in handy when you remove files from your working copy but forget to run git rm.

Classy hData

I've been working on a team that is looking at ways in which we can simplify the exchange of information in Health IT. This effort is called hData. We just released a new version of our packaging and network transport spec, and I would like to talk a bit about how we arrived at this version.

I think it is really important for IT specifications to have a reference implementation available. If you build a spec without code, it's really hard to see where you have gone wrong. To make sure we are on the right track, I built a small web application that implements the spec. I was able to quickly uncover some bugs in our work. Bugs I'm sure we would have missed by just reading the document.

Technology Choices

Since my preferred language of choice is Ruby, it would be natural to think I would want to tackle this project in Rails. However, in hData we make some good use of the HTTP Verbs, and I'm not so sure that they would line up seamlessly with Rails conventions. I decided to go with a much simpler choice. Sinatra is a small web framework that seems perfect for this job. It makes the HTTP Verbs central to your code, so it should be fairly obvious on how we go from the spec to implementation.

There are a few other tools that I used on this adventure. DataMapper was just right for the ORM needs of the project. I could have used ActiveRecord to persist data, but DataMapper has a really nice auto-migration feature, which will save me from writing all of the database creation code. I also used Bundler to manage my application's dependencies.

Getting Started

The best way to get started here is by taking a test driven approach to the spec. For that I will be using Shoulda and Rack Test. With my TDD tools in place, I can take part of the spec that looks like this:

3.1.2 POST

3.1.2.1 Parameters: type, typeId, requirement

For this operation, the value of type MUST equal "extension". The typeId MUST be a URI string that represents a type of section document. The requirement parameter MUST be either "optional" or "mandatory". If any parameters are incorrect or not existent, the server MUST return a status code of 400.

If the system supports the extension identified by the typeId URI string, this operation will modify the extensions node in the root document and add this extension with the requirement level identified by the requirement parameter. The server MUST return a 201 status code.

If the system does not support the extension, it MUST not accept the extension if the requirement parameter is "mandatory" and return a status code of 409. If the requirement is "optional" the server MAY accept the operation, update the root document and send a status code of 201.

Status Code: 201, 400, 409

and turn it into the a Shoulda context block. In the spec above, we're talking about what should happen when you POST to the root of an hData Record. The functionality being described here is how an extension can be added to the record, or how you can register a different type of thing for a record. For example, you could use this feature to add an medications extension to a record, if one did not exist there already. In our test code, we're going to try to register an allergies extension:

As you can see from the code, the combination of Shoulda and Rack Test make it really easy to express the requirements set forth in the specification. The first test tries to POST and incomplete request and should receive an error. The second sends a properly formed request and should get an appropriate response. The last test tries to POST a duplicate extension.

With the tests in place, we can move on to implementation.

I have created a DataMapper Resource to capture all of the information we want to store about an extension. I will also use the validation framework of DataMapper to make sure that all of the requirements for an extension are met. I end up with the resulting code:

With my model in place, I can implement the code to handle the web request:

The code above is pretty typical for Sinatra. The post block handles POST's to the root URL. There I call a method to check and make sure that the type parameter is set. If it isn't I halt the processing and let the user know that the request is malformed with a 400 code. If the type is set to extension, then we drop into the handle_extension method. Inside of the method, I build an Extension object and check it using the DataMapper validation framework.

There is a little bit of funkiness at the end of the handle_extension method where I need to check the type of error. This is due to the fact that I need to return different status codes depending on the error. Unfortunately, with the DataMapper validations, I didn't see any way to return anything with the errors other than a text message, so this seemed like the best way of doing things.

The handle_section at the end of the post block handles another part of the spec. Don't worry, I didn't write it until I had the tests done first.

Lather, Rinse, Repeat

Implementing the rest of the hData Packaging and Transport spec followed the same process. Take the spec and write a matching unit test. Implement the spec and refine the code until the test passed.

In doing this, I found a couple of bugs in our spec. We hadn't provided parameter names for POSTing section documents. Our description of how to add metadata to documents was ambiguous at best. The nice part was that I was able to discover these things before even digging into the implementation.

What still needs to be done

While the Sinatra app that I wrote is a pretty good implementation of the hData Packaging and Transport spec, it still has some gaps. It doesn't support POSTing metadata on documents, it only creates and serves it's own. It also doesn't support nested sections, but that shouldn't be too hard to add.

Wrap Up

You can find the code at eedrummer/classy-hdata on github. Even if you aren't interested in hData, this application should serve as an example Sinatra/DataMapper application. If you dig into the code and the hData spec, I think you'll see that hData is really easy to implement, especially in a classy framework like Sinatra.

Teaching at the Ruby on Rails Workshop for Women

Today I taught the beginner class at the Ruby on Rails Workshop for Women. First of all, I want to thank Mary Tolbert for getting me involved. Also, a big thanks to Sarah Allen and Liana Leahy putting things together and giving me a chance to say that "I taught a class at Harvard."

I was a bit intimidated in teaching the class at first. I was upgraded from TA status to teacher mid-week and was a bit worried that I wouldn't know what to say. Maybe I was right to be worried. Also, many of the TA's were rockstar rubyists and I have to admit that I felt a little silly speaking authoritatively in front of them.

Once I started getting into the material I started to feel pretty comfortable. Watching people have a-ha moments is really rewarding. Trying to explain programming with out falling back on computer science terms is kinda tricky on the fly. Maybe if I do something like this again, I can speak on things a little more smoothly.

It's pretty incredible how much the students were able to pick up in a day. I know that we had some people in the class that had never written code in their life and it was really cool to see them hacking in just a few hours.

If I were to do it over again, I would suggest just a plain ruby class for beginners. Or maybe a two day class where Rails is covered in the second day. When I thought that the students were getting comfortable with Ruby, we only had one session left. I think that everyone was with me when we were modifying the default index page in Rails, but I wanted to make sure that the students saw the ruby bits too. So I rushed through controllers and views in 15 minutes and I would be surprised if anyone got anything out of it. It probably would have been best to punt on the controllers and just explain a little about HTML and CSS.

I noticed that for a lot of beginners, application switching was killer. Switching between the terminal, browser and text editor is second nature for me, but not so much for those just getting started. I wish I had a ton of screen real estate to keep all three visible at once, but I don't know if that would actually help.

We didn't get to really cover git or Heroku in the class. I was able to tell the students enough to use them, but not understand them. The localhost vs. Heroku server was definitely a stumbling block for some. For beginners, I might punt on Heroku and just work from localhost without any source control just to get started.

I got a ton of positive feedback on Twitter and in person. This was super encouraging. I've always been considering teaching at some level later in my career, so it seems like I am on the right track there.

Overall, it was an awesome experience. One thing I was surprised to hear is how intimidating it is for women to participate in local open source meet ups. I suppose that being a 6' 2" person who has played contact sports leaves me in a position where I am generally not to scared of software developers. But getting up in front of the class today with all of the TA's looking at me, I think I'm starting to get it. Hopefully, events like these can start to even out the gender balance so that more women feel comfortable participating in the Ruby and FOSS community.

Building Tokyo Cabinet for use with Java on OS X

I've been really interested in playing with Tokyo Cabinet lately. I thought that it would be fun to take a hack at the GitHub Contest using Scala and Tokyo Cabinet. I then set out to build Tokyo Cabinet and its Java bindings (since I can call those easily from Scala). The Java bindings for Tokyo Cabinet are not pure Java, they use JNI, so you need to compile some C as well as Java. Everything looked fine and dandy until I tried to run some code. I then ran into this stack trace from Scala:



To translate, what is going on here is that by default Tokyo Cabinet will build a 32 bit binaries. Java 1.6 on OS X is 64 bit and will look for a 64 bit version of the library. Here is what I did to make things happy.

When running the configure script for Tokyo Cabinet itself, I added a flag:



I tried the same trick when configuring the Java bindings, but it didn't seem to end up in the resulting Makefile. So I edited the Makefile by hand. In the end, my CFLAGS line looks like this:



After that, I was able to get a small Scala script to create a Hash database.

As an aside, the Scala IDE for Eclipse seems really nice. I had tried it out a few months ago, and it has clearly made a lot of progress since then.

Glassfish is looking speedy

Let me start by saying that most benchmarks should be taken with a grain of salt, so please do the same here.

I work on Project Laika, and for our deployments we are looking to switch to JRuby. We already have java code hanging around, and it looks like we will need some more soon (I'm thinking it will be easier to deal with SOAP web services in java and calling the classes from JRuby). I wanted to run some numbers to make sure that our performance wouldn't fall through the floor.

The Setup

I decided to pit Mongrel 1.1.5 running on MRI 1.8.6 against Glassfish v3 Prelude 1 and JRuby 1.1.5 on Java 5. I'm running Rails 2.0.2 in both setups (I know we are behind the times). I ran them both on OS X 10.5.5 and had the Rails apps hit the same MySQL database.

I used ab to grab the numbers. I had it hit the site 1000 times with 10 concurrent requests. I hacked the Laika app slightly so that you didn't have to log in.

The Test

I wanted to get a feel for how the app would perform, so I did two simple tests: dynamic content from a typical page and static content. The dynamic content was the patient template library in Laika which contains code that you'd expect in a Rails app: AR pulling info from the DB and putting it into ERB templates. I also pulled down the Rails 404 page to get a feel for serving static content. This is probably less meaningful, as you'll probably have Apache or Nginx serve up your static stuff.

The Results

Glassfish, hands down. The results are the average number of milliseconds it took to serve each request.


It beat Mongrel in both static and dynamic content easily. Glassfish v3 also makes it ridiculously easy to deploy Rails apps. You can use the Glassfish gem, and serve up your app with a single command. I installed the Glassfish server, so I could run JEE apps alongside my Rails stuff, there is a single command where you point the app server to the root directory of your app and you're done.

With Rails 2.2 now out and offering thread safety, and JRuby being the only interperter that can take advantage of it... Glassfish and JRuby are really worth checking out.

SezHoo and WikiSym 2008

I'm proud to announce the release of SezHoo! It's a tool we created at MITRE to help establish reputations for authors in wiki sites... specifically those run on MediaWiki (the same software that runs Wikipedia).

The way that SezHoo works is that it goes through a wiki article's history and tracks the authorship of each word in the article. We use a pretty popular technique in text analysis called shingling to help us out.

For each revision of an article, we break it up into shingles. We're using shingles that are 6 words long, so each word will have 6 shingles associated with it. We credit authorship of the word to the author of the earliest shingle.

This allows us to add some information to a MediaWiki installation:

Pages
We show the authors of a page, and the percentages at which they have contributed. We also offer this information at the section level.

The information we compute is better than doing diffs with a standard MediaWiki install. Diffs are line ending based, so changing a word in a paragraph that has a single line ending can make it difficult to determine who wrote the paragraph. In addition, since we are looking for authorship of a word across all revisions at once (as opposed to a revision vs. revision comparison with a diff), this technique is immune to copying and pasting of text and reversion of vandalism.

Authors
We create a reputation for authors with 3 dimensions: quantity, quality and value. For all metrics, we assign a 5 star rating. The way this is done is by computing a raw score for all authors and then ranking them. Authors in the top 20% have a 5-star rating in that dimension.

For quantity, we simply calculate the number of word contributed by each author. This includes words that may no longer be in the current revision of the article. Value takes the percentage authorship of a page, and multiplies that by the number of page views (it's valuable if it's being read).

Quality is a lot more tricky. We assume that wiki articles are a Darwinian environment. That is, quality text will "survive" in that as the article is edited, other authors will leave the text alone. Poor quality text will be "killed" or deleted by other authors. We determine quality by looking at the percentage of text that is expected to be alive for an author after 8 revisions (you can change 8 to whatever you like, but it works well for our internal wiki). To calculate the percentage alive after 8 revisions, we used a Kaplan-Meier estimate. This was necessary due to the fact that authors will have phrases which are "alive" (they are in the current version of an article). So if a word has survived 3 edits and is in the current version of an article, when calculating survival probabilities you don't want to say that it "died" at 3 revisions, but you don't want to lose the data either. Kaplan-Meier handles that for us.

In the end, you get a 5-star rating for each author in the dimensions of quantity, quality and value.



I'll be presenting this work at WikiSym 2008. We're doing some other interesting stuff which we haven't rolled into SezHoo yet, so it's worth coming to my talk even if you've read this blog post.

Using Hadoop

Since I've gotten back from OSCON, I've had a chance to use Hadoop at work. For those who aren't familiar with it, Hadoop is an open source framework for implementing map reduce jobs.

There are plenty of tutorials on Hadoop around the web, so I won't do any of the basic intro stuff. I wanted to write about some of the stuff I didn't find all that easily.

Most of the Hadoop documentation talks about dealing with input where your records are on a single line (like an Apache access log). From using the Google/documentation/experience, we have found that Hadoop works just fine with multi-line records. We're using Hadoop to process XML, specifically a dump from Wikipedia. The dump is a single 33 GB file, where there is a root tag, and then several million child tags (representing Wikipedia pages). Using this code I found on the Hadoop core user mailing list, we can have it so that the mapper gets the XML for one child node (or one Wikipedia page). This is nice, because the XML for a single page is relatively small. We then use JDOM to deal with the contents of the individual pages.

We are using HDFS to store our input and output. By default, it will chop files into 64MB chunks, which get shipped to the mappers so that the file can be processed in parallel. One thing that I was concerned about was how records would be handled that spanned the splits. So far, we haven't seen any issues, and this answer in the Hadoop FAQ seems to indicate that the records spanning the split will be handled. It may be possible that those records get dropped, it would be hard for us to tell at this point... but the good news for us is that it won't affect the data we're trying to collect much.

As for our setup and usage. We have 5 machines in our cluster, most of which are run of the mill dual core Intel machines running Linux. The jobs we're running on the 33GB XML file are taking around 45 minutes, which seems pretty fast to me.

OSCON 2008: Wrap Up

So I've returned from Portland after all of the OSCON activities. The conference was good, but I definitely didn't feel like it was a good as in years past. The keynotes were OK, but there weren't any that were spectacular. I hit a few sessions that were bombs, but I didn't get to one that rocked. Many were good, but nothing off the charts.

Hadoop was big on Thursday. Derek Gottfrid of the New York Times talked about how they used Hadoop and Amazon EC2 to process tons of data. Derek's presentation style is great, which mate the talk entertaining. Some folks from Yahoo were also getting into the nitty gritty details of how the whole thing works too.

The MySQL Proxy talk was good. It seems like a pretty handy tool for performance tuning and all sorts of SQL trickery.

The last talk that stood out to me was Selectricity. The project is a both a site to run elections as well as the software you can use to run elections where ever you want. One point that Benjamin Mako Hill made that I thought was interesting is that most election research goes into government elections... and these are the least likely to change. By building a tool to allow folks to conduct elections for simple things (what movie to see, who will lead the coffee club, etc.) using methods different from plurality, it's a good way to sneak in alternative voting methods to the masses. That way if people get familar with Condercet when voting for the next American Idol, they may be more likely to push for election reform in government elections.

I'm not sure if I'll hit OSCON again next year. I like going because it's nice to get an overview of a lot of different techologies, as opposed to something like Rails Conf. But things did feel pretty shallow this year.

OSCON 2008: Day 3

Today, I gave my first OSCON talk on Laika. I think that the talk went pretty well. I had plenty of good questions from the audience, and I think I may have been able to snag a few people who were interested in contributing.

The speaking experience was pretty cool. I was in a fairly small room, and probably had about 20 to 30 people in the audience, which was pretty non-threatening. I would have been more nervous in one of the more cavernous rooms with 200 people.

As for the rest of the conference today...

XMPP has a lot of buzz for communicating in the cloud. There were a few talks on that today.

There is a lot of Ruby stuff going on outside of web apps. RAD seems to have a lot of buzz for using Ruby to work with Arduino. Ruby's also behind Adhearsion, a tool for building IVR's.

I missed the talk on CouchDB, but some of the folks I'm out here with said it was great.

On a conference logistics note... I was kinda bummed that some of the talks had filled, so I couldn't get in. I wound up missing the talk on Hypertable as well as one on XMPP in the cloud.

OSCON 2008: Day 2

Day 2 at OSCON.... Some of the highlights...

Practical Erlang Programming was great. Francesco Cesarini is a great speaker and delivered a great tutorial. While Erlang does make you look at things differently, I can see how it makes it a lot easier to write concurrent code.

While I was physched to see Mark Shuttleworth give a keynote (given my fondness for Ubuntu), the best keynotes tonight were definitley Robert (r0ml) Lefkowitz and Damian Conway.

R0ml's talk compared various software development methodologies to Quintilian's 1st century works on rhetoric. My take on his talk was that open software has a good development methodology since it doesn't really have a requirements phase. Code gets released early an often. Bugs are filed and patches are submitted. Then users and developers can look at bugs and patches are there to determine what is in the next release. This is different form a typical development methodology, where you need to decide what you want up front. In this model, people do what they want, and you take what you like in the end.

On the other hand Damian Conway is somewhere between insane and brilliant. His talks are hillarious, but the stuff that he is actually able to implement is crazy... I'm sure that we'll be seeing some talk of positronic variables on the tubes in the comming days.

OSCON 2008: Day 1

The first day at OSCON 2008 has come and gone... This year looks to be another good one. Here's what I saw on my first day.

The first session I went to was Python in 3 Hours. While I do most of my work in Ruby, I do try to keep an eye on Python. It seems like a pretty clean scripting language, and quite speedy when compared to Ruby.

The tutorial was good. The material is kinda dry (it's language syntax after all, which is pretty hard to spice up). Steve Holden's presentation was clear and well thought out. I walked away feeling like I could approach Python code now without too much fear. However, I still have some pretty mixed feelings about Python... There are a lot of little things that bother me... having to add a self parameter to instance methods, double under bar naming conventions and the whole significant white space thing. At any rate, I though the tutorial was informative.

The second tutorial I did was Making Things Blink: An Introduction to Arduino. This was a lot of fun. I haven't played with a microcontroller since college... but I've always loved working at the place where software meets hardware.

In the session, we worked through coding for Arduino as well as some basic circuits. The class culminated by building an Etch-a-Sketch. This is accomplished by hooking up two potentiometers to the Arduino, which reads the values and passes them to your computer via USB. We then used Processing to read and visualize the data on the screen. This meant that you could turns the knobs on the pots and draw on the screen pretty cool stuff.

Overall, one of the vibes I'm getting from the conference this year is big data. How do deal with really big databases and how to process tons of data in parallel. We'll see if this continues throughout the conference.

Speaking at OSCON

I'll be speaking about the work I'm doing on Project Laika at OSCON. Come check out the session to see how we're using rails to test electronic health record systems

DBSlayer is taking JSON to the next level

I ran across DBSlayer on the Rails Envy podcast and I was struck by a number of things. One, is that it is really cool that the New York Times has their own open code repository. Another is how far JSON is coming as the choice format for data exchange.

I remember reading a few posts on the tubes months ago that JSON was going to give XML a run for its money and I thought folks were crazy. I don't see JSON totally replacing XML, but I'm definitely seeing it used in many more places where XML would have been the natural choice a couple a of years ago.

DBSlayer also intrigues me because I can see writing some pretty interesting web apps without a traditional app server. I know that DBSlayer is intended to help with scaling your DB layer, but I think it would be cool to hit it directly from the browser. There would be problems with this approach... You'd pretty much only be able to build read only apps where you don't care who sees the data... but I can see plenty of apps fitting that mold (corporate directories, stock data, sports data, etc.)

Spring MVC, Data Binding and JPA

A project that I am working on is using Spring MVC. Naturally, we would like to use its data binding facilities to handle data coming in from HttpRequests into our beans. As mentioned by Matt Fleming in this blog post, the Spring docs don't do a great job of explaining how to deal with data binding when the bean in question has a collection as a property. Something like this:

public class Person
{
 private String name;
 private List<Address> addresses;
}

As Matt mentions, using LazyList found in the Apache Commons Collections is one way to get Spring to bind dynamic collections into your beans (in the example, use a lazy list for the addresses). However, you can run into some issues if your persisting your beans using JPA.

We're using OpenJPA, and when we try to persist an object that has a LazyList in one of its properties, it chokes. The problem is that it can't proxy LazyList because it doesn't have a default constructor. To get around this, we're applying a pretty simple hack. Before Spring does the data binding, we decorate the collection with a LazyList. Before we persist the object with JPA, we copy the collection back to a plain old ArrayList.

An example of doing this can be seen with Spring MVC's SimpleFormController in DocumentLocationCreationController. To help out with the whole process, we've created a LazyListHelper to make the decorating and copying easier.

If you're using Spring MVC with JPA, hopefully this will save you some headaches.

JRuby Compiler Performance

I was pretty excited when I heard that JRuby's compiler was complete. I figured I could run some benchmarks against the C-Based Ruby Implementation to see how it was performing. I've only run this suite once, but I hope to provide enough info so that you could replicate the results yourself. Please let me know if you think I missed something.

The Setup

The benchmarks were run on a Dell D820 laptop. It has an Intel Core Duo running at 2.17GHz and has 1GB of RAM. I'm running Ubuntu 7.04 for an OS. The only wrinkle in my setup is that my root partition is encrypted using LUKS/dmcrypt. This will probably slow down the IO benchmarks, but I'm assuming it would penalize both Ruby implementations equally.

For the C-Based Ruby Implementation, I'm using the Ubuntu packaged Ruby version 1.8.5. For JRuby, I checked out the latest from Codehaus's SVN (Rev 4474) and built the code using Sun's Java version 1.6.0.

For the benchmarks, I pulled the suite from the Ruby Lang SVN (Rev 13608). 

I ran the benchmarks using the run.rb script. The C-Based Ruby Implementation was run with no command line options. I modified the script to run JRuby withe the following options:

• -C to compile the ruby code before running it
• -O to disable ObjectSpace
• -J-Djruby.thread.pooling=true to enable thread pooling
• -J-server to put the JVM in server mode

These options are explained in the Performance Tuning and Compiler pages of the JRuby Wiki.

The Results

The following shows the time it took the Ruby Implementation to perform the benchmark in seconds:

Here is the difference between the two implementations (CRuby - JRuby) again, in seconds:

As you can see, JRuby is performing really well on a lot of these benchmarks. It gets killed on eval, but I suppose that is to be expected. My guess is that JRuby is taking advantage of Java primitives to outperform CRuby at number crunching. Charles Nutter's comments at another blog entry on JRuby benchmarking seem to indicate that this would be the case. Can't wait until JRuby is 1.1