Tracking Hadoop Jobs from Your Mac: There’s an App for That

Our thanks to Etsy developer Brad Greenlee (@bgreenlee) for the post below. We think his Mac OS app for JobTracker is great!

JobTracker.app is a Mac menu bar app interface to the Hadoop JobTracker. It provides Growl/Notification Center notices of starting, completed, and failed jobs and gives easy access to the detail pages of those jobs.

When I started writing Apache Hadoop jobs at Etsy, I found myself wasting a lot of time checking the JobTracker page to see how my job was progressing. The first thing we did to try to solve this problem was to write a Scalding flow listener to announce completed and failed jobs to IRC, but that got a little noisy. So I wrote JobTracker.app.

Installation and Usage

You can download the binary from its GitHub project page. Just unzip it and drop it into your Applications folder. Running it will put a little pith helmet in your menu bar. Clicking that gets you this menu:

You’ll first need to go to Preferences and enter your JobTracker URL:

By default it will track all jobs. You probably don’t want this, so put your username and any other usernames you want to track in the “Usernames to track” field, comma-separated.

Note that this has only been tested with the version of Hadoop that Etsy is running internally. Due to the somewhat horrifying way that the app gets the JobTracker data (by parsing the JobTracker HTML page, since there’s currently no API to JobTracker except via Java), it’s not unlikely that it could break on a different version of Hadoop/JobTracker. If you try it and it doesn’t work for you, file an issue on GitHub and I’ll work with you on fixing it.

Future Development

Next on my list of features is allowing for tracking multiple clusters at once. If you have any requests, please let me know.

Filed under:

No Responses

Leave a comment


three + 2 =