Developing Applications for HUE

Yesterday’s post gave an overview of the HUE (aka. Hadoop User Experience) project which was released in CDH3b2 and available on github. HUE is a graphical “desktop” style web application that runs in modern browsers (Firefox, Chrome, Safari, and IE8+) that allows users to interact with a Hadoop installation as if it were just another computer. They browse the file system, create and manage user accounts, view and edit files, upload files, and then use some Hadoop-specific applications like the Job Browser and Beeswax (our Hive app). Here’s a quick demo from yesterday’s post running through Beeswax. It’s about 10 minutes long, but even if you only watch the first 2 or 3 you’ll get an idea of what HUE is and what it can do.

This post is focused on what it means to develop for HUE, the source of which is available on github for all your forking pleasure.

Developing For HUE On The Server Side

Applications for HUE are usually implemented in Django, a popular MVC web framework that understands the application namespaces. On top of that, the SDK lets the application bundle and start helper daemons which might, for example, talk to various interfaces in Hadoop, HDFS, or one of the numerous other applications that ship with CDH3b2 (such as Hive in the video example above).

To develop applications for HUE developers need only be familiar with basic web development practices. Learning Django is pretty straight forward; there are a lot of terrific resources online, including a free online book, not to mention the excellent documentation on the Django project site itself.

Working with Web 1.0

When we develop applications for HUE here at Cloudera, we often talk about the “Web 1.0″ view. This is because the rich user interface you see when you look at applications in HUE are always built on top of plain old HTML with as little custom JavaScript and styling for each app. A typical HUE application returns tables of data, standard HTML form controls, lists of links for navigation and other simple web components. This makes building, debugging, and testing these web applications easier and also makes life easier for developers who want to build these apps.

Pulling Web 1.0 into HUE With JFrame

These “Web 1.0″ views aren’t that fun to use though. To convert these raw-looking “Web 1.0″ apps into slick user experiences with sortable tables, forms that validate inputs as you enter values, graphs, pretty navigation links and so on, developers pepper their HTML with semantic instructions that HUE knows how to convert into these highly interactive components.

HUE allows users to launch windows that fetch these Web 1.0 views via an AJAX class that we call JFrame. JFrame gets its name because it’s essentially a JavaScript implementation of an IFrame – when the user clicks a link or submits a form, the request is routed back into the same DOM element as if it were a little web browser. Before showing that response to the user, HUE applies filters to the HTML and transforms them into these nice looking widgets:

Developing On the Client Side

Let’s take a simple example and illustrate what we’re talking about here. Let’s say we want our application to have a nice looking table of data with truncated text when things get too verbose as well as sortable columns. These patterns are already defined in the HUE development environment, so developers don’t need to write any JavaScript or CSS to get an output that looks like this:

Instead, we require only that they decorate the HTML they return to have certain properties. Here’s what the HTML returned in the example above looks like:

 

<table class="sortable" data-filters="HtmlTable FitText-Children"
  data-fit-text="td" cellpadding="0" cellspacing="0">
    <thead>
      <tr>
        <th>ID</th>
        <th>TimeZone</th>
        <th>Name</th>
        <th>GEO Latitude</th>
        <th>GEO Longitude</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>22</td>
        <td>New York City, New York, United States of America</td>
        <td>America/New_York</td>
        <td>40.7255</td>
        <td>-73.9983</td>
      </tr>
      <tr>
        <td>23</td>
        <td>San Francisco, California, United States of America</td>
        <td>America/Los_Angeles</td>
        <td>37.7587</td>
        <td>-122.433</td>
      </tr>
    </tbody>
  </table>

Viewed on its own, it’s not that pretty nor is it interactive. To turn this plain HTML into a nice looking interactive interface, HUE uses a MooTools plugin we’ve developed called Behavior (the name being a nod to an old JavaScript library by that name, released back in 2005). MooTools Behaviors convert standard HTML components into attractive, interactive UI components. Things like ajax forms, tool tips, graphs and more. In the example above, the key to turning this table into a sortable table with truncated text is in the table tag:

 

<table class="sortable" data-filters="HtmlTable FitText-Children"
  data-fit-text="td" ...

These HTML5 data tags tell Behavior to run specific filters against these elements. These filters transform it into the screenshot above.

The HUE SDK UI Library

Currently the SDK library in HUE features roughly three dozen of these UI patterns and we add more every day. If you download and install HUE you’ll find an application called “JFrame Gallery” that demos all of them. It’s important to note that applications built in HUE don’t have to use these patterns. It’s easy to add your own if we’re missing something or if you don’t like the way the default one works. Whenever we write new applications we reuse these default patterns, but occasionally we find a need for a pattern that only makes sense for that application. When this happens we still write these HTML transformers, but they remain private to that specific application.

Thus developers who want to use the patterns we have already can crank out applications that look nice without writing CSS or JavaScript (or perhaps writing only a little), while others who want to roll things their own way can customize everything they like.

In today’s post I’ve only scratched the surface of how all this works. If you’d like to get a richer explanation I suggest you read through the SDK walkthrough and maybe download HUE and follow along, creating a simple web app using our tools.

An Invitation

We’re hoping to see people in the Hadoop community jump in and develop more applications for HUE and contribute to the ones already there. One of the benefits to this consolidated environment, aside from usability, the ability to cross-link apps, and not having to re-invent the wheel every time you want to browse the file system, is that anyone can jump in and contribute. If you think HUE could solve your problems if only it had feature X then we hope you’ll consider forking the repository on github and getting involved. If you have an idea for a killer app to make Hadoop better, we hope HUE is the right place for you to develop it. Sign up for the HUE Developer Group on Google Groups and let us hear from you.

If you would like to get started with HUE, download it, and read through the installation guide and user manual.

If you’re looking for ideas of things you might develop on HUE, give one of these a shot:

  • An HBase explorer
  • A Yahoo! Pipes-style UI for Oozie
  • A social gaming app that gives people points for doing interesting things to data (e.g. running a Hive query, using a Combiner, etc.) so you can quickly figure out the data experts in your organization.
  • Surprise us. Go nuts.

Look for more posts in the future that focus more on the server side of HUE development.

Filed under:

3 Responses
  • data recovery boston ma / November 10, 2011 / 5:17 PM

    Oh my goodness! an amazing article. Thank you!

  • Samsung Galaxy Nexus Accessories / November 21, 2011 / 3:00 PM

    It is really a nice and useful piece of info. I’m glad that you shared this useful info with us. Please keep us informed like this. Thank you for sharing.

Leave a comment


6 + two =