Cloudera Desktop and MooTools

At Hadoop World NYC Cloudera announced a new product: Cloudera Desktop. Over the past several months this product has been my principal concern here at Cloudera where I’m the UI lead (actually, until about a week ago, I was the only UI developer).

If you aren’t familiar with Cloudera Desktop, you should check out this brief screencast:

Cloudera Desktop Screencast from Cloudera on Vimeo.

Making Desktop has been a fun, though difficult, task that involved a lot of trial and error (and in many ways we’re just getting started). In the screencast above you can see roughly half a dozen applications running on Desktop. Each of these has code conventions unique to it, and shared with the other applications. Those conventions evolved as we went from application to application, refining and changing direction, building and using software and learning lessons.

MooTools

Cloudera Desktop uses many technologies including Hadoop (obviously), Thrift, Python, Django, and others. On the client side, though, which is where I spend most of my time developing, we use HTML, CSS, and JavaScript (lots of JavaScript) — specifically, the MooTools JavaScript framework.

MooTools is a robust framework that extends well past the basics of DOM manipulation and provides numerous low level utilities and extensions to the JavaScript language (enhancing the native prototypes of Function, Array, etc), as well as exposing JavaScript’s native inheritance model with psuedo-classes.

Design Principles

When we began working on this project to provide tools for Hadoop users, our goal was to make life easier for administrators, developers, and analysts. Hadoop’s principal interface to date has been the command line and a few basic web pages that allow you to see the output and status of jobs. Other projects have started work on making the interface easier. For instance, Yahoo! and Facebook have many non-programmers who use Hadoop to do analysis by way of custom, internal applications. Part of our consideration was that we didn’t want users to have to choose. Ideally, if there’s an awesome tool that help users accomplish a task out there, it should play well with whatever we develop.

picture-4
The File Browser

This led us to consider a Desktop model with numerous windows, with the idea that other developers could expose their applications within our framework and make use of common tools. For instance, a familiar-looking file browser ships with Cloudera Desktop. Other applications can invoke this file browser to allow the user to select a file as input, or a directory as output, or even upload new files easily and quickly. This kind of integration meant making a multitasking environment on which third parties could develop and this lead us to the multi-windowed desktop metaphor.

MooTools Contributions

For the past several months as we’ve been building Cloudera Desktop, we’ve been busy also making a lot of contributions back to MooTools and we wanted to share them with you here. Some of these things are simple UI conventions while others are interesting for their technique.

MooTools Depender

MooTools ships with a dependency map that powers its download builder. The modular nature of the library yields itself to custom builds, putting together a library specific to the task at hand. This allows MooTools to power, for instance, a mobile application with only a small amount of JavaScript. For the Cloudera Desktop, we knew we were going to end up with a LOT of JavaScript, and loading it all on startup didn’t make much sense. Instead, we authored the Depender application. It’s an easy-to-deploy, real-time library builder and dependency mapper. This allows our application to load with a minimum of JavaScript. When users launch specific applications, Depender loads any dependencies for that app that aren’t loaded already, and then display the application. In addition to the server side component (available in both PHP and Python/Django), there are two client side components: a stand alone version to be released in MooTools 1.2.4 and a server side application that ships with a client that talks to the server for you, which lets you do this slickness:

Depender.require({
    scripts: ['DatePicker', 'Logger'], //array or single string for one item
    callback: function() {
        //your code that needs DatePicker and Logger
    }
});
//later, you need to load more dependencies...
Depender.require({
    scripts: 'Fx.Reveal', //array or single string for one item
    callback: function(){
        //if, for some reason, Fx.Reveal is available already,
        //then this function will exeute immediately, otherwise it will
        //wait for the requirements to load
        $('someElement').reveal();
    }
});

The Depender app is now part of the MooTools project and will be powering MooTools.net as well as the forthcoming user-created plugin repository’s dependency builder.

MooTools ART

picture-2
ART Windows

MooTools at the moment doesn’t have an official, public UI system, but that’s changing, and in no small part due to our contributions to the MooTools ART project. MooTools ART is an in-development UI library that currently outputs canvas. It’s an abstraction of the canvas API and it allows developers to make style-able UI elements like buttons, windows, and icons. At the moment it only outputs to canvas (limiting its support to browsers other than Internet Explorer), but we’re working on wrappers for VML and SVG.

In addition to these drawing tools provided by the ART API is a widget-based system that has numerous features including keyboard management, event bubbling, custom styling, and more. This widget system is the foundation for many of our UI elements, though not all of them. While the basic ART API was developed by the core MooTools Team (of which I am a part), we’ve contributed most of the widgets available in the library built with that API, including a window manager, a history interface, pop-ups for alert, confirm, and prompt, split views and more.

MooTools More

The MooTools Core is the basic distribution of the framework. It is minimal and changes infrequently. In addition to this library are official MooTools plugins. Version 1.2.4, soon to be released, has numerous new features and bug fixes, many contributed to by Cloudera, including:

  • Depender (the stand alone version of the afore mentioned dependency loader)
  • Mask – masks elements (including the window) with a semi-opaque overlay
  • Spinner – automates the creation of ajax spinners over DOM elements being updated
  • Form.Request – automates creating ajax forms that update HTML in DOM elements
  • Keyboard – a robust event manager for keyboard groupings

In addition, there were numerous bug patches and other smaller features. It’s important to note that the MooTools developer community helped in the development and integration of these into the official MooTools code base. We contributed much of the code in these features, but they were tested, refined, and optimized in the spirit of open source collaboration.

Third Party Plugin Contributions

In addition to these contributions to the MooTools projects we also made a few other smaller contributions. We contributed to the SubtleTemplate data binding plugin by Thomas Aylott with optimizations and small bug fixes as well as a more robust manager for binding a single data object to numerous template instances.

We also contributed to the Clientcide plugins (which isn’t saying much as these are my personal plugins), mostly with bug fixes and minor features.

Future Open Source Plans

Cloudera remains committed to open source and as such you should expect to see even more output to the MooTools community and JavaScript in general. We recently added another UI developer, Nathan White, to our ranks, which essentially doubles our capacity to crank out useful stuff for others. We hope to release more things from our Desktop environment for others, where it makes sense, and give back to and tap into the thriving MooTools Community. We have numerous little plugins that we want to share but haven’t had time to release. In the mean time, we look forward to continuing to support the things we have released and doing what we can to help the technologies we utilize prosper.

If you’d like to try Cloudera Desktop, you can download it for free at http://cloudera.com/desktop. Other resources include:

Filed under:

7 Responses
  • tofui / October 20, 2009 / 11:23 AM

    The best part IMO was the last bit about cluster health. I want an SMS/email/both sent to the engineers when a node goes red :)

  • Oleg Konovalov / November 05, 2009 / 9:22 AM

    Hi,

    Any plans of creating similar Desktop with Flex/ActionScript ?
    Or any way of interfacing with Flex ?

    I do realize that it is not very difficult to connect JavaScript to ActionScript and vice versa, just fishing if anybody has already had experience (maybe created libs for that). :-)

    TIA,
    Oleg.

  • Aaron Newton / November 05, 2009 / 12:10 PM

    We don’t have Flex/Actionscript on our roadmap. If, as our SDK work matures, there’s a demand for it, we’ll consider it.

Leave a comment


+ nine = 12