I’m visiting Paris, London, and Edinburgh this June. When I travel I like to talk to locals. And, wherever I am, I like to bicycle. So, I thought I might combine these interests and host “data rides” in these three cities.

In each city I’ll name a time and a meeting point, and then ride the local roads for an hour or two with whomever shows up. Afterward, we might need some libations at a local pub. I might even get Cloudera to throw in some schwag.

Ride dates are as follows:

• Paris: Tuesday, June 4

• London: Tuesday, June 11

• Edinburgh: Tuesday, June 18

All rides will start at 5pm. I’ll pick a meeting point closer to the event and tweet it from @cutting. Also, please tweet me if you have ideas of where we should ride.

Doug Cutting is Cloudera’s chief architect, a founder of the Apache Lucene and Apache Hadoop projects, and the current chair of the Apache Software Foundation.

• Search — Google figured out how to index all of the content on the internet so you (internet user) can find what you’re looking for.
• Social — Facebook, Twitter, LinkedIn and other social sites give your friends and other social connections a mechanism to push content you’re interested in to you, so you don’t have to search for it yourself.

So what does Gravity do? Its goal is to drive the third paradigm shift:

• Personal — Creating a web experience that is totally optimized based on your individual interests, behaviors, and preferences. Or, as Jim puts it, “showing you today what you’re going to search for tomorrow.”

Gravity collects and processes more than 10,000 data points every second. All of the data collected is loaded into HDFS, where two Apache Hadoop processes run. The first is a dynamic, real-time system that uses something called “eventual consistency,” meaning it correctly processes as many data points as it can—about both user activity across the web and content that is being published—in real time. 99.99% of that traffic is processed correctly. The second system runs every hour or two, catching the .01% of data points that were missed the first time around. Once the data is processed, it lands in Apache HBase where it is serialized and can be accessed via Apache Hive.

With several Scala engineers in house, the Gravity team decided in 2011 to use the Scala programming language instead of Java. It doesn’t natively integrate with Hadoop or HBase, so the Gravity team wrote its own open source library called HPaste, which allows Scala engineers to take advantage of all the unique features of Scala on top of HBase.

The results of this system?

• Higher click-through rates (CTR) — Gravity has measured CTR of people who engage with their personalized content versus standard segmented or generic content, and they’ve proven that personalized content delivers 300-400% higher CTR.
• Longer sessions — When personalized content is displayed on a web page, users stay on the page longer, which is a strong indication that they like the site more.
• More repeat visitors — If a web visitor sees personalized content their first time visiting a site, the number of times they return to that site afterward is more than 10X higher than when they engage with static content shown to all visitors. Gravity has proven this at scale across some of its largest customers.

Want to learn more?

• Read the full case study.
• Watch Gravity’s Jim Benedetto explain its use case on video.
• Explore the HPaste project on GitHub.
• Learn more about Gravity.

Karina Babcock is Cloudera’s Customer Programs & Marketing Manager.

What led you to your project idea(s)?

Conceptually, Apache Bigtop can actually be traced as far back as me working at Sun Microsystems in 2007-2008. I was assisting the team responsible for coming up with a 100% community-driven, open source Solaris distribution that could also be used as a basis for an enterprise-grade commercial product offering (which eventually became OpenSolaris). I then joined Yahoo! Inc. as a manager of a small team of extremely talented engineers tasked with integration efforts around Yahoo’s internal cloud offering based on Hadoop. Our project was called HIT (Hadoop Integration Testing) and we were known as “HIT-men”.

Aside from doing the initial commit, what is your definition of the project founder’s role across the lifespan of the project? Benevolent dictator, referee, silent partner?

Honestly, my role model is Linus Torvalds. He’s somebody who’s deeply passionate about the state of the community yet he still finds enough time to be involved with most technical aspects of the Linux kernel on a daily basis. But at the end of the day, he’s just plain fun to be around. Of course, the governance framework of Apache Software Foundation is quite different than the governance model of the Linux kernel. I use that excuse when I can’t quite measure up to Linus where influence is concerned.

What has surprised you the most about how your project has evolved/matured?

The elevator pitch for Bigtop has always been: Bigtop is to Hadoop what Debian is to Linux. The most surprising development to me was how well that message resonates with the commercial vendors in the Big Data space. I’m still amazed at how quickly the “Powered by Bigtop” list is growing.

What is the major work yet to be done, from your perspective as the project’s founder?

Developers, developers, developers!

We have to grow the community by leaps and bounds if we want to be remembered as the Debian of Hadoop. This translates into investing in outreach activities, but also (and maybe primarily) into creating enough value in the project itself so that external developers get hooked. Just steer clear of trying to boil the ocean by yourself — make the project interesting enough, and the developer community will come to the party.

What is your philosophy, if you have one, for balancing quality versus quantity with respect to contributions?

That’s a tough one. In the ideally balanced community everybody keeps an eye on all the proposed changes and casts +/-1 as needed. Hence there’s a self-throttling process that also provides a learning opportunity for the newcomers. Fundamentally though, developers don’t like to review patches; they like to write code. Making the review duty appealing is like making the broccoli-eating process appealing to your toddler. The bottom line is: You have to get creative (and personally brace yourself for way more reviews than direct code contributions).

At the same time, accepting somebody’s patches is a great way of keeping that contributor around. Hence, personally, I try and err on the side of openness and community growth over polishing each patch ad infinitum. But my personal coding philosophy is: Commit early, commit often, and re-factor mercilessly.

Any other advice for other potential project founders?

Attachment is the root of all suffering; don’t get attached to your own code or ideas. The only thing that lasts is community. At ASF, we are reminded of the “community over code” mantra all the time, but it’s not just a phrase. It’s for real.

P.S.: Oh, and here’s one more crucial bit of advice: Before naming your project, make sure to check that the vanity license place with its name is available in your state.

Read other “Meet the Project Founders” installments:

- Doug Cutting (Apache Hadoop, Apache Avro, Apache Lucene)

This post covers configuring Eclipse to modify Hadoop’s source. (Developing applications against CDH using Eclipse is covered in a different post.) Hadoop has changed a great deal since our previous post on configuring Eclipse for Hadoop development; here we’ll revisit configuring Eclipse for the latest “flavors” of Hadoop. Note that trunk and other release branches differ in their directory structure, feature set, and build tools they use. (The EclipseEnvironment Hadoop wiki page is a good starting point for development on trunk.)

This post covers the following main flavors:

• The traditional implementation of MapReduce based on the JobTracker/TaskTracker architecture (MR1) running on top of HDFS. Apache Hadoop 1.x and CDH3 releases, among others, capture this setup.
• A highly-scalable MapReduce (MR2) running over YARN and an improved HDFS 2.0 (Federation, HA, Transaction IDs), captured by Apache Hadoop 2.x and CDH4 releases.
• Traditional MapReduce running on HDFS-2 — that is, the stability of MR1 running over critical improvements in HDFS-2. CDH4 MR1 ships this configuration.

The below table captures the releases and the build tools they use along with the preferred version:

Other Requirements:

• Oracle Java 1.6 or later
• Eclipse (Indigo/Juno)

Setting Up Eclipse

1. First, we need to set a couple of classpath variables so Eclipse can find the dependencies.
1. Go to Window -> Preferences.
2. Go to Java -> Build Path -> Classpath Variables.
3. Add a new entry with name ANT_PATH and path set to the ant home on your machine, typically /usr/share/ant.
4. Add another new entry with name M2_REPO and path set to your maven repository, typically $HOME/.m2/repository (e.g. /home/user/.m2/repository).

   

2. Hadoop requires tools.jar, which is under JDK_HOME/lib. Because it is possible Eclipse won’t pick this up:
1. Go to Window->Preferences->Installed JREs.
2. Select the right Java version from the list, and click “Edit”.
3. In the pop-up, “Add External JARs”, navigate to “JDK_HOME/lib”, and add “tools.jar”.

   

3. Hadoop uses a particular style of formatting. When contributing to the project, you are required to follow the style guidelines: Java formatting with all spaces and indentation as well as tabs set to 2 spaces. To do that:
1. Go to Window -> Preferences.
2. Go to Java->Code Style -> Formatter.
3. Import this Formatter.

   

4. It is a good practice to enable automatic formatting of the modified code when you save a file. To do that, go to Window->Preferences->Java->Editor->Save Actions and select “Perform the selected actions on save”, “Format source code”, “Format edited lines”. Also, de-select “Organize imports”.

   

4. For Maven projects, the m2e plugin can be very useful. To install the plugin, go to Help -> Install New Software. Enter “http://download.eclipse.org/technology/m2e/releases” into the “Work with” box and select  the m2e plugins and install them.

   

Configuration for Hadoop 1.x / CDH3

1. Fetch Hadoop using version control systems subversion or git and checkout branch-1 or the particular release branch. Otherwise, download a source tarball from the CDH3 releases or Hadoop releases.
2. Generate Eclipse project information using Ant via command line:
1. For Hadoop (1.x or branch-1), “ant eclipse”
2. For CDH3 releases, “ant eclipse-files”
3. Pull sources into Eclipse:
1. Go to File -> Import.
2. Select General -> Existing Projects into Workspace.
3. For the root directory, navigate to the top directory of the above downloaded source.

Configuration for Hadoop 2.x / CDH4 MR2

Apache Hadoop 2.x (branch-2/trunk based) and CDH4.x have the same directory structure and use Maven as the build tool.

1. Again, fetch sources using svn/git and checkout appropriate branch or download release source tarballs (follow CDH Downloads).
2. Using the m2e plugin we installed earlier:
1. Navigate to the top level and run “mvn generate-sources generate-test-sources”.
2. Import project into Eclipse:
1. Go to File -> Import.
2. Select Maven -> Existing Maven Projects.
3. Navigate to the top directory of the downloaded source.

   

3. The generated sources (e.g. *Proto.java files that are generated using protoc) might not be directly linked and can show up as errors. To fix them, select the project and configure the build path to include the java files under target/generated-sources and target/generated-test-sources. For inclusion pattern, select “**/*.java”.

   

3. Without using the m2e plugin:
1. Generate Eclipse project information using Maven: mvn clean && mvn install -DskipTests && mvn eclipse:eclipse. Note: mvn eclipse:eclipse generates a static .classpath file that Eclipse uses, this file isn’t automatically updated as the project/dependencies change.
2. Pull sources into Eclipse:
1. Go to File -> Import.
2. Select General -> Existing Projects into Workspace.
3. For the root directory, navigate to the top directory of the above downloaded source.

Configuration for CDH4 MR1

CDH4 MR1 runs the stable version of MapReduce (MR1) on top of HDFS from Hadoop 2.x branches. So, we have to set up both HDFS and MapReduce separately.

1. Follow Steps 1 and 2 of the previous section (Hadoop 2.x).
2. Download MR1 source tarball from CDH4 Downloads and untar into a folder different than the one from Step 1.
3. Within the MR1 folder, generate Eclipse project information using Ant via command line (ant eclipse-files).
4. Configure .classpath using this perl script to make sure all classpath entries point to the local Maven repository:
1. Copy the script to the top-level Hadoop directory.
2. Run $ perl configure-classpath.pl
5. Pull sources into Eclipse:
1. Go to File -> Import.
2. Select General -> Existing Projects into Workspace.
3. For the root directory, navigate to the top directory of the above downloaded sources.

Happy Hacking!

Karthik Kambatla is a Software Engineer at Cloudera in the scheduling and resource management team and works primarily on MapReduce and YARN.

The schedule/agenda grid for HBaseCon 2013 (rapidly approaching: June 13 in San Francisco) is a thing of beauty.

If you lacked motivation to register up until this point, we think that this session line-up will convince you otherwise. We repeat: whether you’re an HBase committer or just getting started (or at any level in between), HBaseCon is simply an event that you can’t afford to miss – and with an entry fee of just $350, it’s also one you can easily afford.

Register now while there’s still room!

See also:

- Top 5 Reasons to Attend HBaseCon 2013
- HBaseCon 2013 Speakers, Tracks, and Sessions Announced

Taming Complexity

At Cloudera engineering, we have a big support matrix: We work on many versions of CDH (multiple release trains, plus things like rolling upgrade testing), and CDH works across a wide variety of OS distros (RHEL 5 & 6, Ubuntu Precise & Lucid, Debian Squeeze, and SLES 11), and complex configuration combinations — highly available HDFS or simple HDFS, Kerberized or non-secure, using YARN or MR1 as the execution framework, etc. Clearly, we need an easy way to spin-up a new cluster that has the desired setup, which we can subsequently use for integration, testing, customer support, demos, and so on.

This concept is not new; there are several other examples of Hadoop cluster automation solutions. For example, Yahoo! has its own infrastructure tools, and you can find publicly available Puppet recipes, with various degrees of completeness and maintenance. Furthermore, there are tools that work only with a particular virtualization environment. However, we needed a solution that is more powerful and easier to maintain.

Cloudera’s automation system for Hadoop cluster deployment provisions VMs on-demand in our internal cloud. As cool as that capability sounds, it’s actually not the most interesting part of the solution. More important is that we can install and configure Hadoop according to precise specifications using a powerful yet simple abstraction — using Cloudera Manager’s open source REST API.

Cloudera Manager API

This is what our automation system does:

1. Installs the Cloudera Manager (CM) packages on the cluster. Start CM server.
2. Uses the API to add hosts, installs CDH, defines the cluster and its services.
3. For configuration, we use the API to tune heap sizes, set up HDFS HA, turn on Kerberos security and generate keytabs, customize service directories and ports, and so on. Every configuration available in Cloudera Manager is exposed in the API.
4. The API also gives access to management features, such as gathering logs and monitoring information, starting and stopping services, polling cluster events, and creating a DR replication schedule. We use these features extensively in our automated tests.

The end result is a system that has become an indispensable part of our engineering process. It makes the Hadoop setup easy to maintain. For example, the same API call retrieves logs from HDFS, HBase, or any other service, without the user worrying about the different log locations. The same API call stops any service, without the user worrying about any additional steps. (HBase needs to be gracefully shutdown.) And when Cloudera Manager adds support for more services (e.g. Impala), their setup flows are the same as the existing ones.

Use Cases from Partners and Customers

Many of our customers and partners have also adopted the Cloudera Manager API for cluster automation:

• Some OEM and hardware partners, delivering Hadoop-in-a-box appliances, use the API to set up CDH and Cloudera Manager on bare metal in the factory.
• Some of our high-growth customers are constantly deploying new clusters, and have automated that with a combination of Puppet and the Cloudera Manager API. Puppet does the OS-level provisioning, and the software installation. After that, the Cloudera Manager API sets up the Hadoop services and configures the cluster.
• Others have found it useful to integrate the API with their reporting and alerting infrastructure. An external script can poll the API for health and metrics information, as well as the stream of events and alerts, to feed into a custom dashboard.

Code Samples

A previous blog post gave an example of setting up a CDH4 cluster using the Python API client. Instead of repeating that, let me introduce you to the Java API client. (Although our internal automation tool uses the Python client today, we plan to move to Java to better work with our other Java-based tools like jclouds.) To use the Java client, add this dependency to your project’s pom.xml:

 
<project>
  <repositories>
    <repository>
      <id>cdh.repo</id>
      <url>https://repository.cloudera.com/content/groups/cloudera-repos</url>
      <name>Cloudera Repository</name>
    </repository>
    …
  </repositories>
  <dependencies>
    <dependency>
      <groupId>com.cloudera.api</groupId>
      <artifactId>cloudera-manager-api</artifactId>
      <version>4.5.2</version>      <!-- Or the CM version you work with -->
    </dependency>
    …
  </dependencies>
  ...
</project>

The Java client works like a proxy. It hides from the caller any details about REST, HTTP, and JSON. The entry point is a handle to the root of the API:

RootResourceV3 apiRoot = new ClouderaManagerClientBuilder()
            .withHost("cm.cloudera.com")
            .withUsernamePassword("admin", "admin")
            .build()
            .getRootV3();

From the root, you can traverse down to all other resources. (It’s called “v3” because the currently Cloudera Manager API version is version 3. But the same builder also returns a v1 or v2 root.) Here is the tree view of some of the key resources and the interesting operations they support:

    * RootResourceV3
        * ClustersResourceV3: hosts membership, start cluster
            * ServicesResourceV3: config, get metrics, HA, service commands
                * RolesResource: add roles, get metrics, logs
                * RoleConfigGroupsResource: config
            * ParcelsResource: parcels management
        * HostsResource: hosts management, get metrics
        * UsersResource: users management

Of course, these are all in the Javadoc, and the full API documentation. To give a short concrete example, here is the code to list and start a cluster:

// List of clusters
ApiClusterList clusters = apiRoot.getClustersResource()
                                 .readClusters(DataView.SUMMARY);
for (ApiCluster cluster : clusters) {
  LOG.info("{}: {}", cluster.getName(), cluster.getVersion());
}

// Start the first cluster
ApiCommand cmd = apiRoot.getClustersResource()
                        .startCommand(clusters.get(0).getName());
while (cmd.isActive()) {
   Thread.sleep(100);
   cmd = apiRoot.getCommandsResource().readCommand(cmd.getId());
}
LOG.info("Cluster start {}", cmd.getSuccess() ?
            "succeeded" : "failed " + cmd.getResultMessage());

To see a full example of cluster deployment using the Java client, see whirr-cm. Specifically, jump straight to CmServerImpl#configure to see the core of the action.

You may find it interesting that the Java client is maintained with very little effort. Using Apache CXF, the client proxy comes free, quite magically. It figures out the right HTTP call to make by inspecting the JAX-RS annotations in the REST interface, which is the same interface used by the Cloudera Manager API server. Therefore, new API methods are available to the Java client automatically.

What’s Your Plan?

Overall, we are very happy with our automated deployment capability. I encourage you to try the Cloudera Manager API, and post your questions and feedback on the mailing list.

bc Wong is a Software Engineer on the Enterprise team.

JobTracker.app is a Mac menu bar app interface to the Hadoop JobTracker. It provides Growl/Notification Center notices of starting, completed, and failed jobs and gives easy access to the detail pages of those jobs.

When I started writing Apache Hadoop jobs at Etsy, I found myself wasting a lot of time checking the JobTracker page to see how my job was progressing. The first thing we did to try to solve this problem was to write a Scalding flow listener to announce completed and failed jobs to IRC, but that got a little noisy. So I wrote JobTracker.app.

Installation and Usage

You can download the binary from its GitHub project page. Just unzip it and drop it into your Applications folder. Running it will put a little pith helmet in your menu bar. Clicking that gets you this menu:

You’ll first need to go to Preferences and enter your JobTracker URL:

By default it will track all jobs. You probably don’t want this, so put your username and any other usernames you want to track in the “Usernames to track” field, comma-separated.

Note that this has only been tested with the version of Hadoop that Etsy is running internally. Due to the somewhat horrifying way that the app gets the JobTracker data (by parsing the JobTracker HTML page, since there’s currently no API to JobTracker except via Java), it’s not unlikely that it could break on a different version of Hadoop/JobTracker. If you try it and it doesn’t work for you, file an issue on GitHub and I’ll work with you on fixing it.

Future Development

Next on my list of features is allowing for tracking multiple clusters at once. If you have any requests, please let me know.

HBaseCon 2013 is approaching fast – June 13 in San Francisco. If you’re on the fence about attending – or perhaps your manager is on the fence about approving your participation – here are a few things that you/they need to know (in no particular order):

1. HBaseCon is the annual rallying point for the HBase community. If you’ve ever had a desire to learn how to get involved in the community as a contributor, or just want to ask a committer or PMC member why things are done (or not done) a certain way, this is your opportunity – because this is where those people are. Participating in a mailing list thread is never quite the same once you’ve met the people behind it. 
    
2. HBaseCon is a one-stop shop for learning about the HBase roadmap, as well as other projects across the ecosystem. Current HBase users should be particularly interested in learning about which JIRAs will have the most impact on the user experience – and once again, most of the committers working on those JIRAs will either be leading sessions or otherwise present. Plus, you can learn about how new complementary projects like Impala, Kiji, Phoenix, and Honeycomb are transforming the use cases for HBase and helping to expand its footprint across the enterprise.
    
3. HBaseCon is a feast of real-world experiences and use cases. Sure, maybe you’ve read about the HBase-backed applications used by companies like Facebook, Salesforce.com, eBay, Pinterest, and Yahoo!. But wouldn’t it be helpful to hear technical details and best practices directly from the people who built and run them? I’ll bet it would. And you really can’t do that anywhere else — in the whole world. (Plus, you can take advantage of formal training right before the conference, at a discount.)
    
4. HBaseCon is a pageant of engineer rock-stars. If your company is an HBase user and hungry for talent, there’s no better place to find it: HBaseCon is literally the world’s biggest gathering of HBase experts under one roof.
    
5. HBaseCon is a heck of a blast. Come for the deep-dives and advice, stay for the after-event party. The libations will be extensive!

If you have any interest in HBase whatsoever, whether as a user or prospective user, missing HBaseCon is almost unthinkable. 

Register early, because space is limited and filling up fast. Don’t get left out!

Apache HBase is a distributed big data store modeled after Google’s Bigtable paper. As with all distributed systems, knowing what’s happening at a given time can help  spot problems before they arise, debug on-going issues, evaluate new usage patterns, and provide insight into capacity planning.

Since October 2008, version 0.19.0 (HBASE-625), HBase has been using Apache Hadoop’s metrics system to export metrics to JMX, Ganglia, and other metrics sinks. As the code base grew, more and more metrics were added by different developers. New features got metrics. When users needed more data on issues, they added more metrics. These new metrics were not always consistently named, and some were not well documented.

As HBase’s metrics system grew organically, Hadoop developers were making a new version of the Metrics system called Metrics2. In HADOOP-6728 and subsequent JIRAs, a new version of the metrics system was created. This new subsystem has a new name space, different sinks, different sources, more features, and is more complete than the old metrics. When the Metrics2 system was completed, the old system (aka Metrics1) was deprecated. With all of these things in mind, it was time to update HBase’s metrics system so HBASE-4050 was started. I also wanted to clean up the implementation cruft that had accumulated.

Definitions

The implementation details are pretty dense on terminology so lets make sure everything is defined:

• Metric: A measurement of a property in the system.

• Snapshot: A set of metrics at a given point in time.

• Metrics1: The old Apache Hadoop metrics system.

• Metrics2: The new overhauled Apache Hadoop Metrics system.

• Source: A class that exposes metrics to the Hadoop metrics system.

• Sink: A class that receives metrics snapshots from the Hadoop metrics system.

• JMX: Java Management Extension. A system built into java that facilitates the management of java processes over a network; it includes the ability to expose metrics.

• Dynamic Metrics: Metrics that come and go. These metrics are not all known at compile time; instead they are discovered at runtime.

Implementation

The Hadoop Metrics2 system implementations in branch-1 and branch-2 have diverged pretty drastically. This means that a single implementation of the code to move metrics from HBase to metrics2 sinks would not be performant or easy. As a result I created different hadoop compatibility shims and a system to load a version at runtime. This led to using ServiceLoader to create an instance of any class that touched parts of Hadoop that had changed between branch-1 and branch-2.

Here is an example of how a region server could request a Hadoop 2 version of the shim for exposing metrics about the HRegionServer. (Hadoop 1’s compatibility jar is shown in dotted lines to indicate that it could be swapped in if Hadoop 1 was being used.)

This system allows HBase to support both Hadoop 1.x and Hadoop 2.x implementations without using reflection or other tricks to get around differences in API, usage, and naming.

Now that HBase can use either the Hadoop 1 or Hadoop 2 versions of the metrics 2 systems, I set about cleaning up what metrics HBase exposes, how those metrics are exposed, naming, and performance of gathering the data.

Metrics2 uses either annotations or sources to expose metrics. Since HBase can’t require any part of the metrics2 system in the core classes I exposed all metrics from HBase by creating sources. For metrics that are known ahead of time I created wrappers around classes in the core of HBase that the metrics2 shims could interrogate for values. Here is an example on how HRegionServer’s metrics(the non-dynamic metrics) are exposed.

The above pattern can be repeated to expose a great deal of the metrics that HBase has. However metrics about specific regions are still very interesting but can’t be exposed following the above pattern. So a new solution that would allow metrics about regions to be exposed by whichever HRegionServer is hosting that region was needed. To complicate things further Hadoop’s metrics2 system needs one MetricsSource to be responsible for all metrics that are going to be exposed through a JMX mbean. In order for metrics about regions to be well laid out, HBase needs a way to aggregate metrics from multiple regions into one source. This source will then be responsible for knowing what regions are assigned to the regionserver. These requirements led me to have one aggregation source that contains sudo-sources for each region. These sudo-sources each contain a wrapper around the region. This leads to something that looks like this.

Benefits

That’s a lot of work to re-do a previously working metrics system, so what was gained by all this work? The entire system is much easier to test in unit and systems tests. The whole system has been made more regular; that is everything follows the same patterns and naming conventions. Finally everything has been rewritten to be faster.

Since the previous metrics have all been added on as needed they were not all named well. Some metrics were named following the pattern: “metricNameCount” others were named following “numMetricName” while still others were named like “metricName_Count”. This made parsing hard and gave a generally chaotic feel. After the overhaul metrics that are a counter start with the camel cased metric name followed by the suffix “Count.” The mbeans were poorly laid out. Some metrics we spread out between two mbeans. Metrics about a region were under an mbean named Dynamic, not the most descriptive name. Now mbeans are much better organized and have better descriptions.

Tests have found that single threaded scans run as much as 9% faster after HBase’s old metrics system has been replaced. The previous system used lots of ConcurrentHashMap’s to store dynamic metrics. All accesses to mutate these metrics required a lookup into these large hash maps. The new system minimizes the use of maps. Instead every region or server exports metrics to one pseudo source. The only changes to hashmaps in the metrics system occurs on region close or open.

Conclusion

Overall the whole system is just better. The process was long and laborious, but worth it to make sure that HBase’s metrics system is in a good state. HBase 0.95, and later 0.96, will have the new metrics system.  There’s still more work to be completed but great strides have been made.

Elliott Clark is a Software Engineer at Cloudera and an HBase committer.

At Alteryx we are excited about the release of Cloudera Impala. The impact on Big Data Analytics is that the ability to perform real-time queries on Apache Hadoop will provide faster access and results. This is applicable to our customers, the business users who are running analytics to get access to data, perform analytics, and then follow up with new questions. Insight doesn’t happen all at once. The ability to query and refine quickly is ultimately what will lead business users to insight.

As business users need faster access to data, Alteryx provides a user friendly way to access new solutions like Impala. With Impala support in Alteryx Strategic Analytics, business users can get faster access, and can refine data queries and the corresponding analytics to get the answers they need. They can combine these results with other datasets to provide the context necessary to make the right decision, and they can do it without having to go through months of training to master programming and query languages.

A great example of where Impala can have a big impact is in churn analytics. When customers leave a company or service, there are usually a few interactions prior to leaving that are the cause. In the telecom world, these interactions can be dropped calls, support queries, and rate adjustments. The interactions that lead to churn can happen over the course of just a few hours. To be able to log those events, and then have them show up in an analytics query quickly so customers can be saved, can have a huge impact on an organization. Impala enables Alteryx to iteratively analyze the fast moving data involved in churn analysis and prevention.

At Alteryx, we recognize that organizations not only need to scale technology to address Big Data, they need to scale human capabilities. That is why Alteryx Strategic Analytics provides an easy to use drag-and-drop interface for business users. Subject matter experts and business analysts can quickly build analytics workflows that gather, cleanse, and blend datasets; enrich them with third party data; and then run sophisticated statistical, predictive, or geo-spatial analytics. By giving business users the access to query, analyze, and refine quickly, the analysis takes place at the business user level, where the business impact is understood. Then by having business users run the analytics, it enables the organization to scale.

With Impala support, Alteryx enables business users to benefit from huge innovations in the Big Data market. As the market matures and more ways of accessing data become available, Alteryx provides an easy interface that enables users to benefit from the power of these innovations, while shielding them from the complexity. This makes users more productive, and able to focus on getting the answers they need to make better decisions faster.

For more information about the Alteryx integration with Cloudera, visit www.alteryx.com/cloudera.