It’s been roughly three months since we announced GA of Cloudera Manager 4.0 (CM4) and I wanted to provide an update on its adoption and feedback from customers.

For those new to it, Cloudera Manager is the first and market-leading management platform for CDH (Cloudera’s Distribution Including Apache Hadoop). Enterprise customers are coming to expect an end-to-end tool that manages the entire lifecycle of their Hadoop operations. In fact, in a recent Cloudera customer survey, an overwhelming 95%  emphasized the need for this approach. 

Cloudera Manager sets the standard for enterprise deployment by delivering granular visibility into and control over every part of CDH – empowering operators to improve cluster performance, enhance quality of service, increase compliance and reduce administrative costs. We have also a FREE edition to get started, so try it out today! (BTW, for more information on this subject, you can attend a free Webinar on Wednesday, Sept. 19,  on the topic “How CBS Interactive Uses Cloudera Manager to Effectively Manage Their Hadoop Cluster”.)

We are pleased to announce the availability of Cloudera Manager 4.0.3. This is an enhancement release, with several improvements to configurability and usability. Some key enhancements include:

I’m very pleased to announce the immediate General Availability of CDH4 and Cloudera Manager 4 (part of the Cloudera Enterprise 4.0 subscription).  These releases are an exciting milestone for Cloudera customers, Cloudera users and the open source community as a whole.

Functionality
Both CDH4 and Cloudera Manager 4 are chock full of new features. Many new features will appeal to enterprises looking to move  more important workloads onto the Apache Hadoop platform. CDH4 includes high availability for the filesystem, ability to support multiple namespaces, Apache HBase table and column level security, improved performance, HBase replication and greatly improved usability and browser support for the Hue web interface. Cloudera Manager 4 includes multi-cluster and multi-version support, automation for high availability and MapReduce2, multi-namespace support, cluster-wide heatmaps, host monitoring and automated client configurations.

Other features will appeal to developers and ISV’s looking to build applications on top of CDH and / or Cloudera Manager. HBase coprocessors enable the development of new kinds of real-time applications.  MapReduce2 opens up Hadoop clusters to new data processing frameworks other than MapReduce. There are new REST API’s both for the Hadoop distributed filesystem and for Cloudera Manager.

In this demo video, Philip Zeyliger, a software engineer at Cloudera, discusses the Activity Monitoring and Operational Reports in Cloudera Manager.

Activity Monitoring

The Activity Monitoring feature in Cloudera Manager consolidates all Hadoop cluster activities into a single, real-time view. This capability lets you see who is running what activities on the Hadoop cluster, both at the current time and through historical activity views. Activities are either individual MapReduce jobs or those that are part of larger workflows (via Oozie, Hive or Pig).

Activity Monitoring provides many statistics – both in tabular displays and charts – about the resources used by individual Hadoop jobs and at the aggregate cluster level. The Comparison feature in Activity Monitoring shows the performance of the selected Hadoop job compared with the performance of other similar Hadoop jobs.

In this demo video, BC Wong, a software engineer at Cloudera, discusses the Hadoop Service Monitoring feature in Cloudera Manager. Service Monitoring helps you monitor and manage your Hadoop clusters effectively.

Through the Service Monitoring feature, customers can monitor dozens of service health and performance metrics about the overall service (HDFS, MapReduce, HBase). They can also examine underlying role instances (Namenode, Datanodes, JobTracker, TaskTrackers, Region Servers etc.) in your Hadoop cluster and see what’s going wrong – or what is about to go wrong.

Service Monitoring presents health and performance data in a variety of formats including interactive charts through Cloudera’s new, enhanced user interface. Every Service Monitoring page also includes a widget to enable quick search for relevant Events and Logs associated with the service under consideration. Important Event and Log messages are also highlighted in the various charts. You can also monitor metrics against customizable thresholds, which results in Alerts that operators can pay attention to.

In this demo, Henry Robinson, a software engineer at Cloudera, discusses the Log Management, Event Management and Alerting features in Cloudera Manager that help make sense out of all the discrete events that take place across the Hadoop cluster. He demonstrates how to search the logs valuable information, note important events that pertain to system health and create alerts to warn you when things go wrong.

Log Management

Every process in a Hadoop cluster regularly writes to a log file, which captures valuable data but also creates volumes of information that is difficult to manually sort. Cloudera Manager’s comprehensive log management feature contextualizes all system logs from across the Hadoop cluster and allows the operator to search and filter by service, role, host, keyword and severity. The application also proactively scans the log files for irregularities and warns you before the Hadoop cluster is impacted.

Event Management

With event management, Cloudera Manager proactively reports on important events in the Hadoop cluster such as a change in service health or metrics, log messages with a certain severity or keyword, or abnormal job performance. It creates and aggregates these relevant Hadoop events, and makes them available for searching and alerting.

Alerting

Service and Configuration Management (Part I & II)

We’ve recently recorded a series of demo videos intended to highlight the extensive set of features and functions included with Cloudera Manager, the industry’s first end-to-end management application for Apache Hadoop. These demo videos showcase the newly enhanced Cloudera Manager interface and reveal how to use this powerful application to simplify the administration of Hadoop clusters, optimize performance and enhance the quality of service.

In the first two videos of this series, Philip Langdale, a software engineer at Cloudera, walks through Cloudera Manager’s Service and Configuration Management module. He demonstrates how simple it is to set up and configure the full range of Hadoop services in CDH (including HDFS, MR and HBase); enable security; perform configuration rollbacks; and add, delete and decommission nodes.

Part I

Part I of the Service and Configuration Management demo focuses on managing services and configuring a cluster for optimal performance.  It also demonstrates how to administer users within Cloudera Manager, configure role-based permissions, and better manage security.

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Philip Zeyliger is a software engineer at Cloudera and started the SCM
project.

Two weeks ago, at Hadoop Summit, we released our Service and Configuration Manager (SCM) Express. It’s a dramatically simpler and faster way to get started with Cloudera’s Distribution including Apache Hadoop (CDH). In a previous blog post, we talked in some detail about SCM Express and what it can do for you.

The screencast included in this post demonstrates the simplicity of a CDH installation using SCM Express. The “Directors” conversing in the background are engineers Philip Langdale and Philip Zeyliger and VP of Products, Charles Zedlewski.

Video