Bala Venkatrao is the Director of Product Management at Cloudera.
As many of you know, we recently launched Cloudera Enterprise 3.7. Heres the link to the press release This release marked a transition from Cloudera Management Suite (CMS) to Cloudera Manager (CM), the industrys first and most comprehensive management application for Apache Hadoop. Over the last month we have received very positive feedback from our customers. I want to thank again all the Clouderans who spent countless hours bringing this product to market. I also want to take this opportunity to thank our customers for helping us get here, as many of them helped us to prioritize the key features for this release. Several customers have also shared the challenges/use cases from their Hadoop deployments and the need for specific features (more later) in Cloudera Manager. Many customers were actively involved in usability testing sessions for Cloudera Manager, which were immensely helpful!
At Cloudera, we strive hard to listen to our customers and help build products to address their needs. We hold regular meetings with customers, sharing early design prototypes and feature ideas and then quickly iterate on the feedback we receive. Cloudera Manager has been a result of this amazing collaboration with our customers and we look forward to this continued partnership as we build on our vision to make it even easier for our customers to manage their Hadoop environments.
Here are a few examples of features in Cloudera Manager and how they address some of the pressing needs for our customers:
Hadoop generates a ton of log data and important messages are buried in these log files. However, prior to Cloudera Manager, customers had to write their own scripts to search through these distributed log files attempting to identify which logs contained important messages. Extracting and presenting this information in an actionable manner is even more challenging.
With Cloudera Manager, we made it extremely easy for customers to search through numerous log files for important information. Based on the time range selected and the service type selected, CM will enable customers to search for key messages (filtered by severity or key word) across their Hadoop clusters. A single view presents the consolidated results with key summary information. In addition, CM also provides access to the full log files for further review and analysis.
Customers typically start reviewing the log files when Hadoop clusters are not behaving correctly. With the consolidated log management feature in CM, it is now easier and much more efficient to mine and extract useful information from these log files.
For the most part, Hadoop administrators work in a reactionary mode. They tend to address issues only when things have failed (jobs failed, slow running jobs, services stopped etc.). A constant request from customers is to be more proactive. What this translates to is the need for more actionable insights and recommendations from Cloudera Manager. To this extent, we have added a new feature in Cloudera Manager called Events.
There are several activities (messages, checks, actions etc.) happening in the Hadoop cluster that operators may benefit keeping an eye on. We have synthesized all of this relevant information into Events and made it available in a format that is easy to consume and take action on. Events in CM constitute the following:
Health events These events indicate the occurrence of certain health check activities, or when health check results have met specific conditions (thresholds). For example the overall health of a HDFS service changes from good/green to bad/red or the number of healthy datanodes falls below 95% (set threshold).
Log Message events – These events are generated when a log message matches a certain set of pre-defined rules of interest. The default set of rules is based on Cloudera’s experience in supporting several Hadoop clusters. We have mined through the various support tickets to identify patterns/messages in log files that matter the most and are worthwhile paying attention to. We are constantly adding to this list as we learn more from the various Hadoop clusters we help manage.
Activity events – These are events generated by the Activity Monitor; specifically, for jobs that fail, or that run slowly (as determined by comparison with duration limits).The time duration for these checks is configurable.
Audit events – These are events generated by actions taken through Cloudera Manager, such as creating, deleting, starting, or stopping services or roles. Configuration changes to a service or role also generate audit events that can be tracked.
Many of the above Events can be setup as Alerts and forwarded to the right operator as email notifications. We hope the combination of Events + Alerts feature in CM will enable the Hadoop Operator to be more proactive in managing their Hadoop clusters.
A core value proposition of the Cloudera Enterprise subscription is Support. We employ some of the key contributors on the various Hadoop projects and provide their expertise to address some of the pressing needs of our customers Hadoop problems. From talking to the support team and customers, we have seen that on many occasions, problem resolution gets delayed primarily due to lack of information about the problem and the customers Hadoop environment. Typically, it takes several iterations to get all the pieces of information of the customers Hadoop clusters (configuration files, log files etc.) to triage to the root cause.
With Cloudera Manager, we have greatly simplified this process. Support integration is built-in as part of this offering. What this enables is through a single click, customers can send all the relevant information about their cluster back to Cloudera support. This way, the support team and the customers are literally on the same page when working together to trouble shoot a problem and identify the right solution.
The above features highlight only a subset of the features in Cloudera Manager. In the coming weeks, we will have blog posts/videos that provide more details on the various features and how they help customer to better manage their Hadoop clusters.
For our existing customers I hope you have had the opportunity to try out Cloudera Manager. If not, you can download it from: https://ccp.cloudera.com/display/SUPPORT/Downloads
We are always eager to listen to your feedback as we are constantly looking for ways to further improve the product. Thanks again for all your support in 2011 and look forward to your continued collaboration going forward. If you are interested in participating in our monthly product meetings or for any other feedback regarding Cloudera Manager, please contact me at email@example.com
For prospective customers A free version of Cloudera Manager is available for download. More information available at http://blog.cloudera.com//products-services/tools/ We welcome your feedback at firstname.lastname@example.org