Learn the new features and enhancements in Cloudera Manager 5, including support for YARN, management of third-party apps and frameworks, and more.
The response to the Oct. 2013 release of Cloudera Enterprise 5 Beta has been overwhelming, and Cloudera is busily working closely with several customers to incorporate their feedback.
Cloudera Manager 5 is a key part of this release, and in this post, I will provide a brief overview of some key features in Beta 1 as well as introduce some of those planned for Beta 2 (to be released in early 2014).
Workload and Resource Management
A major theme of the beta release is the notion of supporting multiple workloads on the same data substrate. Effective resource management becomes an important criterion to make this vision a reality. With Cloudera Manager 5, the plan is to have YARN production ready to support dynamic resource allocation (for different applications that leverage YARN). In addition, we continue to support static partitioning (via cgroups) to divide cluster resources (cpu, memory, and so on) among these stand-alone processes. Cloudera Manager 5 adds several knobs and parameters to manage all these resource management aspects in a simplified and streamlined fashion.
Resource Management: static partitioning
Resource Manager: dynamic partitioning
Customers are also asking for an easier way to manage non-CDH services and ISV applications that are deployed on top of, and along with, the CDH stack. Many have already standardized on Cloudera Manager as their Apache Hadoop management platform of choice, so, a natural next step is to use Cloudera Manager to manage these additional/new services.
The Service Extensibility mechanism in Cloudera Manager 5 provides different avenues for non-CDH services and ISV applications to be managed via Cloudera Manager. A good example is Cloudera’s recent collaboration with Syncsort to facilitate the deployment of its DMX-h libraries via Parcels. The complete functionality will enable customers to manage the entire lifecycle of new services (such as Apache Accumulo, Apache Spark [incubating], and so on) via this mechanism. The end goal is to have customers write a simple service descriptor (a JSON file along with set of control scripts) for a new service that gets managed by Cloudera Manager.
The plan is to have good set of examples, documentation, and sample code available as part of Beta 2 (or by GA) for customers to try this on their own for any new service they would like to deploy. In the interim, we continue to work with select partners like SAS, 0xData, Syncsort, and others to fine-tune the implementation.
We have also added multiple new monitoring capabilities in Cloudera Manager. More specifically, data visualization has been beefed up, including the ability to chart the time-series metrics as bar graphs, scatter plots, heat maps, and so on. We have also added more thorough monitoring for Cloudera Impala queries and added support for YARN/MR2 monitoring.
Enhanced charting capabilities
Other New Features & Enhancements
The above lists only a subset of all new features that are part of Cloudera Enterprise 5. For example, significant work was also done to ease the overall upgrade process from CDH 4 to CDH 5 and MR1 to MR2/YARN. On the security side, we added SAML support for SSO access to Cloudera Manager. We also now support JDK 7. (The entire list of updates and enhancements in Beta 1 is available here.)
The team is now busy working on a Beta 2 release, which is currently scheduled to include support for Apache Oozie HA, YARN Resource Management HA workflows, HDFS caching, user-defined triggers, and more. In the meantime, please try out Cloudera Enterprise Beta 1 and give us your feedback via any of the following:
- A beta specific community forum has been set up. Click here to join.
- File a bug through our public JIRA at:
Bala Venkatrao is a Director, Products at Cloudera.