Cloudera Director 2.7 introduces support for LDAP authentication, improved Java 8 support, and instance template level normalization configuration. Continuing improvements have been made to the AWS plugin.
Cloudera Director helps you deploy, scale, and manage Cloudera clusters in AWS, Azure, or Google Cloud Platform. Its enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big-data workloads and applications in a simple, reliable, automated fashion.
Cloudera Director Overview
In this post, you will learn about new functionality in release 2.7, but first, for those new to Cloudera Director, let’s revisit what it does.
- On-demand creation and termination of clusters: Using Cloudera Director, you can allocate and configure Cloudera Manager instances and highly available CDH clusters in the cloud provider of your choice. A single Cloudera Director instance can manage multiple cloud provider environments and the separate lifecycles of multiple Cloudera Managers and clusters.
- Multi-cloud support: Cloudera Director supports creating clusters in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) through its cloud provider plugin architecture. A single Cloudera Director instance can work with multiple cloud providers at once. Because the plugin specification is open source, you can create a plugin to support other providers, either in-house or public.
- On-demand grow and shrink of clusters: One of the main benefits of running Hadoop clusters in the cloud is being able to provision additional instances when demand increases, and to terminate instances when demand decreases. Cloudera Director, in concert with Cloudera Manager, does the work required to add new instances to and remove existing ones from your Hadoop clusters.
- Programmatic and repeatable instantiation of clusters: Cloudera Director can consume cluster definitions specified in HOCON configuration files submitted through the Cloudera Director CLI or in JSON input sent to the Cloudera Director API. A cluster definition can include custom scripts to run after instance provisioning and cluster setup, or before cluster termination, to perform tasks like installing additional packages, configuring system settings, or saving off important data.
- Long running cluster support: Long running clusters often require actions like upgrading CDH and Cloudera Manager, changing the topology of the cluster, and reconfiguring the cluster. Cloudera Director supports such modifications when using Cloudera Manager 5.11 and above.
- Usage-based billing for Cloudera services: Usage-based billing can help you optimize your expenditures for transient clusters. With a pay-as-you-go billing ID from Cloudera, you can use your Cloudera Enterprise license as usual, but you are only charged for CDH services when they are running.
- Security: Cloudera Director, like other Cloudera offerings, is committed to enabling secure deployments and applications. Cloudera Director’s own database is automatically encrypted, and Cloudera Director helps you configure Cloudera Manager and CDH clusters with Kerberos authentication, and now TLS for on-the-wire privacy, as well as deploy Cloudera Navigator for auditing, data lineage, and data discovery
- Auto-TLS: Cloudera Director can work in concert with Cloudera Manager to automatically configure Cloudera Manager and CDH for TLS.
- SSH host key security: Using information from the cloud provider, Cloudera Director can verify SSH host keys to prevent man-in-the-middle attacks.
- Powerful web user interface: Cloudera Director’s user interface provides a single dashboard to assess the health of all your clusters across all cloud providers and all Cloudera Manager deployments. It can also be used to bootstrap new clusters, grow and shrink existing clusters, and terminate clusters that are no longer needed. Exploring the web user interface is a great stepping stone to using the configuration file or API to deploy production-ready clusters.
New Features and Improvements in Cloudera Director 2.7
Many businesses and organizations use LDAP or Active Directory for user authentication and authorization. Cloudera Director 2.7 can be configured to point to your organization’s LDAP server so that your users’ common credentials may be used to login to Cloudera Director. Additionally, groups in LDAP will map to roles in Cloudera Director, which allows administrators to manage access to Director without needing to reconfigure Cloudera Director manually. When enabling LDAP support, Cloudera Director’s built in user management will be disabled.
Cloudera Director has enhanced Java 8 support, which allows Cloudera Director to handle all aspects of Java installation on instances that it allocates and configures for Cloudera Manager and CDH clusters, simplifying the process for users. Previously, Cloudera Director relied on Cloudera Manager’s Java installation mechanisms to install Java for cluster hosts, and only installed Java 7 on Cloudera Manager instances. Now, Cloudera Director itself can handle this installation and can be configured to use Java 8 instead of Java 7. Additionally, if users have specific Java packages that they’d rather use in a custom yum repository, Director can be told to use those packages instead. By setting up the custom yum repositories with bootstrap scripts or pre-baked images, Director can install any Java package that instances can find with yum.
Cloudera Director’s normalization process does a number of things to configure instances for use by Cloudera Manager and its clusters. In very specific circumstances, however, it may be advantageous to turn off one or more parts of the normalization process. For example, users may want to skip installation of Cloudera Director specific packages, leave SELinux turned on, or skip mounting external EBS volumes after allocating an instance. Cloudera Director 2.7 has added configuration flags into instance templates that allow users to enable/disable individual normalization steps per instance.
Cloudera Data Science Workbench (CDSW), for example, manages external volumes on its own. In Cloudera Director 2.6, this would have required application-level configuration that would have impacted all clusters managed by Cloudera Director. With the new instance template normalization configuration options, this can be specified in a Cloudera Director cluster configuration file, allowing for a much simpler deployment of CDSW.
The AWS plugin has added C5, M5, and H1 instance types, which will now show up in Cloudera Director UI’s drop down list of instance types. Additionally, users no longer need to base64 encode user data for instances by using the userDataUnencoded field when creating AWS instances. Plaintext can be supplied, which the AWS plugin will automatically base64 encode for the user data. However, if base64 encoding the user data is preferred for any reason, that is still supported as well.
For more information on what’s new in this release, check out the Cloudera Director 2.7 section of the New Features and Changes in Cloudera Director page in the Cloudera Director documentation.
Using Cloudera Director
If you’re ready to give the latest version of Cloudera Director a try, here are the ways you can get started.
* Use sample configuration files and scripts as starting points for setting up your clusters.
Send questions or feedback to the Cloudera Director community forum.
Michael Wilson is a software engineer at Cloudera.