Cloudera Director 2.6 introduces support for protecting communications with TLS and SSH host keys. Azure support is enhanced with support for Azure Managed Disks and custom images..
Cloudera Director helps you deploy, scale, and manage Cloudera clusters in AWS, Azure, or Google Cloud Platform. Its enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big-data workloads and applications in a simple, reliable, automated fashion.
Cloudera Director Overview
In this post, you will learn about new functionality in release 2.6, but first, for those new to Cloudera Director, let’s revisit what it does.
- On-demand creation and termination of clusters: Using Cloudera Director, you can allocate and configure Cloudera Manager instances and highly available CDH clusters in the cloud provider of your choice. A single Cloudera Director instance can manage multiple cloud provider environments and the separate lifecycles of multiple Cloudera Managers and clusters.
- Multi-cloud support: Cloudera Director supports creating clusters in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) through its cloud provider plugin architecture. A single Cloudera Director instance can work with multiple cloud providers at once. Because the plugin specification is open source, you can create a plugin to support other providers, either in-house or public.
- On-demand grow and shrink of clusters: One of the main benefits of running Hadoop clusters in the cloud is being able to provision additional instances when demand increases, and to terminate instances when demand decreases. Cloudera Director, in concert with Cloudera Manager, does the work required to add new instances to and remove existing ones from your Hadoop clusters.
- Programmatic and repeatable instantiation of clusters: Cloudera Director can consume cluster definitions specified in HOCON configuration files submitted through the Cloudera Director CLI or in JSON input sent to the Cloudera Director API. A cluster definition can include custom scripts to run after instance provisioning and cluster setup, or before cluster termination, to perform tasks like installing additional packages, configuring system settings, or saving off important data.
- Long running cluster support: Long running clusters often require actions like upgrading CDH and Cloudera Manager, changing the topology of the cluster, and reconfiguring the cluster. Cloudera Director supports such modifications when using Cloudera Manager 5.11 and above.
- Usage-based billing for Cloudera services: Usage-based billing can help you optimize your expenditures for transient clusters. With a pay-as-you-go billing ID from Cloudera, you can use your Cloudera Enterprise license as usual, but you are only charged for CDH services when they are running.
- Security: Cloudera Director, like other Cloudera offerings, is committed to enabling secure deployments and applications. Cloudera Director’s own database is automatically encrypted, and Cloudera Director helps you configure Cloudera Manager and CDH clusters with Kerberos authentication, and now TLS for on-the-wire privacy, as well as deploy Cloudera Navigator for auditing, data lineage, and data discovery.
- Powerful web user interface: Cloudera Director’s user interface provides a single dashboard to assess the health of all your clusters across all cloud providers and all Cloudera Manager deployments. It can also be used to bootstrap new clusters, grow and shrink existing clusters, and terminate clusters that are no longer needed. Exploring the web user interface is a great stepping stone to using the configuration file or API to deploy production-ready clusters.
New Features and Improvements in Cloudera Director 2.6
One major focus of the Cloudera Director 2.6 release is on improved support for TLS. Before this release, Cloudera Director was unable to communicate with a Cloudera Manager installation that had been configured for TLS. While this did not inhibit the functioning of Cloudera Manager or its clusters, it did prevent Cloudera Director from working with Cloudera Manager in order to perform operations like growing and shrinking of clusters.
This situation has been improved for Cloudera Director 2.6 with the introduction of a new automatic TLS capability. With a single flag you provide in a Cloudera Director configuration file, Cloudera Manager is automatically configured with its own certificate authority (CA). All of Cloudera Manager’s services, and cluster services such as HDFS, Hive, Hue, and Impala, are then set up automatically with their own server certificates, and configured for TLS communications where they support it. What’s more, Cloudera Director is automatically configured to communicate with Cloudera Manager over TLS, so that you can keep using it to maintain your clusters.
Some organizations would rather have Cloudera Manager and cluster server certificates signed by their own CA. In that case, Cloudera Manager can be configured for the full range of TLS protections as usual, and Cloudera Director can be updated in one API call to communicate with Cloudera Manager over TLS. If you are already used to managing a PKI for your Cloudera Manager deployments and CDH clusters, Cloudera Director can now work with your system too. Check out the documentation on manual TLS setup for details.
Even before the 2.6 release, the Cloudera Director server could be configured to use TLS, but its CLI and client libraries did not support it. Now, Cloudera Director client libraries and the CLI fully support TLS. You can set up a fully protected chain of communication from your Cloudera Director client to a Cloudera Director server, and from there to Cloudera Manager and clusters.
As another part of improving security, Cloudera Director 2.6 adds the ability to retrieve SSH host keys from cloud instances it allocates and verify them when making connections. SSH host keys are one way to prevent man-in-the-middle attacks against SSH connections that Cloudera Director establishes with cloud instances. If the SSH host key for an instance changes, Cloudera Director detects it as a security violation. Refer to Cloudera Director documentation on SSH keys to enable the features listed above.
Cloudera Director 2.6 ships with a new Microsoft Azure plugin, that supports more features of Microsoft Azure, using the latest SDK for Azure. The new plugin supports simplified configurations for Azure environments and instance templates while retaining backwards compatibility with existing configurations. Azure Managed Disks are now supported during cluster provisioning, which removes previous storage account limits and enables the plugin to support custom images for faster bootstrap times. Learn more about these and other improvements to Azure support in our documentation.
For more information on what’s new in this release, check out the Cloudera Director 2.6 section of the New Features and Changes in Cloudera Director page in the Cloudera Director documentation.
Using Cloudera Director
If you’re ready to give the latest version of Cloudera Director a try, here are the ways you can get started.
* Use sample configuration files and scripts as starting points for setting up your clusters.
Send questions or feedback to the Cloudera Director community forum.
Bill Havanki is a software engineer at Cloudera.