Cloudera Director 2.8 introduces a simpler way to create clusters in AWS or Microsoft Azure that requires less information to get started than the standard procedure. A new configuration export capability enables retrieval of a client configuration file for any cluster as a starting point to create new clusters.
Cloudera Director helps you deploy, scale, and manage Cloudera clusters in AWS, Microsoft Azure, or Google Cloud Platform. Its enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big data workloads and applications in a simple, reliable, automated fashion.
Cloudera Director Overview
In this post, you will learn about new functionality in release 2.8, but first, for those new to Cloudera Director, let’s revisit what it does.
- On-demand creation and termination of clusters: Using Cloudera Director, you can allocate and configure Cloudera Manager instances and highly available CDH clusters in the cloud provider of your choice. A single Cloudera Director instance can manage multiple cloud provider environments and the separate lifecycles of multiple Cloudera Managers and clusters.
- Multi-cloud support: Cloudera Director supports creating clusters in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) through its cloud provider plugin architecture. A single Cloudera Director instance can work with multiple cloud providers at once. Because the plugin specification is open source, you can create a plugin to support other providers, either in-house or public.
- On-demand grow and shrink of clusters: One of the main benefits of running CDH clusters in the cloud is being able to provision additional instances when demand increases, and to terminate instances when demand decreases. Cloudera Director, in concert with Cloudera Manager, does the work required to add new instances to and remove existing ones from your CDH clusters.
- Programmatic and repeatable instantiation of clusters: Cloudera Director can consume cluster definitions specified in HOCON configuration files submitted through the Cloudera Director CLI or in JSON or HOCON input sent to the Cloudera Director API. A cluster definition can include custom scripts to run after instance provisioning and cluster setup, or before cluster termination, to perform tasks like installing additional packages, configuring system settings, or saving off important data.
- Long running cluster support: Long running clusters often require actions like upgrading CDH and Cloudera Manager, changing the topology of the cluster, and reconfiguring the cluster. Cloudera Director supports such modifications when using Cloudera Manager 5.11 and above.
- Usage-based billing for Cloudera services: Usage-based billing can help you optimize your expenditures for transient clusters. With a pay-as-you-go billing ID from Cloudera, you can use your Cloudera Enterprise license as usual, but you are only charged for CDH services when they are running.
- Security: Cloudera Director, like other Cloudera offerings, is committed to enabling secure deployments and applications. Cloudera Director’s own database is automatically encrypted, and Cloudera Director helps you configure Cloudera Manager and CDH clusters with Kerberos authentication, TLS for on-the-wire privacy, and Cloudera Navigator for auditing, data lineage, and data discovery.
- Powerful web user interface: Cloudera Director’s user interface provides a single dashboard to assess the health of all your clusters across all cloud providers and all Cloudera Manager deployments. It can also be used to bootstrap new clusters, grow and shrink existing clusters, and terminate clusters that are no longer needed. Exploring the web user interface is a great stepping stone to using the configuration file or API to deploy production-ready clusters.
New Features and Improvements in Cloudera Director 2.8
The Cloudera Director web user interface is a convenient way to work with CDH clusters in the cloud, especially when starting out with Cloudera Director or for scenarios when the full power of the Cloudera Director REST API or client configuration files isn’t necessary. The standard set of forms that you fill out in the user interface works well in many situations, but we wanted to offer something even simpler to use. Cloudera Director 2.8 introduces a new simple cluster setup capability, designed to help you get clusters running more quickly while requiring less information to get started.
The web user interface has been updated as part of the introduction of simple cluster setup. After you create an AWS or Microsoft Azure environment as usual, you can now choose between the new simple setup procedure and the pre-existing (“advanced”) procedure.
When you select “Simple Setup”, you’ll see a new form asking for only a few pieces of information needed for bootstrapping a new cluster, some for your cloud provider of choice, and others for the cluster you’d like to create. All of the other details about how your cluster is configured, like its topology and versions of Cloudera Manager and CDH, are determined for you by Cloudera Director.
A key driver of those details is the Workload Type you choose, which determines the services in the cluster. Today, the choices for Workload Type range from Basic all the way up to an Enterprise Data Hub configuration. The choices here may expand in later releases of Cloudera Director.
After confirming the creation of your cluster and a backing Cloudera Manager installation for it, Cloudera Director starts the bootstrap process as usual, and the end result is a cluster you can manage just like any other. Later on, if you decide to bootstrap another cluster in the first cluster’s environment, you only need to fill in the new simple setup form once more to start the next bootstrap process.
Simple cluster setup works not only through the web user interface, but also through the server REST API and the CLI. You can define a simple client configuration file with the same environment and cluster information that you would supply through the web user interface. The existing “bootstrap-remote” CLI command has been enhanced to support these simple configuration files alongside standard configuration files. And, as with the web user interface, if the environment for your cluster already exists in the Cloudera Director server, you can skip most of the file, leaving you with only a few properties you need to specify.
The clusters that you create using the new simple setup procedure do not include some of the more powerful features that you may be looking for in a production-ready cluster, such as Sentry, Kerberos, TLS or high availability. However, you can use the clusters to help you configure other clusters by using the new cluster configuration export capability. Cloudera Director can now export a client configuration file for any working cluster, either through the web user interface or through the server API. The exported client configuration file reveals the detailed blueprint for the cluster, its environment, and the Cloudera Manager installation managing it. This blueprint can be a starting point for developing a more advanced configuration file with the additional services and features you want.
Along with the Cloudera Director 2.8 release, we are offering a reference configuration for creating a Cloudera Data Science Workbench (CDSW) cluster running in Microsoft Azure using Cloudera Director, to go along with our reference configuration for doing so in AWS.
Cloudera Director 2.8 has improved support for growing CDSW clusters. Prior to this release, it was necessary to manually execute a command in Cloudera Manager for new worker nodes to be added successfully to a cluster. Cloudera Director now handles that step for you, so that growing a CDSW cluster is completely automated.
Finally, as part of our continual updating of AWS support, the AWS plugin bundled with Cloudera Director 2.8 now natively supports the new P3 and G3 EC2 instance types and the new EU (Paris) or eu-west-3 region.
For more information on what’s new in this release, check out the Cloudera Director 2.8 section of the New Features and Changes in Cloudera Director page in the Cloudera Director documentation.
Using Cloudera Director
If you’re ready to give the latest version of Cloudera Director a try, here are the ways you can get started.
* Use sample configuration files and scripts as starting points for setting up your clusters.
Send questions or feedback to the Cloudera Director community forum.
Bill Havanki is a software engineer at Cloudera.