With the launch of the Cloudera Public Cloud 7.2.12, the Streams Messaging for Data Hub deployments have gotten some interesting new features! From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Another notable item is that Streams Replication Manager (SRM) will now support multi-cluster monitoring patterns and aggregate replication metrics from multiple SRM deployments into a single viewable location in Streams Messaging Manager (SMM.) Last but not least is Apache Atlas and Schema Registry (SR) Integration, now you will be able to view Kafka topic schemas in Atlas letting you navigate data lineage from consumers and producers and see the schema of the topic they use without having to navigate back to the SR UI.
Kafka Scaling
Newly deployed Light and Heavy Duty Streams Messaging templates scaling up or down their Kafka brokers will now be possible with the addition of Cruise Control. Cruise Control will automatically rebalance the partition replicas on the cluster making use of the newly added brokers in the event of an up scale, or down scaling will move replicas off the hosts that are targeted to be decommissioned. It should be noted that at the time of writing this, clusters that upgrade to 7.2.12 will not have the ability to downscale and upscaling will not automatically rebalance, requiring the old school manual json assignment files to be created and run by operators.
- Clusters newly provisioned with 7.2.12 or higher
- Support up and downscale operations
- Support automatic partition rebalancing with Cruise Control
- Clusters upgraded to 7.2.12 or higher
- Support upscale operations only
- Partitions must be moved manually to any newly provisioned brokers after upscaling is finished.
This is a manual scaling event done by clicking the “Resize” button and requesting a specific number of brokers to be added to the scalable broker hostgroup. There will now be two specific hostgroups for Kafka Brokers; These are the Core_broker and Broker host groups. During an upscale or downscale operation, new broker nodes are added to or removed from the Broker host group. The Core_broker group contains a core set of brokers and is not scalable.
The Light and Heavy Duty templates change to look like the following deployment topologies.
Light Duty
Heavy Duty (Recommended for Production)
SRM Multi-Cluster Monitoring and Remote Query
Prior to this feature, operators had to navigate to each SMM deployment to view replication metrics for the supporting SRM deployment. In customer environments that involve many separate Kafka clusters, this could result in multiple SRM deployments all of which required their own visits to the local SMM to view and monitor the replication flows. Now, it is possible to have the single SRM service gather the metrics from these other environments. These metrics can then be displayed in a single SMM giving operators a single pane of glass effect for all their replication metrics across participating Kafka clusters.
A single SRM deployment can now monitor all the replication metrics for multiple target clusters. This lets you run a single SRM on a Kafka cluster and be able to verify in the local SMM that all replication flows are working correctly, not just a single cluster replication flow like in prior versions. Additionally SRM can now aggregate metrics from multiple SRM deployments with a feature called SRM Service Remote Querying. This Remote Querying is done by enabling the SRM Service to query other SRM Service daemons and gather the metrics. While it is possible to use a single SRM Service to gather all of these metrics this can result in heavily loaded Service roles and may not be suitable for production deployments. In effect, this feature allows the operator to dedicate a SRM Service demon to act as a monitoring gateway that is used to monitor all other clusters and replications. These monitored metrics then are displayed as mentioned before in a single SMM UI instance.
In the above image, Cluster A has been set up to replicate to Cluster B and to replicate to and from Cluster C with the new multi-cluster monitoring features, then Cluster B is replicating data into Cluster A. The Remote Querying service is then remote querying the metrics for the SRM Service on Cluster B so they can be viewed along with all other cluster metrics in the Cluster A SMM like in the image below.
Atlas Schema Registry Integration
In CDP Public Cloud 7.2.8, an Atlas hook was provided that once configured allows for Kafka metadata to be collected. This enabled data lineage use cases by collecting consumer, producer, topic and consumer group metadata and tracking it within Atlas. Now in 7.2.12, Atlas continues to be enriched by integration of the Schema Registry allowing for the Schemas to be viewed in the Atlas UI without having to navigate to the Schema Registry UI. The lineage between the schema, topic and all versions that the given schema has can be explored with ease.
Summary
In this blog we took a look at some of the new features that came out in CDP Public Cloud 7.2.12. Data Hub Scaling of Kafka clusters with automatic rebalancing of partitions can be done with a click of a button in the management UI. SRM can now monitor multiple cluster replications and also collect metrics from multiple SRM deployments into a single SMM UI providing a much better user experience to cluster operators. Finally Atlas has been integrated with SR allowing users to investigate the schema associated with the Kafka Topics when investigating data lineage.
Editor's Choice
