Category Archives: Kafka

SMM 1.2 Released with Powerful New Alerting and Topic Lifecycle Management Features with Schema Registry Integration

Categories: Kafka Tools

[Editor’s note: Now that the recent merger is complete, the Cloudera Engineering blog will expand to cover products, such as this, originally developed for the Hortonworks platform. Please stay tuned for future product announcements regarding availability of these products on the Cloudera platform.]

Since the release of Streams Messaging Manager (SMM) at the end of last summer, our customers have started to cure the Kafka Blindness within their organizations by using SMM to monitor their Kafka clusters and streaming microservices applications.

Read more

Cloudera Enterprise 6.1.0 is Now Available

Categories: Accumulo CDH Cloudera Manager Cloudera Navigator Kafka Kudu Search Tools

We are pleased to announce the general availability of Cloudera Enterprise 6.1.0, the modern platform for machine learning and analytics optimized for the cloud. This release delivers several new capabilities, improved usability, and better performance.

As usual, the release includes a number of quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list):

Data Engineering

Cloudera Enterprise 6.1 now supports Spark Structured Streaming and enables micro-batch processing at ~100ms increments enabling ingest to query latencies in the Cloudera platform measured in seconds.

Read more

Next Generation Data Warehousing at Santander UK

Categories: CDH HBase HDFS Kafka Kudu Use Case

Timely data is crucial to businesses in the Big Data age: This blog post outlines how Santander UK utilises the latest Cloudera technologies and superior software development capability to create the next generation of data warehousing and streaming analytics to support intelligence that can improve relationships with customers and follow the mantra of ‘we want to help people grow and prosper.

Santander UK’s big data journey started around four years ago.

Read more

Robust Message Serialization in Apache Kafka Using Apache Avro, Part 3

Categories: Avro CDH How-to Kafka

Part 3: Configuring Clients

Earlier, we introduced Kafka Serializers and Deserializers that are capable of writing and reading Kafka records in Avro format. In this part we will going to see how to configure producers and consumers to use them.

Setting up a Kafka Topic for use as a Schema Store

KafkaTopicSchemaProvider works with a Kafka topic as its persistent store. This topic will contain at most thousands of records: the schemas. It does not need multiple partitions,

Read more

Robust Message Serialization in Apache Kafka Using Apache Avro, Part 2

Categories: Avro CDH How-to Kafka

Implementing a Schema Store

In Part 1, we saw the need for an Apache Avro schema provider but did not implement one. In this part we will implement a schema provider that works with Apache Kafka as storage.

In-Memory SchemaStore

First we can implement an in-memory store for schemas. This is useful to understand the requirements for such a store and as the cache of the Kafka backed store. A SchemaStore has to be quick in looking up VersionedSchema entries.

Read more