Offset Management For Apache Kafka With Apache Spark Streaming

No Comments
Categories: CDH Kafka Spark

An ingest pattern that we commonly see being adopted at Cloudera customers is Apache Spark Streaming applications which read data from Kafka. Streaming data continuously from Kafka has many benefits such as having the capability to gather insights faster. However, users must take into consideration management of Kafka offsets in order to recover their streaming application from failures. In this post, we will provide an overview of Offset Management and following topics.

  • Storing offsets in external data stores
    • Checkpoints
    • HBase
    • ZooKeeper
    • Kafka
  • Not managing offsets

Overview of Offset Management

Spark Streaming integration with Kafka allows users to read messages from a single Kafka topic or multiple Kafka topics.

Read More

Solr Memory Tuning for Production (part 2)

No Comments
Categories: CDH

In Part 1 of this blog, we covered some common challenges in memory tuning and baseline setup related to a production Solr deployment. In Part 2, you will learn memory tuning, GC tuning and some best practices.

Memory Tuning

We assume you have read part 1 of the blog and have a stable Solr deployment up running. The next step is memory tuning to get more out of Solr. Before changing any configuration please be aware that playing with some tuning knobs can cause unexpected consequences on the system,

Read More

Apache ZooKeeper Four Letter Words and Security

No Comments
Categories: ZooKeeper

The Security Problem

Four Letter Words (acronym as 4lw) is a very popular feature of the Apache ZooKeeper project. In a nutshell, 4lw is a set of commands that you can use to interact with a ZooKeeper ensemble through a shell interface. Because it’s simple and easy to use, lots of ZooKeeper monitoring solutions are built on top of 4lw.

The simplicity of 4lw comes at a cost: the design did not originally consider security,

Read More

Talend on Why their Partnership with Cloudera Altus is a No Brainer

No Comments
Categories: Altus Cloud

The following article by Ciaran Dynes was reposted from the Talend blog with their permission. 

As you may have read, Talend recently announced its support for Cloudera / TalendCloudera Altus, a newly released Platform-as-a-Service (PaaS) offering that simplifies running large-scale data processing applications in the public cloud. For us, supporting Altus at launch was the absolute easiest decision given that so many of our customers are looking to realize the cost,

Read More

Apache Solr Memory Tuning for Production

No Comments
Categories: CDH HDFS Search

Configuring Apache Solr memory properly is critical for production system stability and performance. It can be hard to find the right balance between competing goals. There are also multiple factors, implicit or explicit, that need to be taken into consideration. This blog talks about some common tasks in memory tuning and guides you through the process to help you understand how to configure Solr memory for a production system.

For simplicity, this blog applies to Solr in Cloudera CDH5.11 running on top of HDFS.

Read More