How-to: Detect and Report Web-Traffic Anomalies in Near Real-Time

Categories: CDH Flume Impala Spark Use Case

This framework based on Apache Flume, Apache Spark Streaming, and Apache Impala (incubating) can detect and report on abnormal bad HTTP requests within seconds.                     

Website performance and availability are mission-critical for companies of all types and sizes, not just those with a revenue stream directly tied to the web. Web pages can become unavailable for many reasons, including overburdened backing data stores or content-management systems or a delay in load times of third-party content such as advertisements.

Read More

How-to: Analyze Fantasy Sports with Apache Spark and SQL (Part 2: Data Exploration)

Categories: Hadoop Spark Use Case

Learn how analyzing stats from professional sports leagues is an instructive use case for data analytics using Apache Spark with SQL. Covered in this installment: data exploration with Apache Impala (incubating) and Hue.

In Part 1 of this series, I introduced the topic of using fantasy sports analytics as an instructive use case for exploring the Apache Hadoop ecosystem. In that installment, we focused on data processing by taking a collection of data from Basketball-Reference.com and enriching it with z-scores and normalized z-scores to analyze the relative value of NBA players.

Read More

Announcing hs2client, A Fast New C++ / Python Thrift Client for Impala and Hive

Categories: Data Science Hive Impala Tools

This new (alpha) C++ client library for Apache Impala (incubating) and Apache Hive provides high-performance data access from Python.

Earlier this year, members of the Python data tools and Impala teams at Cloudera began collaborating to create a new C++ library to eventually become a faster, more memory-efficient replacement for impyla, PyHive, and other (largely pure Python) client libraries for talking to Hive and Impala.

Read More

Untangling Apache Hadoop YARN, Part 4: Fair Scheduler Queue Basics

Categories: Hadoop YARN

In this installment, we provide insight into how the Fair Scheduler works, and why it works the way it does.

In Part 3 of this series, you got a quick introduction to Fair Scheduler, one of the scheduler choices in Apache Hadoop YARN (and the one recommended by Cloudera). In Part 4, we will cover most of the queue properties, some examples of their use, as well as their limitations.

Read More

Best Practices for Enterprise Data Hub Encryption

Categories: Cloudera Navigator Security

Encryption is a key security feature in Cloudera-powered enterprise data hubs (EDHs). This post explains some best practices for deployment of Cloudera Navigator Encrypt for that purpose.

For those unfamiliar with the product, Cloudera Navigator Encrypt provides scalable, high-performance encryption for critical Apache Hadoop data. It utilizes industry-standard AES-256 encryption and provides a transparent layer between the application and filesystem. Cloudera Navigator Encrypt also includes process-based access controls, allowing authorized processes to access encrypted data while simultaneously preventing admins or super-users like root from accessing data that they don’t need to see.

Read More