Category Archives: Hadoop

Deep Learning Frameworks on CDH and Cloudera Data Science Workbench

Categories: CDH Data Science Hadoop

The emergence of “Big Data” has made machine learning much easier because the key burden of statistical estimation—generalizing well to new data after observing only a small amount of data—has been considerably lightened. In a typical machine learning task, the goal is to design the features to separate the factors of variation that explain the observed data. However, a major source of difficulty in many real-world artificial intelligence applications is that many of the factors of variation influence every single piece of data we can observe.

Read more

Cloudera Enterprise 5.11 is Now Available

Categories: CDH Cloud Cloudera Manager Cloudera Navigator Hadoop

Cloudera Enterprise 5.11 is Now Available

Cloudera is pleased to announce that Cloudera Enterprise 5.11 is now generally available (GA). The highlights of this release include lineage support for Apache Spark, Apache Kudu security integration, embedded data discovery for self-service BI, and new cloud capabilities for Microsoft ADLS and Amazon S3.

As usual, there are also a number of quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list):

  • Core Platform and Cloud
    • Amazon S3 Consistency: S3Guard ensures that operations on Amazon S3 are immediately visible to other clients,

Read more

Blacklisting in Apache Spark

Categories: Hadoop Spark

At Cloudera, we’re always working to provide our customers and the Apache Spark community with the most robust, most reliable software possible. This article describes some recent engineering work on [SPARK-8425] that is available in CDH 5.10 and CDH5.11, as well as in upstream Apache Spark starting with the 2.2 release.

The work pertains to the Blacklist Tracker mechanism in Spark’s scheduler. This was the subject of a recent Spark Summit talk,

Read more

How-to: Log Analytics with Solr, Spark, OpenTSDB and Grafana

Categories: Hadoop How-to Search Spark

Organizations analyze logs for a variety of reasons.  Some typical use cases include predicting server failures, analyzing customer behavior, and fighting cybercrime.  However, one of the most overlooked use cases is to help companies write better software.  In this digital age, most companies write applications, be it for its employees or external users.  The cost of faulty software can be severe, ranging from customer churn to a complete firm’s demise, as was the case with Knight Capital in 2012.

Read more

Offheap Read-Path in Production – The Alibaba story

Categories: Hadoop HBase Performance Use Case

This article is syndicated with permission from the Apache HBase blog and highlights a collaboration between our partners at Intel and Alibaba engineering in time for “Singles Day“, the biggest shopping day on the net. For more on HBase, mark your calendars! On June 12th, 2017 the Apache HBase community will be hosting their annual HBaseCon.

Introduction

HBase is the core storage system in Alibaba’s Search Infrastructure.

Read more