Apache Hadoop Ozone – Object Store Overview

Apache Hadoop Ozone – Object Store Overview

This blog post was published on Hortonworks.com before the merger with Cloudera. Some links, resources, or references may no longer be accurate.

1. Introduction

This article is second in the series about Ozone – a distributed key-value store that can efficiently manage small and large files alike.

An earlier article introduced the Ozone design philosophy. The current article is for users who want to try Ozone. We discuss an overview of Ozone concepts, describe how to bring up an Ozone cluster and use the Ozone command shell.

2. Ozone Concepts

Ozone is made up of volumes, buckets and Keys.

  1. Volumes – Volumes are similar to accounts. Volumes can be created or deleted only by administrators. An administrator will typically create a volume for an organization or team.
  2. Buckets – A volume can contain zero or more buckets. Ozone buckets are similar to Amazon S3 buckets.
  3. Keys – Keys are unique within a given bucket and are similar to S3 Objects. Key names can be any string. Values represent the data that you store inside these keys, currently Ozone enforces no upper limit on the key size.

3. Installing Ozone

3.1. Get the Binaries

  1. Download Binary release – get the latest Ozone release from the Ozone website at https://hadoop.apache.org/ozone/.
  2. Build from source code – The instructions are at https://cwiki.apache.org/confluence/display/HADOOP/Try+out+Ozone

3.2. Deploy a cluster

There are multiple ways to setup an Ozone cluster.

  1. Development cluster with Docker – Instructions at https://cwiki.apache.org/confluence/display/HADOOP/Development+cluster+with+docker
  2. Single-node development clusterhttps://cwiki.apache.org/confluence/display/HADOOP/Single-node+dev+cluster
  3. Multi-node Ozone clusterhttps://cwiki.apache.org/confluence/display/HADOOP/Multi-node+Ozone+Cluster

4. Ozone command-line shell

4.1. Working with Volumes, Buckets, Keys

The Alpha release includes the following interfaces:

  2. Command-line shell to perform object store operations.
  3. OzoneFS – A Hadoop Compatible Filesystem interface.

Let’s look at a few examples of using the command-line shell to perform object store operations.

4.1.1. Create a Volume

The following command creates a new volume named videos, owned by the user alice. Creating new volumes is an administrative operation.

4.1.2. Create a Bucket

The following command creates a new bucket named vacation in videos.

4.1.3. Upload a Key

This is the equivalent of ‘putting’ a file into the object store. The following command will upload the ┬álocal file video-001.mp4 to the ozone bucket with the name machu-picchu.mp4.

4.1.4. List Keys in a Bucket

4.1.5. List Buckets in a Volume

4.1.6. List Volumes

5. Further Reading

  1. Ozone web site – https://hadoop.apache.org/ozone/
  2. Ozone Command Shell Reference – https://hadoop.apache.org/ozone/docs/0.5.0-beta/shell.html
  3. HDDS wiki page – https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Contributor+Guide
Arpit Agarwal
More by this author

Leave a comment

Your email address will not be published. Links are not permitted in comments.