Apache Hadoop Ozone - Object Store Overview

by Arpit Agarwal

Posted in Technical | October 10, 2018 3 min read

This blog post was published on Hortonworks.com before the merger with Cloudera. Some links, resources, or references may no longer be accurate.

1. Introduction

This article is second in the series about Ozone – a distributed key-value store that can efficiently manage small and large files alike.

An earlier article introduced the Ozone design philosophy. The current article is for users who want to try Ozone. We discuss an overview of Ozone concepts, describe how to bring up an Ozone cluster and use the Ozone command shell.

2. Ozone Concepts

Ozone is made up of volumes, buckets and Keys.

Volumes – Volumes are similar to accounts. Volumes can be created or deleted only by administrators. An administrator will typically create a volume for an organization or team.
Buckets – A volume can contain zero or more buckets. Ozone buckets are similar to Amazon S3 buckets.
Keys – Keys are unique within a given bucket and are similar to S3 Objects. Key names can be any string. Values represent the data that you store inside these keys, currently Ozone enforces no upper limit on the key size.

3. Installing Ozone

3.1. Get the Binaries

Download Binary release – get the latest Ozone release from the Ozone website at https://hadoop.apache.org/ozone/.
Build from source code – The instructions are at https://cwiki.apache.org/confluence/display/HADOOP/Try+out+Ozone

3.2. Deploy a cluster

There are multiple ways to setup an Ozone cluster.

Development cluster with Docker – Instructions at https://cwiki.apache.org/confluence/display/HADOOP/Development+cluster+with+docker
Single-node development cluster – https://cwiki.apache.org/confluence/display/HADOOP/Single-node+dev+cluster
Multi-node Ozone cluster – https://cwiki.apache.org/confluence/display/HADOOP/Multi-node+Ozone+Cluster

4. Ozone command-line shell

4.1. Working with Volumes, Buckets, Keys

The Alpha release includes the following interfaces:

REST & RPC API
Command-line shell to perform object store operations.
OzoneFS – A Hadoop Compatible Filesystem interface.

Let’s look at a few examples of using the command-line shell to perform object store operations.

4.1.1. Create a Volume

The following command creates a new volume named videos, owned by the user alice. Creating new volumes is an administrative operation.

4.1.2. Create a Bucket

The following command creates a new bucket named vacation in videos.

4.1.3. Upload a Key

This is the equivalent of ‘putting’ a file into the object store. The following command will upload the local file video-001.mp4 to the ozone bucket with the name machu-picchu.mp4.