Introduction to Apache HBase Snapshots

The current (4.2) release of CDH — Cloudera’s 100% open-source distribution of Apache Hadoop and related projects (including Apache HBase) — introduced a new HBase feature, recently landed in trunk, that allows an admin to take a snapshot of a specified table.

Prior to CDH 4.2, the only way to back-up or clone a table was to use Copy/Export Table, or after disabling the table, copy all the hfiles in HDFS. Copy/Export Table is a set of tools that uses MapReduce to scan and copy the table but with a direct impact on Region Server performance. Disabling the table stops all reads and writes, which will almost always be unacceptable.

In contrast, HBase snapshots allow an admin to clone a table without data copies and with minimal impact on Region Servers. Exporting the snapshot to another cluster does not directly affect any of the Region Servers; export is just a distcp with an extra bit of logic.

Here are a few of the use cases for HBase snapshots:

  • Recovery from user/application errors
    • Restore/Recover from a known safe state.
    • View previous snapshots and selectively merge the difference into production.
    • Save a snapshot right before a major application upgrade or change.
  • Auditing and/or reporting on views of data at specific time
    • Capture monthly data for compliance purposes.
    • Run end-of-day/month/quarter reports.
  • Application testing
    • Test schema or application changes on data similar to that in production from a snapshot and then throw it away. For example: take a snapshot, create a new table from the snapshot content (schema plus data), and manipulate the new table by changing the schema, adding and removing rows, and so on. (The original table, the snapshot, and the new table remain mutually independent.)
  • Offloading of work
    • Take a snapshot, export it to another cluster, and run your MapReduce jobs. Since the export snapshot operates at HDFS level, you don’t slow down your main HBase cluster as much as CopyTable does.

What is a Snapshot?

A snapshot is a set of metadata information that allows an admin to get back to a previous state of the table. A snapshot is not a copy of the table; it’s just a list of file names and doesn’t copy the data. A full snapshot restore means that you get back to the previous “table schema” and you get back your previous data losing any changes made since the snapshot was taken.

Operations

  • Take a snapshot: This operation tries to take a snapshot on a specified table. The operation may fail if regions are moving around during balancing, split or merge.
  • Clone a snapshot: This operation creates a new table using the same schema and with the same data present in the specified snapshot. The result of this operation is a new fully functional table that can can be modified with no impact on the original table or the snapshot.
  • Restore a snapshot: This operation brings the table schema and data back to the snapshot state. (Note: this operation discards any changes made since the snapshot was taken.) 
  • Delete a snapshot: This operation removes a snapshot from the system, freeing unshared disk space, without affecting any clones or other snapshots.
  • Export a snapshot: This operation copies the snapshot data and metadata to another cluster. The operation only involves HDFS so there’s no communication with the Master or the Region Servers, and thus the HBase cluster can be down.

Zero-copy Snapshot, Restore, Clone

The main difference between a snapshot and a CopyTable/ExportTable is that the snapshot operations write only metadata. There are no massive data copies involved.

One of the main HBase design principles is that once a file is written it will never be modified. Having immutable files means that a snapshot just keeps track of files used at the moment of the snapshot operation, and during a compaction it is the responsibility of the snapshot to inform the system that the file should not be deleted but instead it should be archived.

The same principle applies to a Clone or Restore operation. Since the files are immutable a new table is created with just “links” to the files referenced by the snapshot.

Export Snapshot is the only operation that require a copy of the data, since the other cluster doesn’t have the data files.

Export Snapshot vs Copy/Export Table

Aside from the better consistency guarantees that a snapshot can provide compared to a Copy/Export Job, the main difference between Exporting a Snapshot and Copying/Exporting a table is that ExportSnapshot operates at HDFS level. This means that Master and Region Servers are not involved in this operations. Consequently, no unnecessary caches for data are created and there is no triggering of additional GC pauses due to the number of objects created during the scan process. The performance impact on the HBase cluster stems from the extra network and disk workload experienced by the DataNodes.

HBase Shell: Snapshot Operations

Confirm that snapshot support is turned on by checking if the hbase.snapshot.enabled property in hbase-site.xml is set to true. To take a snapshot of a specified table, use the snapshot command. (No file copies are performed)

hbase> snapshot ‘tableName’, ‘snapshotName’

 

To list all the snapshots, use the list_snapshot command. it will display the snapshot name, the source table, and the creation date and time. 

hbase> list_snapshots
SNAPSHOT               TABLE + CREATION TIME
 TestSnapshot          TestTable (Mon Feb 25 21:13:49 +0000 2013)

 

To remove a snapshot, use the delete_snapshot command. Removing a snapshot doesn’t impact cloned tables or other subsequent snapshots taken.

hbase> delete_snapshot 'snapshotName'

 

To create a new table from a specified snapshot (clone), use the clone_snapshot command. No data copies are performed, so you don’t end up using twice the space for the same data.

hbase> clone_snapshot 'snapshotName', 'newTableName'

 

To replace the current table schema/data with a specified snapshot content, use the restore_snapshot command.

hbase> restore_snapshot 'snapshotName'

 

To export an existing snapshot to another cluster, use the ExportSnapshot tool. The export doesn’t impact the RegionServers workload, it works at the HDFS level and you have to specify an HDFS location (the hbase.rootdir of the other cluster).

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
SnapshotName -copy-to hdfs:///srv2:8082/hbase

 

Current Limitations

Snapshots rely on some assumptions, and currently there are a couple of tools that are not fully integrated with the new feature:

  • Merging regions referenced by a snapshots causes data loss on the snapshot and on cloned tables.
  • Restoring a table with replication on for the table restored ends up with the two cluster out of synch. The table is not restored on the replica.

Conclusion

Currently the snapshot feature includes all the basic required functionality, but there’s still much work to do, including metrics, Web UI integration, disk usage optimizations and more.

To learn more about how to configure HBase and use snapshots, review the documentation.

Matteo Bertozzi is a Software Engineer on the Platform team, and an HBase committer.

 

Filed under:

11 Responses
  • Ron Buckley / March 08, 2013 / 5:49 AM

    How long is a snapshot ‘good’ for? Is it just the time the HFiles are in the main /hbase directory or the .archive, or is there some other retention at play? Is seems like a snapshot is going to be good for as long as ‘hbase.master.hfilecleaner.ttl’ is configured for?

    • Matteo Bertozzi / March 08, 2013 / 5:54 AM

      A snapshot is ‘good’ until you remove it. Once the files get compacted, instead of being removed they are moved to the archive. And from the archive they are not deleted if a snapshot or a cloned table is referencing them.

  • Ted Yu / March 09, 2013 / 2:44 PM

    bq. Merging regions referenced by a snapshots causes data loss

    Were you talking about offline merge ?
    The online merge tool developed in HBASE-7403 creates reference files of the regions to be merged. Meaning online merge is compatible with snapshotting.

    • Matteo Bertozzi / March 09, 2013 / 2:47 PM

      sure, HBASE-7403 will fix the online merge case. But HBASE-7403 is not in apache trunk or CDH yet

  • digri / March 11, 2013 / 4:36 AM

    Sounds great. Could someone please explain the differences between online/offline/timestamp-consistent/globally-consistent snapshots?

    How is snapshotting of updates present in memstores handled?

    • Matteo Bertozzi / March 11, 2013 / 2:17 PM

      The current implementation has only offline and online snapshot with flush. The timestamp and globally consistent snapshot were discussed, tried and not completed for the first cut.
      Offline means, table disabled. so if your table is disabled and you run the “snapshot” command described in the post all the operations are performed by the master. Online means that your table is enabled, and you may have read and writes going on during the snapshot. In this case the master ask each region server to take a snapshot of the regions that they handle. The RS snapshot procedure is pluggable, but at the moment the only available is the one that flushes the memstore before taking the snapshot.

  • Otis Gospodnetic / March 11, 2013 / 1:51 PM

    Great news! :)

    Can you give us any sense of how fast snapshots are? Maybe a few data points from your experiments?

    How about data size?
    Do they “run at full speed”, thus taking all the disk/net IO and CPU they can get?

    And are there plans for incremental snapshots that contain only the delta since some other snapshot?

    Thanks.

    • Matteo Bertozzi / March 11, 2013 / 2:11 PM

      The time limit for a snapshot to be considered successful is 60sec, but it should take few seconds. At the moment a flush is performed, so it’s the time required by the slowest Region Server to flush, dump the list of hfiles and report back to the Master.

      Since a snapshot is not a copy of the data, there’s not much heavy disk I/O involved. It’s basically just a for loop to store the file names of the store files available in the table. If you take two snapshots and there’re no compactions on the table between the two, you’re sharing the files so it’s kind of incremental, Keeping in mind that snapshots have the hfile granularity.

  • Otis Gospodnetic / March 11, 2013 / 9:21 PM

    Thanks Matteo.
    If a snapshot doesn’t make a copy of the data, what happens when the original data get lost/corrupt?

  • CS / June 17, 2013 / 6:27 PM

    Could you please clarify how to enable snapshots in CDH4.2.

    According to http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_20_12.html, a property “hbase.snapshot.enabled” need to be set in hbase-site.xml. Configuring this in “/etc/hbase/conf.cloudera.hbase1/hbase-site.xml” is not being picked up after restarting hbase1 service. A new file is getting regenerated: on each deploy client configurations. Could you pls point in the right direction?

    Thanks,
    cs

  • Alex / August 05, 2013 / 6:04 AM

    @CS : don’t know if you find the solution for your problem, but if anyone need this, you have to do the change on Cloudera Manager :
    – go to service Hbase –> Configuration –> Service-Wide –> Advanced –> HBase Service Configuration Safety Valve for hbase-site.xml
    And then add those lines (“angle brackets” replaced with @):

    @property@
    @name@hbase.snapshot.enabled@/name@
    @value@true@/value@
    @/property@

    This should do the trick !

Leave a comment


× 5 = thirty