New Apache Phoenix 4.5.2 Package from Cloudera Labs

Categories: Cloudera Labs HBase

[Update: A new package for Apache Phoenix 4.7.0 on CDH 5.7 was released in June 2016.]

New Cloudera Labs packages for Apache Phoenix 4.5.2 (which includes Apache Spark integration) is now available for CDH 5.4.x and CDH 5.5.x.

Earlier this year, Cloudera announced the inclusion of Apache Phoenix in Cloudera Labs.

To recap: Phoenix adds SQL to Apache HBase, the native, highly distributed data store for Apache Hadoop. Phoenix aims to ease HBase access by supporting SQL syntax and allowing inputs and outputs using standard JDBC APIs instead of HBase’s Java client APIs. It lets you perform all CRUD and DDL operations such as creating tables, inserting data, and querying data.

This month, the Cloudera Labs team released new Phoenix packages that allows you to run Apache Phoenix 4.5.2 on CDH 5.4.x (clabs-1.1.0) and CDH 5.5.x (clabs-1.2.0). Aside from bug fixes and performance improvements, new features of Phoenix that are part of this update are:

Compatibility

The CDH version of HBase may contain backports from newer branches to allow developers to test new features or to provide some improvements without having to go through a full rebase. Phoenix also uses some of HBase’s private or evolving APIs, such as coprocessors or methods that directly access HBase server internals. For this reason, you must use the version of Phoenix that has been built against the version of HBase you are using. Note that with this release, there is no guarantee of upgrade compatibility from previous Cloudera Labs Phoenix versions.

The Phoenix Source code with the changes made to run on the specific CDH version is available at https://github.com/cloudera-labs/phoenix.

Quick Start

The Phoenix package is currently distributed only as a parcel. Because Cloudera does not support mixing parcels and packages in the same Cloudera Manager instance, you can only install Phoenix into a Cloudera Manager instance that uses parcels.

Also, if you are configuring HBase to use secondary indexes, no WALs can be present in the HBase log directory. To ensure that this is the case, use a fresh HBase installation or perform a full clean shutdown of HBase before configuring Phoenix.

  1. Install and activate the parcel:
    1. In Cloudera Manager, go to Hosts, then Parcels.
    2. Select Edit Settings.
    3. Click the + sign next to an existing Remote Parcel Repository URL, and add the appropriate URL (http://archive.cloudera.com/cloudera-labs/phoenix/parcels/1.1/ or http://archive.cloudera.com/cloudera-labs/phoenix/parcels/1.2/). Click Save Changes.
    4. Select Hosts, then Parcels.
    5. In the list of Parcel Names, CLABS_PHOENIX should now be available. Select it and choose Download.
    6. The first cluster is selected by default. To choose a different cluster for distribution, select it. Find CLABS_PHOENIX in the list, and click Distribute.
    7. If you do not plan to use secondary indexing, but only plan to use Phoenix for doing simple upsert/select skip this step. If you do plan to use secondary indexing, add the following to the hbase-site.xml advanced configuration snippet. Go to the HBase service, click Configuration, and choose HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml. Paste in the following XML, then save the changes.

      To configure the IndexedWALEditCodec, see https://phoenix.apache.org/secondary_indexing.html.
    8. Click Actions > Restart.
  2. From a shell, execute phoenix-sqlline.py <host>:<port> (e.g. phoenix-sqlline.py localhost:2181). This will open the phoenix shell.
  3. Now you are ready to write some SQL:

Also, take also a look at the “Phoenix in 15 minutes or less” post, which provide an alternative way to execute queries without having to type them in the shell.

More docs and examples are available on the Apache Phoenix website: phoenix.apache.org. As usual, we appreciate your feedback via the Cloudera Labs discussion board.

Matteo Bertozzi is a Software Engineer at Cloudera, and an Apache HBase committer.

facebooktwittergoogle_pluslinkedinmailfacebooktwittergoogle_pluslinkedinmail

28 responses on “New Apache Phoenix 4.5.2 Package from Cloudera Labs

  1. Eric Morich

    Matteo, I had no success getting this up and running on your Quickstart VM. The installation went smooth (after migration to parcels) and all services started after Phoenix activation. Also Hbase works fine until I’m trying to connect through phoenix-sqlline which kills the Region Server. Should Phoenix run on Quickstart VM in general?
    Thanks, Eric

    1. Matteo

      This new package (4.5.2 clabs 1.1.0) should work on every CDH 5.4.x (only the 5.4.x serie)
      You must stop HBase, install the package and restart HBase after the package is installed.
      If you are using the IndexedWALEditCodec you also have to ensure that no WALs in the /hbase directory.

        1. Justin Kestelyn Post author

          We may release a new package for CDH 5.5 in the future (if there is enough demand for it), yes.

          1. Srikanth Bayyannagari

            I am using Phoenix4.5.2 (using parallels) with CDH 5.6 . But not able to connect to phoenix. Can someone please help me here. Getting error as org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
            Wed Apr 13 15:02:41 IST 2016, RpcRetryingCaller{globalStartTime=1460539961058, pause=100, retries=35}, java.io.IOException: java.io.IOException
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
            at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException
            at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getVersion(MetaDataEndpointImpl.java:2655)
            at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:11659)
            at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7388)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1776)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1758)
            at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
            … 4 more

            Wed Apr 13 15:02:41 IST 2016, RpcRetryingCaller{globalStartTime=1460539961058, pause=100, retries=35}, java.io.IOException: java.io.IOException
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2068)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
            at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: java.lang.NullPointerException
            at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getVersion(MetaDataEndpointImpl.java:2655)
            at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:11659)
            at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7388)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1776)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1758)
            at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
            … 4 more

          2. Justin Kestelyn Post author

            Please post details in the Cloudera Labs area at community.cloudera.com.

  2. Buntu

    Thanks for making Phoenix 4.5.2 available!

    How do I get the Phoenix Query server running as a long running process?

  3. HanCheol Cho

    Hi,

    We installed CDH 5.4.3 from command line so needed to install Apache Phoenix instead the one from parcel.
    The installation was not successful at first, but we managed to solve the problem by re-building Phoenix
    based on the information in the following stackoverflow post:
    http://stackoverflow.com/questions/31849454/using-phoenix-with-cloudera-hbase-installed-from-repo

    It seems that the changes are related to hbase regionserver APIs that are called in the following two files:
    /phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/LocalIndexMerger.java
    /phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java

    Can someone tell me what is the intention of these changes? (some customization for performance improvement?)

    Best wishes,

    1. Matteo

      Phoenix uses some of HBase’s private or evolving APIs, such as coprocessors or methods that directly access HBase server internals.

      The CDH version of HBase may contain backports from newer branches to allow developers to test new features or to provide some improvements without having to go through a full rebase.

      so, we need to make some small adjustment in the Phoenix codebase to make use of the new api. In this case the changes to LocalIndexMerger, IndexSplitTransaction are related to an internal API that had a new argument added with the Read-Replicas feature.

  4. Justin Kestelyn Post author

    All,
    Packages for CDH 5.5.x are now available, and this post has been updated accordingly.

  5. Jonathan Leech

    what’s the appropriate maven dependency to use to build a client? I don’t see anything relating to this new version on cloudera’s repo. Should I use apache’s? I had previously been building it locally and using the client-minimal.jar…

    1. Justin Kestelyn Post author

      Agrta,

      There is no tarball; you must install via Cloudera Manager parcel (instructions are in post).

  6. Sunfl

    Hi,
    I am trying to use the parcel for CDH 5.5.1 support but had no luck of connecting to phoenix. The install is good but when I am trying to use phoenix-sqlline.py : to connect to phoenix, it would hang there and the regionserver would all be killed. I cannot find any error and exception messages from the /var/log/hbase/ regionserver log. What can be the cause ?

    1. Justin Kestelyn Post author

      Please post details in the “Cloudera Labs” area at community.cloudera.com.

  7. SAP

    I am trying Dockerized CDH 5.5.0 QuickStart VM.
    It keeps on complaining about the version non compatbility for the phoenix.
    Can you please make Phoenix available for 5.5 asap.

    1. Justin Kestelyn Post author

      Please post this issue in the QuickStart VM area at community.cloudera.com. Per this blog post, a version of Phoenix for CDH 5.5.x is available.

  8. Jim

    “This new package (4.5.2 clabs 1.1.0) should work on every CDH 5.4.x (only the 5.4.x serie)”

    How is that possible if all CDH 5.4.x ship with Hbase 1.0 ??
    In fact, no CDH and Hbase 1.1 as far as I can tell. When they go to CDH 5.7 they skip from Hbase 1.0 to 1.2.
    Of course I’m asking because I can’t get this to work and after countless hours I’m wondering if I’m missing something fundamental.

    1. Justin Kestelyn Post author

      Jim,

      I think you are confusing parcel version numbers with HBase version numbers. v1.1.0 of the Phoenix parcel does not imply “support for HBase 1.1”. In fact, as you point out, there is no HBase 1.1 in CDH.

      Please post whatever issues you’re having in the Cloudera Labs area at community.cloudera.com.

    1. Justin Kestelyn Post author

      Please post this issue in the Cloudera Labs area at community.cloudera.com.