Cloudera Data Platform (CDP) provides an API that enables you to access CDP functionality from a script, or to integrate CDP features with an application. In practice you can use the CDP API to script repetitive tasks, manage CDP resources, or even create custom applications. You can learn more about the API in its official documentation.
There are multiple ways to access the API, including through a dedicated CLI, through a Java SDK, and through a low-level tool called cdpcurl. cdpcurl is designed for applications where a low-level, curl-like access pattern is the best fit.
Why use cdpcurl instead of regular curl? Because the CDP API has its own request signing procedure – described later in this blog post – which curl cannot perform for you. cdpcurl and other CDP API clients each handle request signing.
Before getting started with cdpcurl, you’ll need a user account in a CDP tenant. If your organization already has a tenant in CDP Public Cloud, or has a CDP Private Cloud deployment already established, ask its administrators for a user account.
Installation
cdpcurl is open source, so to get started with it, clone or download its source code from its repository page. Then, set up a Python virtualenv and install cdpcurl into it.
$ git clone git@github.com:cloudera/cdpcurl.git $ virtualenv cdpcurlenv $ . cdpcurlenv/bin/activate $ cd cdpcurl $ pip install .
Configuring credentials
Every call to the CDP API is authenticated by user credentials. If you haven’t done so already, create a key pair for your CDP user account by using the CDP Management Console. Then, place your keys into the file .cdp/credentials under your home directory. The syntax looks like this.
[default] cdp_access_key_id = 6744f22e-c46a-406d-ad28-987584f45351 cdp_private_key = abcdefgh...................................=
If you already have a credentials file from using the CDP CLI, you can reuse that file for cdpcurl.
Calling the CDP API
Now you’re ready to make some API calls. Here’s a good one to start with, which returns information about your user account.
$ cdpcurl -X POST -d '{}' https://iamapi.us-west-1.cdp.cloudera.com/iam/getUser
This command list environments established in your tenant, provided that your user account has the necessary role assigned.
$ cdpcurl -X POST -d '{}' https://api.us-west-1.cdp.cloudera.com/api/v1/environments2/listEnvironments
Type cdpcurl –help to see all of the available options.
To call a specific API operation with cdpcurl, you need to know its URL. There are a couple of different ways to find the URL for an operation.
- Execute the equivalent API call with the CDP CLI, passing the –debug option. The debug output reveals the URL that the CLI called, which you can use with cdpcurl.
- Consult the API reference documentation, which specifies the paths for all API operations. To find the correct hostname for CDP Public Cloud, check the cdpcurl README. For CDP Private Cloud, check with your administrators.
Request Signing
The CDP API authenticates calls by processing a pair of custom, required HTTP headers. The request signing specification is open, and cdpcurl contains a succinct reference implementation. You can use just this part of cdpcurl, through executing cdpv1sign, to generate those headers for an API call
$ cdpv1sign -X POST https://api.us-west-1.cdp.cloudera.com/api/v1/environments2/listEnvironments Content-Type: application/json x-altus-date: Fri, 09 Apr 2021 12:34:56 GMT x-altus-auth: (very long string value)
The output from cdpv1sign lists each required header on its own line. You can then pass those headers to your HTTP client of choice. For example, when the output is saved to a file, you can reference that file with curl‘s –header option.
For More
Learn more about cdpcurl by reading its official documentation and looking through its source code.
Finally, our thanks to the awscurl project, which served as the basis for cdpcurl.
Editor's Choice
