The motivation behind Cloudera Altus SDX is to enable multiple clusters to share the same consistent view of enterprise data hosted on Amazon S3 and Microsoft ADLS. At the heart of Altus SDX is a repository of attributes describing locations and structure of data, access rights, business glossary definitions, lineage and more.
We often hear from our customers about use cases where data is in the cloud and clusters are created on demand to ingest new datasets.
This article was originally posted by Tom Smith Research Analyst and Business Stratgist, DZone, Inc on their website and is being shared here with permission.
Doug Cutting, Chief Architect at Cloudera, shares how the company uses open-source software to help companies use data to improve their business.
What does your company use open-source software to accomplish?
Everything we do.
One of the principal features used in analytic databases is table partitioning. This feature is so frequently used because of its ability to significantly reduce query latency by allowing the execution engine to skip reading data that is not necessary for the query. For example, consider a table of events partitioned on the event time using calendar day granularity. If the table contained 2 years of events and a user wanted to find the events for a given 7-day window,
More of you are moving to public cloud services for backup and disaster recovery purposes, and Cloudera has been enhancing the capabilities of Cloudera Manager and CDH to help you do that. Specifically, Cloudera Backup and Disaster Recovery (BDR) now supports backup to and restore from Amazon S3 for Cloudera Enterprise customers.
BDR lets you replicate Apache HDFS data from your on-premise cluster to or from Amazon S3 with full fidelity (all file and directory metadata is replicated along with the data).
With modern businesses dealing with an ever-increasing volume of data, and an expanding set of data sources, the data engineering process that enables analysis, visualization, and reporting only becomes more important.
When considering running data engineering workloads in the public cloud, there are capabilities which enable different operational models from on-premises deployments. The key factors here are the presence of a distinct storage layer within the cloud environment, and the ability to provision compute resources on-demand (e.g.: with Amazon’s S3 and EC2 respectively).