Since the beginning of our respective partnerships with Dell EMC as legacy companies in 2015, both Cloudera and Hortonworks delivered differentiated consolidated data lake solutions to our joint install base of over a hundred customers running Hortonworks Data Platform (HDP) and Cloudera Distribution of Hadoop (CDH) in a shared storage configuration with Dell EMC Isilon.
Dell EMC Isilon scale-out Network Attached Storage (NAS) has the ability to run HDFS natively and incorporates critical components of the HDFS software stack such as the name-node and data-node inside the OneFS software. For Hadoop analytics, Isilon’s architecture minimizes bottlenecks, rapidly serves petabyte scale data sets and optimizes performance. Dell EMC ECS is a leading-edge distributed object store that supports Hadoop storage using the S3 interface and is a good fit for enterprises looking for either on-prem or cloud-based object storage for Hadoop.
Cloudera’s new streamlined Quality Assurance Test Suite (QATS) certification process is designed to validate HDP and CDH on a variety of Cloud, Storage & Compute Platforms. The validation and certification of Dell EMC’s Isilon and ECS storage solutions is enabling us to deepen and expand our partnership.
QATS functional testing will be performed on the latest HDP/CDH versions on Isilon and ECS. In addition, Stress, High Availability, Reliability, Performance, Operational Readiness, and Integration Testing will be executed to ensure all components accommodate system demands.
What does this mean for our joint customers?
QATS certification on Isilon and ECS provides existing customers with better support and reduced risk. Clusters that use CDH/HDP in conjunction with Cloudera Certified Technologies operate with lower risk and lower total cost of ownership (TCO). Cloudera Certified Technologies have been tested and validated to use supported APIs and to comply with Cloudera development guidelines for integration with Hadoop. Validation includes:
- Overall architecture
- Observance of the Hadoop interface classification system
- Complete integration testing
- Compliance with Cloudera support policies and requirements
- Cluster capability using real-world workloads and micro-benchmarks
Dell EMC Isilon has already been validated with HDP 3.0.1 and CDH 5.14 and Dell EMC ECS has been validated with HDP 2.6.2 and CDH 5.14. These solutions are being jointly supported by Cloudera and Dell EMC. We have a joint support process in place that involves triaging any issue that occurs with our solution, regardless of where it is discovered and directing issues to the appropriate teams, either in Cloudera or Dell EMC.
“Our partnership with Cloudera has provided customers with the data they need to make the most of Hadoop,” said Manuvir Das, Senior Vice President and General Manager of Unstructured Storage at Dell EMC. “We are excited to continue this partnership and bring to market differentiated storage solutions that align with future Hadoop trends, such as independent scaling of compute and storage and containerized Hadoop environments.”
Over the next four months, we plan to work with Dell EMC to get Isilon certified through QATS as the primary HDFS store for both CDH (version 6.3.1) and HDP (version 3.1), with an emphasis to develop joint reference architecture and solutions around Hadoop Tiered Storage. Doing so will enable customers to use Direct Attached Storage (DAS) for hot data and Isilon for cold data within the same logical Hadoop cluster to simultaneously deliver extreme performance and economic scaling.We also plan to certify Dell ECS as the S3 object store using QATS for both CDH and HDP.
| Hadoop Distribution | Hadoop Storage | Configuration |
| CDH | Isilon | Hadoop Primary HDFS Storage |
| HDP | Isilon | Hadoop Primary HDFS Storage |
| CDH | ECS | Hadoop S3 Storage |
| HDP | ECS | Hadoop S3 Storage |
| CDH | Isilon | Hadoop Tiered HDFS Storage |
| HDP | Isilon | Hadoop Tiered HDFS Storage |
How does Dell EMC’s investments in certifying their storage solutions strengthen our partnership? The answer is three-fold:
- It allows us to support our existing install base of HDP/CDH deployments with Isilon as the primary HDFS store.
- Over the past few years, HDP/CDH deployments have gradually matured across multiple industry segments, including large-scale SQL-based analytics. Given the large ecosystem of components that are part of the legacy platforms (CDH/HDP), and its adoption in the market, it is critical for Dell EMC to continue to offer innovative end-to-end solutions for HDP/CDH.
- As HDP/CDH deployments scale in size, companies are starting to separate Hadoop compute and storage so that storage can be scaled independently of compute as data volumes grow. Virtualized and containerized Hadoop deployments are growing in adoption. And finally, object store is gaining popularity as the underlying storage layer for Hadoop. Shared storage configurations utilizing Dell EMC Isilon and Dell EMC ECS aligns well with these industry and technology trends.
If you have questions or feedback on Cloudera’s partnership with Dell EMC, please reach out to your local Cloudera account executive.
Editor's Choice
Wow.. This is interesting ECS (Object Storage) + CDH Validation.. Happy to see that combination, which clears the paths for hadoop compute and storage separation. And also You mention that this will help in “containerized Hadoop deployments”. Could You please share the performance metrics of ECS +CDH and also what are the plans for containerized Hadoop deployments using Object storage..? Would be glad to see any recommended deployment options .
Thanks a lot for the blog post Scott. Will be interested to keep knowing the latest updates on these lines.. Object Storage + Containerized hadoop deployments in production.
Regards
Raja.