One Big Cluster Stuck: Visibility and Transparency

by Shayde Christian

Posted in Technical | May 08, 2023 2 min read

Out of the box Cloudera Data platform (CDP) performs superbly but over time, if data architecture, data engineering, and DevOps best practices are not maintained, you can get stuck maintaining the wild, wild west. In this six-part series, we’re focused on improving the health of your environment.

Visibility and Transparency

Improving environmental health is impossible if you’re flying blind. You’ve got to see the challenges to overcome them. You’ll need to be able to suppress noise in order to focus your efforts on analyzing the root cause of the most significant challenges, and if you can proactively notify impacted users of the incidents, progress, and resolutions, you’ll earn credibility.

We would like to share monitoring, alert, and notification best practices to help you:

Gain visibility into environmental health issues
Manage automated alerts to focus attention on the most important issues
Automate communication to keep users informed
Repair root cause issues instead of merely overcome incidents
Improve the health of the platform, services, and data processes
Move from a reactive to proactive troubleshooting posture

The Instructions, Tools, and Code We Are Providing You

This blog is merely a gateway to many valuable tools, lines of code, and best practices to gain visibility into all that ails your environment. We provide automation, or instructions on how to automate monitoring and alerting processes so that you can dedicate effort not only to recover after incidents, not only to identify and resolve root cause problems, but to become proactive in those discovery and repair processes.

Alternatively, you could adopt Cloudera Observability and get tons of visibility immediately!

In addition to cluster monitoring on Cloudera Data Platform (CDP), we guide you through ways to monitor and troubleshoot issues using commonly used microservices: Cloudera Manager (CM), Hive, Impala, CDSW, Impala, Yarn, Nifi, Airflow, and even Jurassic cron jobs.

Some manuals guide you through proper implementation of out-of-the-box CM monitoring tools while others provide JSON files with steps to launch some of our homegrown monitoring capabilities. Others go into great detail to teach you how to connect your logs to visualizations, step-by-step.

Our great hope is that this will enable you to significantly improve the health of your environment. If we can clarify any steps for you, click the “Ask a Question” button on our Community page. Enjoy!

Links to the Goodies