Connected Manufacturing Insights from the Edge with Cloudera DataFlow

Connected Manufacturing’s Pivot to an Enterprise Data Solution

Connected Manufacturing is at a turning point and it is catalyzed by a real, measurable change and shift in data types – real-time and time-series data is growing 50% faster than latent or static data forms and streaming analytics projected to grow at a 28% CAGR, leaving legacy data platforms that specialize in static historical data solutions, functioning on-prem or in discrete clouds, inadequate in addressing today’s real-time insight needs. Streaming data’s growth is powered from the fact that it enables real-time insight and more importantly, autonomous decision making. 

This shift in manufacturing has been enabled by the proliferation of inexpensive process sensors tailored to the specific use, robust edge computing devices allowing repetitive autonomous decisions, cloud computing performing both analytics and storage, and soon to come – 5G, which opens the lanes of the data superhighway, freeing manufacturing processes from the chains of hard-wired connections.  But the benefits of streaming data create a challenge in managing its massive data volume, its diverse data structure, and its real-time speed of light velocity across manufacturing enterprise business processes.  

Limitations to Traditional Connected Manufacturing Data Ingest Solutions

Today’s manufacturers face limitations addressing the stressful complexities of digitalization, much of these limitations are due to the rapid evolution of new & connected data sources and the massive volume of data spewn out. Some of the critical challenges and key considerations that organizations face with respect to data management for Connected Manufacturing: 

  • The cost of data management: Traditional data management mechanisms tend to be notoriously expensive, do not scale easily and were not built for capturing and processing the petabytes of IoT data streaming from connected devices. Today, organizations need a more flexible and scalable data management & analytics platform that can easily ingest, store, manage, and process streaming data from IoT sources at a lower cost. 
  • Handling the volume and variety of IoT data:   To enable process monitoring & optimization, predictive maintenance and emerging IoT use cases, information architects need a platform that can handle all types of diverse data structures and schemas, including everything from intermittent readings of temperature, pressure, and vibrations per second to handling fully unstructured data (e.g., images, video, text, spectral data) or other forms such as thermographic or acoustic signals, from the edge, delivered through diverse supported drivers and protocols. 
  • Managing the complexity of real-time data: In order to drive continuous process monitoring, throughput optimization or predictive maintenance, a data management platform needs to enable real-time analytics on streaming data. The platform also needs to effectively ingest, store, and process the streaming data in real-time or near-real-time in order to instantly deliver insights and action. 
  • Freeing data from independent silos: Specialized processes (innovation platforms, QMS, MES, etc)  within the value chain reward disparate data sources and data management platforms that tailor to unique siloed solutions. These narrow point solutions limit enterprise value considering only a fraction of the insight cross-enterprise data can offer, in addition, duplicate siloed solutions divide the business, limiting collaboration opportunities. Also, the platform must have the ability to ingest, store, manage, and process streaming data from all points in the value chain, combine it with Data Historians, ERP, MES and QMS sources and leverage it into actionable insights.  

Given the complexity and variety of manufacturing and IoT data, manufacturers are focusing on driving insights from edge to AI.  To do this, a great place to start is naturally at the beginning, where data is ingested into the data lake and enterprise data platform.   

Cloudera Data Platform is addressing these challenges through its portfolio of technologies that reside in Cloudera DataFlow ( CDF).  CDF offers the following solutions:

  • The ability to manage, control, and monitor the edge for all your streaming and IoT initiatives. Cloudera Edge Management (CEM) consists of edge agents and an edge management hub. It manages, controls, and monitors edge agents to collect data from edge devices and push intelligence back to the edge.
  • The ability to ingest and manage real-time streaming data. Cloudera Flow Management (CFM) is a no-code data ingestion and management solution powered by Apache NiFi. With NiFi’s intuitive graphical interface and 300+ processors, CFM delivers highly scalable data movement, transformation, and management capabilities to the enterprise.
  • Advanced messaging and stream processing powered by Apache Kafka. Cloudera Stream Processing (CSP) provides advanced messaging, real-time processing, and analytics on streaming data using Apache Kafka as well as management and monitoring capabilities powered by Cloudera Streams Management.
  • Real-time insights delivered by Cloudera Streaming Analytics (CSA). Powered by Apache Flink, CSA offers stateful, low-latency processing of real-time actionable intelligence of your streaming data from the edge.

 These solutions add up to a powerful combination of technologies that drive data into your enterprise from the edge.  To learn more about tackling the edge to AI challenge,  view the webinar “Connected Manufacturing’s Insight from the Edge, Cloudera DataFlow ” or learn more at

David LeGrand
More by this author

Leave a comment

Your email address will not be published. Links are not permitted in comments.