This post was published on Hortonworks.com before the merger with Cloudera. Some links, resources, or references may no longer be valid.
Deep learning may sound almost mystical, but it’s actually a subset of artificial intelligence (AI). In deep learning, structured and unstructured data is processed in a way that imitates the human brain’s nonlinear processing skills. Deep learning creates neural networks that can process data unsupervised, mostly without human intervention.
The promise of deep learning has been limited by the architecture it runs on. To overcome these limits, deep learning is increasingly being powered by containerization software hosted in easy-to-spin-up environments that run seamlessly over graphical processing units (GPUs). This more flexible architecture offers new opportunities for big data processing that are being realized in sectors generating lots of data. In healthcare, for example, deep learning improves radiologists’ detection of abnormal images, having learned what to look for from thousands upon thousands of scans. That learning allows for better diagnoses and decision-making. It’s also being used with autonomous vehicles to help them recognize objects and model the behavior of other cars on the road, as well as with self-flying drones used for search and rescue missions and building and insurance inspections.
Faster Time to Deployment, Faster Time to Insight
Containers are fully functioning runtime environments that hold an app and all of its dependencies. Being fully contained, they can run in any data center environment and are easy to move and deploy. Once the container itself is created, it’s relatively easy for any end user to access. You don’t necessarily need to ask IT to set aside compute resources or set up a new cluster, which can take up to two months in a physical environment. With containers, there’s no need to deploy a physical machine. You can run a container in a Hadoop environment, utilizing its storage and compute capabilities, with the click of a button. It takes all of five minutes to set up.
Deep learning is compute-intensive, with heavy processing requirements—often necessitating the use of GPUs. The parallel architecture of GPUs can accelerate compute-intensive workloads, using smaller, efficient cores that are designed to manage multiple tasks simultaneously. GPU performance is essential for open-source, deep-learning frameworks. They provide faster processing at a lower cost, making deep learning an option for more companies, and they allow end users to deploy the environment they need very quickly. The combination of containerization software processed via GPUs has led to the ability to achieve data processing at scale. The result is faster learning in an almost self-service manner, delivering insight and answers to end users sooner.
Containerization Software and the Future of Big Data and Deep Learning
Containerization overcomes the limits of data architecture and places applications as close to the actual data as possible. Rather than a hardened infrastructure, containers allow for agility and elasticity in data processing. Containers maximize portability: these self-contained runtime environments can be taken anywhere, to any platform or data repository. This flexibility allows for higher and better use of compute, storage, and memory resources.
Containers also allow for foolproof packaging. You don’t have to be an IT team member to use containers. More tech-savvy team members can set up containers to provide specific functions, and others can use them as needed. As long as the container holds an application and all its dependencies, any user can be assured of a repeatable, reproducible process. This means application development, data processing, and deep learning can happen faster—and be reliably replicated as needed.
The healthcare, finance, retail, and automotive sectors are the current leaders in deep learning. However, as data processing capabilities get faster and the production of data sources grows, big data technology will become more democratized and usable by businesses of all sizes and types. Shortening the time from insight to action will ensure that your business can compete and thrive. Container portability and GPU acceleration are set to be the enablers of your deep learning capabilities.
Learn more about how data science and deep learning can differentiate you from your competitors.
Editor's Choice
