In 2022, it’s hard to believe, that for the first decades of the Information Age, the U.S. military and kept track of health records for millions of active-duty soldiers, sailors, airmen and airwomen, support staff, and retired service people using pens & pencils, typewriters, paper, carbon paper, copy machines, and snail-mail. Unsurprisingly errors were all too common, as people were involved in every transaction. As hard as healthcare staff worked to get it all right, mistakes still happened, records were impacted, and patients often suffered the consequences.
But thanks to cloud services and a next-generation electronic health-care record (EHR) system that has taken more than two decades to develop and deploy, human error has been cut drastically and a system has finally evolved from paper-based to near real-time. Because military families are highly mobile, the new EHR’s “one patient, one record” design makes a person’s medical history available at the point of care at any military medical facility — and within seconds. Those seconds saved could mean life or death for a patient.
The Department of Defense (DoD) began implementing a platform originally tagged as the generic CHCS II (Composite Health Care System), a precursor to the military’s second-generation electronic health records (EHR) system, MHS Genesis. This swift digital information distribution system locks down personal facts, yet makes them available on-demand 24/7 to properly credentialed medical and military personnel, vastly improving safety and service for millions of beneficiaries.
This health-records system emanated from two legacy structures — one serving the Veterans Administration (VA), the other serving the DoD. In 1998, the decision was made to unite it in an all-inclusive — and hopefully more efficient — system. Its evolution to the present-day cloud-based package is a real-world case study that will likely live in IT textbooks for as long as use cases will be referenced.
MHS Genesis has to tackle an almost impossible job in moving and processing petabytes of data, securely and accurately. The DoD is the federal government’s largest agency and one of the most complex organizations in the world. With more than 5,000 locations worldwide, 2.3 million retirees, 1.3 million active-duty service members, 750,000 civilian personnel, and more than 811,000 National Guard and reserve service members, the DoD is the nation’s largest employer.
The DoD’s budget of $703.7 billion for 2022 represents a full 11% of the nation’s total. About 15% of that amount will involve health care. This operation requires a massively scalable records system with backups everywhere, reliable access functionality, and the best security in the world.
The long standing goal of the DoD and VA is to provide a single life-long personal health record, starting with the person’s entry into the military health system and continuing through subsequent updates and eventually on to their retirement and care under VA.
So how exactly is the EHR managing petabytes of data?
Most of the massive data-management tasks DoD faces fall into that area where data, analytics, and the cloud intersect. (Remember, a pedabyte of data is roughly equivalent to 500 billion pages of standard printed text) A solution was needed to backstop those never-ending streams of data into a single, universally available platform, using advanced analytics powered by machine learning optimized for a cloud service. This was the only way such a huge EHR system was going to work.
The DoD/VA team spent four years and $1 billion on a joint records program called iEHR, only to abandon it in 2013 in favor of separate systems (neither one could scale). The VA then announced in June 2017 that it would use DoD’s MHS Genesis system for electronic health records, which is being built under a 10-year contract awarded in 2015 and projected to ultimately cost $10 billion.
The centerpiece of MHS Genesis is Cerner’s Millennium services management platform, which provides hosted software-as-a-service functionality in the cloud. A key reason for selecting Cerner, the DoD said, was the company’s data center allows direct access to proprietary data that it couldn’t obtain from a government-hosted environment. Cerner also had experience in dealing with the massive volume of data created by a health system that serves multiple millions of beneficiaries at hospitals and clinics around the world.
The core of the EHR system
At the core of Cerner Millennium is a next-gen platform powered by Cloudera.
The companies began working together in 2010, when Cerner adopted Cloudera’s open-source platform distribution that enables random access to data. When Cerner, whose services are already used at more than 14,000 locations around the world, set out to create a comprehensive population health platform and needed a way to employ significant computational power while maintaining flexibility in handling its records, it chose Cloudera again.
With Cloudera, the platform takes in data with various formats from multiple sources — electronic medical records (EMRs), Health Level 7 International (HL7) feeds, Health Information Exchange (HIE) information, insurance claims data, and extractions from proprietary or client-owned systems. The platform can absorb data streams in real-time, then pass them on to the right database or distributed file system.
Cloudera’s Shared Data Experience (SDX) provides a single workspace for core functions in EHR administration but can also be utilized for other purposes, such as data engineering, data science, analytics, and operational databases. Its unified model has been described by users as the most cost-effective and fastest system to deploy while being easiest to secure and govern.
SDX solves the problem of isolated data clusters — a constant problem throughout the DoD’s global footprint — by sharing persistent data and metadata across on-demand applications, ensuring that each cluster doesn’t need to be controlled separately. By centralizing security controls, SDX applies policies and practices consistently across all applications, so they don’t have to be reapplied when data is moved or a new application is added into the mix. These are constants in the massive system.
Shared catalog of data, metadata aids compliance requirements
Cloudera Navigator’s user interface uses a shared catalog of data and metadata which makes it easier to meet compliance mandates, something the DoD faces all the time. Mandates may fall under NIST Cybersecurity Framework, the Federal Information Security Management Act, the Health Insurance Portability and Accountability Act (HIPAA), and other regulations. The system supports audits by fast location of data and applications, understanding where data emanated, and tracking how it’s been modified.
Finally, SDX supports multiple public and private cloud environments, as well as single-tenant, bare-metal configurations.
Using Cloudera’s engine, Cerner manages more than 2 petabytes of data in support of both patient and financial matters for numerous clients.
“Cloudera provides a holistic view of our whole environment, and allows us to manage multiple clusters from a central point,” said David Edwards, Vice President and Fellow at Cerner.
Collecting and analyzing data from almost unlimited sources allows clients to create a far more complete picture of patients, their conditions, and treatments. “We’re able to achieve much better outcomes, both patient-related and financial, than we ever could by just looking at pieces of the puzzle individually,” said Ryan Brush, Senior Director and Distinguished Engineer for Cerner.
The EHR is rolling out location by location
Despite the COVID-19 pandemic, MHS Genesis is currently live and operational across more than 600 military treatment facilities with more than 42,000 total active users, according to DoD officials. Twelve more states came online in April, and the project is about one-third of the way finished, but there are still hundreds of installations to go.
The DOD EHR rollouts are being deployed across the U.S. and overseas through a total of 23 waves. Each wave will target a specific region over one year, with an average of three hospitals and numerous physical locations for each wave. This approach enables the DOD to take full advantage of lessons learned from prior waves to maximize subsequent waves’ efficiencies. Full deployment of MHS Genesis is expected by the end of calendar year 2023, according to the DoD.
Even though the DoD feels confident in its deployment process at this point, there’s still plenty more to learn, according to Brig. Gen. George Appenzeller, the Defense Health Agency’s assistant director for combat support.
“I think what we’re learning is less drastic each time. But every time we do something new, we find new things that we can do better, better ways of training, additional things that we need to work on,” Appenzeller said. “That’s why we have the informatics team so heavily involved with our clinicians — with weekly calls leading up to any rollout with all the commanders to go over all of the things that they can and can’t do.
“One of the most impressive things is the data management capabilities: We can actually see how much time people are spending in the record during duty hours, after duty hours. What we’ve found is that what some people colloquially call ‘pajama time’ — after duty charting — has gone way down in the locations that have Genesis. That means that not only is the safety getting better, but providers are getting home to their families. And a rested, happy provider is a much more productive provider who takes better care of our patients.”
Editor's Choice
