The Simple Definitions, Dos, and Don’ts of Installing Middleware

Background

Middleware is a name applied to software that “sits in the middle” between two different sets of devices, systems or software. In the clinical laboratory, middleware applications are commonly found between laboratory instruments and Laboratory Information Systems (LISs), especially in cases where LISs do not have all of the desired functionality needed to manage laboratory data from laboratory instruments. Middleware can be used in both anatomic and clinical pathology laboratories¹ and may help laboratories better and more easily comply with mandated laboratory accreditation programs. It may also help laboratories adhere to internationally recognized best practice standards^2,3.

Middleware applications can perform a variety of functions to assist laboratory personnel including but not limited to:

• Providing more highly customized rules for real-time analysis than are available in the LIS,
• Performing autoverification for laboratory results that meet all autoverification criteria and flagging results for review that don’t meet one or more of these criteria,⁴
• Holding and flagging laboratory results that may need additional inspection or action (e.g., failed delta check, critical value, result outside of technical range of the instrument),
• Better display of data for quality control (QC) review and approval, quality assurance (QA) and analytics, and
• Directing samples on robotic automation lines to instruments for testing and then to storage areas after testing is completed.⁴

The implementation of middleware is complex because, unlike simple software programs, middleware applications are typically database systems residing on one or more servers that connect to multiple laboratory instruments as well as to the LIS. Depending on the number and type of laboratory instruments connected to a middleware system, middleware can be a conduit for a very large portion of a clinical laboratory’s orders and results. Therefore, thorough validation of the functionality of the system including the accurate transfer of data between laboratory instruments, the middleware system and the LIS are critical for patient safety and laboratory operations. Adequate redundancies, cybersecurity and mitigation plans should be in place to eliminate or minimize the possibility and potential adverse impacts of unplanned downtimes.

You are the laboratory director of several laboratories that perform laboratory testing and blood banking services for both inpatients and outpatients at a large, multi-site community health care system. There are several hospitals, each of which has a blood bank laboratory and transfusion service. The manager of your blood bank tells you that your blood banks want new middleware for some of their blood bank instruments because the proposed new middleware will give them certain functions that their current shared blood bank LIS cannot provide. They do not currently have any middleware in the blood banks.

Question to Think About

What additional information do you need to know about the middleware system?

Any new system in your laboratory should be thoroughly investigated to ensure that the system will provide the functions desired. In addition, the system must be secure, compliant with federal, state and local regulations including federal HIPAA security regulations and compliant with laboratory accreditation requirements.⁵ For blood banks and transfusion services, this includes compliance with additional requirements from the FDA.^6-9

Questions that should be asked:

1. What specific functionality does the new middleware server have?
   • Autoverification of results, Quality Assurance tracking (e.g., delta checks, Levy-Jennings charts), alerts, and encounter-agnostic views of data are common functions provided by middleware
2. Do the blood banks already have tools in-house which would serve this need?
   • End-users in a laboratory sometimes request systems or software without first investigating whether existing systems can be leveraged. For example, the blood bank LIS may actually perform some or all of the requested functions either currently or with the next version upgrade which has already been planned for the next year. It is important to examine this in order to save unnecessary expenditure where it can be avoided.
3. If the new middleware is implemented (goes live), and then it is discovered that there is a problem which needs to be fixed in the middleware software, how would the blood bank test software changes prior to making them live?
   • This is a CAP accreditation requirement and also a requirement for any FDA approved system used for medical products of human origin.
   • This will be described in more detail in Part 2.
4. What is this going to cost?
   • Costs for any system should be calculated using a 5-year cost model in which the following items are added together:
     • Implementation fees (year 1) – this should include all of the following:
       • Hardware
       • Software
       • Project management services
       • Training fees
     • Annual maintenance for year 1
     • Annual Maintenance for years 2, 3, 4 and 5
     • Fees for ongoing backups of data.

The health care organization has a total of two hospitals and one outpatient cancer center which has an on-site transfusion service. Each blood bank has three blood typing instruments and two extended antibody typing instruments. The blood bank at the largest hospital has a red cell genotyping / phenotyping system as well. The sales representative for the proposed new middleware states that all of these instruments will connect to it

Question to Think About

Are there any additional hardware and/or software components to the middleware system that should be included besides a middleware server?

There are many components which should be considered in pricing to ensure that there are no surprise costs should the system be implemented. The number of hardware components required will also depend on the type of connection between the laboratory instruments and the middleware.

For example, if each of the blood bank laboratory instruments has to be directly connected to the middleware server through an analog (e.g., RS232) connection, then your organization will need to purchase a server for each blood bank. However, if each laboratory instrument is TCP/IP-enabled or can connect to a terminal server (which converts analog signals into a TCP/IP signal) and if the vendor states that multi-location installations are supported on each of its middleware servers, then you may be able to purchase a single production and single test server for all of your sites.

Other costs that need to be considered, usually on a per-server basis:

• Software licenses (concurrent vs. per user basis)
• Hardware (server) for production (live) orders and results
• Interface costs – two components

• Cost for the middleware vendor to interface to your LIS
• Cost for the LIS vendor to interface to your middleware (yes, this cost is separate and typically has to be obtained from your LIS vendor)
• A test environment for testing changes and fixes prior to putting them into production (making them live). This may include hardware in addition to software. See below.
• Costs of data backup, if any. This is important for middleware that contains data which is not subsequently passed to the LIS.

Test environments have often been considered optional purchases by laboratories until they need to test a change prior to putting it into production (making it live). As the amount of data handled by a system increases, so does the need for a separate system or environment to test changes. Testing changes in a live system is often either not possible or requires a downtime during which testing cannot be resulted. In addition, a change to a system may cause a catastrophic failure of the system as a whole. As such, testing in a live environment is not recommended and, for middleware and other similarly large scale critical systems, should be avoided at all costs.

Test environments can be approached in two ways…

The blood bank manager also states that the vendor wants to be able to access the server from outside the organization so that the server and its software can be supported by the vendor for maintenance and troubleshooting as needed because it is faster and costs less than having a vendor support representative come on site.

Question to Think About

What steps would need to be taken in order to allow the vendor to access the middleware server in a manner that is safe for your patients’ data?

There are several considerations for remote access to any system that contains patient data.

• Secure access using a method approved by your organization’s information technology department which ensures that the vendor representatives can connect through the organization’s firewall via an encrypted connection (e.g., https://, virtual private network (VPN)). Not only are these addressable requirements from the HIPAA Final Security Rule, but insecure connections from outside your firewall are a significant risk for ransomware, attacks and other compromises.
• Under HIPAA, vendors of laboratory instruments who will have any access to protected health information (PHI) must have a business associate agreement (BAA) in place with your health care organization, and Business Associates must also comply with the HIPAA Final Security Rule. This means that access by the vendor’s employee to your system must comply with all of the following:
  • A unique user login with a password directly to your system or to an intermediary login system which allows identification of the individual employee and which patients he/she accessed in your system. It is preferable that such access include date-time stamps.

Note: Per the HIPAA Omnibus Rule (2013),¹⁰ Business Associates have to follow the same HIPAA security practices with patient data that Covered Entities do.

The new middleware system provides an intermediary repository of data from the various blood bank typing instruments and provides better alerts for typing discrepancies and quality assurance functions as well as better ad hoc reporting capabilities.

The blood bank manager wants the middleware server installed in the blood bank so that they can reboot it when needed without having to wait for Information Technology staff or LIS analysts to do it for them. When the blood bank manager was asked where he proposed to put the middleware server in the blood bank, he points to an area that is right behind a large blood typing instrument. He says this will be great because it is so close to the instrument that they use the most.

Question to Think About

Would you put the server in the location that the blood bank manager has identified? Why or why not? What risks would have to be mitigated if the server(s) are put inside the blood bank?

You should not put the server(s) in the blood bank if it can be avoided. There are a number of risks both to the system and to personnel when servers are located in an area with active movement of people and objects, and consequently, it is difficult to provide adequate physical safeguards to servers as required by the HIPAA Final Security Rule. Unlike servers which are located in secure data centers designed for that purpose, servers which are in laboratories or other areas with high foot traffic have the following risks:

• Accidental or intentional damage or disconnection
  • It may be easy for someone to trip over connecting cables to instruments, network and/or power source
  • Reagents could be spilled on the server(s) – unlike instruments, servers are not built to be resistant to liquids
  • Someone could deliberately tamper with or steal the server(s)
• Inadequate cooling systems and protection from outside heat sources
  • Data centers have strong cooling fans and rigid temperature controls in place to ensure that servers operate at optimal temperatures.
  • Laboratory instruments generate a significant amount of heat, and putting a server behind an instrument may cause a local thermal increase that is not good for the server’s performance or longevity.
• Fire protection
  • Defective servers have been known to catch fire and endanger people and other pieces of equipment around them. Data centers are designed to minimize collateral damage to both people and other servers and equipment in the event of fire. Installation of servers in the laboratory afford no such protection, and extinguishing the fire may result in collateral damage to one or more laboratory instruments critical to laboratory operations.
• Lack of sufficient physical access to either the server or the laboratory instrument for maintenance purposes of either device

It is always better to put servers in a secure locked data center if the middleware server can connect to laboratory instruments this way. If direct cabling is needed from the instrument to the server (e.g., extreme bandwidth needed or non-TCP/IP connections such as analog), then a second best option is to put the server(s) in a secure data closet positioned near the laboratory.

While laboratory staff frequently cite the need to be able to access and reboot the server as needed when problems are experienced, this is often not necessary for every problem and can cause additional problems or masking of the original problem if done inappropriately.

The vendor sends you the technical specifications document for the middleware server. You note that the server software is FDA 510(k) cleared, is running on a Windows 2008 server operating system and, for convenience, has included the user login and password for the database administrator right in the technical specification itself.

Question to Think About

Do you have any concerns about the information in the technical specifications document? If yes, what?

You should have lots of concerns.

• Windows 2008 Server Operating System
  • This operating system is no longer supported by Microsoft.
  • This means that Microsoft is no longer providing bug or security patches to the system. This means that the system is already at risk of current and new malware that anti-virus solutions may not be able to effectively mitigate.
  • In addition, servers are devices which are designed to communicate with hundreds of devices at one time, and consequently, they are excellent distributors of malware when infected.
• Generic user login and password for critical database administration functions
  • Anyone with access to this document (i.e., every other laboratory who has installed or considered installing this middleware system) will know the user login and password to your server, and this includes people both inside and outside your institution. This will enable them to:
    • Delete, tamper or steal patient data
    • Destroy your system
    • Propagate malware to other devices without your knowledge.
• FDA 510(k) clearance is good and is required for most blood bank software and applications, but FDA does not check for compliance with the HIPAA Final Security Rule or for basic cybersecurity protections.

The technical specification also states that vendor employees will access the server remotely through the use of a widely available remote desktop application that contacts an application on the server via a “secure http://web address” that is used by all staff at their organization to enable rapid support.

Question to Think About

Do you have any concerns about the method by which the vendor plans to access the server remotely for support? If yes, what?

Again, you should have concerns.

• Cybersecurity teams have reported that non-secure methods of remote access to systems is a major source of introducing malware including ransomware (e.g., WannaCry, Spectre) into your organization behind your firewall.
• Perhaps this was a typo, but “http://” is not secure. This should at least be https://, and even then, “https://” does not always mean that a connection is secure enough to comply with federal regulations. This should be further investigated by personnel at your institution with expertise in cybersecurity.

It is recommended to involve your institutional cybersecurity experts anytime you are bringing in new software, but especially if it is a server-based application because of the above risks.

Middleware can be a conduit for a very large portion of a clinical laboratory’s orders and results. Therefore, thorough validation of the functionality of the system including the accurate transfer of data between laboratory instruments, the middleware system and the LIS are critical for patient safety and laboratory operations. Adequate redundancies, cybersecurity and mitigation plans should be in place to eliminate or minimize the possibility and potential adverse impacts of unplanned downtimes.

If a middleware system has one or more flaws that render it non-compliant with federal or other regulations or with the laboratory’s ability to comply with laboratory accreditation standards, the laboratory’s best hope of getting these corrected is to get written assurance from the vendor that the flaws will be fixed to your institution’s expectations PRIOR to purchase.

• Middleware can provide significant benefits to a laboratory’s operations, but the implementation, function and security of the system must be appropriately planned and scoped prior to purchase.
• Middleware systems must include test environments or preferably separate test systems so that changes can be tested as safely as possible. This has additional cost, but this cost is minimal compared to a catastrophic crash of a live system with inadequate hardware or data backup.
• Physical security and cybersecurity of middleware systems are important considerations and should be addressed in new and existing systems.
• For new middleware systems, get pathologists with expertise in clinical informatics as well as your LIS team, your organization’s information technology department and cybersecurity experts to review the system prior to purchase.
• For existing systems located within the laboratory or which have one or more flaws described previously, getting pathology informatics and organizational information technology experts involved in an optimization project for the system may be helpful, especially if there are considerations to renew the system or purchase a new one.

1. Your laboratory needs a new point-of-care middleware system for all the point-of-care devices throughout the system. The vendor says that you can install the middleware system wherever you choose. Which of the following is the best place to install the middleware system?
   1. In your point-of-care manager’s office.
   2. In one of your organization’s data centers.
   3. In a data center hosted by the vendor.
   4. At the nursing station where most of the point-of-care analyzers are located.
2. A proposed new middleware system for your blood bank is reviewed by your LIS team, and one of the LIS analysts notes that the system uses an outdated operating system. There are no other FDA-approved middleware systems for the functionality provided by the proposed system, and your team did not discover any other flaws to the software or configuration. What is the most appropriate next action to take?
   1. Contact the vendor to discuss the outdated operating system.
   2. Go ahead and purchase the FDA-approved system because the vendor will fix this problem during implementation.
   3. Don’t implement the system and use a different system which is not FDA-approved.
   4. Implement the system because the operating system version doesn’t really make a difference as long as the system is operational.
3. On your first day as medical director of a laboratory, you are informed that the middleware supporting 75% of your chemistry and hematology laboratory orders and results has a vulnerability to a new computer virus. This new computer virus has already adversely affected several other health care organizations, bringing one organization’s computer systems into an unplanned downtime for over 8 hours with losses of large amounts of patient data. The vendor has a patch for the middleware system that will prevent it from getting infected, but it requires changing a fundamental operating system library which affects its entire operation and could impact locally configured rules, thus requiring an estimated four hours of validation. If this weren’t bad enough, one of your LIS analysts points out that there is no test area for this middleware, meaning that any changes will have to be put onto the production (live) server. Which of the following is the best sequence of actions for resolving this issue?
   1. First: After appropriate downtime communications and planning, take the middleware system down and use a backup method for resulting lab tests while the patch is installed and validated, and then resume operations.
      Second: Buy a different middleware system because the current one wasn’t set up appropriately.
   2. First: Purchase a test environment for the middleware from the vendor and install it on the production server to save time and hardware costs.
      Second: Install and validate the patch in the test environment, then install the patch in the production environment.
   3. First: After appropriate downtime communications and planning, take the middleware system down and use a backup method for resulting lab tests while the patch is installed and validated, and then resume operations.
      Second: Purchase and install a separate test server and software for the middleware system.
   4. First: Purchase and install a separate test server and software for the middleware system.
      Second: Install and validate the patch in the test system, then install the patch in the production environment to minimize the amount of downtime.
4. Your laboratory manager says that they need a new middleware system in hematology because the vendor told them that the new middleware will help them pick up aberrant test results that are currently getting missed and verified prior to investigation. The LIS has some capability to perform rule-based flagging of results, but the laboratory manager states that it is not working. Which of the following is the best response to this inquiry?
   1. Ask where you should sign the purchasing order for the new middleware.
   2. Check laboratory accreditation requirements to determine if the laboratory manager really needs this functionality.
   3. Contact the vendor to find out more about the new middleware system.
   4. Get more details about the functionality that appears to be missing from the LIS.
5. Which of the following statements is CORRECT regarding reasons why middleware may have to be installed in a laboratory instead of a data center?
   1. Instruments connect to the middleware system using analog communications.
   2. Instruments connect to the middleware system using TCP/IP communications.
   3. Laboratory staff may need to re-boot the system at a moment’s notice.
   4. Laboratory staff have to have the system in the lab in order to use the software.
   5. It is better to have the instruments and the middleware all in the same location.

1. Riben M. Laboratory Automation and Middleware. Surg Pathol Clin. 2015;8(2):175-186.
2. Marquardt W. AUTO15: Autoverification of Medical Laboratory Results for Specific Disciplines. In. Vol 2019. 1st edition ed: Clinical and Laboratory Standards Institute; 2019.
3. Neeley W. AUTO10: Autoverification of Clinical Laboratory Test Results. In. Vol 2019. 1st edition ed: Clinical and Laboratory Standards Institute; 2010.
4. Levy G. Use of Middleware to Increase Clinical Laboratory Efficiency. LabThruPut 2012; Available at: https://labthruput.com/use-of-middleware-to-increase-clinical-laboratory-efficiency/. Accessed August 16, 2019.
5. Health Insurance Reform: Security Standards (Final Security Rule) (2003), 45 CFR §§ 160, 162, 164. 2003; Available at: https://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/administrative/securityrule/securityrulepdf.pdf?language=es. Accessed February 3, 2017.
6. General Principles of Software Validation; Final Guidance for Industry and FDA Staff. In: U. S. Department of Health and Human Services; Food and Drug Administration; Center for Devices and Radiological Health; Center for Biologics Evaluation and Research; 2002:1-47.
7. ISBT 128: More than Identification – United States Industry Consensus Standard for the Uniform Labeling of Blood and Blood Components Using ISBT 128, Version 2.0.0. In. York, PA: ICCBBA, Inc; 2011:90.
8. Guidance for Industry: Blood Establishment Computer System Validation in the User’s Facility. Food and Drug Administration. Center for Biologics Evaluation and Research. 2013; Available at: http://www.fda.gov/downloads/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/Blood/ucm078815.pdf. Accessed July 10, 2016.
9. Armitage J, Ashford P, Bolton W, et al. ISBT 128 STANDARD: Technical Specification, version 5.5.0. San Bernadino, CA: ICCBBA; 2016.
10. Modifications to the HIPAA Privacy, Security, Enforcement, and Breach Notification Rules Under the Health Information Technology for Economic and Clinical Health Act and the Genetic Information Nondiscrimination Act; Other Modifications to the HIPAA Rules; Final Rule (HIPAA Omnibus Rule) (2013), 45 CFR § 160, 164. Available at: http://www.gpo.gov/fdsys/pkg/FR-2013-01-25/pdf/2013-01073.pdf. Accessed February 3, 2017.

2019
Alexis B. Carter, MD, FCAP
CAP Informatics Committee
Physician Informaticist – Pathology and Laboratory Medicine
Children’s Healthcare of Atlanta
Atlanta, GA

1. b

2. a

3. c

4. d

5. a