Take a Deep Dive into Healthcare IT Challenges, Gaps and Solutions.

The healthcare world is changing rapidly, and change includes ever more pressure to accurately and cost-effectively validate and test software used in laboratory settings, patient management systems and every other aspect of healthcare IT.

We're constantly striving to better understand the stress points and gaps that block the path to success for healthcare organizations.

SmarteSoft's CEO, Gordon MacGregor recently talked with Colin Hope-Murray, the former IT Chief Architect for the American Red Cross, about challenges, opportunities, best practice methodologies and the corporate culture required to overcome obstacles, speed test and deployment of new software tools, while getting the best bang for the buck.

Colin shares his views on cutting edge technology vs. proven solutions that best meet business goals, test cycles and the best method to use, how to efficiently validate software and achieve compliance, the importance of planning for regression testing and how your corporate culture can contribute to improved efficiency and cost containment.

Gordon MacGregor (GM): Colin, can you describe your role at ARC and any insights it gave you into IT’s role in modern blood banking organizations?

Colin Hope-Murray (CHM): As Chief Architect at ARC, I was responsible for the overall system architecture of the ARC MRP system, which included collection, manufacturing and distribution components. This included design, configuration and performance testing of the ARC BECS (Blood Establishment Computer Software) solutions.

The goal in the bio-medical group was primarily to prepare and design the environment that would essentially replace the existing MRP, the manufacturing and retail process, focusing on blood banking concerns: collection, processing and manufacturing into a variety of blood products which are sold to both hospitals and research establishments.

The blood manufacturing industry is unusual in that it is very discrete. There are no international, global or dominating enterprises. Blood Bankers, as they like to be called, are nationally discrete and few have much, if any, competition since in many cases national governments have incorporated the service into their healthcare programs. Consequently there are no large profits or revenue streams such as experienced in more commercial healthcare enterprises.

This has an interesting effect on the use of IT since government bureaucracy and/or the lack of competition produces several interesting reactions. Firstly, it can cultivate an atmosphere of complacency, the “we-have-always-done-it-this-way-and-see-no-need-to-change” attitude, best described as conservative. This applies in no short measure to the blood banking businesses themselves but it is amplified in the IT departments within those enterprises for any IT organization is only as good or as mature as the business it supports.

All too often IT is inhibited even more because whatever funds may be available are channeled toward technology procurements rather than investing in IT resources and management frameworks. Without an imperative to evolve and survive the industry is slow to mature.

But time and tide waits for no man, and the speed of technical innovation, the increase in privacy and health regulations and the pervasiveness of internet accessible information are all pushing these operations into the 21st Century. Just how far some of them have to go can be determined by the amount of paper based operations that remain within this sector of the industry.

GM: Some blood bank organizations are very hesitant to do anything new or different, while others are very forward thinking and open to adopting new technologies and approaches. What are the characteristics you have seen in IT organizations that drive that bias either way?

CHM:What is a successful approach to what would be considered state of the art IT? Not necessarily the most advanced technically, or the most advanced in thinking. I would consider it more important to establish a flexible yet stable architecture and thereby provide the optimal efficiency for the given circumstances, and probably most importantly to any enterprise, an architecture carefully aligned to that business’s goals and objectives. Really being able to understand how all the business, information and technical pieces come together is fundamental to successful architecture design and ultimate IT success.

The enterprise culture has to be as adaptive and forward thinking as possible. It requires a long term adjustment to the many changes within any organization. The organization’s culture has to invite and be supportive of those changes, recognizing the strategic goals, not solely focus on tactical reactions that tend to inhibit long term thinking and evolution.

GM: Can you cite examples of specific objectives and recommended cultural perspective?

CHM:Let’s take this example: you may need to change one component within your ERP/MRP. If you just change it discretely, and yet everything that interfaces and interacts with that component remains the same then there is no real pressure to look far beyond the immediate, tactical reaction to that particular demand.

But as everyone knows, the nature of civilization, the nature of our economy, is that change is constant and it is extremely rare for only one thing to need change in a large and sophisticated system. To recognize this, to be able to react, to be able to be proactive in harnessing those changes whether it is a change at the technical level (which is probably the easiest), to a change at the process level (which tends to be a little more elusive, a little harder to achieve) or a change at the information level (which is perhaps even harder still) is critical to long term success.

Recognizing that changes have a knock-on effect on the organization and its profitability or efficiency is a fundamental challenge to any culture. The cultures that recognize change and adapt and have the ability to plan to address the impact of change in each of these areas of evolution are the ones that tend to be the most successful.

GM: So does that mean you’re never done? Basically you don’t have the perspective that we’ll implement project X and then that will be finished and we can rub our hands together and sit down and relax – we’re finished.

CHM:I think that’s a great point, you’re absolutely right. You are never done. In fact, if you look at many of the frameworks and lifecycle management they are all built around continuous review – that word in itself, lifecycle, suggests that it’s an ever turning wheel. It includes the review and assessment of all of your artifacts, components and frameworks, in fact as much of your business estate as you can digest. The constant review of these items on a regular basis is fundamental to ensure that you are both reacting properly to change and also that you are in a position to forecast change and be able to make longer term plans to adjust to change in the future.

The second part – and I think it’s a key one that took me a very long time to realize – is that because you’re never done, it does not make sense to pursue perfection. I think at some point, there is a tendency to improve performance and efficiency and drive it to be as best as it can be. However the very fact that change is inevitably going to follow hard on the heels of the latest effort suggests that any optimum situation is transient anyway. And therefore it’s better to achieve a good balance between what is perfect and what is good and necessary for the business.

GM: So that raises an interesting thought. It sounds like you are applying almost an agile development mind set in a macro way across all of IT for an Agile culture across the board not just within a specific development team.

CHM:That’s a good way of putting it. I hadn’t thought of it that way. You’re right. If you’ve got all of the frameworks and lifecycle management procedures in place and diligently pursue the regular review of artifacts, essentially what you are doing is preparing yourself to be able to react as quickly as possible to market stimuli and which would, as you said, be an Agile environment.

GM: Unlike Waterfall were you’re trying to design the ultimate thing, spending years building it, with Agile methods, you’re incrementally improving and having sprints and seeing what’s next and so forth. That seems to tie in with what you said about not aiming for perfection every time.

CHM:Correct. Actually what it does do is it exposes sometimes the folly of the Waterfall methodology where you spend a lot of time, a lot of resources, and a lot of effort to achieve something and then when you finally get there you find it really wasn’t quite what is now needed or wanted.

GM: So moving an organization from a waterfall mentality to an Agile mentality, from top to bottom, side to side, is a daunting task. How could you go about doing that?

CHM:Unfortunately it’s not one of those efforts that can really come from the bottom up. It really has to come from pretty much the top down. There has to be a sponsor at the most senior level within the enterprise for the initiative to be recognized, established and then adhered to by the organization. So it does require each and every CEO to recognize – and this is where we come back to the culture questions – that if they do want to have this organism, because that is essentially what the organization is, if they want to keep the whole body healthy, then Agile review of all aspects of the organization has to be a fundamental principle and practice.

For those cultures that don’t have that adaptive mechanism, that appetite for change, they need a CEO with the courage and foresight to see the advantages of the agile enterprise and the skill to articulate the business imperative that will modify the culture to accept it. The cultures that are most resistant to change will be the ones that risk being left behind and becoming redundant.

GM: That makes a lot of sense. I once had a job running a line of business for a computer company in the industrial manufacturing space and industrial automation in particular. And one of the things – this was a long time ago, so the world has changed – but one of the things that surprised me at that time was that the manufacturing organizations I was dealing with didn’t really want the latest and greatest high tech innovation. They wanted just something that worked and always worked and that they probably weren’t going to change for ten years. And that was going to run 24/7 with virtually no downtime every year. So reliability over a long time period was far more important to them than innovation. If I think about the automobile industry today, companies that still think that way wouldn’t last very long because car lifecycles aren’t that long any more.

CHM:Correct.

GM: But in a certain sense, the blood bank industry is a manufacturing industry or at least part of it is. I’m just wondering if there is a similar thought there culturally. Is blood banking typically in the mode of “let’s get something in place and we’ll use it for the next 10 or 15 years” or what is the current culture that you see most prevalent in the blood banking industry?

CHM:Your picture of the automotive industry is probably accurate; for many industries the assumption historically has been that any system that is being developed should and could last for as long as possible. This is partly due to the fact that, at least within the America Red Cross, it’s still largely a paper based operation. Here we are into the second decade of the third millennium and we’re still talking about paper based operations.

The fact that these systems have been there for so long suggests there is a fundamental belief that whatever has been put in place is good enough for now and good enough for as long as can be foreseen. It’s only a major external change such as the arrival of mandatory standards that can cause a disruption in the way systems are developed, maintained and updated. The change demanded by web based applications, for example, is never a direct cause for modification as conservatism would prevent it. Those changes are residual and often overlooked, so that web architectures are incidental and specific more often than they are planned and comprehensive.

One of the things that prompted the American Red Cross to change was the introduction and wholesale adoption of a labeling standard within blood banking, called ISBT128. This standard is very important for providing as much information as possible on the blood product distributed to hospitals and research centers. The standard was established some time ago but it has taken the Red Cross more than 2 or 3 years to get round to implementing it, though they are now almost there.

So part of the challenge is in replacing the current MRP with a new MRP that will allow them to retail or resell products which meet this necessary standard. And yes, there is still a belief that changes made now will be rock solid for the long term.

But if you step back and take a look at all the other changes that are coming – it’s not just the labeling standard, it’s additional HIPPA requirements around personal or donor data. These are also coming down the pike and evolving as well; the HIPPA standard we have today is only a part of the HIPAA standard of tomorrow. HIPAA will be added to and refined in subsequent years.

Every year will bring new standards that will require a response in the form of a change which suggests that standing still is the wrong thing to do. Because if you build your waterfall approach and end up with a new manufacturing system that took you 2½ to 3 years to put together and then found that the standard you now have to meet is not met by your current implementation you’re back to another Waterfall. And that approach is no longer sustainable in this environment.

GM: A new healthcare bill puts all sorts of requirements on electronic medical records.

CHM:Correct. If anything, what the internet does is make us increasingly sensitive to the amount of private and public information that is available online, and to how well that information is secured.

Compliance with standards and regulations is critical to any blood bank. Without compliance an enterprise would be unable to use that software in the country where that compliance is required.

GM: What constraints does the organization face in attaining and retaining compliance? Would this include personnel resources, IT resources?

CHM:It requires a core set of resources both from the operational point of view, from the documentation point of view and, most importantly, resources experienced in short and long term planning. The regulations are there to ensure that any particular process, especially one that has a direct impact on the health and potentially lives of the recipients of blood products, meets requirements of health and safety. Furthermore these regulated processes must also be repeatable, measurable and documented.

If there are any problems, if anything bad happens – say someone has a negative reaction to a blood transfusion – you’ve got a full set of data about the blood product and documentation of the procedures and actions taken for that product, to determine what caused the issue and how we can rectify it to prevent it from happening again. That brings in literally all of your organization, from the business planners and directors to your operators at the blood collection drives and the processors in the manufactories to your IT organization and its set of planners, analysts and operators. Because each one of them has to be able to comply with that set of rigorous standards.

GM: This means validation is a significant issue for blood banks. What are your views on validation?

CHM:Because the collection, manufacturing or processing of blood products must meet strict quality controls (by their very nature) the attendant regulatory requirements then put small to medium sized businesses at a disadvantage. Blood banking organizations often lack the necessary automated governance of their procedures, development and operational management to be able to satisfy most regulatory bodies such as the FDA.
Frequently it falls to the system solution providers to bear the onus of certification or health authority clearance. This automatically rules out some of the larger corporations that deliver ERP and MRP systems, since they generally require that this responsibility remain with the health industry manufacturer or user.

Consequently there are no large vendors in the blood banking space, and fewer still with the comprehensive skill sets to deliver a true multidimensional solution, which might include for example: callable interfaces, modular components, documented data models, comprehensive diagnostic and analytical tools, etc.

These vendors can provide the necessary controls and documentation to satisfy the regulatory requirements, but they do so at a premium. That premium can, of course, be economic and present a high sticker price for their solutions, or alternatively that premium could come at the expense of timely version upgrades and releases that result in project delays and missed opportunities.

One complicating factor is that IT may not be viewed as an asset or a potential edge or advantage by the business it serves. Frequently it is seen as nothing more than the transport or chassis for the business processes and functionality. All too often it is seen as a necessary evil, or one that can be delegated to service vendors. Innovative IT skills and expertise may not be as valued as existing practices – "the-way-we-have-always-done-it" – unmindful of evolving best practice methodology. Common indicators of this mindset might be:

Application Silos, where each solution is a vertical one with a high integration cost for any shared data.

Relying on vendors for skill sets that are outside of their core competencies. This is a particular problem where the market place is full of small vendors and even more so when vendors are reluctant to provide interfaces into their solutions.

Compensating for the many risks associated with small vendors by over configuring resources.

Ignoring the need to establish or nurture infrastructure, partly because the owner of vertical solutions sets don’t want to pay for more than their share of the horizontal systems that support them, even to the point of making any infrastructure investment the first point of cost cutting.

Ignoring the need to establish a long term planning cycle as the costs for each vertical solution continues to spiral.

Ignoring the benefits and needs of standardization.

GM: You seem to have identified a range of issues related to validation and standardization.

CHM:If you look at large commercial organizations there are plenty of MRP systems to suit any pocket or size of company. Let’s take Oracle and SAP as an example, each of those companies can provide solutions for MRP/ERP from the small to the extremely large. What these companies don’t want to do is go through the certification of their product for a particular regulatory requirement. In blood banking, for example, there is the FDA 510K standard – and because it’s a small industry the large vendors don’t want to certify each enterprise against US FDA or other countries regulations.

The onus of that certification, meeting the required national standard falls on the enterprise itself. Most of those enterprises, certainly on their IT side and also on the business side, are fairly small or immature in the sense that they haven’t evolved to the level of understanding that would be more evident in an agile environment. So these enterprises are based more in the waterfall method with large projects and slow moving development. Change is generally difficult and consequently they don’t want to perform that certification themselves. So the only recourse is for blood banks to rely on vendors who will do it for them. And the vendors that are prepared to do that level of service tend to be small organizations.

GM: This puts a need on the blood banking industry to be dependent on their vendors?

CHM:Yes. But it is more because they don’t have the maturity or the frameworks in place to be able to deliver that certification themselves.

GM: One of the things we’ve observed in working with a lot of healthcare organizations around the world is that the testing and validation aspect seems to be a never ending task. There seems to be a cycle of stabilization and then an update from the software vendor and then a retesting and validation process. Is that an accurate observation? And if so, how does the American Red Cross deal with that? Deploy, test, validate, and performance test is done, then change comes from the software vendor and you need to go through the cycle again.

CHM:The industry needs a vendor community that will make planned changes on a regular basis, producing a product and service roadmap. Unplanned or creeping change puts a strain on the testing facilities. Because if the waterfall method is rigorously adhered to, there is a large amount of testing to be done with progression through unit and integration testing to system testing to end-to-end testing to scalability and stress testing, ending with performance testing. That cycle tends to be fairly solid and tends to take a long time to do.

Regression testing of all of those test suites is not necessarily adhered to, partly because of the length of time needed to run them but also because small changes are made albeit with discrete testing. Those small changes soon add up over time and the preference is to wait for the next major waterfall release before running a full regression test suite. I am not suggesting that regression tests are not done; it’s just that they are not always applied comprehensively. There are regression tests but they are applied mainly to system testing rather than to scalability or performance. This is mitigated in many cases by relying on the solution vendor to perform the testing.

You always welcome vendors testing as much as possible and having the vendor provide validation of that testing so that a level of confidence can be established. But more importantly you can use that data to compare against your own tests to ensure that you’ve got the results expected when you bought that particular solution. However there can be an over-reliance on the vendors if the solution and the testing are not fully documented and transparent.

GM: It sounds as if regression testing is an area that, properly planned for, can contribute significantly to speeding validation and implementation as well as minimizing resource drain.

CHM:Yes, and this applies not only to the blood banking industry but almost any industry. For the software market in general, it is best served with solutions that encapsulate, capture and commoditize testing as much as possible – then being able to reuse testing modules and testing scripts in such a way as to minimize the development time increasing the familiarization and practice of regression testing for each iteration.

One of the more traditional challenges of testing is that test scripts would need to be rewritten if there were any changes in procedure or functionality of data structures, a further drain on time and resources. So testing tools that encourage reusability and modularization so that tests can be run and rerun in different scenarios including regression testing is fundamentally a major win if it can be achieved.

GM: Are there other things you have observed in the blood bank space that you think are more generally applicable to healthcare more broadly?

CHM:Yes, while many of these observations are generalized, there are very few healthcare organizations that are at the leading edge across the board, companies that are so advanced both culturally and businesswise that they do everything in the most effective and efficient manner. There is always a part of these organizations that lags behind and may in some way resemble the slow moving blood banking sector.

Every industry has an area in which it is doing well and an area where it is still trying to catch up. From an infrastructure point of view, you can put in a fantastic network environment, you can have it highly secure, you can have full change control and administration done perfectly but perhaps you haven’t sorted out your unstructured file storage.

So essentially, you can look at any part of the blood bank business and say, “Yes, it has these challenges.” It has to respond to certain changes and some if not many of those challenges and changes apply to healthcare in general. Standardizing blood donor data is one of those challenges, both internally and externally, as the formats vary between applications and between organizations.

The same challenge exists with patient data in healthcare. Indeed both patient and donor data are both health records but the adoption of Electronic Health Record (EHR) standards are sporadic and slow, (see following quote):

A new policy brief from Health Affairs and the Robert Wood Johnson Foundation examines electronic health record standards, part of a series of efforts to get doctors and hospitals across the country to adopt EHRs by 2014.

The authors conclude that even with the regulations in place and the incentive payments pending, the technological platform to facilitate the sharing of electronic medical records is still being built. In 2010, HHS has so far awarded $548 million to states and territories to develop state health information exchanges that will link health care providers within their boundaries.

As you pointed out, additional and refined regulations along with more comprehensive privacy requirements from the US government are becoming more and more necessary as the volume and speed at which information is shared or transmitted increases, as do the incidents of hacking and compromise of privacy.

The level at which we expect to deliver product and services is increasing on a regular basis and while there is some small competition, it is there. ARC is not the only blood banker in the US; there are several others and as competition evolves and increases, any blood organization must be able to react quickly and effectively to the level of change.

So the lessons learned within ARC are applicable not just to the healthcare industry but any industry because the need for that sophisticated set of interlocking frameworks and the need to be agile to respond to change is fundamental – a call of nature, almost.
One of the things that we hadn’t discussed is risk management. And how that applies to this need to respond to change and the ability to use a strong framework to support change while minimizing risk and maximizing opportunity. That applies to the testing area as well. Many of the risks are inherent in any change and are best mitigated by comprehensive reactive testing.

Until recently the choices for how to test were limited. And what was worse, they were expensive.

You could do some sporadic testing. You could do limited discrete testing with low cost software or even open source software. But if you really wanted to bring a full testing suite to the project, you required a large framework, a large set of components – many of them un-integrated, many of which would follow separate schemes for automation. Such solutions were both expensive and required a fairly sophisticated set of resources, with high knowledge and skills sets capable of organizing and running the test framework required to ensure comprehensively tested end solutions.

A common way to compensate for the many risks factors associated with either technology or the amount of change on the horizon is by over-configuring resources, that is specifying an overabundance of hardware and software resources. The problem with this is you are cotton-wooling the solution, you never really know what the limitations of the system are but there is enough resource to hide any inefficiency or minor problem. The problem with this approach is obvious; it’s very expensive. However the cotton wool effect gives a false sense of security and risk avoidance and yet another reason to avoid comprehensive testing.

Change might occur in your environment or in your business direction; you might find that you have a runaway successful product and suddenly you are exercising your system in a completely different way. You need to put your system through its paces and understand its characteristics and how it behaves, and this can only be done by acquiring knowledge about the system, how the software operates, where and how the data is manipulated and what are its strengths and weaknesses. Much of this can be acquired from the vendor's training program and their documentation but the best knowledge comes from comprehensive testing.

GM: At SmarteSoft we’ve optimized our test automation software for specific blood bank management testing – in the case of ARC for eProgesa – but we’ve done it for other types of blood bank software as well. What is your perspective on the necessity or the attractiveness of having these highly tailored automation solutions for your specific blood bank uses vs. just taking a generic off the shelf product and trying to apply it?

CHM:Having a customized solution tailor-made to the application is really fundamental to understanding how that piece of software works. The larger vendors, like Oracle and SAP, tend to have very large test suites that they also deliver with their product. They also have a set of diagnostics and analytical tools that help the user understand how the application is operating. It also helps those two companies sell consulting services so they can go in and optimize or improve the level of operational efficiency of their solution within a particular customer installation.

For those users who don’t have a SAP, Oracle or generic enterprise solution, they usually have to rely on their vendor to explain how their system works and for the unfortunate few who are dependent on undocumented proprietary solutions (also known as black boxes) the customized, tailor-made automated solutions is highly desirable almost to the point of being essential.

The vendors who understand that they need to provide a modularized approach with the necessary tools – test scripts and tools, analytical tools and diagnostic tools – those are the people who get it. Those are the ones that will be largely successful. Cerner is an example, that’s one of the things they do.

So where you can get customized scripts that will test the performance, will test for bottlenecks, will test conflict and contention, will test for redundancy – those are really useful if they can be packaged and made available to the smaller user who is dependent on a black box supplier and doesn’t have the resources and the finance or budget to pay for the large testing tools and the resources that go with them.

GM: Of course, there are companies that can do that testing for you; that’s the third option. Companies like SmarteSoft and our partners can certainly go in and provide those capabilities.

CHM:They can and many do. There are several companies out there that do that and to date these companies haven’t been that inexpensive and are potentially luxury services. I think that is changing; open competition and flexible solutions are lowering the cost of entry and driving towards the most welcome word to IT budgets – commoditization.

If test is using standardized methodologies and interfaces it makes it easier to bring the price down. In general, we want to get more bang for the buck. If the cost of test can provide more comprehensive testing, better understanding of my product, better surety against risk, for a lower investment then that is a winning formula.

GM: We’ve observed that each blood bank management system has its own unique quirks as far as automatability goes for testing. We’ve found it has been necessary to have a customized solution for each vendor’s solution to make it as easy as possible to automate the testing of that solution.

CHM:And that certainly helps small organizations or blood banking units in particular because in reality they don’t want to have a large pool of testing resources that are rarely used. One of the challenges (especially with the waterfall method) is you hire a person, bring them on board. Year one, they do planning; year two they wait until development is complete. Third year, they do testing.

And that cycle is rinsed and repeated on the long term basis. That’s a lot of investment to tie up with a skill set that essentially is only being used a third or half of the time. So that approach to products around these solutions is very desirable.

GM: That’s a great set of perspectives you’ve given us. Thank you, Colin.

Colin can be reached at chopemurray@gmail.com for further discussion on this

Click here to download a pdf of this interviewtopic.

We're Interested in Your Perspective on the Healthcare IT Arena

We want to hear about your experience. What IT issues cause recurring pain? If you've solved an IT challenge, you can share it with the healthcare community here. Contact us. We'll help you spread the word.

We're Proud of These Success Stories

SmarteSoft has helped hospitals, blood banks, medical device manufacturers and other healthcare product and services providers achieve software success. Satisfied clients include Hill-Rom, Ohio Health, Scottish National Blood Transfusion Services, Clear Trial, American Red Cross, the US Department of Health and Human Services and New Zealand Blood Services.

At SmarteSoft, we're committed to understanding the challenges healthcare professionals and healthcare software vendors face. Data validation and application testing of the software you use is as important as improved speed, accuracy and cost of test.

For a free consultation call us at (512) 782-9409 or drop us a line at healthcareinfo@smartesoft.com. We'll review your challenges and help formulate an effective test strategy.