Down, Not Out

Don’t wait until your facility’s PACS is on the fritz to consider implementing adequate backup policies and procedures. Plan ahead to head off disaster.

By Herman Oosterwijk*, OTech Inc.








As picture archiving and communication systems have become more widespread, so has dependence on this technology. Administrators who are newer to PACS may not realize the breadth of their facility’s dependency until the system shuts down—a far cry from the best-case scenario of being forewarned and forearmed.


Causes of a shutdown can be multifold. Cabling issues such as a cut line typically are the top cause of network downtime. A server may also overload—for example, someone performing a wide-open query for “Mr. Smith’s images” may create a zillion matches, crashing the database. A technologist may become frustrated with a slow response time, hit the return key 20 times and send the application into a loop. Or perhaps the system is unexpectedly shut down by a software bug.


But a facility’s dependency on high-end technology needn’t be feared; it merely requires proper management. That management includes a reliable, well-tested and relatively mature system; an architecture that provides proper redundancy and backup; and last but hardly least, a proper set of policies and procedures.


Even a highly reliable system with ample redundancy remains vulnerable if the backup procedures aren’t followed. A recent reader’s survey from the OTech e-newsletter, a bimonthly PACS publication, found that only 30 percent of health care institutions have a complete set of PACS policies and procedures in place, 44 percent have a partial set and 26 percent have none. One can imagine the frantic activity of that 26 percent when an ER trauma patient undergoes a computed tomography scan and PACS can’t provide the radiologist with images.


Implementing PACS policies and procedures isn’t rocket science. If they’re in place, surprises shouldn’t occur, patient care can continue and staff won’t need to scramble.


Downtime concerns
Downtime procedures are a relatively small—albeit critical—portion of the overall set of PACS policies and procedures. Sections also deal with data integrity, i.e., how to fix “broken studies” that end up in the PACS without proper identification or the correct relationship to the imaging order. Security-related procedures deal with authentication, screen savers, audit trails and other patient privacy requirements. Included are procedures for importing and exporting images from CDs, proper document scanning, etc. A facility would typically have 40 or 50 different policies and procedures dealing with all the PACS aspects.


Downtime can be scheduled or unscheduled. There is also the disaster recovery procedure in the event of a major catastrophe such as fire, flood or other disasters. While some institutions are down for more than a day, downtime typically lasts from five minutes to eight hours, with an average of about an hour. This doesn’t seem long, but it’s unacceptable when patients are involved.


Typically, scheduled downtime is used for implementing patches or major software upgrades. Most institutions have learned the hard way that they should have a test system in place that can store images for about one-three days and is preconfigured at each modality. The technologist knows to select this test sever as a destination, and the radiologist knows to look for his images on this test source.


As soon as the system is back online, the system administrator will move the images from the test server to the production server. Obviously, the radiologist won’t have access to prior studies while the PACS is running on the test server—but if the scheduled downtime is set to occur during a period of low activity, it should be manageable.


The procedure for nonscheduled downtime is similar to the scheduled downtime procedure. A different strategy would be taken, depending on what’s down. If the PACS is down, one could print a film (assuming a printer is available), burn a CD and carry it to the radiologist, who can use his workstation offline. If the network is down, a network or cable switch could be applied that allows for a second network connection. Some institutions store copies of the images on a separate Web server: If the main PACS is down, physicians can log onto any personal computer and still see the images.

Basic downtime procedures
While the downtime procedure often depends on a facility’s architecture and provisions, the following basic procedures should be in place:

• Backup procedures. Who does the backup and how often? Depending on the risk level one will accept—and the imaging volume—a backup interval could range from hourly to daily for the database and daily to weekly for the images. Remember: Typically, the images also are available for about a week at the disk on the local modality.


• Downtime procedure for modality. This typically consists of a procedure to deal with the worklist being unavailable and one for the PACS store-destination to be unavailable. A solid procedure for lacking worklist capabilities is critical because manual data entry can impact PACS data integrity.


• Infrastructure downtime.These procedures deal with power, network, AC or electronic medical record/health information service outages.


• PACS downtime procedures. A good policy will cover all possible scenarios (PACS workstations down, hospital network down, HIS/RIS down, single modality down, Web server down, etc.).


• Scheduled downtime.This includes the mechanism of user notification.


• Unscheduled downtime. Know how to report it, the steps to take and the escalation policy over time.


• Disaster recovery procedure. This covers the restoration procedures for the database server and image server.

Proper PACS policies and procedures are essential but hardly impossible to create or implement. Rather than reinvent the wheel, use templates and samples from existing PACS policies and procedure manuals. That way, downtime is just another fact of life that can be managed in a professional manner with minimal impact on patient care.






-----------------------------------------

* Herman Oosterwijk is president of OTech Inc., a PACS training and publication company based in Aubrey, Texas. He publishes a PACS policy manual containing softcopy templates to generate the policies and procedures needed to optimally run a PACS system.









**