Professor Tony Avery,

Professor of Primary Health Care

Division of Primary Care

School of Community Health Sciences

University of Nottingham Medical School

Nottingham, UK

tony.avery@nottingham.ac.uk

I am going to be talking very briefly about a project that we have done over the past year for the National Patient Safety Agency (NPSA) on computer errors in primary care. First a little background, Nick was saying that general practice is quite well computerised. There are over 30,000 GP's and there are only 30 practices that do not have a computer and well over 95% of prescriptions are done by computer. There are several suppliers (three main ones) of computers for GP prescribing but all have certain minimal requirement to satisfy funding rules e.g. computerised drug interaction alerts. However the regulation so far have not had a particularly strong safety bias. This is why I think we have picked up quite a number of holes in the system.

Computers have considerable safety potential by providing accurate information about patients and drugs at the point of decision-making. They can provide decision-support, intelligent hazard alerts, precautions, contraindications and allergies. They can help with timely and appropriate monitoring and they can also help with trapping errors. Here is an example of a generic warning system and is one of several screen shots that I will show today to give you an idea of what the systems do. 

The next clearly shows that the patient has a history penicillin allergy, the next an interaction alert between Viagra and Isosorbide Mononitrate. 

This is a major alert on the system that we use in our practice. All you get is a hazard in yellow with three exclamation marks. The later does not really hit you and does not distinguish between different hazard levels and all you have to do is press the escape key to cancel leaving no audit trail. 

This example is a slightly cleverer contraindication alert, picking up that the patient has renal failure. You are trying to prescribe Tetracycline but this is contraindicated and should be used with caution in that situation. So some positives but there are some problems. While computer systems have considerable potential GP's and their staff may not know how to make best use of them including safety features (which they may turn off). Some of these systems allow the interactions alerts to be turned off. GP's may override these alerts as we have found - 25% of our sample overrode these alerts without proper checking. In addition some of the desirable features may be missing from some systems.

The first aim of our project was to identify some of the most important safety issues regarding GP computer systems. We also assessed the systems against these issues. We also wanted to determine GP's knowledge and training needs in relation to these systems. Finally we worked with stakeholders to produce specifications for suppliers and for training practice staff. One of the important aspects of this work is that it has come at a very good time because the NHS is reprocuring I.T. We have been able to recommend improvements to systems when they can be taken up quickly.

How did we identify the most important safety issues? We used two methods: stakeholder interviews usually with senior people from suppliers, database manufacturers, medical defence organisations, Royal Colleges and academics. We did semi-structured interviews that were then transcribed and analysed qualitatively. We also did a two-round Delphi technique with experts in the field from a fairly wide range of disciplines. They were asked to rate the importance of 55 statements that we had derived from the beginning of the project. After two rounds 32 were judged to be either important or very important by over 90% of the participants. Now this slide summarises what came out of that:

1. The importance of computerised alerts.

2. Avoiding spurious alerts. GPs' get really fed up with these and there is the risk of 'cry wolf syndrome' that may cause a really important alert to be overridden.

3. Should not be possible to override the critical alerts where it is possible to kill someone.

4. Audit trails.

5. Ensure the systems do not work unless the users actually record data. For instance a patient with renal failure would not have been flagged up unless the data had been entered that the patient had renal failure.

6. The way patient safety information is presented. The industry has grown up from very small beginnings. There is not a Microsoft equivalent and the industry has not had a strong safety focus. So not a lot of account has been given to the most effective way of presenting safety information to elicit the right response.

7. The need for practitioners to make the best use of computerised systems to ensure that intended actions e.g. patient referrals & monitoring are undertaken. The systems have all got this functionality but we are not very good at using it.

8. Training.

There were some of the key things coming out of the beginning. We then wanted to condense all of this into some test cases e.g. the systems had to recognise age in the context of contraindications. So we developed vignettes e.g. trying to prescribe aspirin to a child, which is contraindicated. We then tested these out in a laboratory setting with the four most commonly used systems. Two independent testers did this and agreed that they had the same results. We then tested these out in general practice to see if we had the same results and also fed the results back to the suppliers to check that they were happy that they had not missed anything. They also agreed with our results. We are going to publish in April.

Here are some of the key points with examples and then I am going to show some screen shots.

1. Lack of alerts in relation to contraindications. I had thought the problem was the database but found when talking to First Databank (Europe) that the company's system could do everything. The problem is that they do not have to do it and it takes programmer time to make the links between the database and the patients' recorded morbidities. So there are already systems that have the potential functionality that we need.

2. Spurious alerts.

3. Drug allergies warnings and I am going to give a very stark example of that.

4. Prescribing drugs with similar names.

5. Lack of warning for methotrexate

6. Hidden alerts.

7. The ease with which alerts can be overridden with a complete lack of an audit trail.

TA (to DB)
You look puzzled about that.

DB
We just have an audit trail for everything and we require people to say why they override.

TA
But what is interesting about the issue of audit trails is that we thought at the beginning that the GP's might be a little worried about introducing audit trails. However through interviews and the questionnaire we found that GP's were fine with the idea and the in some ways audit trails can protect you. If you prescribe a beta-blocker to an asthmatic patient then there may be on balance good reasons for doing so. If the patient suffers an adverse event then you do have a rationale for the audited override. Otherwise the decision looks out-of-the-blue.

So here are some examples. 

This is an example of prescribing aspirin to an eight year-old child. You will see 'no warnings present' because the system does not recognise age when examining contraindications.

This one looks a complicated screen and is a prescription of Microgynon- an oral contraceptive for women with a past history of deep vein thrombosis and the pill is contraindicated for this particular woman. This is one of the systems where this information is deep in the underlying database but the links are missing. You could click on one of these six tabs here and look at all the possible things that you ought to be looking at in terms of contraindications but these are not really going to alert you. What you really want is a warning to flash up and say 'hold on' this patient has a history of deep vein thrombosis.

Now here is a frightening example of an allergy. Some systems allow a doctor to code for an allergy and then find that it does come up as a warning. You can see the recorded history of an allergy on the screen. A GP prescribes penicillin, presses the 'Enter key' and the prescription is printed. If however the GP knows to record this under the reactions & allergies part of the database then an adverse reaction flashes up when the prescription is attempted. This is a crazy situation that the system's supplier recognises and GP's should not need to know the foibles of the system to get the right alert - that is the job of the supplier.

I am trying to prescribe penicillin in this next example. Now we know that the computerise systems have completely eliminated the problem of illegible prescriptions. I and any other GPs I know of in the UK do not have to type out the full name of the drug because the first three or four letters will produce a drop-down menu. This is an example of drugs with similar names. Here Penicillamine comes up instead of Penicillin and is a highly toxic drug used rarely and then only for patients with rheumatoid arthritis. What is worrying about this example is that the dosage - 250 mg - is similar to that for treating tonsillitis. After almost all talks I give to GP's one or two come up and say that they have made that mistake. Now sometimes this sort of mistake is picked up by pharmacists but there are cases of patient deaths from this error. So this is an example of design faults in computerised prescribing. This particular system does default to the most common alert once you have started to use it for a while.

The next is an example of dosage frequency warning. Methotrexate is an immunosuppressive drug that is increasingly being used in general practice under directions from secondary care for conditions like rheumatoid arthritis. The important point is that it is prescribed weekly and the major risk is to prescribe this highly toxic drug daily. It is possible to do this daily without an alert and what is particularly worrying is that the default pack-size is 28, which may give the impression that the drug is taken daily. One of the systems has partially got round this by warning that accompanies an intended prescription that this should normally be given weekly.

Here is what I have called a hidden alert; in this example Propalolol (beta-blocker) is given to an asthmatic patient. Can anyone see the alert? 

You have to click on these two very visible buttons at the bottom of the screen and then the alert comes up. 

Now there is an issue about over-alerting people but this is a contraindication from Committee for the Safety of Medicine. My view is that there are some things that we need to know and not be able to turn off or hide. If we are going to override such an alert then we need to record a very good reason for doing so.

Now to the final part of this whistle-stop tour; we wanted to find out what GP's knew about the safety features of computerised prescribing systems, what use they made of them and their training needs. We developed a questionnaire from interviews, sent this to GP's in two sites in England and got a very acceptable 67% response rate. Very briefly, the key finding is that there is very strong support for alerts of the sort that I have just described. Many GP's are not fully aware of the safety features on their systems. We discovered the very dangerous situation in which 33% of GP's thought their system had contraindication alerts when in fact we had found that this particular system lacks them. Only 20% of GP's had had any training on the safety features of their systems.

So to conclude my introduction to primary care before subsequent discussion. What is the relevance of this work to the Metanetwork Group? Many of the issues that we have identified will be addressed as part of the national I.T. programme. There are issues that do not benefit from large research projects; the issues just need to be tackled. Nevertheless there is considerable scope for thinking about further work that could be done particularly from a multidisciplinary perspective.

Questions

OH
To what extent is the research based on the theory of risk and risk management?

TA
This was a very practical approach. We have a multidisciplinary team that brainstormed seemingly important ideas and we gained evidence from the early interviews.

OH
If the GP's had a very poor understanding of risk and risk management then your concept of risk was based on that same poor understanding of risk.

TA
We identified a practical solution to an identified problem. Do you have any recommendations?

DB
I think this is fabulous. One comment: there should be a core set of things that people should not be allowed to turn off. Our experience is that sometimes doctors turn off all the drug allergy alerts. How can that be right? There is a lot of useful information to be gleaned from a study of the frequency with which individual alerts are overridden and then what happens to patients when this happens. We have some fascinating results and I will talk about that later but we found very high override rates for certain alerts but the patients did just fine. In that instance the doctors were probably right and we were probably over concerned about warnings. We emphasise making the really bad things look very distinct because we found the same override rates for minor and very serious alerts.

BF
We share your concerned about overrides and want to share a little story with you. We know a truck company owner who has 1,000 semitractor-trailors that go all over the West (USA). When one of the drivers misses a delivery an automatic message is sent to the owner who then phones the driver. This has reduced insurance premiums considerably. I heard about this story at the same time that I was helping a hospital introduce a bar-code system. There were many overrides detected and reported to the service manager but there was no follow up.

AJ
I have talked to people in the UK who have done either large hospital systems or many single ward pilots (we have lots of those going on). All agreed that there are many overrides and the messages are not very good. However we do not seem to getting very far in deciding a core set of key messages. If doing this in primary care is difficult then secondary care is much worse. So a warning about Aspirin and Warfarin is deemed unnecessary in a Coronary Care Unit but definitely needed in care-of-the-elderly ward. How do you define and get agreement about a core set of messages?

DB
We are working very hard as a network to try to define a core set. We plan to have a core set medication safety decision support in all our information systems in all our hospitals. We have a group that works on this task. There will be things at the margins that people legitimately disagree about but those are probably not the most important things. I am also trying to galvanise support for a national repository of shared knowledge in the US and there is no reason why this should not be international. We have spent years arriving at a set of drug warning interactions.

TA
We have done the same thing for the NPSA, a scoping exercise using a two-stage Delphi process with group of 12 experts. We found the process difficult because there is a very grey area between interactions where there is very high quality evidence and others where the evidence is very fragmentary.

DB
I agree and the evidence once you get to the granular level is often lousy. We have a paper that is yet to be published in which we looked at ISOL??? interactions in the hospital. About 70% of the people who get an ISOL (a commonly used antifungal) have one or more serious drug-drug interactions. We looked at a very large number of patients who were exposed to those drugs that interacted and none of them had an adverse drug event related to the interaction. This suggests to me that the underlying evidence was just a few case reports.

TA
Looking in detail at the drug-related hospital admissions. Drug interactions do not seem to be a major cause of admissions although from a quality point of view it does not seem to be a good idea to override warnings about interactions.

SG
Picking up a point that David was making: there is an apparently high override frequency and in a lot of cases that might be perfectly reasonable behaviour. In principle computers can track override frequencies and can first indicate when these alerts are inappropriate but can also indicate anomalous patterns of cancellations of alerts by a user. So an extreme level (but not an acceptable level) could be fed back to the user. Here the comparison would be with the peer group override frequency.

DB
That has not yet been done yet but could be very valuable. We have some instances where we almost always know we are right to deliver a warning. This is not mostly about medications but in other areas. We have looked at the distribution of overrides by individual and some are very different from the norm. In a few instances they were then interviewed by their managers with good effect. This is something that we should do on a larger scale.

TA
Are there computer systems in the States that allow pharmacists to pick up errors almost instantaneously?

DB
Yes but we do not check on individuals regularly but in principle it is possible to do this.

ENDS