Why change scores are generally a bad idea for PROMs data: a worked example

Let’s imagine we have a Patient-Reported Outcome Measure (PROM) for surgical outcomes and we give it to patients pre- and post- operatively to see if health status improves after surgery. In common with many questionnaire measures, our PROM has reliability of 0.7.

Let’s also suppose that the surgery is equally effective for everyone, increasing health status by 3 points. In this example, that’s an effect size of d=0.36.

We calculate ‘health gain’ as the difference between pre- and post- operative score:

Health Gain = Postoperative Score – Preoperative Score

We know this should be around 3 points for each person, give or take a bit due to measurement error.

What happens when we plot health gain against preoperative score?

Scatterplot of negative correlation

The correlation between preoperative score and health gain is negative and significant: r = -0.46, p<0.05: can we conclude that surgery is more effective for patients with the lowest initial health status?

NO: because we know that everyone improved by the same amount, 3 points. The correlation should be zero.

Why this happens is quite easy to explain but will have to wait until I work out how to do equations in WordPress.

UPDATE: just in case you were wondering if this sort of thing happens in real life, this graph is taken from a BMJ article on PROMs (1) in which it is concluded that “better preoperative health tends to be associated with smaller, not larger, health gains “.

Look familiar?

1. J. Appleby, “Patient reported outcome measures: how are we feeling today?” BMJ 344, no. jan11 2 (January 11, 2012): d8191-d8191.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s