Remember the Johns Hopkins study published in Lancet last October? The researchers claimed that our invasion of Iraq had led to more than 650,000 "excess" deaths (one in 40 Iraqis) since March 2003. I thought the number was absurd and suspected fraud for political purposes. Quite a few bloggers looked at the study and found it wanting, as did the Prez. But defenders argued that we critics simply lacked the in-depth knowledge of statistics needed to understand the sophisticated methodology employed.
Well, according to The Times of London, a number of people who do have an in-depth knowledge of statistics don’t think much of the study, either (emphasis added):
… The controversy has deepened rather than evaporated. Several academics have tried to find out how the Lancet study was conducted; none regards their queries as having been addressed satisfactorily. Researchers contacted by The Times talk of unreturned e-mails or phone calls, or of being sent information that raises fresh doubts.
…“The authors ignore contrary evidence, cherry-pick and manipulate supporting evidence and evade inconvenient questions,” contends Professor Spagat, who believes the paper was poorly reviewed. “They published a sampling methodology that can overestimate deaths by a wide margin but respond to criticism by claiming that they did not actually follow the procedures that they stated.” The paper had “no scientific standing”. Did he rule out the possibility of fraud? “No.”
Some of the information about how the survey was conducted was new to me. Burnham, Roberts, et al — the Johns Hopkins "researchers" — never set foot in Iraq, did no research themselves, and apparently can’t actually vouch for how the survey was conducted:
They drafted in Professor Riyadh Lafta, at Al Mustansiriya University in Baghdad, as a co-author of the Lancet paper. Professor Lafta supervised eight doctors in 47 different towns across the country. In each town, says the paper, a main street was randomly selected, and a residential street crossing that main street was picked at random.
The doctors knocked on doors and asked residents how many people in that household had died. … Out of 1,849 households contacted, only 15 refused to participate.
A claimed participation rate of over 99%? Gee, there’s nothing suspicious about that!
One of the critics of the study is a former associate, Dr. Richard Garfield. Of course, his complaint seems to be that there was underreporting:
Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a “substantial reporting error” because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.
Professor Rosling was one of several academics who complained about stonewalling:
Professor Rosling says that, despite e-mails, “the authors haven’t provided us with the information needed to validate what they did”. He would like to see a live blog set up for the authors and their critics so that the matter can be clarified.
Another critic is Dr Madelyn Hsaio-Rei Hicks, of the Institute of Psychiatry in London, who specialises in surveying communities in conflict. In her letter to The Lancet, she pointed out that it was unfeasible for the Iraqi interviewing team to have covered 40 households in a day, as claimed. She wrote: “Assuming continuous interviewing for ten hours despite 55C heat, this allows 15 minutes per interview, including walking between households, obtaining informed consent and death certificates.”
I’d say that Hicks has totally destroyed the credibility of this study with one simple little calculation involving only the most basic math. No advanced statistics knowledge required.
Does she think the interviews were done at all? Dr Hicks responds: “I’m sure some interviews have been done but until they can prove it I don’t see how they could have done the study in the way they describe.”
Professor Burnham says the doctors worked in pairs and that interviews “took about 20 minutes”. The journal Nature, however, alleged last week that one of the Iraqi interviewers contradicts this. Dr Hicks says: : “I have started to suspect that they [the American researchers] don’t actually know what the interviewing team did. The fact that they can’t rattle off basic information suggests they either don’t know or they don’t care.”
Burnham told The Times he had "“full confidence in Professor Lafta and full faith in his interviewers.” Well, that settles it. No need to wonder about how those interviews were done. No need to validate the data or clarify the methodology.
Dr. Burnham has assured us that he has "full faith." That’s how they’re doing science at Johns Hopkins these days.