Setting the Record Straight On a Holocaust-Denial Poll

`DOES it seem possible, or does it seem impossible to you that the Nazi extermination of the Jews never happened?''

This terribly confusing question about a tragic historical occurrence triggered a series of responses that tell a story important to all consumers of poll data. The question was posed in a November 1992 Roper Organization survey for the American Jewish Committee (AJC).

The initial error was innocent and accidental. A badly framed question slipped past the review process, was posed in the field, and yielded entirely misleading results. Yet, more than a year passed following the release of the survey data in April 1993 before a full correction was brought before the public. In the meantime, an incorrect account - suggesting that a large segment of the population entertains doubts as to whether the Holocaust actually occurred, with the appalling implication such denial carries - circulated widely. The whole story of this fiasco can now be told.

The double-negative construction in the question shown above - is it ``impossible'' the Holocaust ``never happened'' - confused many respondents. When the results were released - showing 22 percent taking the ``possible that it never happened'' position, and another 12 percent saying ``don't know'' - there was an uproar. Could the anti-Semitic claptrap of Holocaust denial actually be finding at least tentative approval in as much as a third of the United States population?

Survey researchers were distressed by the attention being given to answers to an obviously flawed question. The Gallup Organization, not party to the original error, set about to collect pertinent new data. Gallup posed the question to half the sample as Roper had, and posed a far less confusing version to the other half. Both halves were then asked an additional question designed to examine the level of doubt. This experiment showed the number of doubters to be small - one fifth or less of the proportion that the flawed question had suggested.

The Roper Organization also did corrective survey work. Finally, agreement was reached with the AJC for a full-scale redoing of the study. All of the questions in the original survey were repeated except for the offending item. It was replaced by a better version: ``Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?'' Now only a tiny fraction said it seemed possible that the Holocaust had never taken place!

Burns W. Roper was deeply disturbed by his firm's error when it was brought to his attention upon the April 1993 release of the data. He told his peers at this spring's meeting of the American Association for Public Opinion Research what he had conveyed to the AJC nearly a year earlier. ``We should never have approved the question, and we certainly never should have written it,'' Mr. Roper said. ``This is not the note on which I wanted to conclude my 48-year career in the opinion research field,'' he went on, saying that the confused wording had produced results that ``served to misinform the public, to scare the Jewish community needlessly, and to give aid and comfort to the neo-Nazis who have a commitment to Holocaust denial.''

If the original question was quickly seen by virtually the entire survey research community as badly flawed, and if the survey organization that made the mistake sought to provide the necessary correction, why was a full correction so long in coming? Unfortunately, the client resisted a vigorous public repudiation of the initial work and dragged its feet on new polling.

The AJC's director of research, David Singer, is quoted in the May 20, 1994, New York Times as saying he was ``chagrined'' when he learned of the problems with the first Roper effort, and that data from the new (March 1994) survey had to be carefully reviewed because ``Roper by its own admission provided us with flawed data and created great problems the first time around.'' If Mr. Singer were in fact embarrassed by deficiencies of the first Roper question, why did it take him more than a year to speak publicly about the problem?

The AJC has commissioned Tom Smith of the National Opinion Research Center to examine all the Roper data and other relevant survey findings. The report is scheduled for release in early July. Mr. Smith's efforts are welcome. At the same time, though, it must be stated that no one in the survey research community defends the original Roper question.

The lesson of the flawed Holocaust poll is an important one, reaching far beyond one badly framed question. We are reminded that survey results are often not viewed neutrally by individuals and organizations that sponsor or interpret the work. People care about the results. The ``interested'' character of findings creates potential for distortions.

The most deeply troubling feature of the Holocaust-poll misstep is that even when the survey organization involved fully recognized its error and actively sought a remedy, it was 14 months before a proper acknowledgment of the error was brought to the American people.