Is Bad Evidence Worse than No Evidence

As I mentioned in a recent post, I harbour no animus towards Rav Avi Shafran. Certainly he is an intelligent, educated man and, al pi the Mishnah in Avos, I must judge anything he does favourably but once again I've come across an article of his from Cross-Currents that I feel obliged to comment on, if conly because it really shows how desperate a job as public relations director can be.

The source for his column was a study reported in the American Journal of Psychiatry which concluded that "while observant Jewish women live in a culture defined by a high degree of adherence to specific laws of conduct, including rules designed to regulate sexual contact, sexual abuse of various types still exists among them. " Naturally the article in the Jewish Week News went a little further. The first line of the article pretty much summarizes what the newspaper's interpretation of the study is:

Despite the widespread impression in the Orthodox world that sexual abuse doesn’t happen within its precincts, or happens less than in the “outside world,” a report in the November issue of the journal of the American Psychiatric Association says that Orthodox Jewish women suffer as much of it as other American women do.

The article then goes on to detail the research findings, breaking the numbers down by age and denomination (Modern Orthodox vs. Ultra-Orthodox) and tries to explain how the different numbers might be explained.

There are also the usual disclaimers. For a study of this nature, it is understood there will be limitations and biases within the data. This is a self-reported survey sent to a defined population. People might not answer or they might answer incorrectly to cover up problems. There is no real way to know how much of this happens. So the article includes comments such as:

This was the first study of observant Jewish women’s sexuality, said Friedman. Though face-to-face interviews with randomly selected people on such topics are preferable, even getting Orthodox women to fill out anonymous questionnaires “is a hard thing to do,” she said. “You have to get to the sample, and it requires trust. For them to answer something that’s a study, people have to believe that it’s useful for them, that it’s necessary.”

The study found that the ultra-Orthodox women were more likely to report that their husbands had forced them to have sex — 5 percent compared to 1 percent of the Modern Orthodox women.This could be in part because fervently Orthodox women sometimes view their sexual role differently than Modern Orthodox women do, says Bronya Shaffer. Shaffer teaches marriage education classes to brides, and provides counseling to women about marital and family issues.

In the end, however, there's only so much information one can glean and conclusions one can make from this article. What's the bottom line?

The article hopes to illuminate the need for greater sensitivity to sexual abuse among those who might treat its victims, and also to the reality of its existence in Orthodox Jewish communities.“Religious life is not necessarily protective of the human condition,” said Friedman. “In theory, it’s clearly forbidden. But in practice it happens, and people suffer.”

Naturally, there are those who seem to have a vested interest in not allowing a conclusion like this to pass unchallenged. While it may seem obvious to some that Torah observant Jews are like anywone else with the same virtues and flaws as anyone else. Torah law may demand ethical behaviour but its adherents often fall short. This should not be shocking to anyone. As Chazal say, God wants our best effort and that is the real obligation that is placed upon on. However, for some a commitment to Torah observance and moral perfection seem to be synonymous Thus one cannot accept an accusation that any level of abuse of any kind occurs within the Orthodox community. Hence, Rav Shafran's column. To be fair, some of his points are valid. For example:

Randomized studies, like those that have focused on abuse in the general American population, yield reasonable estimations of the behaviors of their foci. Self-selected surveys of the same populations, however, can easily yield data that diverge substantially from the reality in those groups.

In the medical field, there are different levels of what is called evidence. The best kind is a double-blind randomized control trial, one in which the investigators and the subjects don't know if they're receiving actual treatment or placebo until the end. The least authoritative kinds are ancedotal, self-reports and expert evidence. They are generally only considered in decision making when any other better evidence is lacking. From the description of the study, it would seem that this survey is certainly low-quality evidence at best.

It's where Rav Shafran draws his conclusions that matters get a little tricky:

And so, by comparing the 25%-27% figure for American women claiming (in randomized surveys) to have suffered abuse at some point in their lives with the 26% figure yielded by the recent (self-selected) study of Orthodox women, and concluding that “Orthodox Jewish women suffer as much [abuse] as other American women,” the Jewish Week writer was comparing apples and tractors. If anything, the similar percentages arguably indicate a lower rate of abuse in the Orthodox community. After all, if 26% of a group likely to contain a disproportionate number of abuse victims report they were abused, one would expect a much lower percentage of a randomly selected group from the same population.

No, that's not true. If 26% of such a group was likely to be disproportionate, then the number might actually be higher or lower, or even the same. It is impossible to know but the assumption that the number would be lower isn't tenable. In addition, while the self-selection is a poor feature of the study, the fact that it was sent out to large parts of the Orthodox community does negate that somewhat.

It's its final paragraphs, however, which really set this piece apart in a poor way:

I cannot know that my expectation reflects reality; there are no meaningful statistical data to mine at present. But neither are there any to support the assumptions and speculations of writers like those cited above.

Understand what this means: he's just finished disputing a published study and then admits that in reality he has no more evidence for his own beliefs than the authors of the study! Objectively, that's wrong. The authors have made an effort at gathering data and statistically analyzing it within the limits they specified. Rav Shafran, after trying to dismiss the whole thing as anti-Orthodox media hype then reaches his conclusion:

One thing I do know, though, is that my expectation is based on the quintessential Jewish idea that the study and practice of Torah create more refined human beings. And the others’ assumption is based on their conviction – fueled, perhaps, by wishful thinking – that it does not.
The writers are entitled to their cynicism. But all Jews who respect Torah are entitled – I believe obligated – to expose it, along with offerings of unfounded, bias-born speculations as facts.

It is nice to believe that the study and practice of Torah creates more refine human beings. It is being informed to state that this is more likely in theory than in practice. Not for nothing did our Sages states a person can be a naval b'rshus haTorah, a boor with the permission of Torah. The challenge given to us by God is not only to observe His Torah and mitzvos but to improve ourselves as human beings so that we create an ideal society on Earth.

Too much of maintstream Orthodoxy today seems obsessed with the first part of the challenge while letting the second part fall to the side. As Rav Shafran concludes, "all Jews who respect Torah are entitled to expose it such accusation, along with offerings of unfounded, bias-born speculations as facts." If the Agudah were truly sincere about this, they would immediately sponsor a study that is far more representative of the Torah-observant Jewish community in order to finally create reliable data. If the truth is as Rav Shafran says it is, then the abuse can be seen in its proper context: shameful but rare. If it is as the authors of the study claim it is, then a true Torah-observant organization would spare no effort on correcting it immediately. Lack of interest in that does not speak well of the situation, much to our shame.