Why Survey Questions are Convoluted

There are at least three factors at work that make survey questions particularly convoluted.

First, questions need to be precise in meaning. There is, however, a constant tension between using more precise, technical terms that may be unfamiliar or everyday terms that may have imprecise meanings. Precision, particularly with imprecise everyday terms, often ends up requiring a welter of qualifying clauses that stretch grammar to its breaking point (except perhaps in German, which handles this well), and overrun the brain's ability to hold all the various conditions in mind at once (except, perhaps, for Germans). Negation is employed freely, both because it is sometimes necessary but also to break up what would be unidirectional banks of items. On the other hand, Converse and Presser's near-canonical text tells us to avoid double negatives, which may require its own linguistic acrobatics.

Second, there is extensive switching in subject and object because the first party (the interviewer) must ask the second party (the respondent) questions that refer to the second party sometimes in the first person ("Would you agree or disagree with the following statement: I see myself as more intelligent than average") and sometimes in the second person ("Do you think that people of the same sex should be allowed to marry?"), and of course about third parties, some of whom will be specified by their relationship to the second party (e.g., the relatives we had trouble with in the hand shaking question). Related to this is the tendency of surveys to ask about institutions (e.g., government) and constructs (e.g., shariah) that may raise questions of when to treat as a person vs. an object or treat as singular or plural; this is exacerbated in some instances because treating something as plural or singular, person or thing, may be ideologically charged.

Third, tense varies extensively, with surveys not uncommonly asking about past, present, and future, as well as freely employing conditional tenses ("If X happened, would you do Y or Z?") in close proximity to one another.

And then there is the synergistic effect of these different types of deviations from normal, everyday language happening together. As I think about it, really only legislation and technical manuals approach surveys in the need for precision in language and they, at least, are generally spared the complexities of first, second, and third person that accompany a survey.

Jewish Surveys of Yesterday: Yom Kippur Absence Method

I referred to the abysmal record of New York Jewish population studies in a previous post. The following excerpt from my dissertation on the Yom Kippur absence method for counting Jewish populations provides the missing context.

The Yom Kippur absence method is perhaps the most original of all proxy measures for estimating the size of the Jewish population. The method used is the essence of simplicity. One simply subtracts the average number of absentees of absentees from the public school system from the number absent on Yom Kippur to estimate the school age Jewish population, and multiplies this estimate by the inverse of the expected proportion of school age children in the Jewish population as a whole. The originator of the Yom Kippur absence method is unclear. Ritterband et al. (1988) noted it was first used in London in 1892 and Philadelphia in 1904, citing an unpublished mimeograph of much later date. Originality, unfortunately, does not result in accuracy. Robison’s (1943a) terse and well reasoned critique ably presents the major problems:

The "Yom Kippur method" makes several assumptions, some of which beg the question. Stated briefly, the reasoning behind this technique for calculating the number of Jews in a large city is that all Jewish children attend public schools and that all Jewish parents keep their children out of school on Yom Kippur. The number of absences from each public school on that day, corrected for the expected number of absences for each school, it is assumed, will equal the number of Jewish children at each school. The proportion that the school children bear to the total population in a census year is the co-efficient by which the number of Jewish children is multiplied to obtain the estimate of the total Jewish population. The validity of this calculation is based on the unproved assumption that the proportion of school children in the Jewish population is identical with that of the school population in the total population. (Robison 1943a:2)

Concerns about the accuracy of the Yom Kippur absence method were far from purely theoretical. Taylor (1943) directly compared the number of Jewish students in the Pittsburgh school system (which apparently recorded students’ religion) to survey data, and found the population was underestimated by 20 percent. This was a product of low birth rates. The estimating system expects a normal age distribution for the Jewish population when in fact the age pyramid is elder-biased (the end of mass migration led to a rapid increase in the age of the Jewish population). In fact, school age children constituted about 15 percent of the Jewish population, compared to 22 percent of the general population. Robison's (1943b) own study of Minneapolis Jewry showed that Jews had much notably lower birth rates than native-born whites, which would lead studies based on the Yom Kippur absence method to underestimate the size of the Jewish population.

In certain circumstances, the Yom Kippur absence method did, however, provide accurate estimates. The American Jewish Year Book estimate of New York City in 1953 was based on this method and estimated that Jews constituted 26.4 percent of the population, while a sample study undertaken by the Health Insurance plan of Greater New York estimated the proportion of Jews at 27 percent (Chenkin 1955). One reason for the more accurate New York estimate was undoubtedly that Jews constituted such a high proportion of New Yorkers that census estimates of the city’s age composition were deeply influenced by the Jewish community. This accurate estimate may, however, have been a fluke. The New York Federation of Jewish Charities used the Yom Kippur absence method with 1957 data and arrived at a much higher estimate of the Jewish population than did the Health Insurance Plan (Chenkin 1960, 1961).

Robison rightly directs attention to the problematic assumptions that all Jewish children were in public schools and that all Jewish children were kept out on Yom Kippur. To the extent that the Yom Kippur absence method was ever effective, its utility depended on the social structure of the first half of twentieth century. With the general lessening of prejudice against Jews after the Second World War, it became increasingly possible for Jews to be enrolled in private schools, especially as more such schools were established without religious affiliation (or became less overtly religious). The so-called Jewish love affair with public schools has also waned, with Orthodox Jews the first group to depart in significant numbers with the establishment of day schools following the Second World War, later picked up by the Conservative movement’s establishment of Solomon Schechter schools, and the more recent establishment of Reform and community day schools. The universality of Yom Kippur observance is also a problematic assumption. Some secularists took particular care to disregard halakhah, Jewish religious law. The famous Grand Yom Kippur Balls of the Lower East Side of New York City that celebrated the new year with "Kol Nidre, music, dancing, buffet; Marseillaise and other hymns" (Goodman 1971:330). While ideological secularism may have been confined to a relatively small proportion of the Jewish population, increasingly large portions of the Jewish community arrived at a more workaday version through a general decline in the salience of Jewish identity, further depressing estimates of Jewish population.

As the most significant set of estimates based on the number of Jewish school children, Linfield's (1928) defense of the method is worth considering. He concedes that the ratio of Jewish children to adults probably differs from that of the population at large:

The ratio of Jewish children to the total number of Jews is probably larger than the ratio of the whole number of children to the total population. These errors would have a tendency to render too large the estimate of the number of Jews arrived at on the basis of this method. On the other hand, it is undoubtedly incorrect to assume that fully 100% of the Jewish children in the large cities are recorded as absent on the day in question. A certain percentage are undoubtedly recorded as "present." From this point of view, the number of Jews arrived at would be too small. Is it not possible that the errors neutralized one another or nearly did so in the fall of 1927? (Linfield 1928:172)

Far from the Jewish community having a higher birth rate than the non-Jewish population, Taylor (1943) and Robison's (1943b) studies of a decade later found precisely the opposite, upsetting Linfield’s precariously balanced offsetting biases. That Linfield saw fit to use the same method 10 years later when its problems were well known is deeply problematic. He was harshly critiqued by the chairman of his own advisory committee (Ritterband et al. 1988), as well as his successors at the American Jewish Year Book, who wrote that:

The national estimates of Jewish population made by the Jewish Statistical Bureau in 1926 and 1936 in conjunction with the decennial Census of Religious Bodies of the United States Census Bureau might have been thought by some lay readers to be sufficient, but demographic experts have found it to be adequate only for purposes of rough calculation. (Seligman and Swados 1949:651)

Considering its glaring flaws, the Yom Kippur absence method was used for an unconscionably long time. Its apex appears to have been around 1927, when Linfield (1928) reports the American Jewish Year Book estimates received from local reporters for New York City, Newark, Boston, Detroit, Philadelphia, St. Louis, and Baltimore "checked on the basis of the number of children that abstained from attending school on the Day of Atonement in the fall of 1927" (p. 115). In Cleveland and Pittsburgh the number of Jewish students was estimated by social workers. The only other cities estimated to have 50,000 or more Jews not checked via school records were Baltimore, Chicago, and Los Angeles. The final counts for these cities were apparently leavened with "the number enumerated in the census as persons of foreign birth with Yiddish as their mother-tongue and their children" (p. 118). The manner in which these estimates were combined is not presented. Linfield's estimate is particularly problematic as he took a single day as being representative of the overall level of absenteeism rather than averaging truancy across a longer period. He did not detail the basis by which he extrapolated from the putative population of Jewish children to the total Jewish population other than to refer to it as "the coefficient to the given city," which appeared to assume that Jews had the same age structure as the general population which was deeply problematic and known to be so at the time. The Yom Kippur absence method was used for estimates of larger Jewish communities in conjunction with the 1936 Census of Religious Bodies, a practice that continued throughout the 1950s (Seligman 1950, 1951, 1953). Seligman and Chenkin (1954) used the method to calculate the size of the Jewish population of New York. The final use of the Yom Kippur absence method I am aware of was an estimate of the Jewish population of New York in 1968 (Chenkin 1969). The fact that a method known to be flawed and inaccurate could be used to estimate the size of the largest Jewish community in the world when scientific methods of sampling were well established and had been used for Jewish populations is astonishing and demonstrates a basic disinterest in obtaining accurate data.

References

Chenkin, Alvin. 1955. "Jewish Population of the United States, 1955." Pp. 171-176 in American Jewish Year Book, vol. 56, edited by M. Fine. New York: American Jewish Committee.

Chenkin, Alvin. 1960. "Jewish Population in the United States, 1959." Pp. 3-9 in American Jewish Year Book, vol. 61, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1961. "Jewish Population in the United States, 1960." Pp. 53-62 in American Jewish Year Book, vol. 62, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1963. "Jewish Population in the United States, 1962." Pp. 57-76 in American Jewish Year Book, vol. 64, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1969. "Jewish Population in the United States." Pp. 260-272 in American Jewish Year Book, vol. 70, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Goodman, Philip. 1971. The Yom Kippur Anthology. Philadelphia, PA: Jewish Publication Society.

Linfield, Harry S. 1928. "Jewish Population in the United States, 1927." Pp. 101-198 in American Jewish Year Book, vol. 30, edited by H. Schneiderman. Philadelphia, PA: Jewish Publication Society of America.

Ritterband, Paul, Barry A. Kosmin, and Jeffrey Scheckner. 1988. "Counting Jewish Populations: Methods and Problems." Pp. 204-211 in American Jewish Year Book, vol. 88, edited by D. Singer. New York: American Jewish Committee.

Robison, Sophia M. 1943a. "Methods of Gathering Data on the Jewish Population." Pp. 10-21 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Conference on Jewish Relations.

Robison, Sophia M. 1943b. "The Jewish Population of Minneapolis, 1936." Pp. 152-159 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Seligman, Ben B. 1950. "The American Jew: Some Demographic Features." Pp. 3-52 in American Jewish Year Book, vol. 51, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. 1951. "Jewish Population Estimates of United States' Communities." Pp. 17-21 in American Jewish Year Book, vol. 52, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. 1953. "Recent Demographic Changes in Some Jewish Communities." Pp. 3-24 in American Jewish Year Book, vol. 54, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. and Alvin Chenkin. 1954. "Jewish Population of the United States, 1953." Pp. 3-7 in American Jewish Year Book, vol. 55, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. and Harvey Swados. 1949. "Jewish Population Studies in the United States." Pp. 651-690 in American Jewish Year Book, vol. 50, edited by H. Schneiderman and M. Fine. Philadelphia, PA: Jewish Publication Society of America.

Taylor, Maurice. 1943. "A Sample Study of the Jewish Population of Pittsburgh, 1938." Pp. 81-108 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Jewish Survey of Yesterday: The 1981 New York Jewish community survey

As I wrote about the Rochester study, my mind kept turning back to the 1981 study of the New York Jewish community. I reproduce my description of the study from my dissertation below.

Leo Tolstoy’s (2004 [1877]) opening to Anna Karenina, "All happy families are alike; each unhappy family is unhappy in its own way," applies well to survey research. There are a limited numbers of ways to get a survey right, but countless opportunities for serious error. In many ways, a post mortem of the abject failures among surveys is more instructive than recapitulating the successes. The 1981 New York survey is a study in contrasts. Where the Los Angeles, Denver, and Chicago studies either oversampled low density areas or maintained a constant probability of selection, the Federation of Jewish Philanthropies of New York (1984) continued its long tradition of abysmal methodology by limiting random digit dialed calls to the 40 telephone exchanges (i.e. XXX-NNN-XXXX) with the highest proportion of distinctive Jewish names. At most, these directories could yield 400,000 telephone numbers. In practice, the number of households contained therein was probably much lower, given that many telephone numbers go unused or are ineligible as they belong to businesses or government agencies. These calls were supplemented by a mail survey to a sample of households with distinctive Jewish names. It is unknown whether the Federation went to the trouble of removing duplicate cases. The study’s authors estimated that 31 percent of Jews had distinctive Jewish names. The study itself estimates the Jewish population at 1.67 million. Assuming, for the sake of argument, that this figure is correct, at most the study could have covered only 31 percent of New York Jews (c. 520,000) plus some fraction of less than 400,000 households. In other words, at least half of New York’s Jews--and probably many more--were systematically excluded from the sample. This would have been problematic enough if it were a random half being excluded, but clearly this was not the case. Those excluded lived in less Jewish neighborhoods and did not have identifiably Jewish surnames. It is hardly a leap of faith to assume that all estimates from the study were irreparably biased.

Because the study was not based on a universal sample, the Jewish population size could not be directly estimated by multiplying the proportion of Jewish households found by the survey by the known population of the study area. Instead, the population estimate is extrapolated from the numbers of households with distinctive Jewish names on cantilevers of guesswork and surmise. A contemporary study of Hartford Jewry, comparing estimates from a purely random digit dialed sample with the distinctive Jewish names approach, concluded that: "It seems clear that [distinctive Jewish name] estimates cannot be brought to a reasonable level (i.e. even to 'ballpark figure') by any kind of adjustments" (Abrahamson 1982).

The reason for this travesty appears to be the New York study’s vast sample size of 4,505 Jewish households, which it boasted was "the largest ever single study of a Jewish community outside the state of Israel" (Federation of Jewish Philanthropies of New York 1984:71). (Large as it was, the sample size was still well below the number of interviews conducted for the National Jewish Population Study of 1970-71.) Survey research can be seen as a balancing act between sample quality, sample size, and cost. The Federation of Jewish Philanthropies traded quality and/or cost in order to maximize sample size. The benefit of a large sample is that it decreases the size of the confidence intervals around an estimated value. This is, of course, pointless when the estimate will be biased as a result of the sampling scheme. The fact that the largest Jewish community in the world, with the densest Jewish population in the United States, and presumably the greatest financial resources could not mount a valid study at a time when the cost of survey research was probably at an all-time low beggars all description.

References

Abrahamson, Mark. 1982. "A Study of the Greater Hartford Jewish Population, 1982." Greater Hartford Jewish Federation, Hartford, CT.

Federation of Jewish Philanthropies of New York. 1984. "The Jewish Population of Greater New York: A Profile." Federation of Jewish Philanthropies of New York, New York, NY.

Tolstoy, Leo. 2004 [1877]. Anna Karenina. Translated by R. Pevear and L. Volokhonsky. New York: Penguin.

Jewish Survey of Yesterday: Studies from the 1930s and 1940s

Having commented on community enumerations in the 1930s and 1940s in an earlier post, I reproduce the sections from my dissertation dealing with these studies.

Minneapolis, 1936

The earliest use of lists by a Jewish organization to study a Jewish population is unclear, but apparently predates 1936, as Robison’s (1943a) account of the 1936 census of the Jewish population of Minneapolis refers to following "the methods outlined by the Council for Jewish Federations and Welfare Funds for gather Jewish population data in small communities" (p. 152). For this study she developed a master list from subscribers to the Minneapolis Jewish Federation campaign, members of Jewish women’s organizations, families known to the Jewish Family Welfare Society, Jewish families on public welfare, members of the Downtown Talmud Torah, and members of two (unnamed) large synagogues. To account for households not included on the list, "the 1935 Minneapolis City Directory furnished many additional names of families presumably Jewish which did not appear on organization lists" (Robison 1943a:152). This use of distinctive Jewish names does not cause problems for the sample, except with respect to cost and efficiency, as the Jewish status of each household was directly determined by an interviewer. In addition to these methods, Robison described a house-to-house canvas in the more densely populated Jewish sections of the city and, in the more sparsely settled sections, "a canvass of those households which were designated as Jewish" (p. 152). Beyond these methods, every surveyed household was asked to list the names of other Jewish families who lived in the neighborhood, presumably a labor intensive process for those with many ties. The proportion of households that did not respond is not given.

The feasibility of such an approach depends on factors specific to the time and place. Canvassing all houses and apartments in high density areas requires a ready supply of inexpensive but intelligent interviewers. Two factors came together to make this possible. First, the Depression must have eased labor costs and increased the supply of suitable interviewers. Second, although Robison does not say one way or the other, it is plausible that such studies drew on Jewish women as volunteers. As Sylvia Barack Fishman (1993) has noted, organizations like Hadassah furnished careers for married Jewish women, with a managerial hierarchy ascended by virtue of training, skill and seniority, considerable responsibilities, and long hours, differing from the rest of the workforce mainly in the matter of pay (or the lack thereof). The 1938 studies of Jews in Buffalo, New York and Pittsburgh, Pennsylvania, drew on another kind of cheap labor, using college and graduate students (Engelman 1943; Taylor 1943). Even with an inexpensive source of suitable labor, enumerating a large Jewish community would stretch virtually any organization to the breaking point, all the more so when a large central office staff would be needed for tasks that are now easily computerized: tracking interviewers, making sure all addresses are visited (but not surveyed multiple times), collating completed interview schedules, and maintaining and updating the master list of Jewish households. In addition, the research plan rested on a specific set of characteristics of the Jewish community. The community must be concentrated in certain locales, those areas must be very densely populated by Jews, and few Jews should live beyond these areas. Were the bulk of the Jewish community not to be found in a few neighborhoods, a much larger section of the city would have to be canvassed in its entirety, with concomitant increases in cost and difficulty. It is also necessary that Jews be strongly connected to Jewish organizations and fellow Jews--having "bonding social capital" in Putnam's (2000) terminology--otherwise the sampling scheme would be inadequate. Should a large number of Jews have not belonged to Jewish organizations or had social ties to other Jews, they would have gone unenumerated, causing serious bias. Finally, the use of Jewish names to identify otherwise unknown households assumed that a high proportion of Jews had distinctive names shared by few non-Jews.

While such a procedure may seem unscientific to contemporary eyes, it is important to recall that few alternatives then existed. Neyman's key papers on sampling had just been published and more accessible material would not be available until after the Second World War. Evaluating the adequacy of a contemporary survey is difficult enough, but nigh unto impossible at such a distance. Bearing in mind these limitations, it remains a worthwhile exercise. It is likely that the study was, in fact, quite accurate. Relying on a relatively tight-knit community was appropriate at a time when Jews were still excluded from large portions of non-Jewish society, were relatively close to the immigrant experience, and were concentrated in ethnic neighborhoods (including areas of second or third settlement). The study was well designed to take advantage of these factors, with multiple ways for household to be included (living in a high density area, belonging to any of a number of Jewish organizations, having a distinctive Jewish name, or having Jewish friends). Soon, however, the social environment would change again.

San Francisco, 1938

Similar studies took place in Trenton and Passaic, New Jersey, Norwich and New London, Connecticut, and San Francisco, California. The 1938 San Francisco study (Moment 1943) based its master list on 70 Jewish organizations, supplemented by Jewish hotel guests (presumably determined via their names), and names obtained from other Jewish households by a presurvey mailing and via interview. The study initially intended to enumerate all known Jewish households, but was forced to undertake a systematic sample of one in three Jewish households after the study was about halfway complete, due to financial difficulties. This was accounted for by assigning a weight of three to the sampled cases, the first known use of weighting in a survey of an American Jewish community. Weights are used in surveys to compensate for unequal probabilities of selection and to project to the total target population. In this instance, had these cases not been "weighed" (essentially counting each case three times), they would have ended up being underrepresented in the final sample, together with any characteristics that were more common among this group.

To understand undercoverage, complete enumerations were undertaken in ten groups of four blocks in different census tracts. How these tracts were assigned is not clear, although it does not appear to be determined by Jewish population incidence. In any case, relying on such a small sample introduces considerable uncertainty into the estimates of the total Jewish population, although the concept of sampling error was not widely appreciated among social researchers at the time. Indeed, no list-based study of a Jewish population included any reference to sampling error. Nevertheless, this was a sophisticated and ambitious community study by the standards of the day.

Trenton, 1937

The 1937 study of Trenton, New Jersey forewent a house to house canvass of high density areas, but added Jewish names on the check-lists of ward leaders (the political affiliation of the ward leaders is not mentioned), as were married sons and daughters listed by their parents as living in Trenton (Robison 1943c). The fact that no attempt was made to interview households of unknown status in high density areas is problematic, as the seed of the sample is limited to those with Jewish names or who were members of Jewish organizations. Nevertheless, the breadth of the list sample was likely enough for adequate coverage.

Passaic, 1937

The Passaic study of 1937 cut yet more corners, limiting the initial seed to Jewish names and contributors to the Jewish Welfare Fund (Robison 1943b). Rather than asking for the addresses of all Jewish families, Jewish households were only asked about families on their block. Instead of complete enumeration of blocks known to have Jewish households, only "two or three addresses at which no Jewish family was listed" were canvassed (p. 22). On streets with no known Jewish households, "three houses on each side of the block were canvassed and inquiry made in the stores" (p. 22). This design problematically mixes features of a census (complete enumeration of known Jewish families) with those of a sample (probabilistic sampling of unknown households). Absent weighting for the different probabilities of selection, the results will be biased toward the most connected Jewish households. Had a master list developed from many organizations been used the potential for bias would be mitigated somewhat, but this was not the case. The procedure for selecting households may also have been problematic, if left to the interviewers’ discretion, as they may have opted for the more inviting looking residences.

New London and Norwich, 1938

The 1938 study of New London and Norwich (Wessel 1943) went further in the direction of using the list frame by completely omitting any households not found on organizational lists or known to other Jewish households. Its major innovation was its use of multiple modes of contact, first mailing questionnaires to all known Jewish households and then following up in person with the 75 percent that did not respond.

Summary

The preoccupations of these studies remain true to the social welfare model of their predecessors. Migrant status and naturalization remained a key concern, as did occupation. The move of second and later generation Jews from trade to the professions was noted. Fertility was also of interest, with studies of this era typically noting the low Jewish birthrate. Other than brief mentions of intermarriage (Robison 1943c) and participation in Jewish schools (Moment 1943; Robison 1943b), the focus on Jewish behavior that is the bread and butter of contemporary population studies is entirely absent, speaking to insecurity about the community’s socioeconomic standing coupled with the implausibility of a serious decline in Jewish distinctiveness.

References

Engelman, Uriah Z. 1943. "The Jewish Population of Buffalo, 1938." Pp. 37-50 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Fishman, Sylvia Barack. 1993. A Breath of Life: Feminism in the American Jewish Community. New York: The Free Press.

Moment, Samuel. 1943. "A Study of San Francisco Jewry, 1938." Pp. 160-182 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Putnam, Robert. 2000. Bowling Alone: The Collapse and Revival of American Community. New York: Simon & Schuster.

Robison, Sophia M. 1943a. "The Jewish Population of Minneapolis, 1936." Pp. 152-159 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Robison, Sophia M. 1943b. "The Jewish Population of Passaic, 1937." Pp. 22-36 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Robison, Sophia M. 1943c. "The Jewish Population of Trenton, 1937." Pp. 22-36 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Taylor, Maurice. 1943. "A Sample Study of the Jewish Population of Pittsburgh, 1938." Pp. 81-108 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Wessel, Bessie B. 1943. "A Comparative Study of the Jewish Communities of Norwich and New London, 1938." Pp. 51-80 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Jewish Survey of the Hour: Rochester, NY community study (Jewish Federation of Rochester and Rochester Research Group)

Apparently, today brought a bumper crop of surveys. No sooner had I finished commenting on JESNA's research than a note arrived from the North American Jewish Data Bank heralding a new community study of Rochester, New York. Reading it moved me further away from any attempt at dispassionate analysis. I will continue, however, vaguely organizing my comments around the AAPOR minimum disclosure standards. In case you are wondering whether my comments were fueled by sour grapes, this was the first I was aware of it.

Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.

Research was sponsored by the Jewish Federation of Rochester. Who conducted it isn't clear. The report is credited to Jocelyn Goldberg-Schaible of the Rochester Research Group, but data collection was also recorded as being conducted by volunteers.

The exact wording and presentation of questions and responses whose results are reported.

Mostly. The study does a very good job of this, although item order for things like motivations for charitable giving are reported in order of preference, not order displayed to respondent (if the items were rotated, no indication is provided).

A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population. If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.

Yes. The study's frame is reported, although the efforts by which initial cases were found are not covered in depth.

Even by the combative standards of the field, the authors are extraordinarily quick to praise their own work and damn that of others. The introduction proclaims that "Ours was not a boiler-plate survey designed generically and then tweaked a bit for our community - ours was a survey designed by and for our own community. There is a significant difference there, and part of that difference is its fundamental intent. And that fundamental intent, once again, was not about counting" (p. 2). I'm not sure what Jack Ukeles would say about that. Having spent nine months of my life working on the questionnaire for the 2005 Boston Jewish community study with Combined Jewish Philanthropies' community study committee, I can say that this didn't apply to my work.

In a similar vein, the authors write that the fact that the statistics reported are estimates "...is, by the way, always the case in community demographic surveys, irrespective of methodology and whether it’s telephone-based, online, or even face-to-face, even when charts and graphs imply more specificity than they can actually support, by providing numbers with several decimal places, and by performing convoluted analysis on those numbers" (p. 3; emphasis in original). Apparently lacking a well-developed sense of irony, the authors round their population estimates to the nearest five (while rounding those of earlier Rochester surveys to the nearest hundred, an implicit claim of a twentyfold increase in accuracy), mean years of residency to the nearest tenth of year (p. 37), and mean numbers of Jewish friends to the nearest hundredth (pp. 61-63, 65). (The basis by which population estimates were arrived at is not explained and remains, frankly, a mystery to me.)

A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

It's evident from the material that the study was a nonprobability design--all initial seeds appear to have been self-selected (e.g., hearing about the survey from advertisements, in communal newsletters, and so on) and thus the sample is a nonprobability design.

Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures. The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.

Sample size (n=1,913 plus 421 "usable" partials--"usable" is never defined). Sampling error estimates are reported as well: "With 2,234 respondents [and another 100 college students attending local colleges] our overall sample is conservatively associated with a precision interval (or margin of error) of +/-3% at the 95% confidence level, suggesting that our findings and projections should be within 3% of what we would have found if everyone in our Jewish community had participated" (p. 12).

These "margins of error" are misleading, however. Sampling error is calculable in surveys for which the mode of selection of units is understood. We know how random selection works--with a population of size N and a sample selected of size n, the probability of a unit being selected is n/N. When we mess around with the probability of selection in one way or another (e.g., stratification or clustering), as long as we can calculate the probability of selection, we can work our way back to confidence intervals and the use of inferential statistics. The same holds true when randomize experimental subjects into treatment and control groups. Even if we cannot directly randomize, as long the statistical process generating the observed data can be accurately modeled, we can use the apparatus of inferential statistics. That isn't the case here. Snowball sampling (as occurred here) does not supply it (Berg 1988; Eland-Goosensen et al. 1997; Friedman 1997; Kalton 1983; Kalton and Anderson 1986; Spreen 1992 [citations from Sagalnik and Heckathorn 2004]).

The closest analogy to the methods applied here that has significant levels of acceptance as capable of generating meaningful estimates of sampling error is respondent-driven sampling (RDS; Heckathorn 1997, 2002, 2007; Sagalnik and Heckathorn 2004; Volz and Heckathorn 2008). For RDS to work, though, all sections of the population must be connected with one another, long chains of referrals are needed in order to access parts of the social network with zero probability of selection in the initial wave, respondents must give their "degree" (the number of people with whom they are connected with the characteristic that is salient for sample selection), and numbered coupons must be used to record details of the selection and the social network it was based on (Sagalnik and Heckathorn 2004).

This is not the case here: the initial seeds were self-selected; we have no evidence from the report that long chains of referrals were achieved; respondents were not asked their degree (and it is doubtful whether most people could answer this accurately); and, we do not know which which cases referred other cases. Even RDS, far more credible than the Rochester study, has had doubt cast on its accuracy in recent years (Gile and Handcock 2010; Goel and Sagalnik 2010).

Nevertheless, the authors describe their sample (unjustifiably, in my view) as "extremely solid" (p. 3), "robust" (pp. 7, 9, 17), "highly robust" (p. 11), and "statistically robust beyond all expectations" (p. 12), as well as "inclusive" (pp. 7, 10), "more inclusive" (p. 11), "more inclusionary" (p. 69), "highly inclusive" (pp. 6, 8), and having "broad-based inclusiveness" (p. 11), not to mention being "impressive" (p. 17) and "very clever" (pp. 6, 8). There seem to be two reasons for this extraordinary self-assurance: sample size and inclusiveness.

The sample size is heavily hyped by the authors. "For the sake of comparison, national election polling predictions are based on smaller samples than ours. Communities like Philadelphia, whose Jewish population is roughly ten times larger than our own, recently completed their demographic study with a sample roughly half as large as ours. And ten years ago, when Rochester undertook its last demographic study, our sample was less than one-third as large as ours is today," (p. 12). Sample size is important to a survey inasmuch it reduces sampling error but, as I described above, this survey's attributes do not allow sampling error to be accurately estimated, a large sample is of little avail.

Beyond sampling error, surveys are subject to two serious forms of potential error associated with acquiring a sample: coverage error and nonresponse error. These appear to hold little concern for the authors due to the sample's aforementioned inclusiveness. The only evidence for the study's inclusiveness that I read in the report is that (a) the study had a field period of eight weeks (p. 7); (b) it received "a constant parade of media and PR spotlights" (p. 7); (c) it included Jews of different ages (p. 11); (d) it included interfaith households, GLBT Jews, old Jews, new residents, highly affiliated Jews, marginally affiliated Jews, unaffiliated Jews, Jews living in central areas, and Jews living in outlying areas (p. 7); (e) that volunteers were available to help (p. 11); and (f) that older Jews finished the survey without assistance within a week of it beginning (p. 11). This culminates in the claim (g) that "[i]t seems safe to say that few Jews in the greater Rochester area ended those eight weeks unaware of that the Count Me In survey was taking place" (p. 7).

Nonresponse always carries a concern regarding bias, and it seems entirely conceivable that the people most likely to respond would be those with the greatest interest in the subject-matter. Indeed, these concerns are heightened when the survey is accompanied by a PR campaign that has the effect of increasing the salience of the survey for the most connected Jews; a lower response rate can be a good thing if it lessens bias. With all due respect to the "parade of media and PR spotlights," it is also possible that there was coverage error: Jews who would have been eligible for the survey but were unaware of it and thus were systematically excluded. This is a risk inherent in the survey's approach, which relied on respondents to come to it rather than seeking them out, which is the approach used in virtually all Jewish population studies.

Indeed, nonresponse and/or coverage error apparently did occur: "Might we, as a result of this approach, have ended up with a higher proportion of affiliated Jews than we did in 2000? We probably did. By opening the survey's gates to all who chose to enter, we have over three times as many total participants, and those most involved Jewishly were most apt to take part" (emphasis in original); the estimated proportion of people who were never synagogue members dropped by almost half (p. 68), as did estimates of intermarried household (p. 69), a glaring indicator of bias.

All this is apparently excused by the survey's inclusionary (i.e., self-selecting) nature. "The fact remains," the authors write, "that within this year's sample we also have significantly more non-affiliated participants than we did in 2000" (p. 68; emphasis in original). Further, the reader is told, that "[i]n 2010, we have not turned our backs on the unaffiliated, and have in fact included them in far larger numbers than they were included in 2000 via RDD and DJN [Distinctive Jewish Names] telephone-based approach. It's just that alongside these non-affiliated respondents are a robust cohort of those more affiliated Jews who in the past would never have had the chance to be 'counted in', and this time around, via 2010's more inclusionary online methodology, were provided with that opportunity" (p. 69; emphasis in original). The decision-making is perhaps more clearly seen earlier: "Perhaps the best aspect of this sampling strategy was its broad-based inclusiveness. This year, everyone who wanted to participate had the opportunity to do so. In contrast with a telephone-based survey that works from a pre-determined list and/or Random Digit Dialing [RDD] and/or a set of Distinctively Jewish Names [sic]...no one in our community with ideas and opinions and experiences to share was left out of this survey. Ours was truly a COMMUNITY survey--and everyone who took part now gets to feel that their ideas and opinions and experiences have indeed been COUNTED IN" (p. 11; USE OF CAPITALS in original).

The fact that the survey's estimates of community priorities, age composition, religious behavior, and likely every other topic reported were biased in the direction of the views, behaviors, and characteristics of the more affiliated as a result of the survey's methodology apparently did not enter into consideration. This is utterly wrongheaded for a study with the goal of providing "actionable insight" for organizations' planning processes (p. 2). Bad data are a dangerous basis for decision-making. Inclusivity and feeling counted are indeed virtues, but never, ever, at the expense accuracy.

On first looking at this research, my first reaction was that the authors had done a wonderful job of going back to 1930s. This is not damning with faint praise, as readers of my dissertation (all 1.4 of you) will be aware. As probability samples with close to full coverage of the Jewish community are financially out of reach for most Jewish communities these days, we are in the sample places as we stood in the 1930s and 1940s, before robust methods were developed for sampling human populations. These studies focused on enumerating the Jewish communities rather than estimating (in a probabilistic, statistical sense) with the best of them having well thought out procedures to find as many Jews as possible in a community. Such an approach could reasonably justified. However, the sheer puffery and overweening arrogance--I can think of no other word to describe it--of the authors turned these thoughts into ashes in my mouth. (I'm no slacker when it comes to professional self-regard, and the same goes for other Jewish researchers. Our posturing, boasting, and deprecation of each other is usually confined to private conversations and email exchanges, conference presentations, and the odd journal article rather than community study reports which are, as they ought to be, client focused.)

Better reports (demonstrating my self-regard, I will shamelessly put up my own work on the 2005 Boston Jewish community study as an example) balance their self-promotion with appropriate humility about potential errors. While we said that our study provided "a rich portrait" of the community (p. 1) "breaks new ground" (p. 21) and lauded the research contractor's "high quality work" (p. 21), honestly the only examples of preening I could find in the summary report, we also pointed out the underestimates of young adults and Russian Jews (pp. 21-22).

In the case of Rochester, though, not the slightest hint of potential error (other than the mischaracterized confidence intervals) disrupts the fanfaronade that is this report. And it is a shame. Surveying smaller Jewish communities, never easy, has become extraordinarily difficult and the task of providing representative samples virtually impossible. Collecting nearly 2,000 completed surveys is no mean feat and likely reached much of the affiliated core of Rochester's Jewish population. There was considerable merit in the study's approach had its goal been enumeration, had the authors been forthright in the shortcomings of their data, and had responses by disaggregated by potential bias: affiliated, less affilitated, and unaffiliated Jews...there are, after all, reasons we undertake convoluted analyses.

References

Berg, S. 1988. "Snowball Sampling." Pp. 528-32 in Encyclopedia of Statistical Sciences, vol. 8, edited by S. Kotz and N.L. Johnson. New York: Wiley.

Eland-Goosensen, M., L. Van De Goor, E. Vollemans, V. Hendriks, and H. Garretsen. 1997. "Snowball Sampling Applied to Opiate Addicts Outside the Treatment System." Addiction Research 5(4):317-30.

Friedman, S.R. 1995. "Promising Social Network Research Results and Suggestions for a Research Agenda." Pp. 196-215 in Social Networks, Drug Abuse, and HIV Transmission. NIDA Monograph Number 151. Washington, DC: National Institute on Drug Abuse.

Gile, Krista J. and Mark S. Handcock. 2010. "Respondent-Driven Sampling: An Assessment of Current Methodology." Sociological Methodology 40(1): 285-327.

Goel, Sharad and Matthew J. Sagalnik. 2010. "Assessing Respondent-Driven Sampling." Proceedings of the National Academy of Sciences of the United States 107(15):6743-47.

Heckathorn, Douglas D. 1997. "Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations." Social Problems 44(2):174-99.

Heckathorn, Douglas D. 2002. "Respondent-Driven Sampling II: Deriving Valid Population Estimates from Chain-Referral Samples of Hidden Populations." Social Problems 49(1):11-34.

Heckathorn, Douglas D. 2007. "Extensions of Respondent Driven Sampling: Analyzing Continuous Variables and Controlling for Differential Recruitment." Sociological Methodology 37:151-208.

Kalton, Graham. 1983. Introduction to Survey Sampling. Beverley Hills, CA: Sage.

Kalton, Graham and D.W. Anderson. 1986. "Sampling Rare Populations." Journal of the Royal Statistical Society, Series A 149:65-82.

Sagalnik, Matthew J. and Douglas D. Heckathorn. 2004. "Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling." Sociological Methodology 34:193-239.

Spreen, M. 1992. "Rare Populations, Hidden Populations, and Link-Tracing Designs: What and Why?" Bulletin de Methodologie Sociologique 36:34-58.

Volz, Erik and Douglas D. Heckathorn. 2008. "Probability Based Estimation Theory for Respondent Driven Sampling." Journal of Official Statistics 24(1):79-97.

Jewish Survey of the Day: "Quick Bytes: On the Minds of Teens" (JESNA)

The first of my new series of reviews of Jewish surveys comes to us from JESNA and provides insight into "What is on the minds of Jewish teens?" The italicized sections come directly from the AAPOR Standards for Minimum Disclosure.

Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.

JESNA sponsored the research. Who conducted the research is not clear.

The exact wording and presentation of questions and responses whose results are reported.

No. The survey instrument is not available.

A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population. If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.

The sample appears to come from students attending schools that belong to the North American Association of Community Hebrew High Schools. No information is provided on the sampling frame used.

(This sample certainly doesn't cover all Jewish teens. Many don't attend any formal Jewish education during their high school years, while others--predominantly Orthodox--attend Jewish day schools. Accordingly, it is unlikely to represent "Jewish teens" as a group. Does it represent Jewish teens in supplementary Jewish education in high school? This hinges on how representative the NAACHHS is of "Hebrew High Schools." I can't comment on this, because the portion of their website that lists member schools doesn't work, at least on Firefox. While the introductory paragraph casts the survey's applicability in terms that are too broad for my taste, the fact that the sample is drawn from NAACHHS schools is mentioned multiple times, leaving the reader to reach their own conclusions about its representativeness. How many schools participated? We simply don't know and, on that basis, it's very difficult to know how much weight to give this research.)

A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

No indication of sample design is given.

(We don't have a clue how JESNA reached the students. Does NAACHHS have a comprehensive list of student emails? Was a sample selected or was every student approached? Were students mailed links by their schools? Was parental permission obtained? Again, we don't know these fundamental facts.)

Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures. The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.

Sample size is n=219. No indication of sampling error is provided. (As noted above, we don't even know if this was from a sample.)

Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.

No indication is given.

Method and dates of data collection.

Data was collected by web survey. Dates are a little vague, with "May 2010" given as the timeframe.

Assessment

This survey clearly (to me at least) fails major aspects of the AAPOR minimal disclosure standards. The failure to provide this basic level of information severely inhibits the utility of these data. To give a basic example, it makes a great deal of difference if the results achieved sample size of n=217 was drawn from a sample of 500 cases or from all 15,000 children attending NAACHHS schools (a number I made up on the spot because that information is not reported by NAACHHS).

Jewish Survey of the Day/Month/Week/Year

I am adding a new feature in my periodic blog: reviews of Jewish surveys. As those familiar with me are well aware, I cast a jaundiced eye over much of the work of what used to be my field (through September 2010 I was an associate research scientist at the Cohen Center for Modern Jewish Studies at Brandeis University; since October 2010 I have been working as a senior analyst/project manager at Abt SRBI Inc.). I will vaguely organized this around the Standards of Minimal Disclosure of the American Association for Public Opinion Research (AAPOR) Code of Professional Ethics and Practice [May 2010] which is, to whit, information that shall be included "in any report of research results or make them available immediately upon release of that report" (read the Standards here). This is the basic information required for a reader to make informed use of statistics in the public domain. It is not an "academic" standard (most AAPOR members work in industry, not academia), nor is it particularly difficult to meet. Any survey that exists for purposes other than pure entertainment should adhere at a minimum to these standards. While I initially planned to keep my focus on AAPOR compliance, I immediately fell off the wagon.