Jewish Surveys of Yesterday: Yom Kippur Absence Method

I referred to the abysmal record of New York Jewish population studies in a previous post. The following excerpt from my dissertation on the Yom Kippur absence method for counting Jewish populations provides the missing context.

The Yom Kippur absence method is perhaps the most original of all proxy measures for estimating the size of the Jewish population. The method used is the essence of simplicity. One simply subtracts the average number of absentees of absentees from the public school system from the number absent on Yom Kippur to estimate the school age Jewish population, and multiplies this estimate by the inverse of the expected proportion of school age children in the Jewish population as a whole. The originator of the Yom Kippur absence method is unclear. Ritterband et al. (1988) noted it was first used in London in 1892 and Philadelphia in 1904, citing an unpublished mimeograph of much later date. Originality, unfortunately, does not result in accuracy. Robison’s (1943a) terse and well reasoned critique ably presents the major problems:

The "Yom Kippur method" makes several assumptions, some of which beg the question. Stated briefly, the reasoning behind this technique for calculating the number of Jews in a large city is that all Jewish children attend public schools and that all Jewish parents keep their children out of school on Yom Kippur. The number of absences from each public school on that day, corrected for the expected number of absences for each school, it is assumed, will equal the number of Jewish children at each school. The proportion that the school children bear to the total population in a census year is the co-efficient by which the number of Jewish children is multiplied to obtain the estimate of the total Jewish population. The validity of this calculation is based on the unproved assumption that the proportion of school children in the Jewish population is identical with that of the school population in the total population. (Robison 1943a:2)

Concerns about the accuracy of the Yom Kippur absence method were far from purely theoretical. Taylor (1943) directly compared the number of Jewish students in the Pittsburgh school system (which apparently recorded students’ religion) to survey data, and found the population was underestimated by 20 percent. This was a product of low birth rates. The estimating system expects a normal age distribution for the Jewish population when in fact the age pyramid is elder-biased (the end of mass migration led to a rapid increase in the age of the Jewish population). In fact, school age children constituted about 15 percent of the Jewish population, compared to 22 percent of the general population. Robison's (1943b) own study of Minneapolis Jewry showed that Jews had much notably lower birth rates than native-born whites, which would lead studies based on the Yom Kippur absence method to underestimate the size of the Jewish population.

In certain circumstances, the Yom Kippur absence method did, however, provide accurate estimates. The American Jewish Year Book estimate of New York City in 1953 was based on this method and estimated that Jews constituted 26.4 percent of the population, while a sample study undertaken by the Health Insurance plan of Greater New York estimated the proportion of Jews at 27 percent (Chenkin 1955). One reason for the more accurate New York estimate was undoubtedly that Jews constituted such a high proportion of New Yorkers that census estimates of the city’s age composition were deeply influenced by the Jewish community. This accurate estimate may, however, have been a fluke. The New York Federation of Jewish Charities used the Yom Kippur absence method with 1957 data and arrived at a much higher estimate of the Jewish population than did the Health Insurance Plan (Chenkin 1960, 1961).

Robison rightly directs attention to the problematic assumptions that all Jewish children were in public schools and that all Jewish children were kept out on Yom Kippur. To the extent that the Yom Kippur absence method was ever effective, its utility depended on the social structure of the first half of twentieth century. With the general lessening of prejudice against Jews after the Second World War, it became increasingly possible for Jews to be enrolled in private schools, especially as more such schools were established without religious affiliation (or became less overtly religious). The so-called Jewish love affair with public schools has also waned, with Orthodox Jews the first group to depart in significant numbers with the establishment of day schools following the Second World War, later picked up by the Conservative movement’s establishment of Solomon Schechter schools, and the more recent establishment of Reform and community day schools. The universality of Yom Kippur observance is also a problematic assumption. Some secularists took particular care to disregard halakhah, Jewish religious law. The famous Grand Yom Kippur Balls of the Lower East Side of New York City that celebrated the new year with "Kol Nidre, music, dancing, buffet; Marseillaise and other hymns" (Goodman 1971:330). While ideological secularism may have been confined to a relatively small proportion of the Jewish population, increasingly large portions of the Jewish community arrived at a more workaday version through a general decline in the salience of Jewish identity, further depressing estimates of Jewish population.

As the most significant set of estimates based on the number of Jewish school children, Linfield's (1928) defense of the method is worth considering. He concedes that the ratio of Jewish children to adults probably differs from that of the population at large:

The ratio of Jewish children to the total number of Jews is probably larger than the ratio of the whole number of children to the total population. These errors would have a tendency to render too large the estimate of the number of Jews arrived at on the basis of this method. On the other hand, it is undoubtedly incorrect to assume that fully 100% of the Jewish children in the large cities are recorded as absent on the day in question. A certain percentage are undoubtedly recorded as "present." From this point of view, the number of Jews arrived at would be too small. Is it not possible that the errors neutralized one another or nearly did so in the fall of 1927? (Linfield 1928:172)

Far from the Jewish community having a higher birth rate than the non-Jewish population, Taylor (1943) and Robison's (1943b) studies of a decade later found precisely the opposite, upsetting Linfield’s precariously balanced offsetting biases. That Linfield saw fit to use the same method 10 years later when its problems were well known is deeply problematic. He was harshly critiqued by the chairman of his own advisory committee (Ritterband et al. 1988), as well as his successors at the American Jewish Year Book, who wrote that:

The national estimates of Jewish population made by the Jewish Statistical Bureau in 1926 and 1936 in conjunction with the decennial Census of Religious Bodies of the United States Census Bureau might have been thought by some lay readers to be sufficient, but demographic experts have found it to be adequate only for purposes of rough calculation. (Seligman and Swados 1949:651)

Considering its glaring flaws, the Yom Kippur absence method was used for an unconscionably long time. Its apex appears to have been around 1927, when Linfield (1928) reports the American Jewish Year Book estimates received from local reporters for New York City, Newark, Boston, Detroit, Philadelphia, St. Louis, and Baltimore "checked on the basis of the number of children that abstained from attending school on the Day of Atonement in the fall of 1927" (p. 115). In Cleveland and Pittsburgh the number of Jewish students was estimated by social workers. The only other cities estimated to have 50,000 or more Jews not checked via school records were Baltimore, Chicago, and Los Angeles. The final counts for these cities were apparently leavened with "the number enumerated in the census as persons of foreign birth with Yiddish as their mother-tongue and their children" (p. 118). The manner in which these estimates were combined is not presented. Linfield's estimate is particularly problematic as he took a single day as being representative of the overall level of absenteeism rather than averaging truancy across a longer period. He did not detail the basis by which he extrapolated from the putative population of Jewish children to the total Jewish population other than to refer to it as "the coefficient to the given city," which appeared to assume that Jews had the same age structure as the general population which was deeply problematic and known to be so at the time. The Yom Kippur absence method was used for estimates of larger Jewish communities in conjunction with the 1936 Census of Religious Bodies, a practice that continued throughout the 1950s (Seligman 1950, 1951, 1953). Seligman and Chenkin (1954) used the method to calculate the size of the Jewish population of New York. The final use of the Yom Kippur absence method I am aware of was an estimate of the Jewish population of New York in 1968 (Chenkin 1969). The fact that a method known to be flawed and inaccurate could be used to estimate the size of the largest Jewish community in the world when scientific methods of sampling were well established and had been used for Jewish populations is astonishing and demonstrates a basic disinterest in obtaining accurate data.

References

Chenkin, Alvin. 1955. "Jewish Population of the United States, 1955." Pp. 171-176 in American Jewish Year Book, vol. 56, edited by M. Fine. New York: American Jewish Committee.

Chenkin, Alvin. 1960. "Jewish Population in the United States, 1959." Pp. 3-9 in American Jewish Year Book, vol. 61, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1961. "Jewish Population in the United States, 1960." Pp. 53-62 in American Jewish Year Book, vol. 62, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1963. "Jewish Population in the United States, 1962." Pp. 57-76 in American Jewish Year Book, vol. 64, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Chenkin, Alvin. 1969. "Jewish Population in the United States." Pp. 260-272 in American Jewish Year Book, vol. 70, edited by M. Fine and M. Himmelfarb. New York: American Jewish Committee.

Goodman, Philip. 1971. The Yom Kippur Anthology. Philadelphia, PA: Jewish Publication Society.

Linfield, Harry S. 1928. "Jewish Population in the United States, 1927." Pp. 101-198 in American Jewish Year Book, vol. 30, edited by H. Schneiderman. Philadelphia, PA: Jewish Publication Society of America.

Ritterband, Paul, Barry A. Kosmin, and Jeffrey Scheckner. 1988. "Counting Jewish Populations: Methods and Problems." Pp. 204-211 in American Jewish Year Book, vol. 88, edited by D. Singer. New York: American Jewish Committee.

Robison, Sophia M. 1943a. "Methods of Gathering Data on the Jewish Population." Pp. 10-21 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Conference on Jewish Relations.

Robison, Sophia M. 1943b. "The Jewish Population of Minneapolis, 1936." Pp. 152-159 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Seligman, Ben B. 1950. "The American Jew: Some Demographic Features." Pp. 3-52 in American Jewish Year Book, vol. 51, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. 1951. "Jewish Population Estimates of United States' Communities." Pp. 17-21 in American Jewish Year Book, vol. 52, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. 1953. "Recent Demographic Changes in Some Jewish Communities." Pp. 3-24 in American Jewish Year Book, vol. 54, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. and Alvin Chenkin. 1954. "Jewish Population of the United States, 1953." Pp. 3-7 in American Jewish Year Book, vol. 55, edited by M. Fine. New York: American Jewish Committee.

Seligman, Ben B. and Harvey Swados. 1949. "Jewish Population Studies in the United States." Pp. 651-690 in American Jewish Year Book, vol. 50, edited by H. Schneiderman and M. Fine. Philadelphia, PA: Jewish Publication Society of America.

Taylor, Maurice. 1943. "A Sample Study of the Jewish Population of Pittsburgh, 1938." Pp. 81-108 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Jewish Survey of Yesterday: The 1981 New York Jewish community survey

As I wrote about the Rochester study, my mind kept turning back to the 1981 study of the New York Jewish community. I reproduce my description of the study from my dissertation below.

Leo Tolstoy’s (2004 [1877]) opening to Anna Karenina, "All happy families are alike; each unhappy family is unhappy in its own way," applies well to survey research. There are a limited numbers of ways to get a survey right, but countless opportunities for serious error. In many ways, a post mortem of the abject failures among surveys is more instructive than recapitulating the successes. The 1981 New York survey is a study in contrasts. Where the Los Angeles, Denver, and Chicago studies either oversampled low density areas or maintained a constant probability of selection, the Federation of Jewish Philanthropies of New York (1984) continued its long tradition of abysmal methodology by limiting random digit dialed calls to the 40 telephone exchanges (i.e. XXX-NNN-XXXX) with the highest proportion of distinctive Jewish names. At most, these directories could yield 400,000 telephone numbers. In practice, the number of households contained therein was probably much lower, given that many telephone numbers go unused or are ineligible as they belong to businesses or government agencies. These calls were supplemented by a mail survey to a sample of households with distinctive Jewish names. It is unknown whether the Federation went to the trouble of removing duplicate cases. The study’s authors estimated that 31 percent of Jews had distinctive Jewish names. The study itself estimates the Jewish population at 1.67 million. Assuming, for the sake of argument, that this figure is correct, at most the study could have covered only 31 percent of New York Jews (c. 520,000) plus some fraction of less than 400,000 households. In other words, at least half of New York’s Jews--and probably many more--were systematically excluded from the sample. This would have been problematic enough if it were a random half being excluded, but clearly this was not the case. Those excluded lived in less Jewish neighborhoods and did not have identifiably Jewish surnames. It is hardly a leap of faith to assume that all estimates from the study were irreparably biased.

Because the study was not based on a universal sample, the Jewish population size could not be directly estimated by multiplying the proportion of Jewish households found by the survey by the known population of the study area. Instead, the population estimate is extrapolated from the numbers of households with distinctive Jewish names on cantilevers of guesswork and surmise. A contemporary study of Hartford Jewry, comparing estimates from a purely random digit dialed sample with the distinctive Jewish names approach, concluded that: "It seems clear that [distinctive Jewish name] estimates cannot be brought to a reasonable level (i.e. even to 'ballpark figure') by any kind of adjustments" (Abrahamson 1982).

The reason for this travesty appears to be the New York study’s vast sample size of 4,505 Jewish households, which it boasted was "the largest ever single study of a Jewish community outside the state of Israel" (Federation of Jewish Philanthropies of New York 1984:71). (Large as it was, the sample size was still well below the number of interviews conducted for the National Jewish Population Study of 1970-71.) Survey research can be seen as a balancing act between sample quality, sample size, and cost. The Federation of Jewish Philanthropies traded quality and/or cost in order to maximize sample size. The benefit of a large sample is that it decreases the size of the confidence intervals around an estimated value. This is, of course, pointless when the estimate will be biased as a result of the sampling scheme. The fact that the largest Jewish community in the world, with the densest Jewish population in the United States, and presumably the greatest financial resources could not mount a valid study at a time when the cost of survey research was probably at an all-time low beggars all description.

References

Abrahamson, Mark. 1982. "A Study of the Greater Hartford Jewish Population, 1982." Greater Hartford Jewish Federation, Hartford, CT.

Federation of Jewish Philanthropies of New York. 1984. "The Jewish Population of Greater New York: A Profile." Federation of Jewish Philanthropies of New York, New York, NY.

Tolstoy, Leo. 2004 [1877]. Anna Karenina. Translated by R. Pevear and L. Volokhonsky. New York: Penguin.

Jewish Survey of Yesterday: Studies from the 1930s and 1940s

Having commented on community enumerations in the 1930s and 1940s in an earlier post, I reproduce the sections from my dissertation dealing with these studies.

Minneapolis, 1936

The earliest use of lists by a Jewish organization to study a Jewish population is unclear, but apparently predates 1936, as Robison’s (1943a) account of the 1936 census of the Jewish population of Minneapolis refers to following "the methods outlined by the Council for Jewish Federations and Welfare Funds for gather Jewish population data in small communities" (p. 152). For this study she developed a master list from subscribers to the Minneapolis Jewish Federation campaign, members of Jewish women’s organizations, families known to the Jewish Family Welfare Society, Jewish families on public welfare, members of the Downtown Talmud Torah, and members of two (unnamed) large synagogues. To account for households not included on the list, "the 1935 Minneapolis City Directory furnished many additional names of families presumably Jewish which did not appear on organization lists" (Robison 1943a:152). This use of distinctive Jewish names does not cause problems for the sample, except with respect to cost and efficiency, as the Jewish status of each household was directly determined by an interviewer. In addition to these methods, Robison described a house-to-house canvas in the more densely populated Jewish sections of the city and, in the more sparsely settled sections, "a canvass of those households which were designated as Jewish" (p. 152). Beyond these methods, every surveyed household was asked to list the names of other Jewish families who lived in the neighborhood, presumably a labor intensive process for those with many ties. The proportion of households that did not respond is not given.

The feasibility of such an approach depends on factors specific to the time and place. Canvassing all houses and apartments in high density areas requires a ready supply of inexpensive but intelligent interviewers. Two factors came together to make this possible. First, the Depression must have eased labor costs and increased the supply of suitable interviewers. Second, although Robison does not say one way or the other, it is plausible that such studies drew on Jewish women as volunteers. As Sylvia Barack Fishman (1993) has noted, organizations like Hadassah furnished careers for married Jewish women, with a managerial hierarchy ascended by virtue of training, skill and seniority, considerable responsibilities, and long hours, differing from the rest of the workforce mainly in the matter of pay (or the lack thereof). The 1938 studies of Jews in Buffalo, New York and Pittsburgh, Pennsylvania, drew on another kind of cheap labor, using college and graduate students (Engelman 1943; Taylor 1943). Even with an inexpensive source of suitable labor, enumerating a large Jewish community would stretch virtually any organization to the breaking point, all the more so when a large central office staff would be needed for tasks that are now easily computerized: tracking interviewers, making sure all addresses are visited (but not surveyed multiple times), collating completed interview schedules, and maintaining and updating the master list of Jewish households. In addition, the research plan rested on a specific set of characteristics of the Jewish community. The community must be concentrated in certain locales, those areas must be very densely populated by Jews, and few Jews should live beyond these areas. Were the bulk of the Jewish community not to be found in a few neighborhoods, a much larger section of the city would have to be canvassed in its entirety, with concomitant increases in cost and difficulty. It is also necessary that Jews be strongly connected to Jewish organizations and fellow Jews--having "bonding social capital" in Putnam's (2000) terminology--otherwise the sampling scheme would be inadequate. Should a large number of Jews have not belonged to Jewish organizations or had social ties to other Jews, they would have gone unenumerated, causing serious bias. Finally, the use of Jewish names to identify otherwise unknown households assumed that a high proportion of Jews had distinctive names shared by few non-Jews.

While such a procedure may seem unscientific to contemporary eyes, it is important to recall that few alternatives then existed. Neyman's key papers on sampling had just been published and more accessible material would not be available until after the Second World War. Evaluating the adequacy of a contemporary survey is difficult enough, but nigh unto impossible at such a distance. Bearing in mind these limitations, it remains a worthwhile exercise. It is likely that the study was, in fact, quite accurate. Relying on a relatively tight-knit community was appropriate at a time when Jews were still excluded from large portions of non-Jewish society, were relatively close to the immigrant experience, and were concentrated in ethnic neighborhoods (including areas of second or third settlement). The study was well designed to take advantage of these factors, with multiple ways for household to be included (living in a high density area, belonging to any of a number of Jewish organizations, having a distinctive Jewish name, or having Jewish friends). Soon, however, the social environment would change again.

San Francisco, 1938

Similar studies took place in Trenton and Passaic, New Jersey, Norwich and New London, Connecticut, and San Francisco, California. The 1938 San Francisco study (Moment 1943) based its master list on 70 Jewish organizations, supplemented by Jewish hotel guests (presumably determined via their names), and names obtained from other Jewish households by a presurvey mailing and via interview. The study initially intended to enumerate all known Jewish households, but was forced to undertake a systematic sample of one in three Jewish households after the study was about halfway complete, due to financial difficulties. This was accounted for by assigning a weight of three to the sampled cases, the first known use of weighting in a survey of an American Jewish community. Weights are used in surveys to compensate for unequal probabilities of selection and to project to the total target population. In this instance, had these cases not been "weighed" (essentially counting each case three times), they would have ended up being underrepresented in the final sample, together with any characteristics that were more common among this group.

To understand undercoverage, complete enumerations were undertaken in ten groups of four blocks in different census tracts. How these tracts were assigned is not clear, although it does not appear to be determined by Jewish population incidence. In any case, relying on such a small sample introduces considerable uncertainty into the estimates of the total Jewish population, although the concept of sampling error was not widely appreciated among social researchers at the time. Indeed, no list-based study of a Jewish population included any reference to sampling error. Nevertheless, this was a sophisticated and ambitious community study by the standards of the day.

Trenton, 1937

The 1937 study of Trenton, New Jersey forewent a house to house canvass of high density areas, but added Jewish names on the check-lists of ward leaders (the political affiliation of the ward leaders is not mentioned), as were married sons and daughters listed by their parents as living in Trenton (Robison 1943c). The fact that no attempt was made to interview households of unknown status in high density areas is problematic, as the seed of the sample is limited to those with Jewish names or who were members of Jewish organizations. Nevertheless, the breadth of the list sample was likely enough for adequate coverage.

Passaic, 1937

The Passaic study of 1937 cut yet more corners, limiting the initial seed to Jewish names and contributors to the Jewish Welfare Fund (Robison 1943b). Rather than asking for the addresses of all Jewish families, Jewish households were only asked about families on their block. Instead of complete enumeration of blocks known to have Jewish households, only "two or three addresses at which no Jewish family was listed" were canvassed (p. 22). On streets with no known Jewish households, "three houses on each side of the block were canvassed and inquiry made in the stores" (p. 22). This design problematically mixes features of a census (complete enumeration of known Jewish families) with those of a sample (probabilistic sampling of unknown households). Absent weighting for the different probabilities of selection, the results will be biased toward the most connected Jewish households. Had a master list developed from many organizations been used the potential for bias would be mitigated somewhat, but this was not the case. The procedure for selecting households may also have been problematic, if left to the interviewers’ discretion, as they may have opted for the more inviting looking residences.

New London and Norwich, 1938

The 1938 study of New London and Norwich (Wessel 1943) went further in the direction of using the list frame by completely omitting any households not found on organizational lists or known to other Jewish households. Its major innovation was its use of multiple modes of contact, first mailing questionnaires to all known Jewish households and then following up in person with the 75 percent that did not respond.

Summary

The preoccupations of these studies remain true to the social welfare model of their predecessors. Migrant status and naturalization remained a key concern, as did occupation. The move of second and later generation Jews from trade to the professions was noted. Fertility was also of interest, with studies of this era typically noting the low Jewish birthrate. Other than brief mentions of intermarriage (Robison 1943c) and participation in Jewish schools (Moment 1943; Robison 1943b), the focus on Jewish behavior that is the bread and butter of contemporary population studies is entirely absent, speaking to insecurity about the community’s socioeconomic standing coupled with the implausibility of a serious decline in Jewish distinctiveness.

References

Engelman, Uriah Z. 1943. "The Jewish Population of Buffalo, 1938." Pp. 37-50 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Fishman, Sylvia Barack. 1993. A Breath of Life: Feminism in the American Jewish Community. New York: The Free Press.

Moment, Samuel. 1943. "A Study of San Francisco Jewry, 1938." Pp. 160-182 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Putnam, Robert. 2000. Bowling Alone: The Collapse and Revival of American Community. New York: Simon & Schuster.

Robison, Sophia M. 1943a. "The Jewish Population of Minneapolis, 1936." Pp. 152-159 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Robison, Sophia M. 1943b. "The Jewish Population of Passaic, 1937." Pp. 22-36 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Robison, Sophia M. 1943c. "The Jewish Population of Trenton, 1937." Pp. 22-36 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Taylor, Maurice. 1943. "A Sample Study of the Jewish Population of Pittsburgh, 1938." Pp. 81-108 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Wessel, Bessie B. 1943. "A Comparative Study of the Jewish Communities of Norwich and New London, 1938." Pp. 51-80 in Jewish Social Studies, vol. 3, edited by S. M. Robison. New York: Council on Jewish Relations.

Jewish Survey of the Hour: Rochester, NY community study (Jewish Federation of Rochester and Rochester Research Group)

Apparently, today brought a bumper crop of surveys. No sooner had I finished commenting on JESNA's research than a note arrived from the North American Jewish Data Bank heralding a new community study of Rochester, New York. Reading it moved me further away from any attempt at dispassionate analysis. I will continue, however, vaguely organizing my comments around the AAPOR minimum disclosure standards. In case you are wondering whether my comments were fueled by sour grapes, this was the first I was aware of it.

Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.

Research was sponsored by the Jewish Federation of Rochester. Who conducted it isn't clear. The report is credited to Jocelyn Goldberg-Schaible of the Rochester Research Group, but data collection was also recorded as being conducted by volunteers.

The exact wording and presentation of questions and responses whose results are reported.

Mostly. The study does a very good job of this, although item order for things like motivations for charitable giving are reported in order of preference, not order displayed to respondent (if the items were rotated, no indication is provided).

A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population. If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.

Yes. The study's frame is reported, although the efforts by which initial cases were found are not covered in depth.

Even by the combative standards of the field, the authors are extraordinarily quick to praise their own work and damn that of others. The introduction proclaims that "Ours was not a boiler-plate survey designed generically and then tweaked a bit for our community - ours was a survey designed by and for our own community. There is a significant difference there, and part of that difference is its fundamental intent. And that fundamental intent, once again, was not about counting" (p. 2). I'm not sure what Jack Ukeles would say about that. Having spent nine months of my life working on the questionnaire for the 2005 Boston Jewish community study with Combined Jewish Philanthropies' community study committee, I can say that this didn't apply to my work.

In a similar vein, the authors write that the fact that the statistics reported are estimates "...is, by the way, always the case in community demographic surveys, irrespective of methodology and whether it’s telephone-based, online, or even face-to-face, even when charts and graphs imply more specificity than they can actually support, by providing numbers with several decimal places, and by performing convoluted analysis on those numbers" (p. 3; emphasis in original). Apparently lacking a well-developed sense of irony, the authors round their population estimates to the nearest five (while rounding those of earlier Rochester surveys to the nearest hundred, an implicit claim of a twentyfold increase in accuracy), mean years of residency to the nearest tenth of year (p. 37), and mean numbers of Jewish friends to the nearest hundredth (pp. 61-63, 65). (The basis by which population estimates were arrived at is not explained and remains, frankly, a mystery to me.)

A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

It's evident from the material that the study was a nonprobability design--all initial seeds appear to have been self-selected (e.g., hearing about the survey from advertisements, in communal newsletters, and so on) and thus the sample is a nonprobability design.

Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures. The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.

Sample size (n=1,913 plus 421 "usable" partials--"usable" is never defined). Sampling error estimates are reported as well: "With 2,234 respondents [and another 100 college students attending local colleges] our overall sample is conservatively associated with a precision interval (or margin of error) of +/-3% at the 95% confidence level, suggesting that our findings and projections should be within 3% of what we would have found if everyone in our Jewish community had participated" (p. 12).

These "margins of error" are misleading, however. Sampling error is calculable in surveys for which the mode of selection of units is understood. We know how random selection works--with a population of size N and a sample selected of size n, the probability of a unit being selected is n/N. When we mess around with the probability of selection in one way or another (e.g., stratification or clustering), as long as we can calculate the probability of selection, we can work our way back to confidence intervals and the use of inferential statistics. The same holds true when randomize experimental subjects into treatment and control groups. Even if we cannot directly randomize, as long the statistical process generating the observed data can be accurately modeled, we can use the apparatus of inferential statistics. That isn't the case here. Snowball sampling (as occurred here) does not supply it (Berg 1988; Eland-Goosensen et al. 1997; Friedman 1997; Kalton 1983; Kalton and Anderson 1986; Spreen 1992 [citations from Sagalnik and Heckathorn 2004]).

The closest analogy to the methods applied here that has significant levels of acceptance as capable of generating meaningful estimates of sampling error is respondent-driven sampling (RDS; Heckathorn 1997, 2002, 2007; Sagalnik and Heckathorn 2004; Volz and Heckathorn 2008). For RDS to work, though, all sections of the population must be connected with one another, long chains of referrals are needed in order to access parts of the social network with zero probability of selection in the initial wave, respondents must give their "degree" (the number of people with whom they are connected with the characteristic that is salient for sample selection), and numbered coupons must be used to record details of the selection and the social network it was based on (Sagalnik and Heckathorn 2004).

This is not the case here: the initial seeds were self-selected; we have no evidence from the report that long chains of referrals were achieved; respondents were not asked their degree (and it is doubtful whether most people could answer this accurately); and, we do not know which which cases referred other cases. Even RDS, far more credible than the Rochester study, has had doubt cast on its accuracy in recent years (Gile and Handcock 2010; Goel and Sagalnik 2010).

Nevertheless, the authors describe their sample (unjustifiably, in my view) as "extremely solid" (p. 3), "robust" (pp. 7, 9, 17), "highly robust" (p. 11), and "statistically robust beyond all expectations" (p. 12), as well as "inclusive" (pp. 7, 10), "more inclusive" (p. 11), "more inclusionary" (p. 69), "highly inclusive" (pp. 6, 8), and having "broad-based inclusiveness" (p. 11), not to mention being "impressive" (p. 17) and "very clever" (pp. 6, 8). There seem to be two reasons for this extraordinary self-assurance: sample size and inclusiveness.

The sample size is heavily hyped by the authors. "For the sake of comparison, national election polling predictions are based on smaller samples than ours. Communities like Philadelphia, whose Jewish population is roughly ten times larger than our own, recently completed their demographic study with a sample roughly half as large as ours. And ten years ago, when Rochester undertook its last demographic study, our sample was less than one-third as large as ours is today," (p. 12). Sample size is important to a survey inasmuch it reduces sampling error but, as I described above, this survey's attributes do not allow sampling error to be accurately estimated, a large sample is of little avail.

Beyond sampling error, surveys are subject to two serious forms of potential error associated with acquiring a sample: coverage error and nonresponse error. These appear to hold little concern for the authors due to the sample's aforementioned inclusiveness. The only evidence for the study's inclusiveness that I read in the report is that (a) the study had a field period of eight weeks (p. 7); (b) it received "a constant parade of media and PR spotlights" (p. 7); (c) it included Jews of different ages (p. 11); (d) it included interfaith households, GLBT Jews, old Jews, new residents, highly affiliated Jews, marginally affiliated Jews, unaffiliated Jews, Jews living in central areas, and Jews living in outlying areas (p. 7); (e) that volunteers were available to help (p. 11); and (f) that older Jews finished the survey without assistance within a week of it beginning (p. 11). This culminates in the claim (g) that "[i]t seems safe to say that few Jews in the greater Rochester area ended those eight weeks unaware of that the Count Me In survey was taking place" (p. 7).

Nonresponse always carries a concern regarding bias, and it seems entirely conceivable that the people most likely to respond would be those with the greatest interest in the subject-matter. Indeed, these concerns are heightened when the survey is accompanied by a PR campaign that has the effect of increasing the salience of the survey for the most connected Jews; a lower response rate can be a good thing if it lessens bias. With all due respect to the "parade of media and PR spotlights," it is also possible that there was coverage error: Jews who would have been eligible for the survey but were unaware of it and thus were systematically excluded. This is a risk inherent in the survey's approach, which relied on respondents to come to it rather than seeking them out, which is the approach used in virtually all Jewish population studies.

Indeed, nonresponse and/or coverage error apparently did occur: "Might we, as a result of this approach, have ended up with a higher proportion of affiliated Jews than we did in 2000? We probably did. By opening the survey's gates to all who chose to enter, we have over three times as many total participants, and those most involved Jewishly were most apt to take part" (emphasis in original); the estimated proportion of people who were never synagogue members dropped by almost half (p. 68), as did estimates of intermarried household (p. 69), a glaring indicator of bias.

All this is apparently excused by the survey's inclusionary (i.e., self-selecting) nature. "The fact remains," the authors write, "that within this year's sample we also have significantly more non-affiliated participants than we did in 2000" (p. 68; emphasis in original). Further, the reader is told, that "[i]n 2010, we have not turned our backs on the unaffiliated, and have in fact included them in far larger numbers than they were included in 2000 via RDD and DJN [Distinctive Jewish Names] telephone-based approach. It's just that alongside these non-affiliated respondents are a robust cohort of those more affiliated Jews who in the past would never have had the chance to be 'counted in', and this time around, via 2010's more inclusionary online methodology, were provided with that opportunity" (p. 69; emphasis in original). The decision-making is perhaps more clearly seen earlier: "Perhaps the best aspect of this sampling strategy was its broad-based inclusiveness. This year, everyone who wanted to participate had the opportunity to do so. In contrast with a telephone-based survey that works from a pre-determined list and/or Random Digit Dialing [RDD] and/or a set of Distinctively Jewish Names [sic]...no one in our community with ideas and opinions and experiences to share was left out of this survey. Ours was truly a COMMUNITY survey--and everyone who took part now gets to feel that their ideas and opinions and experiences have indeed been COUNTED IN" (p. 11; USE OF CAPITALS in original).

The fact that the survey's estimates of community priorities, age composition, religious behavior, and likely every other topic reported were biased in the direction of the views, behaviors, and characteristics of the more affiliated as a result of the survey's methodology apparently did not enter into consideration. This is utterly wrongheaded for a study with the goal of providing "actionable insight" for organizations' planning processes (p. 2). Bad data are a dangerous basis for decision-making. Inclusivity and feeling counted are indeed virtues, but never, ever, at the expense accuracy.

On first looking at this research, my first reaction was that the authors had done a wonderful job of going back to 1930s. This is not damning with faint praise, as readers of my dissertation (all 1.4 of you) will be aware. As probability samples with close to full coverage of the Jewish community are financially out of reach for most Jewish communities these days, we are in the sample places as we stood in the 1930s and 1940s, before robust methods were developed for sampling human populations. These studies focused on enumerating the Jewish communities rather than estimating (in a probabilistic, statistical sense) with the best of them having well thought out procedures to find as many Jews as possible in a community. Such an approach could reasonably justified. However, the sheer puffery and overweening arrogance--I can think of no other word to describe it--of the authors turned these thoughts into ashes in my mouth. (I'm no slacker when it comes to professional self-regard, and the same goes for other Jewish researchers. Our posturing, boasting, and deprecation of each other is usually confined to private conversations and email exchanges, conference presentations, and the odd journal article rather than community study reports which are, as they ought to be, client focused.)

Better reports (demonstrating my self-regard, I will shamelessly put up my own work on the 2005 Boston Jewish community study as an example) balance their self-promotion with appropriate humility about potential errors. While we said that our study provided "a rich portrait" of the community (p. 1) "breaks new ground" (p. 21) and lauded the research contractor's "high quality work" (p. 21), honestly the only examples of preening I could find in the summary report, we also pointed out the underestimates of young adults and Russian Jews (pp. 21-22).

In the case of Rochester, though, not the slightest hint of potential error (other than the mischaracterized confidence intervals) disrupts the fanfaronade that is this report. And it is a shame. Surveying smaller Jewish communities, never easy, has become extraordinarily difficult and the task of providing representative samples virtually impossible. Collecting nearly 2,000 completed surveys is no mean feat and likely reached much of the affiliated core of Rochester's Jewish population. There was considerable merit in the study's approach had its goal been enumeration, had the authors been forthright in the shortcomings of their data, and had responses by disaggregated by potential bias: affiliated, less affilitated, and unaffiliated Jews...there are, after all, reasons we undertake convoluted analyses.

References

Berg, S. 1988. "Snowball Sampling." Pp. 528-32 in Encyclopedia of Statistical Sciences, vol. 8, edited by S. Kotz and N.L. Johnson. New York: Wiley.

Eland-Goosensen, M., L. Van De Goor, E. Vollemans, V. Hendriks, and H. Garretsen. 1997. "Snowball Sampling Applied to Opiate Addicts Outside the Treatment System." Addiction Research 5(4):317-30.

Friedman, S.R. 1995. "Promising Social Network Research Results and Suggestions for a Research Agenda." Pp. 196-215 in Social Networks, Drug Abuse, and HIV Transmission. NIDA Monograph Number 151. Washington, DC: National Institute on Drug Abuse.

Gile, Krista J. and Mark S. Handcock. 2010. "Respondent-Driven Sampling: An Assessment of Current Methodology." Sociological Methodology 40(1): 285-327.

Goel, Sharad and Matthew J. Sagalnik. 2010. "Assessing Respondent-Driven Sampling." Proceedings of the National Academy of Sciences of the United States 107(15):6743-47.

Heckathorn, Douglas D. 1997. "Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations." Social Problems 44(2):174-99.

Heckathorn, Douglas D. 2002. "Respondent-Driven Sampling II: Deriving Valid Population Estimates from Chain-Referral Samples of Hidden Populations." Social Problems 49(1):11-34.

Heckathorn, Douglas D. 2007. "Extensions of Respondent Driven Sampling: Analyzing Continuous Variables and Controlling for Differential Recruitment." Sociological Methodology 37:151-208.

Kalton, Graham. 1983. Introduction to Survey Sampling. Beverley Hills, CA: Sage.

Kalton, Graham and D.W. Anderson. 1986. "Sampling Rare Populations." Journal of the Royal Statistical Society, Series A 149:65-82.

Sagalnik, Matthew J. and Douglas D. Heckathorn. 2004. "Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling." Sociological Methodology 34:193-239.

Spreen, M. 1992. "Rare Populations, Hidden Populations, and Link-Tracing Designs: What and Why?" Bulletin de Methodologie Sociologique 36:34-58.

Volz, Erik and Douglas D. Heckathorn. 2008. "Probability Based Estimation Theory for Respondent Driven Sampling." Journal of Official Statistics 24(1):79-97.

Jewish Survey of the Day: "Quick Bytes: On the Minds of Teens" (JESNA)

The first of my new series of reviews of Jewish surveys comes to us from JESNA and provides insight into "What is on the minds of Jewish teens?" The italicized sections come directly from the AAPOR Standards for Minimum Disclosure.

Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.

JESNA sponsored the research. Who conducted the research is not clear.

The exact wording and presentation of questions and responses whose results are reported.

No. The survey instrument is not available.

A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population. If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.

The sample appears to come from students attending schools that belong to the North American Association of Community Hebrew High Schools. No information is provided on the sampling frame used.

(This sample certainly doesn't cover all Jewish teens. Many don't attend any formal Jewish education during their high school years, while others--predominantly Orthodox--attend Jewish day schools. Accordingly, it is unlikely to represent "Jewish teens" as a group. Does it represent Jewish teens in supplementary Jewish education in high school? This hinges on how representative the NAACHHS is of "Hebrew High Schools." I can't comment on this, because the portion of their website that lists member schools doesn't work, at least on Firefox. While the introductory paragraph casts the survey's applicability in terms that are too broad for my taste, the fact that the sample is drawn from NAACHHS schools is mentioned multiple times, leaving the reader to reach their own conclusions about its representativeness. How many schools participated? We simply don't know and, on that basis, it's very difficult to know how much weight to give this research.)

A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

No indication of sample design is given.

(We don't have a clue how JESNA reached the students. Does NAACHHS have a comprehensive list of student emails? Was a sample selected or was every student approached? Were students mailed links by their schools? Was parental permission obtained? Again, we don't know these fundamental facts.)

Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures. The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.

Sample size is n=219. No indication of sampling error is provided. (As noted above, we don't even know if this was from a sample.)

Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.

No indication is given.

Method and dates of data collection.

Data was collected by web survey. Dates are a little vague, with "May 2010" given as the timeframe.

Assessment

This survey clearly (to me at least) fails major aspects of the AAPOR minimal disclosure standards. The failure to provide this basic level of information severely inhibits the utility of these data. To give a basic example, it makes a great deal of difference if the results achieved sample size of n=217 was drawn from a sample of 500 cases or from all 15,000 children attending NAACHHS schools (a number I made up on the spot because that information is not reported by NAACHHS).

Jewish Survey of the Day/Month/Week/Year

I am adding a new feature in my periodic blog: reviews of Jewish surveys. As those familiar with me are well aware, I cast a jaundiced eye over much of the work of what used to be my field (through September 2010 I was an associate research scientist at the Cohen Center for Modern Jewish Studies at Brandeis University; since October 2010 I have been working as a senior analyst/project manager at Abt SRBI Inc.). I will vaguely organized this around the Standards of Minimal Disclosure of the American Association for Public Opinion Research (AAPOR) Code of Professional Ethics and Practice [May 2010] which is, to whit, information that shall be included "in any report of research results or make them available immediately upon release of that report" (read the Standards here). This is the basic information required for a reader to make informed use of statistics in the public domain. It is not an "academic" standard (most AAPOR members work in industry, not academia), nor is it particularly difficult to meet. Any survey that exists for purposes other than pure entertainment should adhere at a minimum to these standards. While I initially planned to keep my focus on AAPOR compliance, I immediately fell off the wagon.

Books I want

Cochran, William G. 1977. Sampling Techniques. 3rd ed. Wiley.
Heeringa, Steven G. 2010. Applied Survey Data Analysis. Chapman & Hall.
Kish, Leslie. 1995. Survey Sampling. Wiley-Interscience.
Lohr, Sharon L. 2009. Sampling: Design and Analysis. 2nd ed. Duxbury Press.
Wolter, Kirk. 2010. Introduction to Variance Estimation. 2nd ed. New York: Springer.

How Did You Get Here? Early Career Decisions and Graduate School

This is a talk I gave to Cohen Center summer interns on July 27, 2010, as part of a series of talks by full-time staff.

My path here began, I suppose, at birth. I was born in 1975 in Sydney, Australia, to a Jewish father and a non-Jewish mother. My father’s father was born in Australia of Lithuanian parents while his mother was born in Baden bei Wein, Austria, and escaped in 1938 following the Anschluß. Both were doctors. My mother’s family was Hungarian. My maternal grandparents walked westward in 1945 away from the Soviet advance and ended up in a displaced persons camp in Wiener Neustadt, Germany, where my Grandpa Fafa served as a doctor. He moved to the then-Australian colony of Papua New Guinea, along with a number of other Hungarian doctors, bringing my grandmother and mother (born in the DP camp) along later. My parents actually met in Papua New Guinea, where Grandpa Fafa had heard that there was a visiting Viennese lady doctor and her two eligible medical student sons. (My paternal grandfather had died before then.) Grandpa Fafa promptly went to see my maternal grandma, Edith, and, European gentleman that he was, clicked his heels, bowed, introduced himself, and explained that he had two daughters and heard that she, my grandmother, had two sons. Thus I came to be.

This level of detail might be seen as self-indulgent, I suppose, except I study Jews, and I study outcomes, and beginnings do count. I also know that my path here was somewhat more winding than normal, so I sketch out my beginnings.

I was not raised as a Jew or a Christian. My religious education, such as it was, consisted of attending an Easter mass at Sydney’s Catholic cathedral, a daily service at an Anglican church, and morning services at an Orthodox synagogue. (Most synagogues in Australia are Orthodox.) I felt I belonged in the synagogue in a way that I did not in the cathedral or the church. I don’t know why. Ever afterwards, I felt Jewish. I don’t think it was because I felt closer to that side of my family. I spent far more time with my Hungarian grandparents than my Jewish grandmother. The ethnic food I cook is Hungarian. The language I curse in is Hungarian. Nor was Grandma Edith a pillar of Jewish practice. She had a hostile relationship with Judaism, her observant family having barred her from attending Jewish tutoring with her brothers. Despite living but a door away from her observant parents, Jewish holidays were not celebrated.

Until college (or university, as we would say in Australia), there wasn’t much I could do about my feelings of Jewishness. I didn’t live in a Jewish neighborhood. I didn’t know that Jewish youth groups existed. There was no web for me to search things out on. I did read the Jerusalem Report, which my grandmother passed onto me, and books on Israel and occasionally Jewish life in general. At college, however, I had options. During orientation week, I joined the Australian Union of Jewish Students (we don’t have Hillels). When signing up for classes, at the last minute I chickened out from taking Chinese and decided to sign up for Jewish Civilisation, Thought, and Culture. I only planned to take it for a year, dropping it and keeping up with my “serious” subjects of political science and modern history, as well as my classes as a law student. (Like Canada and the rest of the Commonwealth, law is an undergraduate degree in Australia.) As it turned out, Jewish Civ was the only class I enjoyed, and I dropped history without a further thought at the end of the year. At the end of the year, I spent the summer holidays in Israel on program ran by the Union of Jewish Students and with my relatives.

By the time I came back, I knew I wanted to resolve my status. I certainly I wasn’t Jewish by Orthodox or Conservative standards, nor was I Jewish under the Reform movement’s standard of “appropriate and timely public and formal acts of identification with the Jewish faith and the Jewish people.” No exception or loophole covered me. Thus, I began the process of conversion. It was clear to me that I couldn’t “accept the yoke of the mitzvot,” in an Orthodox sense, so I converted under Reform auspices (there being no Conservative movement in Australia) after a year of studying material I had already learned at university. And so, I became a Jew.

I kept taking classes in Jewish Civilisation and started enjoying political science more. I became secretary and then vice-president of the Union of Jewish Students at Sydney University. At the end of my second year, I went back to Israel for the summer for the Union of Jewish Students’ Leadership Development Program. It developed me in ways I never expected, as I met and fell in love with an American at the Jerusalem youth hostel in which my group was staying. Back in Australia, I never did enjoy law and dropped my law degree at the end of my third year, keeping studying for my B.A. If I wasn’t going to be a lawyer, what would I do? With the innocence (read “foolishness”) of youth, I decided I would go to grad school in Jewish studies. Because I was in a long-distance relationship with an American, clearly it would be in the U.S. As it happened, because I dropped Criminal Law, I was a few credits shy of a three year B.A., which in any case was not accepted by U.S. universities. I thus spent something of a gap year in my fourth year of college: working at a department store, catching up on my honours year entry courses in political science, and taking Hebrew classes outside my degree. I also planned a topic for my honours thesis. This was quite difficult. It had to be on a Jewish and political science topic; it couldn’t be on something purely Israeli, because my Hebrew was not up to reading scholarly literature, to put it mildly; it had to involve original research; it couldn’t have been studied before; and it had to involve travel to the U.S. Eventually, I worked out that I could study American Jewish lobbying of Israel over the then-current (1998) debate over the status of non-Orthodox converts for the Law of Return. Because I had a quasi-gap year, I was able to do my bibliographic research in advance and work out who I wanted to interview. I then spent three months in the U.S. over my summer holidays doing all my reading of the existing literature and interviewing heads of major American Jewish organizations. Nothing to do with my girlfriend, of course. When I came back to Australia and classes began, I was able to write my thesis with ease.

This marked the end of my involvement in political science. I was interested in studying Jews with social scientific methods, but couldn’t stand being locked into political questions. While all Jewish studies interested me, I found that the more recent the topic was, the more fascinating it was. The pivotal point for me was reading Jewish Choices: American Jewish Denominationalism. This is actually an unremarkable book, a heavily statistical analysis of Jewish religious identification and behavior using the National Jewish Population Survey of 1990, but it showed me what could be done with social scientific methods and Jewish topics.

Degree in hand, I took my GRE’s, doing abysmally in math, poorly in logic, and brilliantly in English. Again, with the foolishness of youth, I only applied to two grad programs: sociology at Columbia and Near Eastern and Judaic Studies and sociology at Brandeis. I found out about the latter because I called the Brandeis sociology department and spoke to the grad student administrator. She heard what I was interested in and said, Well, actually, we have this joint Ph.D. I was sold. I was accepted to Brandeis but not Columbia. I’m not sure my qualifications would have been enough for a straight sociology or Jewish studies degree. Foolishly, once again, I took the Brandeis offer even though I was not funded (my application was sent at the very last minute). I got funding in my second year, but I was extraordinarily lucky to do so. I don’t know if I would have applied to grad school in the U.S. had my fiancée not been American. I might well have ended up in exactly the same place, but the odds certainly would have been lower.

I arrived here on August 2, 1999, less than a year after finishing college, started grad school a month later, and married my fiancée in June of 2000. It took me a while to adjust to life as a grad student. Some classes were great, others not so much. The classes that were central to my academic development were those that were out of department. I took a class in survey research at the Heller School. I also had to take a module on stats in Heller as part of my NEJS requirements. Being a module, it wasn’t listed in the course catalog. Most people would have asked for help and eventually enrolled. Not me. I’m shy and hate talking to people. I took the nearest class listed in the course catalog, Applied Regression Analysis. I almost died when I open the text and saw the statistical formulae in all their Greek glory. I’d had a little exposure to stats in a sociology class, though, and took to the material very rapidly. In my final semester, I took the 21st and 22nd of my required 21 classes by taking Applied Econometrics and Applied Multivariate Analysis. I also TAed. I ended up getting quite depressed. In retrospect, I should have worked at the Cohen Center rather than just trying to study and be a TA. (I did work for the Cohen Center during vacations as a participant observer on Birthright trips and at summer camps.) I like having structure in my life, and work keeps me busy and gives me a sense of worth. While it didn’t fit my plans, it’s probably better to spend a year or two in the real world before signing yourself up for another three year or more years of institutionalization in an educational institution—you will already have spent at least 17 years in school by this point.

With coursework over, I should have been studying hard for my sociology accreditation committee and NEJS comprehensive exam. As it was, I was very depressed (lack of structure, once again), and my wife was pregnant and we would need more income. I screwed up my courage and emailed Len to see if he had work for me at the Cohen Center. He did. I was meant to work 20 hours a week and spend the rest of my time studying. My initial assignment was a small independent project to do some literature reviews. This is typical. A grad student doesn’t necessarily make a good employee. Some are know-it-alls for whom drudge work is beneath them. Others are just flaky. We give assignments that don’t matter much to see what kind of person a grad student is. My record, looking back, gave clear signs of flakiness. I hadn’t worked as a grad student and I didn’t have a record of analogous research projects. As it turned out, I wasn’t a flake. Just as lack of structure depressed me, having structure in the form of a job energized me and I was rapidly working 40 hour weeks. I spent the next year essentially working full-time and neglecting my exams. While not advancing my studies, it was therapeutic. I was then able to get back to my exams and treat them as if they were a job, while keeping working at the Cohen Center. Having reframed them as work, I was able to get through them with minimal trauma. I had progressed through steadily more meaningful jobs at the Cohen Center and found my niche at the quantitative end of the spectrum. It’s far more difficult to find people interested in Jewish topics with quantitative predilections than qualitative ones, so I fit in very well.

Working as a researcher is very different from studying to be one. Grad school provides a way of thinking about the world and provides a basic level of knowledge. But that’s it. I remember sitting in my survey research class talking about data collection and the professor said “and then you clean the data.” Little did I know that cleaning the data is more than 95 percent of the effort expended, with analysis being 5 percent at most. Nobody teaches you in grad school to clean data or any of the other elements of what I will call the art or practice of social research. These are learned by doing and take place in the context of an apprenticeship. When I look at resumes, I look for people with practical experience far more than people with higher degrees. Somebody who has practiced research, even in a humble way, takes far less effort to train and represents a much lower level of risk as a hire.

Having gotten my exams out the way, I needed a topic for my dissertation. I had a role model in Shaul Kelner, who wrote his Ph.D. at CUNY on his work as the Cohen Center’s fieldwork coordinator for Birthright studies in Israel. I was determined to do the same and have my job and my dissertation be one and the same. At that time, Len was talking to Combined Jewish Philanthropies about their forthcoming community survey. I told Len that I wanted to take the project on and be involved in every single aspect I possibly could and write about methods used in Jewish population studies. This built on the work I had been doing at the Cohen Center. For the nine months the North American Jewish Data Bank was at Brandeis, I served as the day to day staff person and became familiar with the extensive history of Jewish population studies. I was also responsible for distributing the National Jewish Population Survey of 2000-01, a deeply flawed piece of research. Understanding its limitations gave me a keen sense of research methods. The Boston study was a wonderful experience and I learned a vast amount about the theory and practice of survey research. Large sections of my dissertation were drawn with little editing from methodological material I prepared during the course of the project and other work I had conducted on the National Jewish Population Survey. In turn, large sections of my dissertation ended up in the methodological report with little editing. It was a win-win situation, with the added bonus that I had a $500,000 budget for my dissertation research.

Having received my Ph.D., why am I still working here? There aren’t many faculty positions in contemporary Jewry. In the past decade, I recall the following: A tenure-track position in sociology at the University of Judaism, Los Angeles. The person in the position left and it appears the position no longer exists. A tenure-track position in the sociology department at Yeshiva University which seems never to have been filled. A tenure track position in Jewish studies and any social science department at the University of Michigan, the funding for which never came to fruition. A tenure-track position in sociology and Jewish studies at Vanderbilt University, which was filled by my former colleague Shaul Kelner. A position for an “established scholar” (i.e., not me) at Monash University in Australia. They ended up hiring a Yiddishist. To teach contemporary Jewry. A tenure-track position at Brown University, which was written for a specific candidate who, not surprisingly, was hired. A position at open rank at the Hornstein Jewish Professional Leadership Program at Brandeis. No-one was hired for it and the position wasn’t filled. A position at the professorial level in the Hornstein Program which was written for Len, who was hired. A tenure-track position at USC in sociology and Jewish studies, filled by a Ph.D. student in sociology from Columbia whom I’m not familiar with. I might be forgetting one or two, but that’s about it. I was one of three people to interview for the Vanderbilt position, but that’s the furthest I have advanced. There has also been a post-doctoral position at the University of Michigan. I can’t recall a single person from either the contemporary Jewry track of the NEJS Ph.D. program or from the joint Ph.D. I studied for who is a professor. If you want to be a professor of Jewish studies, you appear to be better off with a Ph.D. in modern Jewish history. If you want to be a professor in a social scientific discipline who studies Jewish topics, you’re probably better off doing a straight Ph.D. in the discipline of your choice, being careful not to let your Jewish interests put you outside the mainstream.

What about other careers? People with my training seem to go three ways. They work for a philanthropic foundation focused on Jewish life, they work for a Jewish organization, most commonly in research or evaluation, or they work here.

I don’t know what it’s like to work for a foundation, but I can’t imagine it’s much fun to be a program officer, the person who supervises projects for the philanthropy. We work with them all the time, and they’re administrators with extremely limited scope for innovation or creativity. Research and evaluation for a foundation might be a little better, but your life will be spent conducting small-scale evaluations of programs and contracting for larger projects. The professional heads of foundations have backgrounds in Jewish organizations, usually federations, so it’s not clear if a career path exists from evaluation to senior roles.

Work for Jewish organizations seems to have little career path, either, if you work on the research side. People who have had these positions seem to leave in under a decade or have their jobs downsized. Research tends not to be taken very seriously in the Jewish organizational world. Some senior professionals do have Ph.D.’s, but they are on the managerial track and don’t engage in much research. You can also start with a master’s in Jewish communal service, an MBA, rabbinical ordination, or perhaps a Jewish studies MA. It’s not entirely clear why being a rabbi qualifies you to work in a complex service organization, but many rabbis end up there, so apparently it does. The coin of the realm for the highest positions in Jewish organizations is the ability to raise money.

This leaves the Cohen Center and other research organizations. Entry level for people with masters degrees is the research analyst or research associate level, depending on skills and experience. This involves working for a principal investigator, albeit with a considerable degree of autonomy once you’ve proved yourself (and assuming you work for a good P.I.). You are typically given a job and work out how to complete it. You are usually working on several projects at once and need to juggle these. Some analysis work will be involved, but writing is minimal. The intangibles we look for are maturity, being hard-working, capable of working well without constant supervision, being a quick study, the ability to work well as part of a team, and generally not being a prima donna. The specifics of education and even intelligence in general are less important. Ph.D.’s with limited experience will come in at the research associate level. Most Ph.D.’s will start as senior research associates. Initially, work will be under or in conjunction with a more senior researcher and will involve similar work to a research associate, but with greater responsibilities, autonomy, and supervision of junior staff. Analysis and writing is a major part of the job. Advance sufficiently high on the totem pole to be at the principal investigator level and a major part of your job is preparing research proposals in response to RFPs (requests for proposals), a good number of which will fail. At this level, working with clients, writing reports, and supervising staff are major responsibilities. Substantive involvement in research and analysis is typically limited to setting up paradigms for more junior researchers to follow, resolving difficult issues, and conducting particularly complicated work. Much more time is spent looking over the work others have done and advising them on how to refine it.
Working at the Cohen Center enables one to conduct a different type of research to academics. Most working professors don’t actually do particularly exciting research. They don’t have the staff to conduct complicated research projects. Consequently, if they are quantitative types, they spend their lives analyzing datasets that other people have created. If they are qualitatively oriented, they will do small-scale interview and participant observation research. Teaching responsibilities take up some time (more earlier in your career when you’re still creating courses), particularly if you don’t work at a research university. You’re also expected to publish in peer-reviewed journals and need to produce a book to be eligible for tenure and move from assistant professor to associate professor.

What should you learn from all this? In some ways, the most helpful advice is: don’t be me. Consider working for a while before going to grad school. Apply to plenty of grad schools. Visit them and talk to faculty and students. Get your applications in early. Apply for fellowships like Wexner and other funding opportunities. Work while you study. Prepare for your qualifying classes while you’re still doing coursework. Understand the career options available to you. I shudder to think at all the places where things could have fallen apart but fortuitously did not. But if I were to offer this advice to myself as an undergraduate, I might have made fewer errors, but it wouldn’t have changed my plans. It was the only thing I wanted to do…I couldn’t avoid it any more than I could have decided not to become a Jew. The flip side of the coin is that I’m a living example of Herzl’s line that “if you will it, it is no dream,” anachronistically rendered in Hebrew as “im tirtzu ein zo agadah.” I had no Jewish background to speak of, grew up in Australia, had a decent but not outstanding undergraduate education, and no appreciable mathematical abilities, but I wanted to conduct social research on American Jewry. Yet today I am an “expert” on American Jewry, a competent survey researcher, and a very good statistical analyst. I’ve also discovered my interests on the border of survey design, statistics, and mathematics, which I didn’t know I would become interested in. I can’t complain about the life I lead, but if you are planning to apply to grad school in a field of purely academic interest (astronomy, English, linguistics, Jewish studies), do so because you can’t help yourself, not because you want a career in academia.

20¢ = 25¢

Hailing from Australia, I've often been puzzled by American currency mores. Why "nickel" and "dime"? (I still can't remember which is which after a decade in the U.S.) Why is a dime (10¢ if you're wondering) smaller than a nickel (5¢)? What do Americans have against their 50¢ and $1 coins? Why do all banknotes look the same? (A practice that has been ruled to be in violation of the Disabilities Act.) Why don't Americans use $50 and $100 bills?

Besides all of these questions, I've wondered whether quarters (25¢) or 20¢ coins (as we have in Australia) are more efficient. I define efficiency for a system of coinage as requiring the fewest coins for an arbitrary amount of change between 1¢ and 99¢. Although Australia withdrew 1¢ and 2¢ coins from circulation in 1992, I made the comparison as close as possible by looking at a system with 1¢, 5¢, 10¢, 20¢/25¢, and 50¢ coins. The result, as the title of this piece suggests, is that 20¢ and 25¢ coins are equally efficient--the average number of coins required for amounts of change between 1¢ and 99¢ is 4.24 in both cases (SD is 1.71 in both cases).

As I mentioned earlier, though, Americans have a strange aversion to 50¢ coins, while pre-1992 Australia had 2¢ coins. If we take this into account, the average number of coins required for change in the U.S. rises to 4.75 while the Australian average falls to 3.43, meaning that you get on average more than a coin extra in change in the U.S.

The next step, of course, is to ask whether it is possible to do better than 5¢, 10¢, 20¢/25¢, and 50¢ coins altogether. Might we in fact better be served by 4¢, 12¢, 37¢, and 74¢ coins with respect to minimizing change? (Actually, yes. This would require 4.02 coins on average.) I've cleared most hurdles to specifying this problem in AMPL (a mathematical programming language for optimization problems) and will return with my "rational" scheme for change in the near future.

Of course, not all amounts of change are created equal. In currency regimes with 1¢ coins, prices tend to end in nines (Basu 1992, 2006; Demery and Duck 2007), while in Australia prices end in fives. Ideally, we would weight the number of coins by the probability of receiving change of that amount in a transaction. However, a cursory search has not revealed a useful dataset for these calculations.

References

Basu, Kaushik. 1997. "Why are so many goods priced to end in nine? And why this practice hurts the producers." Economics Letters 54:41-44.

-----. 2006 "Consumer cognition and pricing in the nines in oligopolistic markets." Journal of Economics & Management Strategy 15:125-141.

Demery, David and Nigel W. Duck. 2007. "Two plus two equals six: an alternative explanation of why so many goods prices end in nine." Discussion Paper No. 07/598, Dept. of Economics, University of Bristol, Bristol, UK.)

A Checklist for Critically Reading Quantitative Research

The following is an attempt to create a heuristic for evaluating the quality of quantitative social scientific research with zero assumed background in social scientific methods.

1. Is the information contained solely in a press release and/or interview or is a report or article in a scholarly journal available? If a report or scholarly journal, was the research reviewed by knowledgeable peers?

2. Is the basic methodological information required by the American Association for Public Opinion Research's Standards for Minimal Disclosure available, either in the report or a methodological appendix?

a. Who sponsored the survey, and who conducted it. Commentary: Search for information on the researcher and the company that actually carried out the data collection, if different. Have either been cited by AAPOR for violating its Code of Professional Ethics and Practices (e.g., )? Alternately, have company principals held posts in professional associations like AAPOR (c.f. Tom Smith, Mark Shulman) and/or publish methodological articles and give presentations at meetings like the American Statistical Association? Does the research firm used primarily conduct political polls, market research, or academic research? Naturally, look for commentaries on the specific piece of research. Where research is sponsored by an organization with an agenda that might influence research (i.e., just about all Jewish communal research), does the researcher explain the nature of their relationship with the sponsor?

b. The exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response. Commentary: Are the questions worded in a way that might bias the answer? If the results are very different to previous research, has the author used standard questions? If not, do they justify their items? Are there any skip patterns that fail to collect information from important populations? If the researcher combines items to form a scale or index, does s/he explain exactly how it was done? Does s/he report measures of scale reliability (e.g., Cronbach’s/coefficient alpha > .75)? Does the index/scale actually seem to reflect what the researcher says it does? Quite often, it doesn’t.

c. A definition of the population under study, and a description of the sampling frame used to identify this population. Commentary: Is there any reason to think that the sample might differ systematically from the population it is intended to represent? Was it a convenience sample (a.k.a. open sample)? Was it from an opt-in Internet panel? Does the sample just constitute Jews by religion? If there is a reasonable probability of bias, is it likely to have increased or decreased the reported results of the study? Does the author address any possible limitations of the sample?

d. Sample sizes and, where appropriate, eligibility criteria, screening procedures, and response rates computed according to AAPOR Standard Definitions. At a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed. Commentary: The lack of a response rate is a major red flag. The response rate should be accompanied by a specification of the exact response rate formula used (e.g., AAPOR RR3). Better research will not only list the response rate but will address possible biases.

e. A discussion of the precision of the findings, including estimates of sampling error, and a description of any weighting or estimating procedures used. Commentary: If the survey uses an opt-in Internet panel or a convenience sample, estimates of sampling error are inappropriate and should never be reported. Discussion of weights may distinguish between design weights and poststratification weights (which correct for biases). If poststratification weights are used, the discussion should specify what variables were used and where how targets for adjustment were derived.

f. Which results are based on parts of the sample, rather than on the total sample, and the size of such parts. Commentary: If analyses exclude parts of the sample, is there a reasonable justification for their exclusion?

g. Method, location, and dates of data collection. Commentary: How long was the field period? How many contact attempts were made? Did the researcher try to convert refusals? Overall, does it appear that sufficient effort was put into collecting the data?

3. If the researcher describes something as a well-established fact, is it in fact widely supported? Often, it is not. Do a quick fact-check.

4. Does the researcher actually present evidence that directly supports their conclusions? Is there any evidence in the research that contradicts the researcher’s assertions? (This is surprisingly common.) Look carefully for situations where some related data is shown, “hand-waving” takes place (i.e., unsupported assertions are made), and a conclusion is stated definitively.

a. For processes involving changes over time, is there actually evidence of change over time or does the researcher base her/his analyses on differences between age groups? If so, can one reasonably expect there to be lifecycle effects?

b. For processes involving individuals, does the researcher base their conclusions on aggregated data?

5. Does the researcher omit any relevant outcomes? First, were there questions asked that aren’t reported? Second, were any topics simply not asked about? Is there a reason to expect that the omitted topics might diverge from the reported results?

6. If the researcher asserts that X caused Y, does s/he control for other factors that might be expected to influence the outcome?

a. Simple bivariate analyses are far more prone to biases than are regression analyses with suitable controls, which tend to be quite robust, even for biased samples.

b. How important are exact estimates to the researcher’s findings? The more research depends on a specific figure (e.g., the number of Jews in the United States), the more vulnerable it is to shortcomings in data collection.

c. Did the researcher omit any relevant explanatory variables? First, were there any topics asked about that weren’t included in the analysis? (Typically, where a variable is not significant, a researcher will say something like “X, Y, and Z did not have a significant effect and are omitted; model not shown.”) Second, were there any potentially relevant explanatory variables omitted? If so, what explanatory variables included in the analysis might be picking up the omitted variables’ effects?

d. Does the researcher report whether effects were statistically significant? This is especially important where sample sizes are small. (Note that statistical significance isn’t appropriate in cases where all relevant units were surveyed, like many Birthright reports, or for convenience and other nonprobability samples.)

e. If the sample size is very large, are the effects reported large enough to be meaningful?

7. For evaluation research on the effectiveness of a program or policy:

a. How does the researcher estimate the program effect? In declining order of rigor, these are:

i. True experiments that randomly allocate individuals or other units like communities into treatment or control groups.

ii. Quasi-experimental designs that have nonrandom treatment and control groups and measure outcomes before and after the treatment intervention begins. Does the researcher document whether systematic differences exist between treatment and control groups? If there are differences, what steps does the researcher take to take account of these?

iii. Quasi-experimental designs that either measure participants before and after the treatment but do not have a control group or measure treatment and control groups only after intervention. For participant-only pre/post designs, are there other factors such as aging or a major event like a terrorist attack that could also explain the results? For treatment and control post-only designs, does the researcher model the characteristics of the groups and control for them when analyzing outcomes.

iv. Treatment-only post-only designs that ask retrospective questions about attitudes and behavior. Are the events being recalled memorable? Is there a reason to believe that respondents may have recall errors, like telescoping events?

v. Treatment-only post-only designs that ask about program satisfaction and self-perceptions of program effect. These are very weak.

b. Does the researcher generalize about the program’s effect beyond the type of people who actually participated? For instance, does a study of a program of outreach to intermarried Reform synagogue members claim that the program will work for all intermarried families? Is this a reasonable assumption?