The first of my new series of reviews of Jewish surveys comes to us from JESNA and provides insight into "What is on the minds of Jewish teens?" The italicized sections come directly from the AAPOR Standards for Minimum Disclosure.
Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.
JESNA sponsored the research. Who conducted the research is not clear.
The exact wording and presentation of questions and responses whose results are reported.
No. The survey instrument is not available.
A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population. If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.
The sample appears to come from students attending schools that belong to the North American Association of Community Hebrew High Schools. No information is provided on the sampling frame used.
(This sample certainly doesn't cover all Jewish teens. Many don't attend any formal Jewish education during their high school years, while others--predominantly Orthodox--attend Jewish day schools. Accordingly, it is unlikely to represent "Jewish teens" as a group. Does it represent Jewish teens in supplementary Jewish education in high school? This hinges on how representative the NAACHHS is of "Hebrew High Schools." I can't comment on this, because the portion of their website that lists member schools doesn't work, at least on Firefox. While the introductory paragraph casts the survey's applicability in terms that are too broad for my taste, the fact that the sample is drawn from NAACHHS schools is mentioned multiple times, leaving the reader to reach their own conclusions about its representativeness. How many schools participated? We simply don't know and, on that basis, it's very difficult to know how much weight to give this research.)
A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.
No indication of sample design is given.
(We don't have a clue how JESNA reached the students. Does NAACHHS have a comprehensive list of student emails? Was a sample selected or was every student approached? Were students mailed links by their schools? Was parental permission obtained? Again, we don't know these fundamental facts.)
Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures. The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.
Sample size is n=219. No indication of sampling error is provided. (As noted above, we don't even know if this was from a sample.)
Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.
No indication is given.
Method and dates of data collection.
Data was collected by web survey. Dates are a little vague, with "May 2010" given as the timeframe.
Assessment
This survey clearly (to me at least) fails major aspects of the AAPOR minimal disclosure standards. The failure to provide this basic level of information severely inhibits the utility of these data. To give a basic example, it makes a great deal of difference if the results achieved sample size of n=217 was drawn from a sample of 500 cases or from all 15,000 children attending NAACHHS schools (a number I made up on the spot because that information is not reported by NAACHHS).
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment