A CRITIQUE OF A MIXED RESEARCH STUDY

By

George H. Adams

B.A., University of Connecticut, 1977

A RESEARCH CRITIQUE SUBMITTED IN

PARTIAL FULFILLMENT OF THE

REQUIREMENTS FOR

RESEARCH IN EDUCATION (EDU 508)

Instructor: Dr. Hari Koirala

DEPARTMENT OF EDUCATION

EASTERN CONNECTICUT STATE UNIVERSITY

Summer 2007

A CRITIQUE OF A MIXED RESEARCH STUDY

One current trend in education is curriculum reform, and Parker and Gerber (2000) evaluate an approach to restructuring curricula through a science intervention program and its effects on science achievement and attitudes of middle school students.  Specifically, their study examines how a five- week summer enrichment program (independent variable) acts upon the science achievement and attitudes (dependent variables) of 11 middle school students.  Theirs is a mixed-methodology research design; employing a hypothesis that the science intervention program will promote middle school achievement and positive attitudes toward science, quantitative data was generated by a pre- and post- criterion-referenced test and survey measuring attitudes toward science, while qualitative data included a teacher/researcher daily log that recorded narrative descriptions of student behavior.  Driving the research are two related questions: What is the effect of a science intervention program of fifth- and sixth-grade students’   (a) science achievement, and (b) their attitudes toward science?  The intent of the study is to validate the intervention program, showing that it performs a necessary service by raising student achievement in science and their attitudes toward the discipline as well.  Such validation is of practical importance should other school systems need to improve science achievement and attitude.  The researchers’ reason for wanting this to occur is so that youth can assume science-related careers in business and industry in the future to prevent anticipated shortages of scientists and engineers.  While the researchers do not state any personal bias, or offer personal information other than school affiliation, the fact that both are associated with academia would seem to increase the desire for successful results.  However, the qualitative aspect of the study requires an observer who can make insightful observations about educational factors.

Parker and Gerber support the validity of conducting their study with national education goals from the United States Department of Education that call for improved science education.  They point out the imperative that science curriculum in the state of Georgia is based on the objectives detailed in Georgia’s Quality Core Curriculum, as well as national standards.  The research they cite deals with attempts to create relevancy for students by connecting science curricula with environmental and social issues, and interest through hands-on, inquiry based methodology.  They point out that the instructional model that incorporates both is the learning cycle, and detail research on learners’ behaviors in the cycle.  This information is relevant to the study because the intervention program employed is using the learning cycle methodology.  Cited research follows indicating that quality classroom instruction has the greatest effect on improving science achievement and attitude as opposed to self-image or home environment.  The importance of this research rests on the nature of the sample for this study.  The authors do not identify any flaws in the research cited; the overall effect of the literature is to bolster the validity of the rationale and design of the research study.

Parker and Gerber create a purposive sample by selecting 11 African American students from a summer enrichment program in rural Georgia.  Four of the students have been promoted to fifth grade, 7 to sixth grade. Five are boys, six, girls. These students are identified as having academic difficulties in reading and math, and were enrolled initially in the summer program to improve the overall quality of their lives through increased academic performance.  The science intervention program will run five weeks, and will incorporate selected content that is aligned with Georgia’s state mandated instructional objectives for sixth grade.  While the sample does seem to be appropriate for a science enrichment program, the researchers may have compromised their study by combining the two grade levels.  These students are already at risk; the fifth graders in particular may have a difficult time with sixth grade level material, and there is no information that describes results by grade level.  Any thoughts of expanding the results from this sample to include a much larger population should be reconsidered when taking into consideration the grade-level composition of this relatively small sample.

Evaluation of the sample is  in both quantitative and qualitative formats.  The quantitative data was obtained from a pre- and post- criterion-referenced test,(CRT)  consisting of fifteen multiple-choice questions about elements of the intervention program.  Content validity was determined by the researchers themselves as being high because of the test’s direct relationship to the instructional objectives of the intervention course.  However, the statistical validity of the pretest scores (.52 alpha coefficient) and posttest scores (.47 alpha coefficient) was low, and the researchers speculate that this was caused by the small sample size.  An additional issue might be the low number of questions on the test.

Additionally, a pre-and post- “Attitudes Toward Science Survey” (ATSS) (based on an existing, Likert-type questionnaire with a coefficient alpha of .84) was administered.  The actual coefficient alphas for the administered tests were high (pre - .79, post-.83).  While instructions for completing this survey were written on the survey, these instructions and test items were read aloud to the students.  Here, the researchers open themselves to a charge of bias because of this type of presentation.  Clearly, the pre- and posttest readings could not be identical. 

Adding to the quantitative studies, a qualitative teacher’s log recorded student behaviors.  These observations were compared with quantitative results, creating triangulation based on an opportunity to “simultaneously collect both quantitative and qualitative data, merge the data, and use the results to understand a research problem...{O}ne data collection form supplies strengths to offset the weaknesses of the other form.”  (Cresswell, 2005. p. 514)  Regrettably, the researchers do not describe the format used in the log .

Because of the small sample, a t-test was appropriately used on the CRT and ATSS .  The researchers provided the data on the t-tests.  The t-value (10) on the CRT was 5.52 with a p<.001, a statistically significant result.  In addition to an overall evaluation of the ATSS ( t-value(10) 2.68, p=.023, statistically significant) The ATSS was divided into two parts for two further t-tests: a Science Motivation Scale (t-value(10) 2.90, p=.016, statistically significant), and a Science Importance Scale (t-value(10) 1.484, p=.168, not statistically significant.)  The hypotheses apply to the current sample; such a claim cannot be made about a much larger sample, which detracts from the usefulness of the study.  Additionally, it should be noted that the posttest mean for the CRT jumped to 8.64 (SD 2.20) from the pretest mean of 3.0 (SD 2.10).  However, one score went up 13 points, and two others 8 points.  A fourth gained 7 points, while all other scores showed increases of 4 points or less.  Once again, the small sample creates problems for interpreting the overall success of the research study.

In summary, several areas create concern about the validity of this research project, and the subsequent report.  As noted earlier, the sample size, and its composition make extrapolation of results difficult.  The initial coefficient alphas are troubling, and while this critique can comment on the small sample size, it seems self-defeating for the researchers to use it to explain the low coefficient alpha values. The lack of information about research processes, specifically the rationale for reading the pre- and posttest instructions and questions, and the logging procedure (and the log’s subsequent “validation” of the quantitative elements of the study) is less than ideal.  The only data presented in table form are the pre- and posttest results from the CRT, and while the t-test results are supplied, the researchers do not provide an in-depth evaluation of their own.  Another question centers around the time span of the intervention; general curriculum studies in regular classrooms use nine weeks or more to complete units of study.  This project cuts that time span nearly in half.  While this study may present an element of hope to science teachers and those who employ science majors, the study is limited to the specific individuals involved in the study, and any type of educational enhancement might have achieved the same results.  As both researchers are educators, it would appear that at least philosophically their desire is for the study to produce positive results.  Unfortunately, the elements of concern combined cast doubt on the studied procedure as the only answer to the research questions. While the results support the hypotheses, there are enough concerns to warrant further research study in this area.  

To References