Manuscript Details

Pfannenstiel, J. (2015). Evaluation of the i3 validation of improving education outcomes for American Indian children. Unpublished manuscript. Overland Park, KS: Research & Training Associates, Inc.

Moderate rating
This manuscript received a rating of moderate because it is a randomized-controlled trial with high-attrition that satisfies the baseline equivalence requirement.

Study reviewed under: Handbook of Procedures and Standards, Version 1
Child Development and School Readiness
Outcome Measure Timing of Follow-Up Rating Direction of Effect Effect Size (Absolute Value) Stastical Significance Sample Size Sample Description
Boehm Test of Basic Concepts -3 Preschool, Total Score 3 years Moderate
0.34 Not statistically significant, p = 0.09 67 families RCT: BabyFACE vs. comparision
Boehm Test of Basic Concepts-3 Preschool - Total Score 3 years Moderate
Not available Not statistically significant, p=0.050 67 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Attachment 2 years Moderate
0.07 Not statistically significant, p = 0.69 66 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Attachment 3 years Moderate
0.15 Not statistically significant, p = 0.26 68 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Behavioral concerns 3 years Moderate
Not available Statistically significant, p = 0.03 68 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Behavioral concerns 2 years Moderate
0.19 Not statistically significant, p = 0.22 66 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Behavioral concerns 3 years Moderate
0.47 Not statistically significant, p = 0.56 68 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Initiative 2 years Moderate
0.27 Not statistically significant, p = 0.56 66 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Initiative 3 years Moderate
1.04 Not statistically significant, p = 0.19 68 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Self-control 2 years Moderate
0.38 Not statistically significant, p = 0.82 66 families RCT: BabyFACE vs. comparision
Devereux Early Childhood Assessment (DECA) - Self-control 3 years Moderate
0.10 Not statistically significant, p = 0.35 68 families RCT: BabyFACE vs. comparision
Effect rating key
Favorable finding / Statistically significant
Unfavorable finding / Statistically significant
Ambiguous finding / Statistically significant
No effect / Not statistically significant
Positive Parenting Practices
Outcome Measure Timing of Follow-Up Rating Direction of Effect Effect Size (Absolute Value) Stastical Significance Sample Size Sample Description
Frequency of home literacy activity 3 years Moderate
0.64 Statistically significant, p = 0.01 67 families RCT: BabyFACE vs. comparision
Hours parent reads to child 3 years Moderate
0.49 Statistically significant, p = 0.04 67 families RCT: BabyFACE vs. comparision
Protective Factors Survey - Nurturing and Attachment 2 years Moderate
0.08 Not statistically significant, p = 0.55 66 families RCT: BabyFACE vs. comparision
Protective Factors Survey - Nurturing and Attachment 3 years Moderate
0.13 Not statistically significant, p = 0.56 68 families RCT: BabyFACE vs. comparision
Protective Factors Survey - Parent's Child Development Knowledge 3 years Moderate
0.27 Not statistically significant, p = 0.13 68 families RCT: BabyFACE vs. comparision
Protective Factors Survey - Parent's Child Development Knowledge 2 years Moderate
0.16 Not statistically significant, p = 0.21 66 families RCT: BabyFACE vs. comparision
Effect rating key
Favorable finding / Statistically significant
Unfavorable finding / Statistically significant
Ambiguous finding / Statistically significant
No effect / Not statistically significant

This study included participants with the following characteristics at enrollment:

Race/Ethnicity

The race and ethnicity categories may sum to more than 100 percent if Hispanic ethnicity was reported separately or respondents could select two or more race or ethnicity categories.

American Indian or Alaska Native
100%

Maternal Education

Less than a high school diploma
24%
High school diploma or GED
35%
Some college or Associate's degree
34%
Bachelor's degree or higher
5%
Unknown
3%

Other Characteristics

Indigenous population
100%
Enrollment in means-tested programs
73%

This study included participants from the following locations:

  • State not reported or international
Study Participants
  • The study sample included families from one Bureau of Indian Education school catchment area taking part in a randomized controlled trial. A total of 129 families were randomly assigned in this area, with 63 families assigned to the home visiting intervention group and 66 assigned to the comparison group.
  • Seventy-three percent of families in the study received Supplemental Nutrition Assistance Program benefits. Twenty-four percent of mothers had completed less than a 12th grade education, 37 percent had a high school diploma or GED, and 39 percent had some college or a degree. 
Setting

The study took place in a single, unnamed Bureau of Indian Education school catchment area.

Intervention condition
Comparison Conditions

The comparison group was eligible to receive existing business-as-usual services that were available to them within their communities.

Subgroups examined

• Maternal education (high education, level undefined) • Household poverty status (below poverty threshold) • Intergenerational household (yes or no)

Author Affiliation

Study authors are not model developers or distributors. They were contracted by the implementers to conduct an independent evaluation of the home visiting intervention.

Funding Sources

Institute of Education Sciences (IES) U.S. Department of Education, Grant #U396B100189.

Study design characteristics contributing to rating
Design Attrition Baseline equivalence Confounding factors? Valid, reliable measures?
Randomized controlled trial High

Established on race/ethnicity and SES; outcomes not feasible to assess at baseline

None

Not assessed in manuscripts reviewed under Handbook of Procedures and Standards, Version 1
Notes from the review of this manuscript

This study reports findings for both a RCT and a quasi-experimental design (QED). In the RCT, outcomes that were not assessable at baseline received a moderate rating. Outcomes that were assessable at baseline received a low rating because the study did not establish baseline equivalence for these outcomes.

In the QED, all outcomes received a low rating because treatment and comparison groups were not enrolled in the study at the same time, creating a confounding factor.