Frequently Asked Questions
How does a model get on the HomVEE list?
The answer to this question has two parts: (1) how a model is selected for review; and (2) whether HomVEE determines a model meets the criteria established by HHS for an “evidence-based early childhood home visiting service delivery model.”
HomVEE uses a systematic process to select models for review. To identify new research, HomVEE conducts a literature search and issues a call for research. Using the identified research, models are then assigned points based on the (1) number and design of eligible impact studies, (2) sample sizes and sample characteristics of the impact studies, and (3) characteristics of the model. (For prioritization points, the sample size refers to the total number of participants in both the treatment and comparison conditions, and the largest analytic sample size reported in the manuscript being reviewed after any attrition.) Models are ranked by accumulated points and those with higher rankings are selected for review. For more information on the review process, please see Chapter 2 of the Version 2 Handbook of Procedures and Evidence Standards located on the Methods and Standards section of the website.
After a model is reviewed, HomVEE assesses whether it meets the HHS criteria for an evidence-based model. Regardless of whether the model meets the HHS criteria, information about each reviewed model can be accessed on the HomVEE website via the Model Effectiveness Research Reports section. A list of models HomVEE has reviewed, and whether they meet the HHS criteria, is available in the Home Visiting Model Effects Table.
How does HomVEE screen research for review?
HomVEE screens research to identify which manuscripts are eligible for review according to HomVEE’s published procedures and standards. Once manuscripts are screened in, manuscripts about impact studies contribute to prioritization rankings (implementation research does not contribute prioritization points). Research on models that are not prioritized for review in a given year remains in the HomVEE database for consideration in future years.
HomVEE starts screening research every October and continues through early January of the following year. HomVEE conducts a broad annual search for research on home visiting models serving pregnant women or families with children from birth to kindergarten entry (that is, up through age 5) and carefully screens the resulting manuscripts for eligibility. Specific screening criteria can be found in Chapter 2 of the Version 2 Handbook of Procedures and Evidence Standards located on the Methods and Standards section of the website.
The annual review is HomVEE’s main product, but it is not the only one. HomVEE may refine the screening approach as appropriate for specific stand-alone products, such as the review of research on tribal populations and the Evidence Says series of briefs on topics of special interest.
May I submit research on my model for review?
Yes, HomVEE issues a call for research every year in November that is open through early January. The call is sent to relevant electronic mailing lists and is also posted on the HomVEE website homepage with instructions on how to submit research. Research can be submitted at any time, but if it is submitted outside of the call window it will be screened the following year.
HomVEE aims to treat all models consistently and fairly and to have a complete picture of all observed effects for a model. Therefore, HomVEE has a systematic process for identifying all relevant research and prioritizing models for review based on the available empirical evidence. Accepting new information, studies, or research during the course of the review would significantly delay HomVEE’s annual review process. For these reasons, HomVEE cannot review specific manuscripts on request. More details on this process are available in Chapter 2 of the Version 2 Handbook of Procedures and Evidence Standards Methods and Standards section of the website.
Please sign up for the HomVEE mailing list to be notified of the call for research and other HomVEE-related news. To join the email list, find the "Join the HomVEE email list" box on the bottom right side of the Home page or the top left side of the What's New page, enter your email address and click the Subscribe button.
May I appeal the review results?
You may request a re-review if you think that (1) in applying the criteria to determine whether a particular model is evidence based, HomVEE made one or more errors; and (2) if these errors were addressed, the model would be evidence based. Specifically, you may request reconsideration of the evidence-based determination based on misapplication of the HHS criteria, missing information, or errors on the HomVEE website. These concerns should be described in detail and submitted to: HomVEE@acf.hhs.gov. Pending approval from HHS, an independent re-review team will conduct a new review of the manuscript(s) in question. Re-reviews are conducted using the procedures and standards in place at the time of the original review. For more information on the re-review process, see Chapter 2 of the Version 2 Handbook of Procedures and Evidence Standards on the Methods and Standards section of the website.
How long does it take to re-review models that have submitted an appeal?
HomVEE aims to issue a final decision as to whether the standards were accurately applied within 60 days of the submission of the request for review. All requests must be submitted to HomVEE@acf.hhs.gov. Once a decision is made, the requester will be notified and any necessary adjustments will be made to reports on the HomVEE website.
Are models on the HomVEE list automatically eligible for MIECHV funding?
A HomVEE designation as an evidence-based model does not guarantee that a model is eligible to be implemented with MIECHV funding. To be eligible for implementation as an evidence-based model with MIECHV funding, a model must both meet HHS criteria for evidence of effectiveness (as determined by HomVEE) and meet all other statutory requirements for model eligibility (as determined by the Health Resources and Services Administration [HRSA] at HHS). In addition, MIECHV’s authorizing statute allows awardees to utilize a portion of their MIECHV funding for a model that qualifies as a promising approach. For additional information on the MIECHV Program, see the HRSA website
Which study designs and analyses are eligible for review?
HomVEE considers two types of study designs to be reliable for answering the question of whether an early childhood home visiting model impacts the outcomes of children and families: (1) randomized controlled trials (RCTs) and (2) three quasi-experimental designs (QEDs), including, single-case designs (SCDs), regression discontinuity designs (RDDs) and non-experimental comparison group designs (NEDs), which are described in more detail below.
Why can research with some designs rate higher than research with other designs?
Study designs vary in their relative risk of bias – that is, they vary in the extent to which researchers can be confident that impacts are due to home visiting rather than other factors. HomVEE accounts for risk of bias when assigning ratings to research. Research with a higher risk of bias receives a lower rating. For more details, please see Chapter 3 of the Version 2 Handbook of Procedures and Evidence Standards located on the Methods and Standards section of the website. HomVEE routinely contacts authors if additional details are needed to assess the appropriate rating to assign to the manuscript.
What types of analyses or contrasts are not eligible for review?
HomVEE focuses its resources on reviewing manuscripts about impact studies that answer the review’s core question of whether an early childhood home visiting model is effective. Research that answers other questions is not eligible for review by HomVEE. For example, questions about for whom and under what circumstances a model is effective are valuable to the field but beyond the scope of the HomVEE review.
Research evaluating the impact of an isolated model feature or group of features is generally ineligible for HomVEE’s annual review. Knowing that a certain feature of a model is effective in isolation does not establish that a model consisting of multiple features (including the effective one) is effective overall. Therefore, research on single features should not contribute to the evidence base for an entire model.
Analyses designed to answer questions other than whether a model is effective are generally ineligible for review. Once HomVEE has confirmed that a manuscript reports on a study that uses an eligible design, a reviewer examines whether any analysis in the manuscript uses an eligible analytic approach. Specifically, two types of approaches are generally ineligible for review by HomVEE: most mediating and moderating analyses and all analyses that control for endogenous characteristics. A longer discussion of these types of analyses and exceptions in which HomVEE would review them can be found in Chapter 3 of the Version 2 Handbook of Procedures and Evidence Standards located on the Methods and Standards section of the website.
HomVEE may explore questions of which models are effective and for whom in other products beyond the annual review, such as Evidence Says briefs on special topics and HomVEE’s review of research on tribal populations.
Does HomVEE provide guidelines authors can refer to when writing up their research findings?
HomVEE’s Reporting Guide for Authors, provides guidance about how to describe each eligible research design and how to report findings in a way that is clear, complete, and transparent. Reporting the information described in the Reporting Guide for Authors is considered a best practice in general, but it can also help HomVEE reviewers assess the appropriate rating to assign to the manuscript.
How does HomVEE define an early childhood home visiting model?
For the purposes of the HomVEE review, an early childhood home visiting model is an intervention in which trained home visitors meet with expectant parents or families with young children to deliver a specified set of services through a specified set of interactions. These programs are voluntary interventions that are either designed or adapted and tested for delivery in the home. During the visits, home visitors aim to build strong, positive relationships with families to improve child and family outcomes. Services may be delivered on a schedule that is defined or can be tailored to meet family needs. A model has a set of fidelity standards that describe how the model is to be implemented.
Models reviewed by HomVEE must serve pregnant women or families with children from birth to kindergarten entry (that is, through age 5), and the primary service delivery strategy must be home visiting. In addition, the model must have research that examines its effects in at least one of eight outcome domains.
Are maternal/infant health home visiting models, not just early childhood ones, included in the review?
Yes, any model that meets the definition of the early childhood home visiting model is included in the review, and this may include models that enroll women during pregnancy and aim to improve maternal health, infant health, and birth outcomes. Detailed information about the theoretical approach, intended population, and targeted outcomes can be found on the model overview page of each implementation profile. Profiles can be accessed through the Model Implementation Menu on the HomVEE website. Detailed information about maternal and child health outcomes can be found in the Effects shown in research tab of each model effectiveness research report. That page also has a search function that allows the user to filter the models by target population. The reports can be accessed through the Model Effectiveness Menu on the HomVEE website.
How are models prioritized and reviewed by HomVEE?
Models are eligible for review as long as they meet HomVEE’s criteria for inclusion. HomVEE will select models to review by calculating a prioritization score for each model using a combination of model and manuscript characteristics. Prioritization scoring assigns points based on various factors, including (1) number and design of eligible impact studies, (2) sample sizes and sample characteristics of the impact studies, and (3) characteristics of the model. (In prioritization scoring, the sample size refers to the total number of participants in both the treatment and comparison conditions, and the largest analytic sample size reported in the manuscript being reviewed after any attrition. Note that when HomVEE prioritizes a model, the review includes both the model and related versions.) After a model is selected for review, HomVEE will review all previously unreviewed research on the model. See the Version 2 Handbook of Procedures and Evidence Standards located on the Methods and Standards section of the website for more information on these processes.
How does HomVEE treat virtual and telehealth interventions and adaptations?
HomVEE is working with federal agencies and other stakeholders to develop guidelines for HomVEE’s review of research on virtual and telehealth interventions and adaptations, including model adaptations made in response to the COVID-19 pandemic. Pending finalization of these guidelines, HomVEE is collecting and holding manuscripts about interventions that are purely or primarily virtual.
HHS criteria for evidence-based models
What are the HHS criteria for evidence-based models?
HomVEE uses the term “evidence-based model” to refer specifically to a model that meets HHS criteria developed based on statutory requirements in the authorizing legislation for the MIECHV Program. HomVEE recognizes that other systematic reviews may use different criteria to evaluate evidence of effectiveness. Thus, an evidence-based model in the context of HomVEE might or might not meet requirements for evidence of effectiveness according to other systematic reviews. The HHS criteria for an evidence-based early childhood home visiting service delivery model are designed to ensure confidence in model impacts through replicated findings from well-designed research.
How does HomVEE apply HHS criteria for evidence-based early childhood service delivery models?
HomVEE assesses whether each model meets the HHS criteria for an “evidence-based early childhood service delivery model.” To operationalize the HHS criteria related to studies, and because study findings may be reported across several manuscripts, HomVEE rates manuscripts based on the highest rated finding reported in that manuscript. Any high- or moderate-rated finding from a study about a model is considered as part of the evidence base for that model. Notably, for models with research solely from either RCT or SCD studies, additional criteria apply (see the HHS criteria). The additional criteria for RCTs to be from peer-reviewed journal articles and to have sustained findings align with MIECHV Program statutory requirements. The HHS criteria for an evidence-based model have no additional requirements for RDD or NED studies. A model may be evidence based on the strength of subgroup findings alone only if the research about it satisfies all of the subgroup criteria.
Is a model required to have evidence from an RCT in order to be evidence-based according to HHS criteria?
No. A model may be evidence-based, according to HHS criteria, on the strength of either RCT research or QED research (including non-experimental comparison group, single-case, and regression discontinuity designs). The research must be of moderate or high quality based on HomVEE’s published standards. Additional HHS criteria apply when the research uses an RCT design. For details, please see the HHS criteria for evidence-based early childhood home visiting service delivery models.
Where can I find a list of models that meet the HHS criteria for evidence-based models?
A full list of models reviewed by HomVEE, and whether they meet the HHS criteria, is available in the Model Effects Summary Table. The information is also available in the summary brief, available on the HomVEE Summary Page. Notably, a HomVEE designation as an evidence-based model does not guarantee that a model is eligible to be implemented with MIECHV funding (see above, “Are models on the HomVEE list automatically eligible for MIECHV funding?”).
Is there any consideration given to those models that may not demonstrate a large impact because the sample members are difficult to treat (for example, parents with severe mental illness)?
The HHS criteria do not consider the magnitude of impacts in determining whether a model has evidence of effectiveness.
Even though magnitude is not considered in the HHS criteria, HomVEE reports standard effect sizes for findings rated high and moderate when they were available in the manuscripts or the HomVEE team had enough information to calculate them. Effect sizes can be found in the Effects shown in research section of each model effectiveness research report. The reports can be accessed through the Model Effectiveness Menu.
Information available on the website
Is there a list of models reviewed but that did not meet the requirements?
Is information on implementation of models available?
Information on implementation of models is available in the implementation profiles. The implementation profiles give an overview of the model and information about developers’ input on the following: prerequisites for implementation, training, materials and forms, program costs, and model contact information. These profiles can be accessed on the Model Implementation Summary Report page.
Does HomVEE categorize research that is in progress on these models?
No. HomVEE does not have consistent information about research underway on home visiting models. Stakeholders seeking information about research in progress on specific models should contact the model developer directly.
Study sample diversity
What is the racial/ethnic and income diversity of study samples in the HomVEE review?
Overall, the manuscripts about high- and moderate-quality impact studies included in the HomVEE review have diverse study samples that include families from a range of racial/ethnic groups and families with low incomes. The majority of manuscripts include families from multiple racial/ethnic groups and most of the samples include families with low incomes. More information on study characteristics can be found for each high- and moderate-rated manuscript about a model, via the Information about research reviewed tab in any Model Effectiveness Research Report.
Does HomVEE provide information on the effectiveness of early childhood home visiting models for specific racial/ethnic groups?
Although study samples included in the HomVEE review are quite diverse, most of the manuscripts about high- and moderate-quality impact studies aggregate the results across racial/ethnic groups. Thus, the available evidence generally does not reveal whether these models are more or less effective with one group or another.
Further, the sample sizes of most studies may not provide sufficient statistical power to detect effects for specific racial/ethnic subgroups. In other words, even if there are sizeable effects of the intervention for a particular racial/ethnic group, these may not be detected because of small sample sizes for that subgroup.
Because HomVEE’s mission is to identify evidence-based models, and HHS’ criteria require replication before subgroup research can be part of the evidence base, the annual review only reports findings from impact research on replicated subgroups. Replication provides more confidence that findings were not observed by chance. HomVEE uses the same set of standards to rate the quality of research for all findings on a replicated subgroup. For example, if the subgroup replication straddled the timing of an update to HomVEE standards, HomVEE would review both sets of subgroup findings using the newest standards (even if other, full-sample findings from the older manuscript were reviewed using older HomVEE standards).
HomVEE recognizes that the effects of models for different populations is of interest to the home visiting field. Although HomVEE only reports findings for replicated subgroups, HomVEE lists the subgroups examined in each manuscript beginning with its 2021 annual review. HomVEE does not report details on the findings or the quality of the study design until the subgroup is replicated.
More information on study characteristics, including the race/ethnicity of a study sample, can be found for each manuscript about high- or moderate-rated study on a model via the Information about research reviewed tab in any Model Effectiveness Research Report. If a model has been tested with a replicated subgroup, HomVEE summarizes the subgroup findings as part of that model’s effectiveness research report.
Are there any studies specific to American Indian/Alaska Native populations?
In addition to the main HomVEE review, HomVEE periodically updates a review of models that were implemented in tribal communities or included substantial proportions of American Indian Alaska Native families in the study samples. The results of this review are presented in a two-volume report and related research briefs. These reports can be accessed on the Tribal Home Visiting page.
Were studies in the HomVEE review conducted in both rural and urban communities?
Several of the models that meet the HHS criteria for an “evidence-based early childhood home visiting service delivery model” have been studied with families in a mix of urban and rural communities. Results for families from rural and urban areas are generally aggregated across groups. More information on study characteristics, including the setting, can be found for each manuscript about high- or moderate-rated study on a model via the Information about research reviewed tab in any Model Effectiveness Research Report.
What is a single-case design (SCD)?
Studies that use a single-case design (SCD) measure outcomes for cases (such as a child or family) repeatedly during multiple phases of a study to determine the success of an intervention. The number of phases in the study will depend on the research questions, intervention, and outcome(s) of interest. For the Home Visiting Evidence of Effectiveness (HomVEE) review, the interventions of interest are early childhood home visiting models, and outcomes of interest must fall within at least one of eight domains.
SCDs rigorously test the effectiveness of an early childhood home visiting model using a relatively small sample. Researchers can compare (a) outcomes measured when the child or family is receiving services from the home visiting model with (b) outcomes measured before and after the child or family receives services from the home visiting model. This allows each child or family to serve as its own comparison. This feature of SCDs enables researchers to isolate the impact of the home visiting model on the outcome of interest from changes that could occur naturally because of child development or for other reasons.
How does HomVEE interpret findings from SCDs?
Whenever possible, the Home Visiting Evidence of Effectiveness review (HomVEE) will use SCD findings across cases to calculate a design-comparable effect size (D-CES). The D-CES produces a measure of the difference between the intervention and comparison conditions. An effect size is a standardized measure of magnitude that researchers can calculate for group design studies, in which a group of participants is offered home visiting services and a comparison group of participants does not receive home visiting services (HomVEE uses Hedges’ g to calculate effect size); researchers cannot directly compute effect sizes for SCD research. However, a D-CES produces an effect size that is comparable to the effect size statistic that HomVEE uses for group design studies (such as randomized controlled trials [RCTs] and non-experimental group designs [NEDs]). Producing this calculation will enable HomVEE to use findings from both SCD and group designs to characterize the magnitude of an effect of an intervention. To date, researchers can only calculate a D-CES for treatment reversal/withdrawal designs and multiple baseline/multiple probe designs across at least three cases. Because of methodological limitations, researchers cannot calculate a D-CES for changing criterion designs, multiple baseline/multiple probe designs across settings, or alternating treatment designs. As calculations for these types of designs become available in the methodological literature, HomVEE might calculate the D-CES for these designs in the future.
How does HomVEE apply the U.S. Department of Health and Human Services (HHS) criteria to SCDs? Can these study designs contribute to the summary of evidence for a particular model?
HomVEE will apply the HHS criteria to SCDs in the same way it applies the criteria to other group designs. If an SCD manuscript receives a rating of moderate or high based on the HomVEE standards, it means the study has at least some (and perhaps strong) internal validity—that is, evidence shows the effects researchers reported are attributable, at least partly, to the home visiting model examined. This is the same definition HomVEE uses for manuscripts about group design studies rated moderate or high. HomVEE can confidently add SCD research to the summary of evidence with well-designed research from group design studies because the moderate or high rating indicates a similar level of internal validity across research designs, according to the HomVEE standards.
Are SCDs as rigorous a study design as RCTs or NEDs?
Yes, they can be. HomVEE developed review standards for SCDs (as it has for group designs) to ensure that research contributing to a model’s summary of evidence has a sufficiently rigorous design to confirm that the findings are attributable, at least partly, to the home visiting model the researchers examined. The standards ensure that the researchers were responsible for manipulating the intervention (that is, exposure to the home visiting model), more than one assessor systematically measured the outcomes over time, and researchers collected a sufficient number of data points across a sufficient number of phase changes. When a manuscript about an SCD receives a rating of moderate or high, it is deemed to have the same level of internal validity as research about group design studies that achieve the same rating.
Can researchers use SCDs to address research questions related to child development? How do SCDs differentiate between changes resulting from development and changes resulting from an intervention?
Researchers can use SCDs to measure outcomes related to child development even though the study design does not have a separate comparison group. That is, there are features of SCD research that provide comparison conditions, so researchers can determine whether the effects they observe are a result of an intervention rather than expected child development over time. For example, in multiple baseline designs, researchers collect repeated measurements for multiple cases during the same period, starting with baseline phases (before the intervention). They then introduce the intervention to one case at a time in a staggered manner, while repeatedly measuring outcomes. Rigorous multiple baseline design studies must collect repeated measurements across conditions for at least three cases, staggering introduction of the intervention at a minimum of three different time points. This way, any observed changes in outcomes can be attributed to the intervention, as opposed to maturation or developmental changes. Similarly, in treatment reversal/withdrawal designs, the researchers alternate between intervention and non-intervention phases across time. The non-intervention phases occur before and after the intervention; together, they serve as a comparison condition. See the “Single-Case Design Research Methods” brief on the HomVEE website for more information.
Does it matter to the evidence base that SCD research typically has far fewer participants than other types of effectiveness research?
Well-designed SCD research can help researchers learn whether the home visiting model is effective based on a small number of participants. Well-designed RCT and NED research can help researchers learn whether a model is effective based on small or large numbers of participants (study sizes vary).
HomVEE assesses the evidence of effectiveness for early childhood home visiting models. That is, it focuses on the question: Does the model work? A separate, interesting line of inquiry explores for whom the model works (and whether someone implementing it outside its tested setting would get similar results). Researchers typically refer to this concept as external validity or generalizability; it is related to the number of participants in the research and the diversity among those participants. HomVEE acknowledges that the generalizability of findings is useful for the field, yet addressing that topic is not the primary focus of the HomVEE review process.
Even so, SCDs can sometimes address concerns related to external validity. The particulars of which research participants responded to an intervention under which conditions might be obscured when reporting only group means and associated effect sizes from a large group design study (Horner et al. 2005). When using SCDs, researchers can provide detailed documentation of the characteristics of those cases that responded to an intervention and those that did not.
Horner, Robert H., Edward G. Carr, James Halle, Gail McGee, Samuel Odom, and Mark Wolery. “The Use of Single-Subject Research to Identify Evidence-Based Practice in Special Education.” Exceptional Children, vol. 71, no.2, 2005, pp.165–179.
Can researchers use SCDs to evaluate adaptations of a home visiting model?
Yes, researchers can use SCDs to explore model adaptations and innovations and compare various modes of delivery (for example, virtual versus in person). However, these research questions are not the focus of HomVEE’s assessment of whether a model is effective. HomVEE will only review SCD research on a model version if it meets the definition of eligible research for review outlined in the HomVEE procedures. See the HomVEE Handbook of Procedures and Evidence Standards for more information.