Invitation for public comment

In the coming months, HomVEE will release Version 2.3 of its Handbook of Procedures and Standards. In alignment with our commitment to a rigorous and transparent review process, HomVEE invites public comment on these key proposed changes.

Please submit comments via email to HomVEE@acf.hhs.gov no later than Friday, July 12, 2024.

Proposed changes to HomVEE Handbook of Procedures and Standards (Version 2.3)

HomVEE is proposing new procedures and standards related to model adaptations and enhancements in the forthcoming Version 2.3 Handbook. It is seeking public comment about these changes, described below.

In addition, the updated handbook will include updates to reliability standards for findings to earn a rating of high or moderate. HomVEE is providing advance notice of those changes below.

Request for public comment on HomVEE approach to model adaptations

Background

HomVEE’s mission is to assess which early childhood home visiting models meet the U.S. Department of Health and Human Services’ criteria for an evidence-based early childhood home visiting model. To achieve this goal, HomVEE must synthesize findings from well-designed research on the same model to determine that model’s effectiveness. 

Models are constantly evolving to meet the changing needs of families, including to serve new populations or deliver new content. Models also change in response to lessons learned through continuous quality improvement. With its proposed procedures and standards, HomVEE is striving to find a solution that does not impede model evolution or continuous quality improvement.

To correctly group research about the same model when assessing effectiveness, and to allow space for continuous model growth and development, HomVEE must establish a process for defining and identifying adapted and enhanced models.

Summary of proposed approach

HomVEE’s proposed approach, described in greater detail below, will generally be to confer with model developers to determine whether research about a change to the model should be reviewed separately as an adapted model or grouped together with other research on the model.

This request for public comment proposes definitions for adapted model, enhanced model, and related terms; it also proposes how HomVEE will review and report research on adapted and enhanced models. HomVEE invites public comment on these proposed updates, including new ideas and suggestions for alternative approaches to identifying adapted and enhanced models.

Proposed definitions of key terms

HomVEE proposes the following definitions for use in the review:

  • Core components are the components of a model that, according to that model’s developer, are critical to achieving the model’s intended outcomes. 
  • Fidelity standards are requirements developers create to document how to operationalize their model’s self-defined core components. They are designed to ensure implementation achieves a model’s intended outcomes. Fidelity standards may allow for flexibility to meet the needs of families or communities.

Models may have different terminology, structure, and content for their fidelity standards. Fidelity standards may change over time in ways that do not alter a model’s core components. However, when a model’s core components do change, HomVEE assumes this would be reflected in changes to fidelity standards.

  • An adapted model, incorporates an intentional change that fundamentally alters a model’s core components. This change (an adaptation) is characterized by a significant change to the model’s fidelity standards. The model’s developer determines whether the change is significant.

Models may differ in which components the developer defines as core and in the flexibility that the developer permits. Intentional changes that fall within a model’s defined areas for flexibility are not adaptations. For example, when a model allows local providers to select a curriculum, different curriculum choices are not model adaptations. 

  • An enhanced model, incorporates an intentional change that results in either (1) no changes to a model’s fidelity standards or (2) changes that are not significant enough to alter the model’s core components. HomVEE would characterize either of these types of changes as an enhancement. The model’s developer determines whether the change is significant.
  • A referent model is the model for which HomVEE has eligible research that qualifies for reporting. It is the focus of a HomVEE report and is assessed with the HHS criteria to determine whether it qualifies as an evidence-based early childhood home visiting model. Occasionally, as discussed later on this page, HomVEE may treat a grouping of models as a combined referent model.

The following illustrative example describes changes to a fictional early childhood home visiting model called Healthy Launch, to demonstrate what might constitute an adapted or enhanced model. 

Description of referent model: Fictional example

Healthy Launch serves children up to age 36 months. Its core components include coaching parents of infants and toddlers to improve outcomes related to language development and child-caregiver relationships. 

Illustrative example of adapted model: Healthy Launch Pre-K

Healthy Launch Pre-K is an adapted model that focuses on children ages 4 to 5 years old. This adaptation adds a new focus on school readiness outcomes. The new focus on school readiness outcomes alters the model’s core components. As a result, the developer makes substantial changes to several fidelity standards. The developer indicates that this is an adapted model because these changes significantly alter the model’s core components.

Illustrative example of enhanced model 

Healthy Launch adds an element to increase the coaching parents receive by having them submit videos of their child-parent interactions. This change builds on extensive coaching that already exists as part of the model’s core components. As a result, the developer makes minor changes to existing fidelity standards related to coaching. The developer indicates that this is an enhanced model because it does not significantly alter the model’s core components. 

Proposed approach to applying new definitions

Typically, HomVEE’s approach to applying these new definitions will be to defer to the developer’s classification of adapted and enhanced models when grouping the research under review, because HomVEE recognizes the essential role that developers play in translating evidence-based interventions (Freire et al., 2015).

When reviewing research about a model:

  • HomVEE will consult the developer of the model to understand whether research under review is about a referent model (including any enhanced model) or an adapted model. This consultation could happen for up to three reasons. HomVEE might:
    1. Ask developers to send a list of enhanced and adapted models related to the developer’s referent model. HomVEE will ask this of developers for all models it reviews. For this first request, HomVEE will alert developers at least two weeks in advance that a request about their model is coming and will allow the developer four weeks to respond once that request arrives.
    2. Determine if new research is about the referent model, or an adapted or enhanced model. HomVEE would contact the developer about this as the review progresses and only if the need arises.
    3. Request a copy of the document(s) the developer uses as fidelity standards. HomVEE will only make this request if needed to review a developer’s classification (see next section on proposed exceptions). The review team will keep copies confidential but may consult the fidelity standards if questions arise during the review.
  • HomVEE will review research about an enhanced model as part of the evidence base for the referent model. Where necessary for clarity, HomVEE reports will explain the extent to which a model’s evidence base relies on research from the referent model, any enhanced model, or both. In some cases, an enhanced model may be labeled as the referent model. For example:
    • If the only eligible research about the model is on an enhanced model
    • If the only well-designed research about the model is on an enhanced model
    • If the only research that allows the model to meet HHS criteria is on an enhanced model
  • HomVEE will continue to report research about adapted models separately, and assess whether an adapted model is evidence based using only research about the adapted model. An adapted model can be evidence based independently from a referent model.

Proposed exceptions to general approach

As described above, HomVEE will generally defer to a model developer as to whether or not something is an adapted or enhanced model. Rarely, HomVEE might revisit the developer’s classification. To be consistent and systematic in identifying these rare cases, HomVEE proposes that the following types of changes would warrant a closer look if the developer classifies them as nonsignificant (model enhancements), because these changes are especially likely to require independent evidence of effectiveness (as a model adaptation): 

  1. Changes that alter intended outcome domains; 
  2. Changes in the intended service population; 
  3. Changes in staff qualifications; and 
  4. Changes that add, eliminate, or substantially reduce a component that HomVEE leadership believe may be core to achieving outcomes. 

HomVEE reserves the right to look more closely at other types of changes as the field evolves. Therefore, the list above covers most anticipated cases but is not intended to be exhaustive. HomVEE may also more closely review other changes if project leadership have concerns that a change may not be reasonable to characterize as an enhancement and instead may be an adaptation. 

Process to review a developer’s recommendation 

When HomVEE revisits a developer’s classification of a model’s research, the team will: 

  1. Follow up with the model developer to better understand the nature and rationale of the changes and how they impact core components; 
  2. Carefully review the research, fidelity standards documentation, and correspondence with the developer; 
  3. Consult with federal and external experts for additional perspectives to inform HomVEE’s decision. These experts may include staff from the Administration for Children and Families’ Office of Planning, Research, and Evaluation (OPRE), the Health Resources Services Administration (HRSA), and subject matter experts, including those with backgrounds in home visiting and implementation science 

Based on these efforts, HomVEE might then decide to treat a developer-designated enhanced model as an adapted model, which would mean separate reporting and separate assessment of the evidence base for the adapted model. HomVEE would label its report clearly, describe the evidence base, and indicate whether the developer had characterized the model change differently than the HomVEE team. 

Public notice of updates to reliability standard 

The Version 2.3 Handbook update will revise HomVEE’s reliability standard with the following changes: 

  1. Updating HomVEE’s list of measures that are assumed to be reliable to include self-reported, single-item measures of counts or dichotomous events—for example, the number of times the parent reads to their child in a day or whether the parent breastfeeds their child 
  2. Clarifying that for measures in the reductions in child maltreatment domain, substantiated reports from administrative records and data from medical records do not require reliability statistics (unsubstantiated reports remain ineligible for review) 
  3. Raising the threshold for internal consistency reliability from 0.50 to 0.70 (for example, for Cronbach’s alpha, Pearson’s r and Spearman’s rho) 
  4. Replacing the current inter-rater reliability threshold of 0.50 with thresholds of 0.70 or higher for inter-rater percent agreement and 0.60 or higher for kappa 

When does HomVEE intend to finalize and apply Version 2.3 changes? 

HomVEE intends to begin applying the changes described on this page with the 2025 review. HomVEE will finalize the revisions and publish an updated handbook on the HomVEE website in late 2024 or early 2025. A summary of the substantial changes will appear at the front of the updated handbook so HomVEE users can easily identify what is new. HomVEE will notify website users by an email blast after it posts the updated handbook. If you would like to be notified when the Version 2.3 handbook is released, please join the HomVEE email list. 

References 

Freire, K. E., Perkinson, L., Morrel‐Samuels, S., & Zimmerman, M.A. (2015). Three Cs of translating evidence‐based programs for youth and families to practice settings. New Directions for Child and Adolescent Development, 2015(149), 25–39.