WP No. WR-A782-1

• Report is also available in Portable Document Format (PDF)

Independent Interim Evaluation of the New York State

1115 Waiver Amendment: The Children´s Design Preliminary Report

Submitted to:

Department of Health
Attention: Nichole Katz
New York State Department of Health
Office of Health Insurance Programs
99 Washington Avenue, Suite 720
Albany, New York 12210

Email: nichole.katz@health.ny.gov

Submitted by:

RAND Corporation
1776 Main Street
Santa Monica, CA 90407
Lisa Wagner
Phone: (703) 413-1100 Ext. 5067

Email: LisaW@rand.org

Submitted on:
November 13, 2020

---

TABLE OF CONTENTS

1. Executive Summary

2. Demonstration Description

Evaluation Objective
Timeline and Progress to Date

3. Evaluation Design and Methods

Goal 1. Effect of Managed Care on HCBS Population Outcomes: Improve the health outcomes for individuals under 21 receiving HCBS (HCBS Child/Youth) with access to the Medicaid managed care delivery system
Goal 2. Effect of Timely Access to Early and Periodic Screening, Diagnostic, and Treatment Benefits: Improved timely access to the additional Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefits that address early behavioral health needs and health needs of children will improve health outcomes and long-term financial savings
Goal 3. Effect of Access to HCBS: Increase appropriate access to the uniform HCBS benefit package for children who meet level of care criteria to achieve improved health outcomes while recognizing that children´s needs, including the duration, scope, and frequency of services, change over time
Goal 5. Effect of Access to Health Home Model: Improve access to the integrated Health Home model for all children to improve the coordination of care for children and increase access to services

4. Findings

5. Policy Implications

6. Interactions with Other State Initiatives

Appendix A. Tentative Evaluation Timeline and Milestones

---

1. EXECUTIVE SUMMARY

As part of ongoing redesign efforts with input from the Medicaid Redesign Team (MRT), New York State (NYS) developed a set of coordinated amendments to its 1115 Medicaid Redesign Team Waiver and its 1915(c) Children's Waivers. Together, these changes are called the Children´s Design. The Children´s Design aims to consolidate and streamline care for children and youth under age 21 who have needs for Behavioral Health (BH) and Home and Community-Based Services (HCBS). The main research goals relevant to this interim evaluation are to assess:¹

• Goal 1. Effect of Managed Care on HCBS Population Outcomes;
• Goal 2. Effect of Timely Access to Early and Periodic Screening, Diagnostic, and Treatment Benefits on health outcomes and long-term financial savings;
• Goal 3. Effect of Access to HCBS on health; and
• Goal 5. Effect of Health Home Model on care coordination and access to services.

Two additional goals, Goals 4 and 6, are not included in the interim evaluation but will be included in the final summative evaluation. To address these research goals, the evaluation team at RAND will conduct approximately 10-15 interviews with key stakeholders of the Children´s Design to understand implementation barriers and successes. The team will supplement these interviews with a review of State policy documents and existing meeting minutes from the Department of Health (DOH) with stakeholders.

In addition, the evaluation team will assess access to and quality of care primarily using baseline data. Post-implementation data are not readily available but will be included to the extent possible. A fee-for-service (FFS) group may be used as the comparison group to those enrolled in mainstream Medicaid managed care (MMMC).

The evaluation design received approval from CMS in April of 2020 when the coronavirus-19 (COVID-19) pandemic started. The significant impact of the pandemic on the NYS health care system required DOH personnel to shift attention, resources, and priorities. This shift created contract execution and data access delays. The independent contractor selection was completed and the evaluation contract was signed in October 2020. As a result, the independent evaluation timeline has changed, and no findings are available at the time of this writing.

Despite COVID-19-related administrative delays, a number of meetings have been held to plan the evaluation and to answer operative questions about the program. After significant scoping efforts, DOH has shared all required data except those on program enrollment and a quality of care measure, which are delayed due to the coronavirus-19 (COVID-19) pandemic. NYS has also shared a list of candidates for interviews, notes for meetings with various stakeholders, and State policy documents. The findings of this interim evaluation are expected to be available in Spring 2021.

---

2. DEMONSTRATION DESCRIPTION

Since 1997, the NYS MRT has worked to create an efficient managed care delivery system that will extend high-quality health care coverage to individuals needing long-term services and supports. The redesign has been updated multiple times, including coordination with the Affordable Care Act Medicaid Expansion and the addition of the Delivery System Reform Incentive Payment (DSRIP) program in 2014. As part of ongoing redesign efforts, NYS proposed, and the Centers for Medicare & Medicaid Services (CMS) approved, concurrent amendments to the 1115 MRT Waiver and the 1915(c) Children´s Waiver that aim to consolidate and streamline care for children and youth under age 21 who have needs for BH services and HCBS. Together, these waiver amendments are called the Children´s Design. Implementation of the Children´s Design started in August 2019 for the following four groups of children who were already covered by the State´s 1915(c) Children´s Waiver:

1. Medically fragile children
2. Children with a behavioral health diagnosis
3. Children with medical fragility and developmental disabilities
4. Children with developmental disabilities who are in foster care

The Children´s Design streamlines the care for these groups of children by authorizing NYS to require enrollment in MMMC for children receiving HCBS under the 1915(c) Children´s Waiver and include children´s HCBS, previously reimbursed through FFS, in managed care organization benefit packages. The Children´s Design also allows NYS to target eligibility to medically needy Family of One (Fo1) children who meet clinical criteria but are not enrolled in the 1915(c) Children´s Waiver.

The streamlined model of care aims to achieve broad improvements in the care that children with behavioral health and HCBS needs receive through the NYS Medicaid system. Specific goals include improved clinical and recovery health outcomes; timely access to health care services during childhood that can improve functioning and reduce health care needs in adulthood; improved integration of care that is commonly fragmented across behavioral health, general medical, and community support systems; and increased capacity of provider networks to deliver community-based recovery-oriented services and supports.

Evaluation Objective

The objective of this evaluation is to examine the early implementation period of the approved Children´s Design. The goals of the interim evaluation are to:

1. Identify the facilitators of and barriers to program implementation
2. Describe and delineate the baseline (i.e., pre-implementation) trends in the outcomes of interest
3. Assess the feasibility of identifying comparison groups and conducting difference-in-differences (DD) analyses or comparative interrupted time series analyses for the final summative evaluation.

Timeline and Progress to Date

The information provided in this report includes information pertaining to the design and implementation of the Children´s Design interim evaluation. Due to contractual delays, all findings and conclusions will be discussed in a subsequent interim report, expected in early 2021. A final summative evaluation will be conducted at a later date.

Revised Timeline and Next Steps

Due to delays, the evaluation timeline was reevaluated to allow for additional time for data collection, analysis, and report writing. The COVID-19 response within the NYS DOH and other state partners, along with other related factors, delayed the execution of the contract, hampering the ability of the evaluation team to begin conducting interviews and to access the data necessary to conduct analyses. As discussed in the methodology below in Section 3, the ability to gather qualitative data and assess the client data is integral to responding to the evaluation questions. Figure 1 in Section 4 below provides the planned timeline for the completion of the interim evaluation. Appendix A outlines next steps, including the client interviews and data analysis.

---

3. EVALUATION DESIGN AND METHODS

To conduct the interim evaluation of the Children´s Design, the evaluation team will use a mixed-methods approach to answer the research questions outlined by NYS. Specifically, the team will conduct semi-structured interviews with various stakeholders to examine implementation barriers and successes. Quantitative assessments of access to and quality of care will use only baseline data, given that the post-implementation data are not readily available. In addition, due to the timing of the interim evaluation, the observation window may not be long enough for the evaluation team to observe outcome changes resulting from the Children´s Design. But, to the extent possible, post-implementation data points, such as the number of individuals enrolled in the Children´s Design or outcome measures that can be derived from the Medicaid Data Warehouse, will be included. The resulting interim report will lay a foundation for the final summative evaluation.

The qualitative interviews will be conducted with a mix of key informants representing diverse stakeholders in the Children´s Design implementation. Informants, who will include representatives of advocacy organizations, plan administrators, and care providers. Drawing on suggestions from DOH, the sampling goal will be to ensure that a broad range of perspectives is represented in the study sample, including diverse advocacy groups and providers from New York City (NYC), as well as both urban and rural regions upstate. The evaluation team anticipates conducting approximately 10-15 key informant interviews. In addition to key informant interviews, the analysis will be informed by review of documents that have been provided to the research team by DOH. The documents include policy documents, which describe how the program was administered, and meeting minutes, which describe public stakeholder meetings at which views of the Children´s Design were discussed.

The interviews and documents will be analyzed by the RAND team to identify issues that have arisen in the course of the implementation of the Children´s Design. For instance, we will ask advocacy organizations whether the implementation has gone according to expectations, whether they have concerns about barriers to successful implementation, and whether there are aspects of the implementation that have been particularly promising. Issues raised by key informants will be summarized and compared across the categories of informants. While the key informant interviews cannot provide definitive information on the impact of the Children´s Design, they can be extremely helpful in identifying common areas of concern. The results will inform the interpretation of the quantitative results and the analytic plan for the summative evaluation report.

To the extent possible, a comparison group will be included in the quantitative assessments of the baseline data for the interim report, which will allow us to compare the demographics, medical acuity, the level of HCBS needs, and outcomes between the program target population and comparison populations. The evaluation team is working with DOH to find the best way to address this challenge.

• Option A is to use children or youth under 21 who are on FFS Medicaid as the comparison. The downside of this comparison is that most individuals are on FFS Medicaid on a temporary basis before they are enrolled in a MMMC plan. In other words, we would be comparing a relatively stable program target population to a comparison population that changes over time.
• Option B is to find a population that has always been in an MMMC plan as the comparison. The concern is that this comparison population is likely to be healthier and have no or a lower level of need for HCBS.
• Option C would consider using data from other states for a population that is similar to that of the Children´s Design, e.g., for outcomes that are part of the Core Set of Children´s Health Care Quality Measures for Medicaid. However, there might be significant variation in how states address such a population´s needs, and the barriers to accessing data would be difficult to overcome within the evaluation timeline.

In part due to a tight timeline for this interim evaluation, aggregate data points for both the target population and the comparison population will be used in the analysis. Depending on specific outcome measures, we will stratify our analyses based on the three subpopulations: HCBS, Health Home Serving Children, and FFS. Given the constraints in the timeline and data, the interim evaluation will be largely descriptive in nature. Although some questions will not be fully addressed in the interim evaluation, this work will provide a foundation for the summative evaluation.

Research Goals and Questions

The research goals for the interim evaluation are illustrated in Table 3.1 below. Note that as outlined in the approved Evaluation Design, Goals 4 and 6 are relevant only to the final summative evaluation, as are some research questions and hypotheses under Goals 1, 2, 3, and 5. These will not be addressed in the interim report and are thus excluded from the table and the summaries that follow.

Table 3.1. Evaluation Goals, Hypotheses, Measures, Data Sources, and Analytic Approaches

Goal	Research Question	Hypothesis	Measure	Data Source	Status
Goal 1. Improve the health outcomes for individuals under 21 receiving HCBS (HCBS Child/Youth) with access to the Medicaid managed care delivery system	1.1 What are the consequences of targeting availability of HCBS to a more narrowly-defined population than the criteria in the State Plan?	1.1.1 Targeting HCBS availability to a more narrowly-defined population will improve the health outcomes of the population most needing supports to remain in the community, as measured by Potentially Preventable Emergency Room Visits (PPVs) and stakeholder observations about the consequences of targeting HCBS availability to a more narrowly-defined population	Implementation barriers and successes; stakeholders´ views of the consequences of targeting availability of HCBS to a narrowly-defined population	Semi-structured key Informant Interviews with advocates, plan administrators, and providers	Protocol in development; state has shared a list of potential candidates for interviews and relevant notes for the meetings with stakeholders
	1.3 To what extent are children with special needs accessing primary care providers who understand the child´s needs?	1.3.1 Parents of children with special needs will report being satisfied with primary care providers´ understanding of their children´s special conditions (CPC-CH questions 44 and 45)	1. Does your child´s personal doctor understand how your child´s medical, behavioral, or other health conditions affect your child´s day-to-day life? 2. Does your child´s personal doctor understand how your child´s medical, behavioral, or other health conditions affect your family´s day-to-day life?	2018 CAHPS CCC survey data	Aggregate data received (10/15)
		1.3.2 Number of children in MMMC/HH/HCBS receiving child/adolescent well-care visits will increase (W15-CH, W34-CH and AWC-CH).	1. W15-CH: Well-child visits in the first 15 months of life 2. W34-CH: Well-child visits in the third, fourth, fifth, and sixth years of life 3. AWC-CH: Adolescent well-care visits	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
Goal 2. Improved timely access to the additional EPSDT benefits that address early behavioral health needs and health needs of children will improve health outcomes and long-term financial savings	2.1 To what extent are MMMC enrollees accessing community- based specialty services in a timely manner?	2.1.1 MMMC child enrollees will report being satisfied with their access to community-based specialty services for children with chronic conditions (CPC- CH)	1. In the last 6 months, how often was it easy to get special medical equipment or devices for your child? 2. In the last 6 months, how often was it easy to get this therapy for your child? 3. In the last 6 months, how often was it easy to get this treatment or counseling for your child?	2018 CAHPS CCC survey data	Aggregate data received (10/15)
	2.2 To what extent are MMMC enrollees accessing community- based health care or integrated health/behavioral health care in a manner that results in improved health care outcomes?	2.2.1 MMMC child enrollees will have improved follow up after hospitalizations (FUH-CH) compared to non-enrollees	Follow-up after hospitalization for mental illness among children or adolescents ages 6 to 17	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
		2.2.2 MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased follow-up for children prescribed ADHD medication (ADD-CH)	Follow-up care for children prescribed ADHD medication	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
		2.2.3 MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased metabolic monitoring for children and adolescents on antipsychotics (APM-CH)	Metabolic monitoring for children and adolescents on antipsychotics	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
Goal 3. Increase appropriate access to the uniform HCBS benefit package for children who meet level of care criteria to achieve improved health outcomes while recognizing that children´s needs, including the duration, scope, and frequency of services, change over time	3.1 How has enrollment in HCBS increased over the length of the Demonstration?	3.1.1 Enrollment in HCBS will increase over the length of the Demonstration	The number of children enrolled in HCBS	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
Goal 5. Improve access to the integrated Health Home model for all children to improve the coordination of care for children and increase access to services	5.1 To what extent are Health Home/HCBS enrollees accessing primary care?	5.1.1 Stakeholders will report improved care coordination	Stakeholders´ views of care coordination	Semi-structured key Informant interviews with advocates, plan administrators, and providers	Protocol in development; state has shared a list of potential candidates for interviews and relevant notes for the meetings with stakeholders
		5.1.2 The number of child/adolescent immunizations will increase (CIS-CH and IMA-CH)	Childhood immunization status Immunizations for adolescents	2017-2019 Medicaid Data Warehouse	Aggregate data received (10/15)
	5.3. Are Health Home/HCBS enrollees accessing necessary services such as health monitoring and prevention services? Are chronic health and behavioral health conditions being managed appropriately?	5.3.2 The receipt of services in an integrated managed care setting will result in increased weight assessment and counseling for nutrition and physical activity for children/adolescents (WCC-CH)	Weight assessment and counseling for nutrition and physical activity for children/adolescents – body mass index assessment for children/adolescents	2017-2019 NYS Quality Assurance Reporting Requirements (QARR) data	Aggregate data expected by 12/15/2020
		5.3.3 MMMC enrollees with chronic conditions will report that someone helped them coordinate care (CPC-CH questions 21, 24, 27, and 30)	Did anyone from your child´s health plan, doctor´s office or clinic help you get special medical equipment or devices for your child? Did anyone from your child´s health plan, doctor´s office or clinic help you get this therapy for your child? Did anyone from your child´s health plan, doctor´s office or clinic help you get this treatment or counseling for your child? In the last 6 months, did anyone from your child´s health plan, doctor´s office, or clinic help coordinate your child´s care among these different providers or services?	2018 CAHPS CCC survey data	Aggregate data received (10/15)

Goal 1. Effect of Managed Care on HCBS Population Outcomes: Improve the health outcomes for individuals under 21 receiving HCBS (HCBS Child/Youth) with access to the Medicaid managed care delivery system.

Research Question 1.1: Targeting HCBS Availability to a Narrowly-defined Population

What are the consequences of targeting availability of HCBS to a more narrowly-defined population than the criteria in the State Plan?

Hypothesis 1.1.1: Potentially Preventable Emergency Room Visits

Targeting HCBS availability to a more narrowly-defined population will improve the health outcomes of the population most needing supports to remain in the community, as measured by stakeholder observations about Potentially Preventable Emergency Room Visits (PPVs) and the consequences of targeting HCBS availability to a more narrowly-defined population.

Key Stakeholder Observations

Semi-structured interviews will be conducted with between 10 and 15 key informants representing three types of stakeholders: advocates, plan administrators, and providers. The interviews will address Hypothesis 1.1.1, concerning the consequences of targeting HCBS availability to a more narrowly defined population. As we describe below, the same interviews will also be used to address Hypothesis 5.1.1, concerning care coordination. It is important to note that qualitative methods cannot formally test these hypotheses. Rather, they will reveal stakeholders´ views of implementation and explanations for challenges and successes.

Protocol Development

The RAND evaluation team will develop semi-structured interview protocols for each category of stakeholder. Each of the protocols will be designed to elicit key stakeholders´ views regarding the success or lack of success of the Children´s Design in achieving the goal of improving health outcomes and reducing PPVs. Stakeholders will be asked to describe barriers to implementation of the Children´s Design as well as unanticipated challenges to successfully achieving the implementation goals. The protocols will be informed by review of documents provided by DOH that include minutes from stakeholder meetings and presentations related to implementation of the Children´s Design.

Key Informant Selection

Between 10 and 15 key informant interviews will be conducted with individuals selected from a list of stakeholders provided by DOH, additional recommendations from DOH, suggestions by informants recommended by DOH, or identified through review of documents including minutes of stakeholder meetings. Informants will be selected from different regions of the state, ensuring representation of NYC, urban areas outside of NYC, and rural areas.

Key Informant Recruitment

The evaluation team will schedule all interviews. The team may obtain contact information for some informants from DOH. In addition, DOH may facilitate introductions to potential informants to facilitate timely recruitment.

Interviewer Training

In anticipation of conducting interviews, the qualitative team has received training on the Children´s Design and the context of the NYS Medicaid policy for children. The training included a review of documents provided by DOH, participation in discussions with DOH subject matter expert staff, and internal discussions with the project leads and technical advisors who have experience with NYS Medicaid. The training ensured that the interviewers are aware of issues relevant to implementation when conducting interviews.

Conducting Interviews

Interviews will be conducted by phone, with audio recording if informants consent. At least two evaluation team staff will participate in each interview. One staff person will be the designated interviewer, and another will be the designated note taker. Interviews are expected to take 60 minutes on average.

Qualitative Data Analysis

Immediately after each interview, the note taker will summarize the interview using a structured template. The summary will describe the key points that were raised by the interviewee, highlighting implementation barriers and successes. The summaries, along with the interview notes and audio recordings, will then be uploaded to a qualitative data analysis platform called Dedoose that will enable the entire team to jointly read and analyze their contents. Analyses will focus on the summaries, drawing on the more detailed notes and audio recordings as needed for clarification. The research team will identify themes in each interview and compare and contrast themes that arise across interviews. To take one example, we expect that administrative procedures involved in the transition to the Children´s Design will be a theme that emerges from key informants´ comments during the interviews. The team will identify all the ways in which this theme arose, including positive and negative experiences with the transition. The informants´ perspectives on these procedures and their impact on PPVs will then be summarized in the report.

Reporting of Results

The results of the qualitative analysis will be reported in the interim report. The report will include a section on the qualitative analysis addressing Research Question 1.1 that describes the themes that arose in the qualitative interviews and compares the views of different stakeholders and stakeholder groups. The discussion will also aim to identify implementation issues that should be taken into account in the analysis plan for the summative evaluation. No names or identifiable information will be included in the report.

Research Question 1.3: Access to Primary Care in Children with Special Needs

To what extent are children with special needs accessing primary care providers who understand the child´s needs?

Hypothesis 1.3.1: Satisfaction with Primary Care

Parents of children with special needs will report being satisfied with primary care providers´ understanding of their children´s special conditions.

Study Population and Data Sources

The evaluation team is testing this hypothesis for children with special needs using the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) 5.0 data for children with chronic conditions (CCC) for 2018. The CAHPS CCC questionnaire asks parents or caretakers of children in health plans about their experiences with access to care, health care providers, and health plans. The survey is conducted every two years. For the interim evaluation, the 2018 CAHPS survey data is the baseline to reflect the implementation of the Children´s Design in 2019.

Outcome Measures

Primary outcomes include parent reports of satisfaction with primary care providers´ understanding of children´s special conditions:

1. Does your child´s personal doctor understand how your child´s medical, behavioral, or other health conditions affect your child´s day-to-day life?
2. Does your child´s personal doctor understand how your child´s medical, behavioral, or other health conditions affect your family´s day-to-day life?

Analytic Approach

The evaluation team will describe the differences in the measure between Medicaid FFS and MMMC populations. A chi-square test may be used to test the difference. The limitation of this analysis is that we will not be able to identify the target populations of the Children´s Design–– HCBS and Health Home-and the population surveyed at baseline (2018) may be different from those surveyed in later years when the target populations will be included.

Hypothesis 1.3.2: Well-care Visits

The number of children enrolled in MMMC/Health Home/HCBS who are receiving child/adolescent well-care visits will increase.

Study Population and Data Sources

The evaluation team will use aggregate measures (W15-CH, W34-CH, and AWC-CH, discussed below) generated by DOH using the July 2017 to July 2019 Medicaid Data Warehouse for the target populations of the Children´s Design, as well as the comparison population, comparable Medicaid FFS children or MMMC children.

Outcome Measures

Depending on a child´s age, one of the following measures will be used:

1. W15-CH: Well-child visits in the first 15 months of life
2. W34-CH: Well-child visits in the third, fourth, fifth, and sixth years of life
3. AWC-CH: Adolescent well-care visits

Analytic Approach

For the Interim Report, the evaluation team will use a FFS population as the comparison group. For the Final Summative Evaluation Report, we will determine the feasibility of using a comparison group who have been in FFS or in MMMC both prior to and after the Children´s Design implementation, potentially using a propensity score matching approach based on demographics, medical conditions, the level of HCBS needs, past medical utilizations, and other individual-level characteristics.

The trends in the outcome measure and the differences at baseline between the target population (e.g., HCBS or HHSC) and the comparison group (e.g., FFS) will be described. A c2 test or a logistic regression may be used to test the difference. If the number of well-child visits is available, a Poisson or negative binomial regression will be conducted. In addition, the analysis will be stratified for each of the target populations.

---

Goal 2. Effect of Timely Access to Early and Periodic Screening, Diagnostic, and Treatment Benefits: Improved timely access to the additional Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefits that address early behavioral health needs and health needs of children will improve health outcomes and long-term financial savings

Research Question 2.1: Access to Community-based Specialty Services

To what extent are MMMC enrollees accessing community-based specialty services in a timely manner?

Hypothesis: 2.1.1: Satisfaction with Access to Community-based Specialty Services

MMMC child enrollees will report being satisfied with their access to community-based specialty services for children with chronic conditions.

Study Population and Data Sources

This hypothesis will be tested for children with special needs in the 2018 CAHPS CCC data.

Outcome Measures

Questions related to access to community-based specialty services for children with chronic conditions will be used, including:

1. In the last 6 months, how often was it easy to get special medical equipment or devices for your child?
2. In the last 6 months, how often was it easy to get this therapy for your child?
3. In the last 6 months, how often was it easy to get this treatment or counseling for your child?

Analytic Approach

The evaluation team will describe the differences in the measure between Medicaid FFS and MMMC populations. A _X² test may be used to test the difference.

Research Question 2.2: Effect of Access to Community-based Integrated Health/Behavioral Health Care

To what extent are MMMC enrollees accessing community-based health care or integrated health/behavioral health care in a manner that results in improved health care outcomes?

The data source and analytic approach are similar across the four hypotheses under this research question; they are described together below.

Hypothesis 2.2.1: Follow-Up after Hospitalization for Mental Illness

MMMC child enrollees will have better follow up after hospitalizations compared to non-enrollees.

Hypothesis 2.2.2: Follow Up for Children Prescribed ADHD Medication

MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased follow up for children prescribed attention-deficit/hyperactivity disorder (ADHD) medication.

Hypothesis 2.2.3: Metabolic Monitoring for Children on Antipsychotics

MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased metabolic monitoring for children and adolescents on antipsychotics.

Study Population and Data Sources

The evaluation will use aggregate measures produced by DOH using the July 2017 to July 2019 Medicaid Data Warehouse for the target populations of the Children´s Design, as well as for the comparison population, which may be comparable Medicaid FFS children or MMMC children.

Outcome Measures

1. Hypothesis 2.2.1: Follow-up after hospitalization for mental illness among children or adolescents ages 6 to 17
2. Hypothesis 2.2.2: Follow-up care for children prescribed ADHD medication
3. Hypothesis 2.2.3: Metabolic monitoring for children and adolescents on antipsychotics

Analytic Approach

The trends in these outcome measures and the differences at baseline between the target population and the comparison population will be discussed. Because these outcome measures come from the Medicaid Data Warehouse, a comparison population is likely feasible. Care will be taken in identifying such a comparison population, as the general MMMC population or FFS population may have fewer or less severe conditions and a lower level of HCBS needs. The propensity score matching approach may be used to find similar comparison individuals.

A _X² or t-test or a logistic, Poisson, negative binomial regression may be used to test the difference, as appropriate. For multiple data points for both groups, logistic regression may be used for dichotomous outcomes using the number of individuals in each group for each time period as the frequency weight. The analysis will be stratified for each of the target populations.

Goal 3. Effect of Access to HCBS: Increase appropriate access to the uniform HCBS benefit package for children who meet level of care criteria to achieve improved health outcomes while recognizing that children´s needs, including the duration, scope, and frequency of services, change over time

Research Question 3.1: HCBS Enrollment

How has enrollment in HCBS increased over the length of the Demonstration?

Hypothesis 3.1.1: Increase in HCBS Enrollment

Enrollment in HCBS will increase over the length of the Demonstration.

Study Population and Data Sources

Medicaid Data Warehouse data will be used to identify children enrolled in HCBS, as well as the timing of enrollment and disenrollment. The same group of children may be used for a pre-and post-implementation comparison.

Outcome Measures

The number of children enrolled in HCBS.

Analytic Approach

Enrollment changes over time and the patterns of enrollment of the target populations will be delineated. If the enrollment timings are available at the aggregate level, enrollment patterns will be examined using survival analysis techniques as appropriate to describe the enrollment duration and compare the pre- and post-implementation patterns.

Goal 5. Effect of Access to Health Home Model: Improve access to the integrated Health Home model for all children to improve the coordination of care for children and increase access to services

Research Question 5.1: Access to Primary Care

To what extent are Health Home/HCBS enrollees accessing primary care?

Hypothesis 5.1.1: Improved care coordination

As noted above, Hypothesis 5.1.1 will be addressed in the interim report using qualitative methods. Data will be collected following the methods described under Hypothesis 1.1.1. It is important to reiterate that these methods cannot provide a formal test of the hypothesis and are not intended to do so. Rather, through key informant interviews, the evaluation team will collect and analyze stakeholder perspectives on whether the Children´s Design has met this important goal. In the interviews, stakeholders will be asked about their impressions of whether the Children´s Design implementation has improved care coordination or not and the evidence that has led them to these opinions. As evaluators, the aim is not to assess the validity of the stakeholders´ beliefs about the effects of the Children´s Design. However, it may be possible to ascertain whether stakeholders base their claims on their own clinical experience or on a more systematic assessment of evidence. As described above, the qualitative analysis of key informant interviews will summarize the themes that arise during the interviews and will compare and contrast these views across informant types (advocate, plan administrator, or provider).

Hypothesis 5.1.2: Increase in Immunization

The number of child/adolescent immunizations will increase.

Study Population and Data Sources

The immunization measures for children and adolescents will come from the Medicaid Data Warehouse.

Outcome Measures

1. Childhood immunization status
2. Immunizations for adolescents

Analytic Approach

Immunization status of children and adolescents will be analyzed over time. A comparison population may be constructed to compare the measures between the two groups. A _X² test or a logistic regression may be used to test the difference. If the sample size allows, the analysis can be conducted for each of the subpopulations.

Research Question 5.3: Effect of Health Home Model on Quality of Care

Are Health Homes Serving Children/HCBS enrollees accessing necessary services such as health monitoring and prevention services? Are chronic health and behavioral health conditions being managed appropriately?

Hypothesis 5.3.2: Improved Weight Management and Nutrition Counseling

The receipt of services in an integrated managed care setting will result in increased weight assessment and counseling for nutrition and physical activity for children/adolescents.

Study Population and Data Sources

The measure will be derived using NYS Quality Assurance Reporting Requirements (QARR) data for the population eligible for the Children´s Design and the comparison population.

Outcome Measures

Weight assessment and counseling for nutrition and physical activity for children/adolescents – body mass index assessment for children/adolescents will be included.

Analytic Approach

A FFS population will be used as a comparison. The differences in the receipt of weight management and counseling for nutrition and physical activity will be described. A _X² test or a logistic regression may be used to test the difference. The analysis will be conducted for each of the three subpopulations.

Hypothesis 5.3.3: Care Coordination

MMMC enrollees with chronic conditions will report that someone helped them coordinate care.

Study Population and Data Sources

The 2018 CAHPS CCC survey data will be used to test this hypothesis among children and adolescents with chronic conditions.

Outcome Measures

Questions that elicit parents´ reports of care coordination provided by a health plan or doctor´s office or clinic will be used, including:

1. Did anyone from your child´s health plan, doctor´s office, or clinic help you get special medical equipment or devices for your child?
2. Did anyone from your child´s health plan, doctor´s office, or clinic help you get this therapy for your child?
3. Did anyone from your child´s health plan, doctor´s office, or clinic help you get this treatment or counseling for your child?
4. In the last 6 months, did anyone from your child´s health plan, doctor´s office, or clinic help coordinate your child´s care among these different providers or services?

Analytic Approach

Differences in the measure between Medicaid FFS and MMMC children with chronic conditions will be described. A _X² test may be used to test the difference between the two groups.

---

4. FINDINGS

Beginning in March 2020, the significant impact of the COVID-19 pandemic on the NYS health care system required DOH personnel to shift attention, resources, and priorities. This understandable shift created contract execution and data access delays. As a result, the evaluation timeline has changed; no findings are available at the time of this writing given the recent receipt of the data, which did not allow for adequate analyses to develop findings.

Despite contractual delays, a number of meetings have been held between DOH and RAND to discuss and plan the evaluation and to answer RAND´s questions related to program implementation. DOH has made significant progress on data curation and has shared all data except those on program enrollment and a quality of care measure (please see the "Status" column in Table 3.1). NYS has also shared a list of candidates for interviews, notes for meetings with various stakeholders, and State policy documents. The findings of this interim evaluation are expected to be available in Spring 2021. The timeline for the interim evaluation is presented in Figure 1.

Figure 1. Children´s Design Evaluation Timeline

---

5. POLICY IMPLICATIONS

No policy implications are available to report at this time. A more thorough discussion will be included in the interim report, once the results and conclusions are available.

---

6. INTERACTIONS WITH OTHER STATE INITIATIVES

Interactions of Children´s Design implementation with other state initiatives will be described in the interim report.

---

APPENDIX A. TENTATIVE EVALUATION TIMELINE AND MILESTONES

Milestones	2020 - 2021
	Prelim	Month 1	Month 2	Month 3	Month 4	Month 5	Month 6
Institutional Review Board approval
Request submission	X
Approval	X
Data Access
Request submission	X
Access to aggregate data	X
Obtain a list of key informants	X
Goal 1: HCBS Population Outcomes
Hypothesis 1.1.1: Potentially Preventable Emergency Room Visits and Stakeholder Observations
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 1.3.1: Satisfaction with Primary Care
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 1.3.2: Well-care Visits
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Goal 2: EPSDT Benefits
Hypothesis: 2.1.1: Satisfaction with Access to Community-based Specialty Services
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 2.2.1: Follow-Up after Hospitalization for Mental Illness
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 2.2.2: Follow Up for Children Prescribed ADHD Medication
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 2.2.3: Metabolic Monitoring for Children on Antipsychotics
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 2.2.4: Emergency Department Visits and Hospitalizations
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Goal 3: Access to HCBS
Hypothesis 3.1.1: HCBS Enrollment
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Goal 5: Access to Health Home
Hypothesis 5.1.1: Stakeholder Observations on Care Coordination
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 5.1.2: Immunization
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 5.3.2: Weight Management and Nutrition Counseling
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Hypothesis 5.3.3: Care Coordination
Preliminary analyses		X	X
Final analyses				X
Interpretation of findings				X
Reports
Monthly progress report		X	X	X	X	X	X
Draft interim report				X
RAND Quality Assurance					X	X
DOH-approved interim report					X
CMS-approved interim report							X

___________________________________

^{1. Note that Goals 4 and 6 will be addressed in the Final Summative Evaluation. 1}