New York 1115 Waiver Amendment: Children's Program Evaluation Design
- Document also available in Portable Document Format (PDF)
Revised March 20, 2020
Start Date of Demonstration Period (Children's Design): August 2, 2019
End Date of Demonstration Period: March 31, 2021
Overview
On August 2, 2019, New York (the State) began implementation of the Children's Design, an approved amendment to the existing 1115 waiver program. This amendment streamlined the model of care for children and youth under 21 years of age with behavioral health and home and community-based services (HCBS) needs, including the following children covered under the 1915(c) Children's Waiver: medically fragile children, children with a behavioral health diagnosis, children with medical fragility and developmental disabilities, and children with developmental disabilities who are in foster care. The
Children's Design includes the following authority under the Medicaid Redesign Team Waiver:
- Authority to mandatorily enroll the children receiving HCBS under the State's newly consolidated 1915(c) Children's Waiver in managed care (implemented 10/1/2019, except for foster care children, for whom it will be implemented 7/1/2020);
- Authority to include current fee-for-service (FFS) 1915(c) Children's HCBS authorized in managed care organization (MCO) benefit packages (implemented 10/1/2019);
- Authorities to target eligibility to medically needy "Family of One" children (Fo1 children) who meet the risk factors, targeting criteria, and clinical eligibility standard for the Children's Waiver but are not otherwise enrolled in the 1915(c) Children's Waiver. Children under this authority receive Health Home Comprehensive Care Management and no HCBS, or are eligible under a non-Supplemental Security Income category;
- Authority to institute an enrollment cap for Fo1 children who attain Medicaid eligibility via the 1115 waiver (implemented 8/2/2019); and
- Authority to provide customized goods and services, and financial management services, under the Demonstration's Health and Recovery Plan's self-direction pilot for Fo1 children (will be implemented no earlier than 9/30/2020).
This amendment created a streamlined children's model of care for children and youth under 21 years of age with behavioral health and HCBS needs. A streamlined children's model of care will improve clinical and recovery health outcomes for children and youth with behavioral health and HCBS needs; improve timely access to services that address needs early in childhood and before they escalate and become more costly and complex in adulthood; improve access to integrated Health Homes and managed care models that integrate the delivery and care planning of behavioral health, health services, and community supports; and increase access and network capacity to deliver community-based recovery-oriented services and supports.
This amendment improves the continuity of care for transition-age youth and preserves Medicaid eligibility for many medically needy Fo1 children who would otherwise lose their Medicaid eligibility because they no longer received at least one 1915(c) service because case management is now covered outside of the 1915(c) Children's Waiver or are eligible under a non-Supplemental Security Income eligibility category.
This document outlines an overall evaluation plan for the Children's Design. This includes specifications for what can be included in the upcoming 1115 Waiver Interim Evaluation Report (due September 2020). Given the short time between the start of the Children's Design and the due date for the Interim Evaluation Report, the report will include only high-level metrics and qualitative data. The overall evaluation plan also includes a more comprehensive evaluation design that can be implemented as a Summative Evaluation Report to be completed in the future with the agreement of both the State and the Centers for Medicare & Medicaid Services (CMS).
Background
The New York Medicaid Redesign Team (MRT) Demonstration (formerly known as "Partnership Plan") allows New York to implement a managed care delivery system to provide benefits to its Medicaid recipients, create efficiencies in the Medicaid program, and enable the extension of coverage to many individuals needing long-term services and supports (LTSS). The Demonstration was originally approved in 1997 to enroll most of the state's Medicaid recipients into MCOs and it has been amended numerous times, including through the following notable amendments:
- In 2010, an HCBS expansion program was added.
- In 2012, an improved care coordination model of managed LTSS was added.
- In 2013, modifications were approved to coordinate with the Medicaid expansion and other changes under the Affordable Care Act–including a) transitioning childless adults and parents and caretaker relatives with incomes up to, and including, 133 percent of the federal poverty limit (FPL) into State Plan coverage; and b) mandating them into managed care arrangements.
- In 2014, a Delivery System Reform Incentive Payment (DSRIP) program was added.
- In 2015, Health and Recovery Plans were approved to integrate physical, behavioral health, and HCBS for beneficiaries diagnosed with severe mental illness and/or substance use disorder.
- In 2019, a waiver of comparability was added to exempt Medicaid Mainstream Managed Care (MMMC) enrollees from cost sharing–except for applicable pharmacy co-payments.
- In 2019, CMS approved the Children's 1115 MRT waiver amendment concurrently with the 1915(c) Children's Waiver amendments to consolidate and streamline children's HCBS services delivery in New York.
Larger Comprehensive Design Timeframe Limitations
One primary limitation to the evaluation methodology is the timing of the approval, the Children's
Design implementation, and the overall Demonstration evaluation.
The MRT Waiver Special Terms and Conditions (STCs) require that a draft Interim Evaluation be submitted with the 1115 renewal application. Because of the timing of the approval and implementation of the amendment,1 New York intends to stratify existing performance metrics for children's Health Home populations to the extent that the metrics are relevant to the larger Comprehensive Design. Other Interim Evaluation metrics for children's populations will not include stratification for HCBS and Fo1 populations because of timing and the limited availability of data at this time. The Interim Evaluation will be analyzed and interpreted by a conflict-free independent evaluator.
MRT Waiver Evaluation Target Dates:
- August 2019 – January 2020:Contractor orientation, data applications, and other preparatory steps take place.
- January 2020:Dataset for evaluation period becomes available and can be provided to contractor. Review and analysis begin.
- June 2020:Draft Interim Evaluation Report is submitted to the New York State Department of Health.
- July 2020:State publicly posts Interim Evaluation Report.
- September 2020: 1115 renewal application is submitted with Interim Evaluation Report to CMS.
Due to delays with the approval of the Children's Design authorities, the State's implementation timeline was significantly compressed. The State recognizes this Interim Evaluation Report has an aggressive timeline associated with the 1115 renewal application given the schedule necessary to procure an independent evaluator. Updates and risks will be shared with CMS via the quarterly 1115 report.
Other descriptive analyses directly relevant to the Children's Design–as noted in this design–will be added to the Interim Evaluation Report in an addendum, once the design is approved by CMS. The addendum will be primarily descriptive in nature because of the timing and data limitations. Most children will be newly enrolled during the Interim Evaluation data collection period; most measures will require participation over a longer time period in order to observe outcomes. This means any quantitative data in the Interim Evaluation Report will be baseline data only. No data showing effects of the waiver will be reported at that time. The addendum will be reviewed and analyzed by a conflict-free independent evaluator.
Consumer Assessment of Healthcare Providers and Systems (CAHPS®) baseline data collected in 2018 is currently available for the Interim Evaluation Report. This survey will be re-administered in fall 2020.
The Demonstration ends in 2021. Because the implementation of the Children's Design will only have begun in fall 2019, the impact of the Children's Design will have limited time to affect the satisfaction of members in managed care, as measured by the CAHPS®.
The Summative Evaluation Report will incorporate all metrics and the entire evaluation design plan and will be submitted consistent with Section XI of the approved STCs. The State will include any changes in the evaluation design in the 1115 quarterly report to CMS.
Research Questions and Hypotheses
The following section outlines the research questions, organized by the six goals outlined in the original Children's Design application. Each question includes one or more hypotheses to be tested for this evaluation. Note: the approved Children's Design is slightly different than the design proposed in the original application. Slight changes to the wording of goals and research questions have been made as a result of these changes and to reflect information available to the evaluator. For example, children served only in the FFS delivery system were omitted from the approved Demonstration, so goal six in the original application that addressed the health status of FFS HCBS-enrolled children was not incorporated into this evaluation design. Another example is that children meeting level of need criteria will not be enrolled under the Demonstration at this time.
These research questions and hypotheses reflect the complete, Summative Evaluation design. Hypotheses that can be at least partially addressed in the Interim Evaluation Report have been noted and italicized in blue print.
Goal 1: Improve the health outcomes for individuals under 21 receiving HCBS (HCBS Child/Youth) with access to the Medicaid managed care delivery system.
- Research Question 1.1 (Access to Care) – What are the consequences of targeting availability of HCBS to a more narrowly-defined population than the criteria in the State Plan?
- Hypothesis 1.1.1: Targeting HCBS availability to a more narrowly-defined population will improve the health outcomes of the population most needing supports to remain in the community, as measured by a reduction in Potentially Preventable Emergency Room Visits (PPVs) and stakeholder observations about the consequences of targeting availability of HCBS to a more narrowly-defined population. [Interim Evaluation Report will include qualitative data only.]
- Research Question 1.2 (Costs) – What are the per member per month (PMPM) costs of HCBS for children enrollees who receive services and how have they improved health outcomes?
- Hypothesis 1.2.1: The PMPM costs of HCBS for children enrollees will decrease because more children are eligible to receive former HCBS services under State Plan authority in an integrated managed care setting. [The Summative Evaluation Report will incorporate.]
- Hypothesis 1.2.2: The receipt of services in an integrated managed care setting will improve outcomes among Health Home/HCBS/Fo1-enrolled children, as demonstrated by a stable or decreasing percentage of the Health Home/HCBS/Fo1 population who have had an emergency room visit (AMB-CH). [The Summative Evaluation Report will incorporate.]
- Research Question 1.3: To what extent are children with special needs accessing primary care providers who understand the child's needs?
- Hypothesis 1.3.1: Parents of children with special needs will report being satisfied with primary care providers' understanding of their children's special conditions (CPC-CH, questions 44 and 45). [Interim Evaluation Report will include baseline data only.]
- Hypothesis 1.3.2: The number of children enrolled in MMMC/Health Home/HCBS/Fo1 who are receiving child/adolescent well-care visits will increase (W15-CH, W34-CH and AWC-CH) [Interim Evaluation Report will include baseline data for the entire MMMC population only.].
Goal 2: Improved timely access to the additional Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefits that address early behavioral health needs and health needs of children will improve health outcomes and long-term financial savings.
- Research Question 2.1: To what extent are MMMC enrollees accessing community-based specialty services in a timely manner?
- Hypothesis: 2.1.1: MMMC child enrollees will report being satisfied with their access to community-based specialty services for children with chronic conditions (CPC-CH questions 20, 23, 26). [Interim Evaluation Report will include baseline data only.]
- Hypothesis 2.1.2: MMMC child enrollees will have improved access to behavioral health care, as demonstrated through increased use of first-line psychosocial care for children and adolescents on antipsychotics (APP-CH) [The Summative Evaluation Report will incorporate.]
- Research Question 2.2 (Access to Care): To what extent are MMMC enrollees accessing community-based health care or integrated health/behavioral health care in a manner that results in improved health care outcomes?
- Hypothesis 2.2.1: MMMC child enrollees will have better follow up after hospitalizations compared to non-enrollees (FUH-CH) [Interim Evaluation Report will include baseline data only.]
- Hypothesis 2.2.2: MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased follow up for children prescribed ADHD medication (ADD-CH). [Interim Evaluation Report will include baseline data only.]
- Hypothesis 2.2.3: MMMC child enrollees will have enhanced integrated health/behavioral health care, as demonstrated through increased metabolic monitoring for children and adolescents on antipsychotics (APM-CH)). [Interim Evaluation Report will include baseline data only.]
- Hypothesis 2.2.4: Children who have these behavioral health interventions (follow up after hospitalizations, or prescribed ADHD medication, or increased metabolic monitoring) will have lower numbers of emergency department visits and fewer hospital admissions, compared with children who do not. [The summative evaluation will incorporate.]
Goal 3: Increase appropriate access to the uniform HCBS benefit package for children who meet level of care criteria to achieve improved health outcomes while recognizing that children's needs, including the duration, scope, and frequency of services, change over time.
- Research Question 3.1: How has enrollment in HCBS increased over the length of the Demonstration?
- Hypothesis 3.1.1: Enrollment in HCBS will increase over the length of the Demonstration. [Interim Evaluation Report will include baseline data on the number of children enrolled in HCBS only.]
- Research Question 3.2: What are the demographic, social, functional, and clinical characteristics of the HCBS population and do they change over time?
- Hypothesis 3.2.1: The relative number of children within each target group in the 1915(c) Children's Waiver/1115 waiver will remain the same over time. Target groups include HCBS Serious Emotional Disturbance (SED), HCBS Medically Fragile (MF), HCBS Developmentally Disabled (DD) with Foster Care, HCBS Developmentally Disabled and Medically Fragile (DD & MF), children in foster care, children eligible under Family of One. [The Summative Evaluation Report will incorporate.]
Goal 4: Increase access to HCBS under the Demonstration and reduce the number of children being referred and diverted to more costly institutional levels of care. More children will remain in the community and be diverted from institutional services if HCBS are delivered prior to the child meeting an institutional level of care.
- Research Question 4.1: To what extent has the Demonstration improved the availability of HCBS for children? What are their health outcomes, and have they been able to remain in the community?
- Hypothesis 4.1.1: Children are being admitted to institutional settings (i.e., psychiatric hospitals, general hospitals, intermediate care facilities for individuals with intellectual disabilities [ICF-ID], nursing facilities, and psychiatric residential treatment facilities [PRTFs]) less frequently and for shorter lengths of stays after the implementation of the Children's Design. [The Summative Evaluation Report will incorporate.]
- Research Question 4.2 (Costs): To what extent are HCBS cost effective? What are the PMPM costs of inpatient psychiatric services, substance use disorder (SUD) ancillary withdrawal, hospital-based detox, and emergency room services for the children's HCBS population? Are these costs decreasing over time?
- Hypothesis 4.2.1: PMPM costs for inpatient psychiatric services, SUD ancillary withdrawal, hospital-based detox, and emergency room services for the children's HCBS population will decrease during the Demonstration period. [The Summative Evaluation Report will incorporate.]
Goal 5: Improve access to the integrated Health Home model for all children to improve the coordination of care for children and increase access to services.
- Research Question 5.1: To what extent are Health Home/HCBS enrollees accessing primary care?
- Hypothesis 5.1.1: Stakeholders will report improved care coordination. [Interim Evaluation Report will include.]
- Hypothesis 5.1.2: The number of child/adolescent immunizations will increase (CIS-CH and IMA-CH). [Interim Evaluation Report will include baseline data only.]
- Research Question 5.2 (Access to Care): To the extent there is capacity for HCBS services, to what extent are Health Home/HCBS/Fo1 enrollees accessing community-based health care or integrated health/behavioral health care?
- Hypothesis 5.2.1: Health Home/HCBS/Fo1 child enrollees will have increased utilization of first-line psychosocial care for children and adolescents on antipsychotics (APP-CH). [The Summative Evaluation Report will incorporate.]
- Hypothesis 5.2.2: Rates of follow-up for Health Home/HCBS/Fo1 child enrollees prescribed ADHD medication will increase (ADD-CH). [The Summative Evaluation Report will incorporate.]
- Hypothesis 5.2.3: Metabolic monitoring for Health Home/HCBS/Fo1 child enrollees who are prescribed antipsychotics will increase (APM-CH). [The Summative Evaluation Report will incorporate.]
- Research Question 5.3 (Quality of Care): Are Health Home/HCBS enrollees accessing necessary services such as health monitoring and prevention services? Are chronic health and behavioral health conditions being managed appropriately?
- Hypothesis 5.3.1: The receipt of services in an integrated managed care setting will result in an increased asthma medication ratio among Health Home/HCBS/Fo1 enrolled children (AMR-CH). [The Summative Evaluation Report will incorporate.]
- Hypothesis 5.3.2: The receipt of services in an integrated managed care setting will result in increased weight assessment and counseling for nutrition and physical activity for children/adolescents (WCC-CH). [The Summative Evaluation Report will incorporate.]
- Hypothesis 5.3.3: MMMC enrollees with chronic conditions will report that someone helped them coordinate care (CPC-CH questions 21, 24, 27, and 30). [Interim evaluation will have baseline data only.]
Goal 6: Improve continuity of care for youth as they transition into the adult Medicaid services system, specifically to the Health and Recovery Plan from the children's Medicaid Mainstream Managed Care benefits.
- Research Question 6.1: Are chronic health and behavioral health conditions for young adults (e.g., ages 21–25) who transition to adult HCBS and other Medicaid services in the Demonstration being managed appropriately?
- Hypothesis 6.1.1: Young adults transitioning to HCBS and other Medicaid services in the Demonstration have their chronic conditions properly managed, as measured by lower rates of emergency department visits (AMB-CH). [The Summative Evaluation Report will incorporate.]
Study Design
The overall evaluation of the Children's Design will include a mixed-methods approach, using primarily quantitative performance metrics to test hypotheses. However, it will also include a qualitative component designed to describe the process of implementing the Children's Design, including any challenges to implementation that may have an impact on expected outcomes.
Interim Evaluation Report Design: The majority of this design relates to a future, potential Summative Evaluation Report of the project. For the Interim Evaluation Report, due in September of 2020, only qualitative measures and select baselines for some high-level quantitative measures will be included.
Overall Design: Because children will not be randomly assigned to receive (or be eligible for) Demonstration services, an experimental evaluation design is not feasible. Any Summative Evaluation Report of the Children's Design will utilize a quasi-experimental pre-post design that compares trends in performance metrics before implementation of the waiver amendment to the time period directly after. In order to attribute any observed changes over time to the amendment, a comparison group will be matched to the target population, where possible. Comparison groups will be utilized on a measure-by- measure basis when an adequate comparison pool is available. This is discussed for each measure in the Research Design table at the end of this document.
In cases where the evaluation question is either descriptive in nature or a comparison group is not available, the Summative Evaluation Report will use a descriptive time series analysis to illustrate changes over time both before and after implementation of the amendment.
The primary limitation of the design is the short time frame between implementation of the amendment and end of the waiver. Where possible, data will be reported on a rolling-year quarterly basis. However, even rolling-year quarterly data reporting limits the number of data points available for analysis. The table below shows the evaluation time frame, assuming an Interim Evaluation Report is due September 2020 and that a Summative Evaluation Report may be required by future approved STCs.
| EVALUATION PERIOD | REPORT | TIME FRAME | NUMBER OF DATA POINTS |
|---|---|---|---|
| Pre-Implementation Period | Limited qualitative and high-level quantitative data for Interim Evaluation Report Summative Evaluation Report |
Q1 2018 through Q2 2019 for Interim Evaluation Report Q1 2018 through Q3 2019 |
6 data points 7 data points |
| Waiver Amendment Period Implementation | No data available for Interim Evaluation Report Summative Evaluation Report |
Q4 2019 |
1 data point |
| Post-Implementation Period | No data available for Interim Evaluation Report Summative Evaluation Report |
Q1 2020 through Q1 2021 |
5 data points |
This limited data availability means that more sophisticated time-based regression analysis, such as that used with Interrupted Time Series analysis (ITS) or difference-in-difference testing is not likely to show significant differences due to a small number of data points. To the extent that HCBS or Fo1 stratification is limited in either the sample size or availability of data pre-implementation, the state may utilize a post-only comparison group design or a descriptive interrupted time series analysis.
The short evaluation window for the project also leaves very little time for significant changes to occur and be observed. As changes to the waiver are implemented through the Children's Design, it is expected that there will be a time delay between the official "start date" for those changes and when changes in most of the performance measures proposed here are implemented, particularly those related to health outcomes. Therefore, the main value of this design will be to provide a description of changes that happen following implementation. Any observed differences (even if not statistically significant) between Children's Design participants and any available comparison group or national/regional benchmarks could be an indicator of positive impact. This design, however, is limited in its ability to directly attribute change to the program. This limitation is addressed somewhat through the use of comparison groups, provided that any confounding events (e.g., other health reform efforts or interventions that could also influence outcomes, in addition to the Children's Design activities) have an equal impact on both the target population and the comparison groups.
The time frame also necessitates the use of existing performance measures to address research questions. There is not enough time between the potential approval of the evaluation design and the end of the waiver to collect new data. Performance measures–drawn from metrics already being reported by the state–that closely match the questions have been selected. However, they may not align completely, limiting the ability of the evaluation to directly address the original research questions.
When it is possible to use comparison groups, the evaluation will do so. In most cases, an attempt will be made to use children in fee for service (FFS) Medicaid to draw a comparison group. For some measures, we will rely on FFS children receiving similar services, which will help to ensure that the groups are similar. For some of the other measures, there are likely to be some differences between the FFS and the evaluation population, based on differences in program eligibility requirements. Because of this, we will use matching techniques (e.g., propensity score matching or coarsened exact matching) to ensure that the comparison group is similar to the target population on various demographic variables (age, race/ethnicity, gender, location) as well as health indicators (diagnoses, health history, level of care etc.). This selection will also consider, as discussed above, ways to ensure the use of a population that is equally likely as the target population to be affected by outside factors.. If an adequate comparison group cannot be derived for any of the measures, we will use data from similar measures in other states for comparison.
Future, summative evaluation designs will also consider the degree to which additional states have data on comparable children's populations that can be used either in place of, or in addition to FFS comparison groups.
Most performance measures can be stratified by specific population groups (e.g., entire MMMC population versus HCBS, Health Home, and Fo1 populations). This will strengthen the design by establishing exact dates (specific to a calendar quarter) that waiver activities began for that population and examining change from time periods directly before and directly after that change. This is a technique that uses an interrupted time series design when comparison groups are not available.
Limited data points or challenges in stratifying data to these populations may restrict a regression analysis of change over time, but these descriptive, observable changes will help to indicate whether changes could be attributed to the Children's Design. Further, stratifying the populations into these specific subgroups will allow for more precise matching to the FFS population. In cases where stratifications can only be done post-intervention, the analysis will utilize a post-only comparison group design in addition to the descriptive interrupted time series for the treatment population.
Performance Measures
As stated previously, the timeframe for this evaluation is very short due to the 2021 end date of the current 1115 waiver. The State will utilize existing measures in order to address this concern. Because these measures are already routinely collected and reported by the State, there is certainty that measures will be available, even without adequate time for data collection. Specific performance measures for each research question are fully described in the research design table at the end of this document. Because these measures are standardized Medicaid core measures for children's programs, the evaluation will include a comparison to national/regional trends for each measure.
Data Collection
There will be three main data sources for this evaluation: (1) the Medicaid Data Warehouse, (2) the Consumer Assessment of Healthcare Providers and Systems 5.0H Children with Chronic Conditions questionnaire, and (3) qualitative data collected from key informant interviews and document reviews regarding Children's Design implementation.
Medicaid Data Warehouse
This robust dataset includes enrollment and eligibility data as well as claims and managed care encounters. Several 3M products are used to evaluate members' clinical risk (Clinical Risk Groups) and preventable event measures, such as prevention quality indicators. These data will be used to evaluate patterns of care and health outcomes for the sub-populations of interest.
The Office of Quality and Patient Safety will calculate aggregate performance rates from the data warehouse. These rates will be provided to evaluators in February 2020 for analysis for the Interim
Evaluation Report and inclusion in the Summative Evaluation Report. Evaluators will use the data provided to calculate each identified performance measure on an annual rolling-quarter basis from the first quarter (Q1) of 2018 through the fourth quarter (Q4) of 2019 (interim reporting period) and from Q1 2018 through Q1 2021 (Summative Evaluation reporting period).
As previously mentioned, the primary limitation for this data set is the limited timeframe. However, because this is a data set from which the study measures are generally calculated, it is anticipated that there will be no challenges for the state to provide the data to the evaluator and for the measures to be calculated.
CAHPS® 5.0 CCC Questionnaire
The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) 5.0 Children with Chronic Conditions (CCC) questionnaire is a comprehensive tool designed to assess consumers' experience with health care and health plans. CAHPS® CCC is the questionnaire that asks parents/caretakers of child health plan members about experiences with access to care, health care providers, and health plans.
The CCC component of the questionnaire is a supplement to the CAHPS® Child Medicaid questionnaire, which allows health plans to identify children with chronic conditions and evaluate their experience of care. The New York State Department of Health (NYSDOH) sponsored the CAHPS® CCC survey in response to CMS Children's Health Insurance Program Reauthorization Act requirements. Results will be used to determine variation in parent/caretaker satisfaction among the plans.
CAHPS® baseline data collected in 2018 are currently available for the Interim Evaluation Report. Aggregate data files will be provided to the evaluators for reporting at baseline. This survey will be re- administered in fall 2020.
The primary limitation of using CAHPS® data to analyze research questions related to satisfaction is that the data are not linked to specific Medicaid clients. This means that while some comparisons can be made, generally, between the enrollees of different MMMC plans as well as between Medicaid FFS and MMMC populations, the data cannot be stratified for HCBS or other Children's Design targeted population. To the extent that the CCC version of the CAHPS® allows respondents to identify if they have special needs or chronic conditions, New York is utilizing those responses for this evaluation. Additional satisfaction data cannot be linked to any outcome measures.
There are no anticipated challenges with providing these data sets to the evaluators.
Qualitative Data
In order to understand the perceived challenges and benefits of activities under the Children's Design, key informant interviews will be conducted with stakeholders, including advocates, plan administrators, providers, and families for any Summative Evaluation conducted. For the Interim Evaluation report, Department of Health staff will use existing advocate/stakeholder meetings to gather information on the evaluation questions. In addition, key stakeholders will be interviewed for the Interim Evaluation Report regarding the nature of the implementation. Questions will center on specific barriers to implementing the planned activities under the Children's Design, any challenges or barriers to children accessing needed services, and perceived outcomes associated with delivered services. Both the interim and summative evaluation reports will include a complete list of Key Informant and meeting questions as an appendix. Topics for these questions include:
- Eligibility determinations
- Service array
- Provider qualifications
- Accessibility of care
- Care Management
- Appeals and grievances
- Critical incidents, restrictive interventions, seclusion and restraint
- Quality of care
- Fiscal accountability
In addition, documentation will be reviewed to confirm that the Children's Design was implemented as intended and to identify any challenges or delays to implementation.
Qualitative data is generally limited in that it only reflects the level of information available to a key informant and can reflect individuals' biases. These limitations can be minimized by using standardized interview protocols and consistent interview techniques. These data can be used to provide important context for quantitative data analysis findings and may also be able to provide supporting evidence to demonstrate that observed changes in performance metrics could reasonably be inferred to be results from demonstration activities.
Qualitative data will be included in both the Interim Evaluation and Summative Evaluation Report drafts.
Assurances Needed to Obtain Data
This report utilizes measures already being calculated by the state. The state will provide aggregated rates to the independent evaluator for analysis.
Data Analysis
The following table outlines, for each hypothesis, the specific measures to be used, the study and comparison populations, the level of analysis, the measure steward or descriptions of the numerator and denominator, the data sources, and the analytic methods to be used.
Two quantitative analytic methods will be used, depending on the measure and availability of a comparison group. Difference-in-difference (DID) testing will be used to articulate the hypotheses "counterfactual" (what would have happened without implementation of the Children's Design) and to estimate the effect (difference in the change over time between the target population and control group). "DID is typically used to estimate the effect of a specific intervention or treatment (such as a passage of law, enactment of policy, or large-scale program implementation) by comparing the changes in outcomes over time between a population that is enrolled in a program (the intervention group) and a population that is not (the control group)."2
Descriptive time series analysis will be used to describe trends over time for measures when a comparison group is not appropriate or not available. The graphic below shows an example of a descriptive time series analysis, utilizing an interrupted time series analysis approach. As previously mentioned, there will not be enough data points to conduct a regression analysis; the graphical presentation of the data will allow for a general description in changes over time. We also propose that difference-in-difference pre-post regression analysis be conducted for those variables where a comparison group and sufficient data are available. This is indicated for the appropriate measures in the detailed research design (Table 1).
Qualitative data analysis will include a summarization of key informant interviews and any reviewed documents and an analysis of this narrative content to describe the implementation of the Children's Design.
Figure 1: Sample of Time Series Analysis
Table 1: Detailed Research Design Table
| PROCESSOR OUTCOME MEASURE(S) | STUDY POPULATION & COMPARISON | LEVEL OF ANALYSIS | MEASURE STEWARD OR NUMERATOR AND DENOMINATOR | DATA SOURCE(S) | ANALYTIC METHODS |
|---|---|---|---|---|---|
| Goal 1: Improve the health outcomes for individuals under 21 receiving HCBS (HCBS Child/Youth) with access to the Medicaid managed care delivery system. | |||||
Research Question 1 (Access to Care): What are the consequences of targeting availability of HCBS to a more narrowly-defined population than the criteria in the State Plan? |
|||||
|
|||||
|
MMMC enrollees with HCBS (children and youth) FFS matched comparison group, or metrics from similar group from another state |
Individual Summative Evaluation Report only |
3M | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
|
Plan administrators and services providers | Demonstration – Children's Design Interim Evaluation Report | N/A Qualitative data | Key informant interviews | Narrative analysis |
Research Question 1.2: (Costs) What are the PMPM costs of HCBS for children enrollees who receive services and how have they improved health outcomes? |
|||||
|
|||||
|
HCBS children and youth Fee-for-service HCBS costs for FFS population (estimate a PMPM)) or PMPM from similar group from another state | Individual Summative Evaluation Report only |
Numerator: Total HCBS cost of care Denominator: Number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
Stratify: Health Home (HH) children and youth HCBS children and youth Fo1 children and youth Comparison Group: FFS population (estimate a PMPM), or PMPM from similar group from another state |
Individual Summative Evaluation Report only |
AMB-CH | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
Research Question 1.3: To what extent are children with special needs accessing primary care providers who understand the child's needs? |
|||||
|
|||||
|
Parents/caregivers of children needing chronic care | Individual Interim Evaluation Report—baseline data only |
CPC-CH questions 44 and 45 | CAHPS® | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison group: FFS population, or metrics from similar group from another state |
Individual Interim Evaluation Report—baseline data only for the entire MMMC population only. |
NCQA #1392, NCQA #1516, NCQA | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
| Goal 2: Improved timely access to the additional EPSDT benefits that address early behavioral health needs and health needs of children will improve health outcomes and long-term financial savings. | |||||
Research Question 2.1: To what extent are MMMC enrollees accessing community-based specialty services in a timely manner? |
|||||
|
|||||
|
MMMC child population where parent reported that the child received Durable Medical Equipment, therapies, or behavioral health services | Interim Evaluation Report—baseline data only | Questions 20, 23, 26 on CPC-CH | CAHPS® | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
MMMC enrollees Comparison group: FFS children and youth, or metrics from similar group from another state | Individual Summative Evaluation Report only |
NCQA 2801 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
Research Question 2.2: To what extent are MMMC enrollees accessing community-based health care or integrated health/behavioral health care in a manner that results in improved health care outcomes? |
|||||
|
|||||
|
MMMC enrolleesComparison group: FFS children and youth, or metrics from similar group from another state | Individual Interim Evaluation Report—baseline data only |
NCQA #0576 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
MMMC enrollees Comparison group: FFS children and youth, or metrics from similar group from another state | Individual Interim Evaluation Report -Baseline data only. |
NCQA #0108 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
MMMC enrollees Comparison group: FFS children and youth, or metrics from similar group from another state | Individual Interim Evaluation Report—baseline data only | NCQA #2800 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
|
|||||
|
MMMC enrollees FFS comparison group, or metrics from similar group from another state | Individual Interim Evaluation Report—baseline data only | NCQA (various) | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Pearson's R correlation (with follow-up measures above) |
| Goal 3: Increase appropriate access to the uniform HCBS benefit package for children who meet level of care criteria to achieve improved health outcomes while recognizing that children's needs, including the duration, scope and frequency of services, change over time. | |||||
Research Question 3.1: How has enrollment in HCBS increased over the length of the Demonstration? |
|||||
|
|||||
|
Children enrolled with HCBS | Individual Interim Evaluation Report—baseline data only |
N/A | Medicaid data warehouse | Descriptive time series analysis |
Research Question 3.2: What are the demographic, social, functional, and clinical characteristics of the HCBS population and do they change over time? |
|||||
|
|||||
|
Children enrolled with HCBS | Individual Summative Evaluation Report only | N/A | Medicaid data warehouse | Descriptive time series analysis |
| Goal 4: Increase access to HCBS under the Demonstration and reduce the number of children being referred and diverted to more costly institutional levels of care. More children will remain in the community and be diverted from institutional services if HCBS are delivered prior to the child meeting an institutional level of care. | |||||
Research Question 4.1: To what extent has the Demonstration improved the availability of HCBS for children? What are their health outcomes, and have they been able to remain in the community? |
|||||
|
|||||
|
Children's HCBS population in MMMC Comparison group: FFS children receiving comparable services | Individual Summative Evaluation Report only | Numerator: total days | Medicaid data warehouse | Pre-post with comparison Difference-in-difference testing, if available |
Research Question 4.2: Costs: To what extent are HCBS cost effective? What are the PMPM costs of inpatient psychiatric services, SUD ancillary withdrawal, hospital-based detox, and emergency room services for the children's HCBS population? Are these costs decreasing over time? |
|||||
|
|||||
|
Children's HCBS population in MMMC Comparison group: FFS children receiving comparable services |
Individual Summative Evaluation Report only | Numerator: total costs for HCBS managed care enrollees Denominator: total number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison Difference-in-difference testing, if available |
|
Children's HCBS population in MMMC Comparison group: Fee for service children receiving comparable services |
Individual Summative Evaluation Report only | Numerator: total costs for inpatient psychiatric services Denominator: total number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison Difference-in-difference testing, if available |
|
Children's HCBS population in MMMC Comparison group: FFS children receiving comparable services |
Individual Summative Evaluation Report only | Numerator: total costs for SUD ancillary withdrawal Denominator: total number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison Difference-in-difference testing, if available |
|
Children's HCBS population in MMMC Comparison group: FFS children receiving comparable services |
Individual Summative Evaluation Report only | Numerator: total costs for hospital-based detox Denominator: total number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison Difference-in-difference testing, if available |
|
Children's HCBS population in MMMC Comparison group: FFS children receiving comparable services |
Individual Summative Evaluation Report only | Numerator: total costs for emergency room services Denominator: total number of member months | Medicaid data warehouse (372 data) | Pre-post with comparison Difference-in-difference testing, if available |
| Goal 5: Improve access to the integrated Health Home model for all children to improve the coordination of care for children and increase access to services. | |||||
Research Question 5.1: To what extent are Health Home/HCBS enrollees accessing primary care? |
|||||
|
|||||
|
Plan administrators and services providers | Demonstration—Children's Design Interim evaluation—qualitative data |
N/A Qualitative data | Key informant interviews | Narrative analysis |
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison group: FFS population, or metrics from similar group from another state |
Individual Interim Evaluation— baseline data only |
NCQA #0038 NCQA #1407 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note: To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
Research Question 5.2 (Access to Care): To the extent there is capacity for HCBS services, to what extent are Health Home/HCBS/Fo1 enrollees accessing community-based health care or integrated health/behavioral health care? |
|||||
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison group: FFS children, or metrics from similar group from another state |
Individual Summative Evaluation Report only |
NCQA 2801 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note: To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison Group: FFS population, or metrics from similar group from another state |
Individual Summative Evaluation Report only |
NCQA #0108 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note:To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison Group: FFS population, or metrics from similar group from another state |
Individual Summative Evaluation Report only | NCQA #2800 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note: To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
Research Question 5.3 (Quality of Care): Are Health Home/HCBS enrollees accessing necessary services such as health monitoring and prevention services? Are chronic health and behavioral health conditions being managed appropriately? |
|||||
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison Group: FFS population, or metrics from similar group from another state |
Individual Summative Evaluation Report only |
NCQA #1800 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note: To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
|
|||||
|
Stratify: HH children and youth HCBS children and youth Fo1 children and youth Comparison Group: FFS population, or metrics from similar group from another state |
Individual Interim Evaluation - baseline data only. |
NCQA #0024 | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available Note: To the extent that HCBS or Family of One stratification is limited, the state may utilize a post-only comparison group design or a descriptive time series analysis. |
|
|||||
|
MMMC child population where parent reported that the child received DME, therapies, or behavioral health services | Interim Evaluation— baseline data only | Questions 21, 24, 27, and 30 on CPC-CH | CAHPS® | Descriptive time series (pre-post with no comparison group) |
| Goal 6: Improve continuity of care for youth as they transition into the adult Medicaid services system, specifically to the Health and Recovery Plan from the children's Medicaid Mainstream Managed Care benefits. | |||||
Research Question 6.1: Are chronic health and behavioral health conditions for young adults (e.g., 21–25) who transition to adult HCBS and other Medicaid services in the Demonstration being managed appropriately? |
|||||
|
|||||
|
Young adults (21-25) transitioning from HCBS Young adults (21-25) in FFS Medicaid | Individual Summative Evaluation Report only | NCQA | Medicaid data warehouse | Pre-post with comparison group Difference-in-difference testing, if available |
___________________________________________
1. The Children's Design was approved in August, 2019. Many elements were not implemented until October, 2019, and some elements will not be implemented until July 2020. 1
2. Columbia University Mailman School of Public Health. (n.d.). Population health methods, difference-in-difference estimation. 2
