WP No. WR-A783-1

• Report is also available in Portable Document Format (PDF)

Independent Evaluation of the New York State
Health and Recovery Plans (HARP) Program

Interim Report

Submitted to:

Department of Health
Attention: Dianne Kiernan
New York State Department of Health
Office of Health Insurance Programs
99 Washington Avenue, Suite 720
Albany, New York 12210

Email: Dianne.Kiernan@health.ny.gov

Submitted by:

RAND Corporation
1776 Main Street
Santa Monica, CA 90407
Lisa Wagner
Phone: (703) 413-1100 Ext. 5067

Email: LisaW@rand.org

Submitted on:
November 13, 2020

---

TABLE OF CONTENTS

1. EXECUTIVE SUMMARY

2. DEMONSTRATION DESCRIPTION

2.1 Introduction
2.2 Background
2.3 Behavioral Health Demonstration
2.4 Evaluation Timeline and Progress to Date

3. Evaluation Design and Methods

3.1 Discussions with Experts to Refine Approach
3.2 HARP Goals and Research Questions
3.3 Quantitative Methods
3.4 Qualitative Methods

4. Findings

5. Policy Implications

6. Interactions with Other State Initiatives

References

Appendix A. Key Stakeholder Interview Protocol

---

FIGURES

Figure 2.1. HARP Eligibility, Target Criteria, and Risk Factors

Figure 2.2. Determination of BH HCBS Service Eligibility

Figure 2.3. HARP Evaluation Progress Timeline

---

TABLES

Table 1.1: Proposed Timeline for Remaining Evaluation Tasks

Table 3.1: HARP Goals and Evaluation Methods and Research Questions

Table 3.2: BH Demonstration Timeline

Table 3.3: Goal 1 Program Goals, Data Sources, and Outcome Measures

Table 3.4: Goal 2 Program Goals, Data Sources, and Outcome Measures

Table 3.5: Goal 3 Program Goals, Data Sources, and Outcome Measures

---

ACRONYMS
ACT	Assertive Community Treatment
ANOVA	Analysis of variance
BH	Behavioral health
CAHPS	Consumer Assessment of Health Providers and Systems
CMH	Community Mental Health
CMS	Centers for Medicare and Medicaid Services
COVID-19	SARS-CoV-19
DOH	Department of Health
DSRIP	Delivery System Reform Incentive Payment
ECHO	Experience of Care and Health Outcomes
FEP	First Episode Psychosis
FFS	Fee-for-Service
F-SHRP	Federal-State Health Reform Partnership
GLMM	Generalized Linear Mixed Model
HARP	Health and Recovery Plans
HH	Health Home
HHS	Health and Human Services
HCBS	Home and Community-Based Services
MCO	Managed Care Organization
MHARS	Mental Health Automated Record System
MMC	Medicaid Managed Care
NYC	New York City
NYS	New York State
OASAS	New York State Office of Addiction Services and Supports
OMH	New York State Office of Mental Health
OTNY	OnTrackNY
PMPM/Y	Per Member per Month/Year
PPS	Performing Provider System
PROS	Personalized Recovery Oriented Services
ROS	Rest of the State
SMI	Serious Mental Illness
SNP	Special Needs Plans
SSI	Supplemental Security Income
SUD	Substance Use Disorder

---

1. EXECUTIVE SUMMARY

Through the New York Medicaid Redesign Team Section 1115 Demonstration, New York has pursued the goal of improving access to and quality of health care for the Medicaid population through a managed care delivery system. The Section 1115 Demonstration included reforms specifically targeted to beneficiaries with behavioral health (BH) needs (hereafter, BH Demonstration); one of them is the Health and Recovery Plans (HARP) program.

New York State contracted with the RAND Corporation to conduct an independent evaluation of the BH Demonstration programs, including a HARP program evaluation (New York State Department of Health, 2019).

The HARP program evaluation uses a mixed methods approach to determine the extent to which three goals of the Behavioral Health Demonstration have been achieved since implementation (October 2015 in New York City [NYC]; and July 2016 in Rest of State [ROS]). The three goals are as follows:

1. Improve health and BH outcomes for adults enrolled in Mainstream Medicaid Managed Care (MMC) plans whose BH care was previously covered under a fee-for-service (FFS) payment arrangement.
2. Improve health, BH, and social functioning outcomes for adults enrolled in the HARP program.
3. Develop BH home and community-based services (HCBS) focused on recovery, social functioning, and community integration for HARP enrollees meeting eligibility criteria for such services.

Beginning in March 2020, the significant impact of SARS-CoV-19 (COVID-19) pandemic on the NYS health care system resulted in the shift of NYS Department of Health (DOH) personnel, attention, resources, and priorities. This shift in focus resulted in understandable and unavoidable delays in providing the evaluation team with access to data and necessitated elongated timelines compared to those proposed prior to the COVID-19 epidemic. RAND, as the Independent Evaluator, and the NYS DOH are continuing to make progress in the sharing of data to allow RAND to complete the analysis of the HARP program evaluation research questions. At this time, there are no preliminary analyses available, and the proposed timeline to continue evaluative tasks is presented in Table 1.1 below:

TABLE 1.1. PROPOSED TIMELINE FOR REMAINING EVALUATION TASKS

Proposed Timeline	Remaining Tasks
November & December 2020	Complete Data Access for HARP Research Questions
January 2021	Data Analysis
February 2021	Data Interpretation
March 2021	Report Findings to DOH
April 2021	Summative Evaluation Report to CMS

This interim report describes RAND´s understanding of the Behavioral Health Demonstration as it pertains to the MMC and HARP programs, the questions the HARP program evaluation aims to answer, and the proposed methodology RAND will use to conduct the evaluation. The final summative report, available in 2021, will provide a full discussion of the HARP program evaluation findings and their implications for policy.

---

2. DEMONSTRATION DESCRIPTION

2.1 INTRODUCTION

Through the New York Medicaid Redesign Team Section 1115 Demonstration, New York pursued the goal of improving access to and quality of health care for the Medicaid population through a managed care delivery system. The Section 1115 Demonstration included reforms specifically targeted to Medicaid beneficiaries with BH needs (hereafter, Behavioral Health Demonstration). These included the MMC carve-in of BH specialty services for Supplemental Security Income (SSI) beneficiaries and the creation of the HARP program.

The RAND team is conducting a comprehensive, statewide independent evaluation of the Behavioral Health Demonstration. This interim report describes RAND´s understanding of these reforms, the questions the evaluation is aimed to answer, and the proposed methodology to conduct the HARP program evaluation. The final report will provide a full discussion of the HARP program evaluation findings.

The HARP program evaluation was designed to determine the extent to which the following three goals of the Behavioral Health Demonstration have been achieved since the program was implemented (October 2015, NYC; July 2016, ROS):

1. Improve health and BH outcomes for adults enrolled in Mainstream Medicaid Managed Care plans whose BH care was previously covered under an FFS payment arrangement.
2. Improve health, BH, and social functioning outcomes for adults enrolled in the HARP program.
3. Develop BH home and community-based services (HCBS) focused on recovery, social functioning, and community integration for HARP enrollees meeting eligibility criteria for such services.

The evaluation uses both primary (qualitative) and secondary (quantitative) data in a mixed methods empirical investigation of the beneficiary-and system-level impacts of the programs. The evaluation seeks to examine research questions related to a variety of intermediate and long-term outcomes of the Behavioral Health Demonstration.

Intermediate outcomes include access to outpatient services (primary and preventive services, BH specialty services including services for individuals experiencing first episode psychosis, BH HCBS, crisis services); quality of BH and physical health care; social outcomes, including functioning and recovery; satisfaction with care; and utilization of acute care, namely inpatient and emergency department (ED) services.

Long-term outcomes include BH and chronic physical health status; quality of life; social circumstances; Medicaid spending; and cost shift from spending on acute care to community-based services.

2.2 BACKGROUND

The New York Medicaid Redesign Team Section 1115 Demonstration (hereafter, Section 1115 Demonstration) was originally approved in 1997 with the goal of improving access to and quality of health care for the Medicaid population through a managed care delivery system (New York State, 2020).

The Demonstration has been amended numerous times since the initial design. As part of the renewal in 2006, authority to require some disabled and aged populations to enroll in mandatory managed care was transferred to the Federal-State Health Reform Partnership (F-SHRP). In April 2014, as F-SHRP was phased out, this authority was transferred to the Section 1115 demonstration. An amendment to the Demonstration approved on April 14, 2014, allowed NYS to take its first steps toward a major reform in the financing and delivery of health care to Medicaid beneficiaries through a Delivery System Reform Incentive Payment (DSRIP) program. The amendment provided funds to incentivize provider participation in DSRIP transformation activities beginning in 2015. Under the DSRIP program, all providers are required to form provider partnerships, known as Performing Provider Systems (PPSs), and collaborate to achieve system transformation goals. The DSRIP program also includes a value-based payment reform targeting both PPSs and MMC plans.

2.3 BEHAVIORAL HEALTH DEMONSTRATION

In August 2015, NYS amended its Section 1115 Demonstration to enable qualified managed care organizations (MCOs) to comprehensively manage BH care for SSI beneficiaries whose BH benefit was previously covered under an FFS payment arrangement. Additionally, the amendment provided for BH HCBS to be made available to eligible individuals meeting defined functional needs criteria. The goals of the BH Demonstration were to improve health care quality, costs, and outcomes for the State´s Medicaid BH population and to transform the BH system from an inpatient-focused system to a recovery-focused outpatient system.

The BH benefits were made available through all mainstream MMC plans and a separate coverage product, the HARPs, which are specialty lines of business operated by qualified mainstream MMC plans and available statewide. Mainstream MMC plans began to cover expanded BH benefits statewide on October 1, 2015; while HARPs also launched on October 1, 2015 in NYC, they launched in July 2016 for ROS. The BH HCBS were offered beginning in January 2016 in NYC and in October 2016 for ROS.

COMPONENTS OF THE BEHAVIORAL HEALTH DEMONSTRATION

The mainstream MMC carve-in of BH specialty services (MMC BH carve-in program) covers Medicaid state plan and Demonstration benefits (i.e., the full medical and BH benefit) through a managed care delivery system comprised of MCOs and primary care case management arrangements for adult MMC-eligible beneficiaries, except those with dual Medicare-Medicaid eligibility and certain other populations. The expanded BH benefit, under the MMC BH carve-in, includes psychiatric services (inpatient and outpatient) previously carved out in the Medicaid FFS program for the SSI population, SUD inpatient rehabilitation (previously carved out for the SSI population), SUD outpatient (previously carved out in the Medicaid FFS for both SSI and Non SSI), along with community-based BH specialty services such as Assertive Community Treatment (ACT), Personalized Recovery Oriented Services (PROS), and First Episode Psychosis (FEP) programs, some of which were previously covered only by the FFS program.

The HARP program covers a benefit package of BH HCBS in addition to the existing mainstream MMC benefit package for non-dual Medicaid beneficiaries meeting eligibility criteria. HARP benefit eligibility includes being age 21 or over; meeting eligibility for mainstream MMC; having serious mental illness (SMI) and/or substance use disorder (SUD) diagnoses (HARP Target Criteria); and meeting HARP Risk Factor criteria, most of which are based on BH utilization patterns (Figure 2.1).

FIGURE 2.1. HARP ELIGIBILITY, TARGET CRITERIA, AND RISK FACTORS

*Adult TCM Transition to Health Home ended on 12/1/2015 and PMHP ended on 12/31/2015; both are no longer funded programs.

Being an SSI beneficiary is not, in itself, an eligibility criterion. The HARP criteria have not changed since the launch of the program. HARP-eligible individuals are identified through Medicaid data reviews of BH service utilization conducted every two months by Plans and/or NYS indicating that specific pre-determined criteria have been met (HARP algorithm).

The HARP benefit package may be accessed through HARPs or HIV SNPs. HARP-eligible individuals who are already enrolled in an HIV SNP receive the enhanced HARP benefits while enrolled in their current plan. Though these individuals may disenroll from an HIV SNP into a HARP, this is not encouraged as this would entail loss of the HIV SNP benefits.

Eligible beneficiaries are passively enrolled into HARPs; however, they are able to opt out within the first 90 days following passive enrollment and return to their original Mainstream MMC plan. Following the 90 day opt out-period, HARP beneficiaries may not change plans again until the remainder of the 12-month lock-in period has lapsed. HARP eligible individuals enrolled in a Mainstream MMC plan whose MCO does not operate a HARP line of business can voluntarily enroll in a HARP, with the MCO assisting with the transfer to the HARP.

Upon enrollment, the HARPs and HIV SNPs work with Health Homes or other state-designated entities to develop a person-centered care plan that includes assessment for BH HCBS eligibility and to provide care management for all services, including BH HCBS. The plan of care, including eligibility for BH HCBS, is reassessed at least annually; reassessment will also occur when the individual´s circumstances or needs change significantly or at the request of the individual.

BH HCBS are delivered to HARP and HARP-eligible HIV SNP enrollees in residential and non-residential settings located in the community under a two-level tier structure determined by the person-centered plan of care. Tier 1 services include Individual Employment Support, Education Support, and Peer Services. Tier 2 services include all Tier 1 services plus additional services for beneficiaries with a higher level of need.

Eligibility for BH HCBS is assessed through the BH HCBS Eligibility Assessment, a standardized clinical and functional assessment tool derived from the interRAI™ Community Mental Health (CMH) Assessment (Hirdes et al., 2000), also referred to as CMH Screen. The eligibility threshold for Tier 2 services, higher relative to Tier 1 services, requires evidence of at least "moderate" level of need as indicated by a state-designated score on the CMH Screen (see Figure 2.2). While these are the current criteria, the original criteria were more stringent (Table 3.1 provides a timeline of key events). Until June 2018, eligibility for Tier 2 services required moderate need on at least four domains (or extensive need on at least one domain). A third criterion was added in June 2019 that permitted previously eligible BH HCBS users to continue receiving services.

FIGURE 2.2. DETERMINATION OF BH HCBS SERVICE ELIGIBILITY

NYS DOH had expected that, by the end of the evaluation period, 75 percent of HARP enrollees would be eligible for Tier 1 BH HCBS, with fewer, 70 percent, eligible for Tier 2 services, and that among those deemed eligible, 75 percent would be utilizing BH HCBS. However, the new expectation based on recent fiscal discussions is that 30 percent of HARP enrollees would utilize BH HCBS (Marleen Radigan/OMH, 2/27/2020).

In addition to BH HCBS, all HARP enrollees, regardless of BH HCBS eligibility or tier, are eligible to receive crisis respite services, including intensive crisis respite and short-term crisis respite in a dedicated facility.

2.4 EVALUATION TIMELINE AND PROGRESS TO DATE

Due to significant impacts of COVID-19 on NYS DOH staff, this interim report only includes information pertaining to the design and implementation of the HARP program evaluation to date. All findings and conclusions will be discussed in a final summative report available in Spring 2021.

REVISED TIMELINE AND BARRIERS TO DATA ACCESS

The original evaluation timeline was revised to allow for additional time for analysis. The progress to date is presented in Figure 2.3. The COVID-19 response within the NYS DOH, along with other related factors, delayed the execution of data use agreements, hampering the ability of the evaluation team to access the data necessary to conduct analysis. As discussed in Section 3, the ability to access and analyze the person-level data is integral to responding to the evaluation questions.

FIGURE 2.3. HARP EVALUATION PROGRESS TIMELINE TO DATE

NEXT STEPS

All evaluation components will be completed and will be published in a final summative report in 2021, as noted in Table 1.1.

---

3. EVALUATION DESIGN AND METHODS

RAND is conducting a comprehensive, statewide independent evaluation of the BH Demonstration implemented in 2015 as part of the latest amendment to the New York Medicaid Redesign Team Section 1115 Demonstration, with a focus on the MMC BH carve-in and the HARP programs (HARP program evaluation). The independent evaluation adheres to the evaluation standards set forth in the Special Terms and Conditions for the Demonstration (New York State, 2020, Section XI, Evaluation Requirements).

The evaluation design is a mixed-method investigation driven by research questions and testable hypotheses that address the goals of the BH Demonstration, including the beneficiary-and system-level impacts of the MMC BH carve-in and HARP programs. Quantitative methods will be used for descriptive purposes and for the outcome evaluations, and qualitative methods will be used to provide context for the quantitative findings and to inform the process evaluation with administrative, provider, and beneficiaries´ perspectives on HARP program functioning and effectiveness. Each type of method will be used as feasible and necessary to address the research questions.

The data sources for the HARP program evaluation include qualitative data collected during the course of the evaluation, and a variety of administrative and survey data previously collected by the NYS DOH, the NYS OMH, and NYS New York State Office of Addiction Services and Supports (OASAS) during the course of health care administrative or clinical operations and quality improvement initiatives. The evaluation team has also planned to integrate data describing county-level characteristics that have the potential to affect program outcomes.

The length of time to be covered by this evaluation-about three years or more (depending on region) after the launch of the BH Demonstration-ensures sufficient program maturity and adequate availability of post-policy patient populations (e.g., comparisons of eligible HARP enrollees receiving BH HCBS with those who have opted out or those deemed ineligible). Hence, RAND expects that the findings of this evaluation will be a valuable resource for NYS DOH and CMS in determining whether and what kinds of changes or corrections to the implementation of the BH Demonstration are needed.

Table 3.1 presents an overview of the goals of the evaluation, the original research questions related to each goal, and the methods proposed to answer each research question. Each will be discussed in Section 3.2; the data sources will be discussed more thoroughly in Sections 3.3 and 3.4.

TABLE 3.1 HARP GOALS AND EVALUATION METHODS AND RESEARCH QUESTIONS

Goals	Methods	Research Questions
1. Improve health and BH outcomes for adults in Mainstream MMC whose BH care was previously carved out in a FFS payment arrangement.	Analyses of Medicaid claims and encounter data and data from the OTNY system; key informant interviews with BH providers.	1. To what extent are MMC enrollees accessing community-based BH specialty services (e.g., ACT, PROS, and FEP programs)? 2. To what extent are MMC enrollees accessing community-based health care or integrated BH/physical health care?
2. Improve health, BH, and social functioning outcomes for adults in the HARP program (HARP eligible, HARP enrolled).	Analyses of Medicaid claims, encounter, and enrollment data; data from CMH screens; plan-reported HEDIS/QARR quality measures; Consumer Assessment of Health Providers and Systems (CAHPS) and HARP PCS patient experience data; interviews with HARP enrollees.	1. How has enrollment in HARP plans increased over the length of the demonstration? 2. What factors are associated with individuals choosing to opt out of HARP plans? 3. What are the demographic, social, functional, and clinical characteristics of the HARP population? Are they changing over time? 4. What are the educational and employment characteristics of the HARP population? Are they changing over time? 5. To what extent are HARP enrollees accessing primary care? 6. To what extent are HARP enrollees accessing community-based BH specialty services? (ACT, PROS, OMH Outpatient Clinic, Continuing Day Treatment, Partial Hospitalization, OASAS Opioid Treatment Program, OASAS Outpatient Clinic, and FEP programs) 7. To what extent are HARP enrollees accessing Health Homes for care coordination? 8. To what extent is HARP quality of care improving, especially related to the HEDIS measures of health monitoring, prevention, and management of BH conditions, cardiovascular disease, asthma, diabetes, and other selected chronic health conditions? 9. To what extent are HARP enrollee experiences with care and access to health and BH services positive? 10. To what extent are HARP enrollees satisfied with the cultural sensitivity of BH providers and their wellness, recovery, and degree of social connectedness? 11. To what extent are HARPs cost effective? What are the PMPM cost of inpatient psychiatric services, SUD ancillary withdrawal, hospital-based detox, and ER services for the HARP population? Are these costs decreasing over time?
3. Develop HCBS focused on recovery, social functioning, and community integration for individuals in HARPs meeting eligibility criteria.	Analyses of Medicaid claims and encounter data; data from CMH screens; data from the MMC HCBS Provider Network Data System; Complaints and Appeals data; interviews with HARP enrollees; key informant interviews with BH HCBS providers, Home Health and HARP administrators, and state officials.	1. To what extent are HARP enrollees deemed eligible to receive HCBS? 2. To what extent are HARP enrollees who are deemed HCBS-eligible receiving HCBS? 3. To what extent has the demonstration developed provider network capacity to provide BH HCBS for HARPs? 4. Does targeting of BH HCBS more narrowly lead to increased numbers of members without access to appropriate BH care? (What are the consequences of targeting availability of BH HCBS to a more narrowly defined population than the criteria in the State Plan?)

The evaluation approach described below is the approach as planned; additional modifications may be made if necessary, during analysis.

3.1 DISCUSSIONS WITH EXPERTS TO REFINE APPROACH

To better understand the policy context, objectives, and challenges to the implementation of the BH Demonstration, the RAND team held calls with subject matter experts within NYS DOH, NYS OMH, NYS OASAS, and OnTrackNY (OTNY) to discuss the background and implementation of the MMC BH carve-in and HARP programs. In addition, the evaluation team held discussions with data experts within these agencies to review the feasibility of fully addressing the research questions given constraints of the available data.

The evaluation team has been using the information thus gathered to inform the qualitative component of the evaluation and revise and enhance the planned quantitative analyses. Some research questions and outcome measures have already been refined to reflect the information available in the data. Moreover, at the time of this writing, it is not yet clear whether the evaluation team will have access to data as far back as 2011, which would be required to address some of the research questions. Additional modifications to the evaluation plan may be necessary.

Using the information gathered in these calls along with publicly available NYS DOH documents, a timeline was developed to indicate key events of the BH demonstration with the potential to impact the implementation and outcomes of the MMC BH carve-in and HARP programs. Table presents these key events and associated dates.

TABLE 3.2. BH DEMONSTRATION TIMELINE

Year	Month	Event
2015	April	DSRIP (Performing Provider Systems)
	August	Amended 1115 Waiver includes BH reform initiatives: (a) qualified MCOs may manage BH benefits for SSI beneficiaries through MMC plans and HARPs (BH carve-in) (b) eligible individuals meeting defined functional needs criteria may access BH-HCBS
	October	MMC BH Carve-in launches in NYC
	October	HARP program launches in NYC (also for eligible HIV SNP enrollees)
2016	January	BH-HCBS become available in NYC (for eligible HARP & HIV SNP enrollees)
	July	MMC BH Carve-in launches in ROS
	July	HARP program launches in ROS (also for eligible HIV SNP enrollees)
	October	BH-HCBS become available in ROS (for eligible HARP & HIV SNP enrollees)
	December	DOH pauses Health Homes (HH) billing to Plans for payment for BH-HCBS assessment and authorizes direct FFS billing to DOH
2017	March	BH-HCBS assessment process was streamlined
	October	Quality Funds become available to MCOs to promote access to BH-HCBS for their HARP enrollees (awards retained based on number of new BH HCBS recipients)
	October – March 2019	BH-HCBS Infrastructure Funds added to the HARP premium for MCOs and providers to develop capacity, connectivity, and innovative service delivery
	October	Revision of BH-HCBS Workflow Guidance for HH-enrolled HARP enrollees
2018	January	Funds for BH-HCBS (including assessments and plans of care) are included in the HARPs´ premium rates (NYC)
	February	Beneficiary-targeted BH-HCBS educational initiatives implemented (e.g., peer focused outreach & training about BH-HCBS)
	April	HARPs may contract with State Designated Entities (RCAs) to conduct BH-HCBS assessments and care planning for enrollees not enrolled in HHs
	May	Expansion of ´Health Home Plus´ to include high-need SMI individuals
	June	HARP becomes an option on the NYS of Health (Exchange)
	June	Changes to eligibility criteria for BH-HCBS Tier 2 services
	July	DOH resumes payments to HHs for BH-HCBS assessment via HARPs´ capitated budgets
	July	All health plans contracted with HHs need to submit Engagement & Enrollment (outreach) Optimization Proposal to enroll high-risk enrollees
	August	Launch of HARP performance measures for HHs
	October	Funds for BH-HCBS (including assessments and plans of care) are included in the HARPs´ premium rates (ROS)
2019	January	Updated HH re-designation policy and chart review and scoring tools (including HARP performance)
	June	Addition of new criterion to eligibility criteria for BH-HBCS
	September	Update of (a) staff qualifications to serve ´Health Home Plus´ SMI enrollees and (b) assessor qualifications for administering the BH-HCBS assessments
	September	Care managers and/or supervisors may request a waiver of education/experience qualifications

3.2 HARP GOALS AND RESEARCH QUESTIONS

The HARP program evaluation was designed to determine the extent to which three goals of the BH Demonstration have been achieved since the program was implemented (October 2015, NYC; July 2016, ROS). These include improving health outcomes (1) in mainstream MMC, (2) among HARP-enrolled beneficiaries, and (3) among BH HCBS-using beneficiaries. These three goals are described below:

GOAL 1: IMPROVE HEALTH OUTCOMES IN MAINSTREAM MMC

The first goal of the BH Demonstration is to improve health and BH outcomes for adults enrolled in Mainstream MMC plans whose BH care was previously carved out in an FFS payment arrangement. As presented in Table 3.3, this goal is broken into two research questions focused on determining the extent to which health and behavioral health outcomes changed. The data sources for this question are Medicaid data and OTNY data, coupled with key informant interviews.

TABLE 3.3: GOAL 1 PROGRAM GOALS, DATA SOURCES, AND OUTCOME MEASURES

Program Goals	Data Sources	Outcome Measures
1. Improve access to BH specialty services, including OTNY (pre: 2011-9/2015; post: 10/2015- 2019; OTNY-based outcomes are only possible 2015-2019)	Medicaid Data (Claims and Encounters)	Percentage of Mainstream MMC enrollees receiving non-FEP BH specialty services (any, specific, average units), by annual period, pre and post (statewide)
	OTNY Data System	Percentage of Mainstream MMC receiving OTNY services, by annual period from baseline (statewide)
	Key informant interviews with BH Providers	Barriers and facilitators to BH specialty care under mainstream MMC
2. Improve access to primary and/or preventive services (pre: 2011-9/2015; post: 10/2015- 2019)	Medicaid Data (Claims and Encounters)	Percentage of MMC enrollees not receiving primary and/or preventive services, by annual period, pre and post (statewide)
	Key informant interviews with BH Providers	Barriers and facilitators to primary and preventive care under mainstream MMC

GOAL 2: IMPROVE HEALTH OUTCOMES AMONG HARP-ENROLLED BENEFICIARIES

The second goal of the BH Demonstration is to improve health, BH, and social functioning outcomes for adults enrolled in the HARP program. This goal has 11 research questions described in Table 3.4.

TABLE 3.4: GOAL 2 PROGRAM GOALS, DATA SOURCES, AND OUTCOME MEASURES

Program Goals	Data Sources	Outcome Measures
1. Increase HARP Enrollment (10/2015-2019)	Medicaid Data (Enrollment Data)	Percentage of HARP eligible beneficiaries enrolled in MMC, HARP, or HIV SNP, by annual period, NYC and ROS
2. Describe characteristics of HARP eligible beneficiaries electing to or declining enrollment in HARP and reasons for declining enrollment in HARP (10/2015- 2019)	Medicaid Data (Claims and Encounters)	Population-level differences in person-level characteristics (demographics and health status/clinical characteristics including BH service utilization) for HARP eligible enrollees who opt-in versus opt-out of HARP, by annual period, NYC and ROS
	Medicaid Choice Enrollment Data	Reasons for opting out of HARP, by annual period, NYC and ROS
	Key informant interviews with BH providers, care coordinators, and state officials	Barriers and facilitators to HARP enrollment, access to specialty BH, primary, and preventive care and use of care coordination services
3. Describe demographic, social, functional, and clinical characteristics of the HARP population (10/2015-2019)	Medicaid Data (Claims and Encounters)	Percentage of HARP enrollees with specific socio- demographics, by annual period, NYC and ROS: population level and individual level Percentage of HARP enrollees with Risk and Protective factors, by annual period, NYC and ROS: population level and individual level
	CMH Screen
	Interviews with HARP enrollees	Barriers and facilitators to HARP enrollment, access to care and care coordination
4. Improve educational and employment characteristics of the HARP population (10/2015- 2019)	CMH Screen	Educational and employment attainment for HARP enrollees, by annual period, NYC and ROS: population level and (risk-adjusted) individual level
5. Improve access to primary and/or preventive services for the HARP population (NYC pre: 10/2013-9/2015; NYC post: 10/2015; ROS pre: 7/2014- 6/2016; ROS post: 7/2016- 6/2018)	Medicaid Data (Claims and Encounters)	Percentage of HARP eligible enrollees not receiving primary or preventive health services, by annual period, NYC and ROS
6. Improve access to BH specialty services for the HARP population (NYC pre: 10/2013- 9/2015; NYC post: 10/2015; ROS pre: 7/2014-6/2016; ROS post: 7/2016-6/2018)	Medicaid Data (Claims and Encounters)	Percentage of HARP eligible enrollees receiving BH specialty services, by annual period, NYC and ROS
	OTNY Data System	Percentage of HARP enrollees receiving OTNY services, by annual period from baseline (statewide)
7. Increase access to care coordination (Health Homes) for the HARP population (NYC pre: 10/2013-9/2015; NYC post: 10/2015; ROS pre: 7/2014-6/2016; ROS post: 7/2016-6/2018)	Medicaid Data (Claims and Encounters)	Percentage of HARP eligible enrollees engaged in Health Home services, by annual period, NYC and ROS
8. Improve quality of care related to health monitoring, prevention, and management of chronic health conditions for the HARP population (NYC pre: 10/2013-9/2015; NYC post: 10/2015; ROS pre: 7/2014- 6/2016; ROS post: 7/2016- 6/2018)	Plan-reported HEDIS® / QARR quality measures	Quality of care among HARP eligible enrollees, by annual period, NYC and ROS
	Medicaid Data (Claims and Encounters)
9, 10. Improve experiences with and satisfaction with care for the HARP population (10/2015-9/2019)	CAHPS	Percentage of HARP enrollees who: report it was easy to get BH treatment; report it was easy to get SUD treatment; rated their BH treatment positively; rated their SUD treatment positively; rated items related to communication with health care providers positively. By annual period when data are available, NYS and ROS.
	HARP PCS	Percentage of HARP enrollees who: report that BH care was responsive to their cultural background; had a positive overall rating of quality of life; had overall positive beliefs about health and wellness; rated PCS questions in the social connectedness domain positively. By annual period when data are available, NYS and ROS.
11. Decrease utilization and PMPM cost of acute care BH services (inpatient psychiatric services, SUD ancillary withdrawal, hospital-based detox, and ER services) for the HARP population (NYC pre: 10/2013-9/2015; NYC post: 10/2015; ROS pre: 7/2014- 6/2016; ROS post: 7/2016- 6/2018)	Medicaid Data (Claims and Encounters)	Risk-adjusted utilization of acute care and non-acute care (outpatient) BH services among HARP eligible enrollees, by annual period (PMPM/Y), NYC and ROS Risk-adjusted PMPM cost of acute care and non- acute care (outpatient) BH services among HARP eligible enrollees, by annual period (PMPM/Y), NYC and ROS
	MHARS

GOAL 3: IMPROVE HEALTH OUTCOMES AMONG BH HCBS-USING BENEFICIARIES

The third goal of the BH Demonstration is to develop BH HCBS focused on recovery, social functioning, and community integration for HARP enrollees meeting eligibility criteria for such services. This goal, presented in Table 3.5, has four research questions focused on assessing the level of enrollment in BH HCBS by HARP enrollees and cost reduction. The data sources for this question are Medicaid claims and encounters data, CMH Screen data, MMC data, complaints and appeals data, coupled with key informant interviews.

TABLE 3.5: GOAL 3 PROGRAM GOALS, DATA SOURCES, AND OUTCOME MEASURES

Program Goals	Data Sources	Outcome Measures
1. Increase the number of HARP enrollees assessed for eligibility to receive BH HCBS and describe those deemed BH HCBS- eligible (10/2015- 2019)	Medicaid Data (Claims and Encounters)	Percentage of HARP enrollees who are assessed for BH HCBS eligibility, by annual period, NYC and ROS Percentage of HARP enrollees who are deemed BH HCBS eligible (any, by Tier), by annual period, NYC and ROS Population-level characteristics of HARP enrollees deemed eligible for BH HCBS – these include HARP Plan membership, socio-demographics (including geographical region), health status/clinical characteristics, and functional status). By annual period, NYC and ROS.
	CMH Screen
2. Increase the number of BH HCBS-eligible HARP enrollees who are receiving BH HCBS (2016-2019)	Medicaid Data (Claims and Encounters)	Percentage of BH HCBS-eligible HARP enrollees receiving any BH HCBS, by month and annually, at the HARP plan level, regionally (NYC, ROS, by county) and statewide; and annual percent change Eventually: Risk-adjusted percentage of BH HCBS- eligible HARP enrollees receiving BH HCBS (any, at least 6 months) (compared to those receiving none, less than 6 months)
	CMH Screen
	Interviews with HARP Enrollees	Barriers and facilitators to accessing BH HCBS services
3. Develop provider network capacity to provide BH HCBS for HARPs (2016-2019)	Medicaid Data (Claims and Encounters)	Number of providers contracted for BH HCBS in HARP plans, by HARP plan, by annual period, regionally (NYC, ROS, by
	MMC HCBS Provider Network Data System	county) and statewide Rate of BH HCBS providers per 1000 BH HCBS eligible enrollees, by annual period, regionally (NYC, ROS, by county) and statewide
	Complaints and Appeals Data	Rate of complaints and appeals due to denial of BH HCBS services per 1000 BH HCBS eligible enrollees, by annual period, regionally (NYC, ROS, by county) and statewide
	Key informant interviews with BH HCBS providers, Health Home and HARP administrators, State officials	Barriers and facilitators to provision of BH HCBS services and the effectiveness of the services provided
4. Reduce total PMPM costs for BH HCBS recipients through reductions in higher cost (acute-care) services (NYC pre: 10/2013-9/2015; NYC post: 10/2015; ROS pre: 7/2014-6/2016; ROS post: 7/2016-6/2018)	Medicaid Data (Claims and Encounters)	Risk-adjusted total Medicaid PMPM costs, by annual period (PMPM/Y), NYC and ROS Risk-adjusted PMPM costs for acute care BH services, by annual period (PMPM/Y), NYC and ROS Percentage using acute care BH services, by annual period, NYC and ROS Percentage using non-acute (outpatient) BH services, by annual period, NYC and ROS

3.3 QUANTITATIVE METHODS

This evaluation will adopt a rigorous analytic approach that combines descriptive statistical analyses with state-of-the-art methods, allowing for unbiased inferences. These methods include quasi-experimental methods that can allow for causal inference of the impact of the BH Demonstration while also utilizing the temporal trends in the data. Where possible, RAND plans to strengthen the validity and robustness of the analyses by leveraging features of the BH Demonstration including the regional and temporal phasing-in of the HARP program and the BH HCBS benefit package, HARP enrollees´ ability to opt out of the program, and the ability of those who are eligible to receive BH HCBS to opt in or out.

Our approach will permit minimizing threats to valid causal inferences posed by the effect of other ongoing health care policies (e.g., other Medicaid redesign initiatives, provisions of the Affordable Care Act). Concurrent policies and other unobserved factors could affect estimates of program effects if they are correlated with the BH Demonstration and specifically HARP. This possibility will be investigated in three ways: (1) examining the relative timing of other key policies with HARP implementation, (2) including controls for other policies in the causal models, and (3) estimating models with time period indicators in difference-in-differences model settings to account for other time invariant unobserved policies or idiosyncratic effects.

A critically important task of the HARP program evaluation is to identify comparison beneficiaries for several of the analytic tasks. Because HARP-eligible beneficiaries can opt out, those who opt out provide a potential comparison group. The evaluation team will assess whether, at the time of the initiation of the HARP program, the identified comparison group is comparable to the specific population of HARP enrollees being considered (the "treatment" group). RAND will begin this task with the definition and identification of comparison group beneficiaries that will be matched to the treatment group with respect to person-and small area-level characteristics prior to the implementation of the program. To adjust for differences across measured variables in these treatment and control settings, RAND plans to apply propensity weights in order to achieve good balance across treatment and comparison groups. To further assure comparability between treatment and comparison groups, RAND will examine trends over time in both groups during the years prior to the program implementation to assess the model assumptions that trends in utilization and access, process (quality of care), and costs are parallel.

DATA SOURCES

A variety of secondary data sources will be used to construct study variables (outcome measures and covariates for risk adjustment) for the quantitative component of the HARP program evaluation. Data will be provided by the NYS DOH and OMH and will include:

1. ONTrackNY (OTNY) Data System. Patient and program-level information collected by the OTNY Coordinated Specialty Care program, a statewide program that began in earnest in 2015. The data system includes socio-demographics, clinical history and treatment, and program outcomes of enrolled patients with FEP, and OTNY program components. These data will be used primarily for the assessment of access to OTNY services for patients with FEP (outcome measure); they may also be used for risk adjustment in regression models.
2. CMH Screen data. A mix of lifetime and current patient self-reported information and assessor-gathered information collected as part of the assessment of BH HCBS eligibility with the BH HCBS Eligibility scale, brief and full¹, a standardized clinical and functional assessment tool derived from the interRAI™ Community Mental Health Assessment (Hirdes et al., 2000). The CMH Screen is required annually for all HARP and HARP-eligible HIV SNP enrollees, but not for HARP eligible beneficiaries who opt out and return to Mainstream MMC plans. Domains include socio-demographic characteristics (e.g., marital status, homelessness); health status (BH and chronic health conditions); functional status (independent living skills, cognitive skills, social relations, employment, education and finances); BH service utilization; risky behaviors (substance use, harmful/self-injurious behaviors); traumatic events; and criminal justice system involvement. As such, the data may be used to describe program outcomes (e.g., health status, functional status), as well as risk factors (e.g., traumatic life events, homelessness, criminal justice involvement, substance use, chronic physical health conditions) and protective factors (e.g., social relations, education, employment, adequate finances). The CMH screen is required annually for all HARP and HARP-eligible HIV SNP enrollees; the number of individuals who have been assessed with the CMH screen has varied over the years but has not met expectations. These data, as available, will permit assessment of sociodemographics, health status/clinical, and recovery-related outcomes (outcome measures); they may also be used for risk adjustment in regression models.
3. HEDIS®/QARR Plan-Reported Metrics. Person-level quality of care information in the form of HEDIS®/QARR quality measures collected by Mainstream MMC plans, HARPs, and HIV SNPs and reported annually to NYS DOH. These data will permit assessment of quality of care (outcome measures).
4. CAHPS® survey data. De-identified patient self-reported information on experience with access to care and experiences with health care providers and health plan staff, assessed through the Health Plan version of the CAHPS® survey and collected every other year from a sample of adults enrolled in all MMC product lines; BH-specific questions include need for BH/SUD treatment, access to BH/SUD treatment, satisfaction with BH/SUD treatment, and self-rating of overall BH. These data will permit assessment of self-rated need for care, experiences/satisfaction with care, and self-rated BH (outcome measures).
5. HARP Perception of Care (PCS) survey data. Patient self-reported information on perception of outcomes, access and quality of care, appropriateness of services, social connectedness, wellness, and quality of life collected through a survey of randomly selected enrollees in HARPs or HIV SNPs; demographics are also collected. The survey was adapted from the Experience of Care and Health Outcomes (ECHO) Survey, the Mental Health Statistics Improvement Program (MHIP)/OMH Consumer Assessment of Care Survey, and others. It was piloted in early 2017 and implemented in the fourth quarter of 2017 and again in 2019. These data will permit assessment of experience and satisfaction with care, enrollees´ satisfaction with their BH providers´ cultural sensitivity, and enrollees´ satisfaction with their wellness, recovery, and degree of social connectedness (outcome measures).
6. Medicaid Data. Information maintained by the Medicaid Data Warehouse containing billing records for health care services, including pharmacy, for individuals enrolled in Medicaid in a given year, whether under FFS arrangements or MCOs (i.e., claims and encounters). Source of information on Medicaid enrollment status, plan membership, BH HCBS eligibility status, demographic, health status (diagnoses including BH and chronic physical health conditions; Clinical Risk Group categories), service utilization, provider associated with the billed services, and cost of health care for all Medicaid enrollees; available with a six-month lag. These data will permit assessment of HARP enrollment, BH HCBS eligibility, diagnostic characteristics, service utilization patterns, including BH HCBS and cost of health care (outcome measures), and may also be used for risk adjustment in regression models.
7. Medicaid Choice Enrollment Data. Information on the HARP enrollment process collected on an ongoing basis by New York Medicaid Choice, the enrollment broker, and available since program implementation. Data include passive enrollment, opt-out acknowledgement letters distributed and returned, number of beneficiaries who are enrolled, number of beneficiaries who opt out, and reasons for opting out. These data will permit assessment of reasons for opting out of HARPs (outcome measure).
8. Complaints and Appeals Data. Complaint and appeal information pertaining to denials of access to BH HCBS. Complaint information collected through a designated email address available to BH HCBS providers since October 2015 has been systematized to allow for tracking of a number of fields (e.g., type of inquiry, fields for MCOs to indicate if they are part of the inquiry, etc.). This information is monitored and acted upon by NYS DOH, OMH, and OASAS; OMH is able to generate complaint reports from a linked database. These data will permit assessment of the number of complaints and appeals related to access to BH HCBS (outcome measure).
9. MMC HCBS Provider Network Data System. Information on providers who have applied to provide BH HCBS, including contact information, location, services provided, staff qualifications, and funding information. These data will permit assessment of BH HCBS provider availability to meet the need, and HARP/HIV SNP contracts by geographic area (outcome measures).
10. Mental Health Automated Record System (MHARS) data. Information maintained by OMH on inpatient, residential, and outpatient utilization in NYS Psychiatric Centers, used to identify psychiatric inpatient utilization not captured in the Medicaid data. These data permit a complete assessment of the number of inpatient admissions and inpatient days (outcome measure).

In addition to these NYS DOH/OMH data, the evaluation will incorporate contemporaneous data from Area Health Resource Files (ARF), a collection of publicly available data assembled by the Health Resources & Services Administration (HRSA) or PolicyMap, a web-based data warehouse. Both datasets aggregate information from multiple sources including the Centers for Disease Control and Prevention, HRSA, the U.S. Census, and other neighborhood-level datasets. Small area-level information being considered include sociodemographic characteristics (e.g., urbanicity, household income) and characteristics of the healthcare infrastructure (e.g., psychiatrists per 1,000 population, HRSA-designated health professional shortage area). This information is available at various geographic levels, including ZIP code and county.

ANALYTIC APPROACHES

Throughout the evaluation, different analytic approaches will be used depending on the research questions of interest. They include descriptive methods as well as quasi-experimental state-of-the-art methods to enable causal inferences.

1. Descriptive Statistics. This approach will be used for simple population-level, year-to-year comparisons in NYC and ROS during the evaluation period. With it, RAND will examine the characteristics of HARP enrollees in NYC and ROS in each annual period since program implementation. For categorical variables, this will consist of Chi-square test and McNemar´s chi-square test (to compare binary outcomes between correlated groups for each region before and after implementation). For continuous variables on the other hand, we will use the Analysis of Variance (ANOVA) test; paired t-test (to compare pairs of years); and the Bonferroni adjustment for multiple pair comparisons. Whenever repeated measures are analyzed with ANOVA for yearly changes within each region, the RAND team will evaluate whether the sphericity assumption of this method is violated.
2. Interrupted Times Series. This pre-post approach will be used for the evaluation of trends/trajectory of outcomes over an extended period of time that covers the implementation of the HARP program. Depending on the research question, the period was two or four years before, and two or four years after program implementation. For the HARP evaluation, the outcome domains to use are health status, functional status, and service utilization. This quasi-experimental method will be utilized when non-BH/non-HARP control groups are not available as it minimizes the confounding effect of other potential drivers of observed effects, including ongoing health care reform initiatives. The RAND team will also utilize a segmented regression (Wagner et al., 2002) to analyze the interrupted time series data. Variables to include in the regression adjustment potentially include health status (diagnostic history), prior service utilization patterns (inpatient, ED, primary care), and other resource use. This analysis will enable the evaluation of changes in the level and trend in the outcome variable from pre-to post-intervention and use the estimates to test causal hypotheses about the HARP program. In the post-intervention period, actual rates for the various metrics for each month will be compared to expected rates, while controlling for patient-level confounders, secular trend, serial autocorrelation, and seasonal fluctuation in the outcome variable.
3. Difference-in-Differences. This pre-post approach will be employed when concurrent comparison groups are available, thus enabling a robust assessment of program outcomes. For the HARP program evaluation, the outcome domains are quality, service utilization, and cost. The treatment and control groups will be:
   1. HARP-eligible individuals who opt into HARP (treatment), versus those who opt out of the HARP (HARP-Opt Out) and were enrolled in mainstream MMC (control)
   2. HARP enrollees who are BH HCBS eligible who opt for BH HCBS services (treatment) versus those who do not opt for BH HCBS and received only traditional (non-BH HCBS) services (control)
   The outcomes of interest were measured over consecutive periods of two (2) years before/after program implementation:
   • Period 1: 10/2013-9/2015 (NYC), 7/2014-6/2016 (ROS)
     Period 2: 10/2015-9/2017 (NYC), 7/2016-6/2018 (ROS)
   This quasi-experimental approach accounts for any secular trend/changes in the outcome metrics as it eliminates fixed differences not related to program implementation; thus, remaining significant differences may be validly attributable to the impact of program implementation (Harman et al., 2011). The difference-in-differences approach requires that pairs of "treatment" and "control" individuals comparable on key observed confounders be identified through Propensity Score Matching – see below.
4. Longitudinal Mixed Effect Regression. This approach uses a Generalized Linear Mixed Model (GLMM) to estimate an average program effect while adjusting for key covariates when examining change trajectories (Diggle et al., 2002; Tooze, Grunwald and Jones, 2002). For the HARP evaluation, the outcome domains are health status, functional status, and service utilization. This quasi-experimental approach separates the effects of time from that of the HARP program implementation, accommodating the heterogeneity in the program implementation effect, and accounting for serial correlations within individuals and variation of risk/protective factors and outcomes over time due to strong temporal trends. The multivariable mixed effects regressions to be used will include fixed effects, namely demographics (age, gender, and race/ethnicity) and time, and random effects assessed at each annual time point, namely risk and protective factor levels as assessed with the CMH Screen. Random effects will be incorporated in the models on two (2) levels: for persons within areas/site and for change over time within persons. The HARP evaluation research questions to be addressed with GLMM are the one that were assessed following program implementation.
5. Propensity Score Matching. This approach controls for potential confounding by identifying individuals with similar characteristics belonging to the treatment and control groups, thus enabling the use of quasi-experimental causal models (Austin, Grootendorst and Anderson, 2007). In the HARP evaluation, propensity score matching will be used in combination with the difference-in-differences approach to examine the impact of the HARP benefit on health outcomes and to examine the impact of the BH HCBS on recovery outcomes. The method uses a logistic regression or a generalized boosting method (GBM) to estimate each individual´s conditional probability (or propensity score) of belonging to the treatment group. Predictors include variables included in the HARP algorithm and others related to sociodemographics, health status/clinical characteristics, and functional status not included in the algorithm. A greedy matching algorithm with an appropriate matching ratio of treatment to control individuals will be used to create a matched analytic cohort based on the estimated propensity score and other variables such as service utilization variables assessed prior to program implementation. The RAND team will a priori select the confounding variables for inclusion in the models using the team´s subject matter expertise and also consulting with other subject matter experts. Balance in covariate distribution between treatment and control individuals in the matched analytic cohort will be assessed with weighted standardized difference.

3.4 QUALITATIVE METHODS

The qualitative component of the HARP evaluation seeks to provide additional context and multiple perspectives on program implementation, including barriers and facilitators to implementation success and insight into potential mechanisms of impact on program outcomes. As described below, the qualitative data collection component of the HARP evaluation is near complete. Interviews with key informants other than the HARP enrollees have been completed. Due to the COVID-19 pandemic, procedures for interviews with HARP enrollees had to be revised. The interview protocol, recruitment methods, and institutional review board (IRB) approval for the interviews with HARP enrollees are being developed.

DATA SOURCES

For the completed interviews, the evaluation team has employed a combined purposive and snowballing sampling approach to recruit key informants. Through maximum variation sampling, the evaluation team sought to maximize the diversity of organizations represented by key informants and considered factors such as agency type, geographic region within NYS, degree to which areas served were urban or rural, and the program size and number of beneficiaries served (e.g., number of HARP enrollees within an MCO, number of BH HCBS enrollees served by a provider organization). Publicly available data and state agency reports were reviewed to identify and sample potential agencies and stakeholders in order to capture variation along key factors. This was complemented by snowball sampling, wherein several key informants identified other stakeholders who could provide additional perspectives and who were subsequently invited to participate (e.g., Health Home organizations identifying Care Management Agencies in different regions with varying numbers of HARP enrollees).

The key informants that have already been interviewed represent organizational leadership staff, from the program director to senior executive management levels, in organizations including MCOs, Health Homes, Care Management Agencies, providers of BH services (e.g., ACT, PROS, BH HCBS), statewide groups (e.g., patient, provider, and trade associations), and NYS agencies (e.g., OMH, OASAS). The evaluation team has already conducted 32 key informant interviews. The interview tool is described below and in Appendix A.

A similar approach will be taken for the interviews with HARP enrollees. To identify HARP enrollees for participation, evaluators will utilize purposive and convenience sampling strategies. To capture a range of perspectives, the evaluation will seek to maximize the diversity of HARP enrollees who participate, considering factors such as geographic region within NYS, location in urban or rural areas, status of enrollment in BH HCBS, and a range of demographic characteristics (e.g., gender, race, diagnosis). The evaluation team anticipates conducting approximately 10 interviews with HARP enrollees.

RESPONDENT RECRUITMENT

Potential key informants received an e-mail inviting them to participate in the evaluation interview and to contact the evaluators if they were interested in participating. An information sheet was e-mailed to key informants in advance of scheduled interviews and reviewed prior to commencing the interview.

For the HARP enrollee interviews, provider agencies will identify potential HARP enrollees and provide them with information about the evaluation. HARP enrollees interested in participating will contact the evaluators directly or inform the provider agency staff that they consent to have the evaluators contact them.

INTERVIEWER TRAINING

Prior to beginning the key informant interviews, the qualitative team received training on the MMC BH carve-in, the HARP Program, the BH HCBS program, and the roles of various stakeholder agencies involved in the implementation and operation of these initiatives and programs. The training included a review of documents, participation in discussions with DOH, OMH, and OASAS subject matter expert staff, and internal discussions with the project leads and technical advisors who have experience with NYS Medicaid and the development and implementation of these initiatives. The training ensured that the interviewers were aware of issues relevant to the program implementation for each type of key informant.

DATA COLLECTION/CONDUCTING INTERVIEWS

A semi-structured interview guide for key informants representing a diversity of (non-HARP enrollee) stakeholders was developed (Appendix A) and covered the MMC BH carve-in, the HARP program, and the BH HCBS program. The interview guide focuses on understanding the implementation and operation of each initiative/program, including barriers and facilitators to implementation, as well as factors that may influence program access and outcomes.

Interviews with key informants other than the HARP enrollees were conducted virtually and lasted one hour, on average. The majority of data collection consisted of individual interviews with one identified key informant; in several cases the originally recruited key informant suggested additional informants to be included in the interview.

Interviews are conducted by one qualitative researcher, with an additional researcher taking notes concurrently that are used to produce a written interview summary. Interviewers cover core topic areas but flexibly maneuver through the interview guide and probe certain topics more in-depth as appropriate. Interviews are audio-recorded and transcribed verbatim. The IRB of the NYS Psychiatric Institute determined that data collection with key informants who were not HARP enrollees did not constitute human subjects research and was thus exempt from review. Review of data collection with HARP enrollees is pending.

Interviews with HARP enrollees will be conducted individually by phone or virtually. A semi-structured interview guide for HARP enrollees is being developed. Interview guides for HARP enrollees will focus on HARP and BH HCBS barriers and facilitators to program enrollment and access to care management and services, as well as satisfaction with providers/services, and perceived impact on individual outcomes (e.g., recovery, functioning, community integration).

ANALYSIS

Analytic methods, aligned with recommendations of Bradley, Curry, and Devers (2007), will follow a grounded theory approach to developing coding structures that emphasize inductive codes emerging directly from the data (Bradley, Curry and Devers, 2007). Consistent with grounded theory, qualitative analysis begins concurrently with data collection, allowing interviews to be shaped by preliminary concepts and themes emerging from the data. The analysis will proceed in a series of steps: developing initial codes (open-coding), validating & using the codes (i.e., coding all transcripts with a final code list), clustering and interpreting the codes, and developing broader findings and themes. Strategies to ensure rigor include weekly data collection and analysis debrief meetings, development of interview summaries and memos, and the use of multiple coders. As described below, analyses of the qualitative data will inform evaluation of each of the HARP program evaluation goals.

Goal 1 (Improve health and BH outcomes for adults in Mainstream MMC whose BH care was previously carved out in an FFS payment arrangement): This goal will be addressed using data from key informant interviews with MCOs, Health Homes, Care Management Agencies, providers of BH services (e.g., ACT, PROS, substance use treatment), statewide groups (e.g., patient, provider, and trade associations), and NYS agencies (e.g., OMH, OASAS). Analyses will be informed by interview content that focuses on how the mainstream MMC BH carve-in has affected stakeholders´ work, and barriers and facilitators that, according to these informants, may impact Medicaid beneficiaries´ access to services.

Goal 2 (Improve health, BH, and social functioning outcomes for adults in the HARP): In addition to the key informants in Goal 1, this goal will also draw on interviews with HARP enrollees, who will provide additional perspectives on barriers and facilitators to enrollment, accessing primary/preventive services, specialty behavioral health care services, and care coordination. In addition, the RAND team will explore HARP enrollees´ perceptions of care quality, including experiences interacting with providers and receiving services, satisfaction with these services, and how these services are aligned with educational, employment, wellness, recovery, social functioning, and community integration outcomes. Analyses will focus on identifying factors that, in the view of key informants, affect the impact of the HARP program on enrollee health, BH, and social functioning.

Goal 3 (Develop BH HCBS focused on recovery, social functioning, and community integration for individuals in HARPs meeting eligibility criteria): Data from all key informant interviews will be used to address Goal 3. Analysis will examine informant perspectives on assessment of BH HCBS eligibility, linkages between MCOs, Health Homes and BH HCBS providers; BH HCBS providers´ assessment processes for specific services; and ongoing approval processes from Health Homes providers and Managed Care Organizations.

Analysis of interviews with HARP enrollees and with HARP enrollees receiving BH HCBS will explore their experiences with qualifying and using BH HCBS. Finally, a systematic document review will be used to examine complaints and appeals related to BH HCBS services.

---

4. FINDINGS

Due to the delays in initiating the HARP program evaluation, findings for the three Goals and aligned Research Questions listed in Table 3.1 are not yet available for presentation. The proposed timeline for remaining evaluative tasks is presented above in Table 1.1. All findings will be presented in the final summative report, available Spring 2021.

---

5. POLICY IMPLICATIONS

Because there are no findings yet available, no policy implications can be provided at this time. A thorough discussion of the policy implications of the evaluation findings will be included in the final summative report.

---

6. INTERACTIONS WITH OTHER STATE INITIATIVES

An in-depth empirical investigation of the manner in which the implementation and effects of the BH demonstration, namely the MMC BH carve-in and the HARP programs, were affected by other state initiatives is out of scope for the HARP program evaluation as proposed and executed under the RFP and RAND contract. Information on other policy initiatives implemented by the state and potentially affecting the BH demonstration was alternatively collected to assist with the design of the analyses and to interpret and provide context to the pending findings. Potential statistical interactions will be explored and discussed in the final summative report, available Spring 2021.

The state initiatives that will be reviewed for potential interactions with the implementation of the HARP program in the final summative report include:

• Other provisions of the DSRIP Program, including payment reform in the form of a Value Based Payment (VBP) Roadmap
• Provisions of the Affordable Care Act (ACA), including the Medicaid Health Home program and Medicaid access expansion.

---

REFERENCES

Austin, Peter C, Paul Grootendorst, and Geoffrey M Anderson, "A Comparison of the Ability of Different Propensity Score Models to Balance Measured Variables Between Treated and Untreated Subjects: a Monte Carlo Study," Statistics in medicine, Vol. 26, No. 4, 2007, pp. 734-753.

Bradley, E. H., L. A. Curry, and K. J. Devers, "Qualitative Data Analysis for Health Services Research: Developing Taxonomy, Themes, and Theory," Health Serv Res, Vol. 42, No. 4, 2007, pp. 1758-1772.

Diggle, Peter, Peter J Diggle, Patrick Heagerty, Patrick J Heagerty, Kung-Yee Liang, and Scott Zeger, Analysis of Longitudinal Data: Oxford University Press, 2002.

Harman, Jeffrey S, Christy H Lemak, Mona Al-Amin, Allyson G Hall, and Robert Paul Duncan, "Changes in Per Member Per Month Expenditures after Implementation of Florida´s Medicaid Reform Demonstration," Health Serv Res, Vol. 46, No. 3, 2011, pp. 787-804.

Hirdes, John P, Mounir Marhaba, Trevor Frise Smith, Leah Clyburn, Lori Mitchell, Ra Lemick, N Curtin Telegdi, Edgardo Perez, Peter Prendergast, and Terry Rabinowitz, "Development of the resident assessment instrument–mental health (RAI-MH)," Hosp Q, Vol. 4, No. 2, 2000, pp. 44-51.

New York State, "MRT Plan Current STCs - August 10, 2020," September, 2020. As of October 29

New York State Department of Health, Office of Quality and Patient Safety, "Independent Evaluation of the New York State (NYS) Health and Recovery Plans (HARP) Program and Self-Directed Care (SDC) Pilot Program: Request for Proposal #20024," February, 2019.

Tooze, Janet A, Gary K Grunwald, and Richard H Jones, "Analysis of Repeated Measures Data with Clumping at Zero," Statistical methods in medical research, Vol. 11, No. 4, 2002, pp. 341-355.

Wagner, Anita K, Stephen B Soumerai, Fang Zhang, and Dennis Ross-Degnan, "Segmented Regression Analysis of Interrupted Time Series Studies in Medication Use Research," Journal of clinical pharmacy and therapeutics, Vol. 27, No. 4, 2002, pp. 299-309.

---

APPENDIX A. KEY STAKEHOLDER INTERVIEW PROTOCOL

HARP & HCBS:
Interview Guide: Non-Client Stakeholder

Participant ID: ___________________________________ Interview Date: _______________________________

Region: Central ________ Hudson River ________ Long Island ________ NYC ________ Western ________

Providers Only Number of HCBS Clients Served: 1-10 11-20 21-40 41-60 61-80 81-100 100+

Stakeholder Type: _____________________________

Agency Type: _______________________________________________________

Interviewer: ____________________________________

The purpose of this interview is to explore your perspective and experience regarding the shift of behavioral health services for adults with Medicaid into Managed Care in New York State. This included enrolling eligible adults with Medicaid and significant behavioral health (BH) needs into Health and Recovery Plans (HARPs). HARPs sought to offer an enhanced benefits package that would expand access to specialized services and care coordination of physical health, mental health, and substance use services. HARP members work with Health Home agencies, or other state-designated entities, to develop a person-centered plan and to meet wellness goals, including accessing an array of specialty services, such as BH Home and Community Based Services (HCBS). BH HCBS seek to help people move forward in their recovery and life goals, such as improving quality of life, finding employment, going to school, managing stress, and living independently.

The interview will take approximately 60 minutes to complete. Again, the goal is to learn about your views and experience of the shift in behavioral health services to Medicaid Managed care, and in particular the implementation of HARPs and HCBS in New York State. There are no right or wrong answers to these questions. We are only interested in your honest opinions. Any questions before we begin?

INTERVIEWER PROBES

1. Enrollment issues
2. Administrative issues/burden - billing? Paperwork/documentation?
3. Developing plans of care?
4. Care coordination/integration – coordinating care among mental illness, substance use, and physical healthcare providers
5. Communication with other agencies (e.g., OMH, Health Homes, Managed Care)
6. Clients´ access to services?
   1. What services are most accessible? What services are now available to clients that didn´t used to be?
   2. What services are harder to access or are under-utilized? What services are no longer available to clients?
7. Quality of services/care?
8. Impact/Measuring impact; recipient/enrollees/client outcomes?
9. Funding/Financing

<< BEGIN RECORDING >> << BEGIN RECORDING >> << BEGIN RECORDING >>

Role

What is your role in this organization/agency?

1. How do your responsibilities relate to HARPs and HCBS?
2. How familiar are you with HARPs and HCBS?

I. Behavioral Health Carve-in for Adults in Mainstream Managed Care Goal One: Improve health and BH outcomes for adults in Mainstream MMC whose BH care was previously carved out in an FFS payment arrangement

Now I´m going to ask you questions about your experience and thoughts on transitioning behavioral health services to mainstream managed care.

2. What has your experience been with the transition to mainstream managed care for individuals whose behavioral health benefits were previously carved out in a Fee for Service arrangement?
   1. How has it been different from when behavioral health had been carved out through a fee-for-service arrangement?
3. How has the transition to Medicaid Managed Care for behavioral health impacted your agency?
   1. SEE PROBES
4. How has the switch to mainstream Medicaid Managed Care impacted Medicaid recipients with behavioral health needs?
   1. How has it impacted recipients´ administrative burden (e.g., paperwork, applications)?
   2. How has it impacted recipients´ access to services?
   3. How has it impacted recipient outcomes (e.g., health, recovery, wellness goals, quality of life, stress management, employment, school, community involvement/integration, functioning)?
5. What have been some of the benefits of having mainstream Medicaid Managed Care plans manage behavioral health for adults in New York State?
   1. For recipients? Are there certain recipients who have benefited more/less?
   2. For your organization? Are there certain organizations who have benefited more/less?
   3. For systems of care? Are there certain systems of care who have benefited more/less?
   4. SEE PROBES
6. What have been some of the challenges of having behavioral health managed by mainstream Medicaid Managed Care?
   1. For recipients?
   2. For your organization?
   3. For the system of care?
   4. SEE PROBES
   5. What can be done to address those challenges?
   6. If not addressed: What can be done to improve access to services? Quality of services? Coordination or integration of care? Client outcomes?

II. HARP Goal 2: Improve health, BH, and social functioning outcomes for adults in the HARP

Now I´m going to ask you some specific questions about Health and Recovery Plans.

7. What has been your experience with the HARP program?
   1. Experiences with HARPs in general and care management?
   2. Experiences specifically with HCBS aspects of HARP?
8. How has the implementation of HARP impacted your agency´s work?
   1. SEE PROBES
   2. What has made your agency´s work easier? More difficult?
9. How would you describe your interactions with other agencies/organizations involved in HARPs?
   1. Managed Care Companies
   2. Health Homes
   3. DOH, OMH, OASAS
   4. Service Providers
      1. Mental Health
      2. Substance use
      3. Primary care
      4. Other psychiatric services (ACT, PROS)
      5. Other services/providers?
10. How has belonging to a HARP program impacted enrollees?
    1. Ability to access care?
    2. Quality of care received?
    3. The degree to which their care is integrated?
    4. Enrollee outcomes (e.g., health, recovery, wellness goals, quality of life, stress management, employment, school, community involvement/integration, functioning)?
    5. In what areas have you seen the biggest improvement for enrollees?
    6. In what areas have you seen less improvement for enrollees?
    7. Are there any potential long-term benefits for enrollees?
11. What have been some of the benefits of having the HARP program? What has gone well?
    1. For HARP enrollees? Are there certain enrollees who have benefited more/less?
    2. For your organization? Are there certain organizations who have benefited more/less?
    3. For systems of care? Are there certain systems of care who have benefited more/less?
    4. SEE PROBES
    5. How would you define or measure HARP success?
12. What have been some of the challenges of the HARP program?
    1. For HARP enrollees?
    2. For your organization?
    3. For systems of care?
    4. SEE PROBES
    5. What could be improved? What would help address some of the challenges?
    6. If not addressed: What can be done to improve access to services? Quality of services? Coordination or integration of care? Client outcomes?
13. What other changes would you suggest making to the HARP program?
    1. SEE PROBES

III. HCBS Goal 3: Develop HCBS focused on recovery, social functioning, and community integration for individuals in HARPs meeting eligibility criteria

Finally, I´m going to ask you some questions specifically about Home and Community Based Services:

14. What has been your experience with HCBS?
    1. With Tier 1 HCBS?
    2. With Tier 2 HCBS?
15. How has the implementation of HCBS affected your agency?
    1. SEE PROBES
16. How would you describe your interactions with other agencies/organizations involved in HCBS?
    1. Managed Care Companies
    2. Health Homes
    3. DOH, OMH, OASAS
    4. Service Providers
       1. Mental Health
       2. Substance use
       3. Primary care
       4. Other psychiatric services (ACT, PROS)
       5. Other services/providers?
17. How has HCBS impacted individuals with behavioral health needs?
    1. How well is HCBS meeting clients´ needs?
    2. Ability to access services?
    3. Quality of services received?
    4. The degree to which their care is integrated?
    5. Enrollee outcomes (e.g., health, recovery, wellness goals, quality of life, stress management, employment, school, community involvement/integration, functioning)?
    6. In what areas have you seen the biggest improvement for enrollees?
    7. In what areas have you seen less improvement for enrollees?
    8. Are there any potential long-term benefits for enrollees?
18. What have been some of the benefits of having HCBS? What has gone well?
    1. For people with behavioral health needs? Are there certain people who have benefited more/less?
    2. For your organization? Are there certain organizations who have benefited more/less?
    3. For systems of care? Are there certain systems of care who have benefited more/less?
    4. SEE PROBES
    5. How would you define or measure the success of HCBS?
    6. To what degree are clients receiving the care they need through HCBS?
19. What have been some of the challenges of HCBS?
    1. For HARP enrollees?
    2. For your organization?
    3. For systems of care?
    4. SEE PROBES
    5. What could be improved? What would help address some of the challenges?
20. What do you see as the future for HCBS services?
21. We are also interested in speaking with HARP/HCBS enrollees to get their perspective on the program. Do you have any suggestions on how best to recruit and/or contact HARP/HCBS enrollees to get their perspectives?
22. Is there anything else that we did not ask that is important for us to know?

___________________________________

^{1. The BH HCBS Full Assessment ceased to be required in March 2017. 1}