WP No. WR-A783-2

• Report is also available in Portable Document Format (PDF)

Independent Evaluation of the New York State
Self-Directed Care (SDC) Program

Interim Report

Submitted to:

Department of Health
Attention: Dianne Kiernan
New York State Department of Health
Office of Health Insurance Programs
99 Washington Avenue, Suite 720
Albany, New York 12210

Email: Dianne.Kiernan@health.ny.gov

Submitted by:

RAND Corporation
1776 Main Street
Santa Monica, CA 90407
Lisa Wagner
Phone: (703) 413-1100 Ext. 5067

Email: LisaW@rand.org

Submitted on:
November 13, 2020

---

TABLE OF CONTENTS

Tables

Figures

Acronyms

1. Executive Summary

2. Self-Directed Care and the Larger Demonstration

2.1 Introduction to the SDC Pilot Program Evaluation
2.2 The Self-Directed Care Pilot Program
2.3 Services Eligible for Self-Direction
2.4 Evaluation Timeline and Progress to Date

3. Evaluation Design and Methods

3.1 Overview
3.2 Discussions with Experts to Refine Approach to the Evaluation
3.3 Quantitative Data Sources
3.4 Analytic Approaches
3.5 Qualitative Methods
3.6 Integration of Quantitative and Qualitative Methods

4. Findings

5. Policy Implications

6. Interactions with Other State Initiatives

References

Appendix A. List of SDC Goods and Services

Appendix B. Key Stakeholder Interview Protocol

---

TABLES

Table 1.1: Proposed Timeline for Evaluation Tasks

Table 3.1: Outcome Measures by Goal and Research Question

Table 3.2: Timeline of SDC Implementation

---

FIGURES

Figure 2.1. Determination of BH HCBS Service Eligibility

Figure 2.2. SDC Independent Evaluation Timeline of Activities to Date

---

ACRONYMS
ACA	Affordable Care Act
BH	Behavioral Health
CMH	Community Mental Health
COVID-19	Severe Acute Respiratory Syndrome Coronavirus 19 (SARS-CoV-19)
DOH	Department of Health
DSRIP	Delivery System Reform Incentive Payment
ED	Emergency Department
FFS	Fee for Service
GLMM	Generalized Linear Mixed Models
HARP	Health and Recovery Plans
HCBS	Home and Community-Based Services
HIV	Human Immunodeficiency Virus
HRSA	Health Resources and Services Administration
IRB	Institutional Review Board
MCO	Managed Care Organization
MHARS	Mental Health Automated Record System
MMC	Medicaid Managed Care
NYC	New York City
NYS	New York State
NYSHF	New York State Health Foundation
OASAS	Office of Addiction Services and Supports
OMH	Office of Mental Health
PCS	Perception of Care Survey
PSM	Propensity Score Matching
ROS	Rest of the State
SAMHSA	Substance Abuse and Mental Health Services Administration
SDC	Self-Directed Care
SMI	Serious Mental Illness
SNP	Special Needs Plans
VBP	Value Based Payment

---

1. EXECUTIVE SUMMARY

Through the New York Medicaid Redesign Team Section 1115 Demonstration, New York State (NYS) pursued the goal of improving access to and quality of health care for the Medicaid population through a managed care delivery system. The Self-Directed Care (SDC) pilot program was implemented as part of the behavioral health (BH) reforms included in the larger Section 1115 Demonstration. In 2019 NYS contracted with the RAND Corporation to conduct an independent evaluation of the SDC pilot program.

This SDC pilot program evaluation uses a mixed methods approach to determine the extent to which three goals of the program were achieved during the first two years of the pilot (January 1, 2018 to December 31, 2019):

1. Implementation of a viable and effective SDC program for Health and Recovery Plans (HARP) enrolled/ BH Home and Community-Based Services (HCBS) eligible individuals throughout NYS
2. Improvement in recovery, health, BH, social functioning, and satisfaction with care for SDC participants
3. Maintenance of Medicaid cost neutrality overall and reduction of BH inpatient and crisis service utilization and cost for SDC participants.

The impacts of the SARS-CoV-19 (COVID-19) pandemic have affected the implementation of the SDC evaluation. The significant strain on the health care system required NYS Department of Health (DOH) staff to shift their priorities. These shifts delayed the execution of the evaluation contract and data access activities. At the time of this writing, NYS is working to make data available to the evaluation team to address the evaluation research questions. The original timeline for the evaluation has also shifted, and a list of ongoing tasks and a new timeline are proposed below.

TABLE 1.1. Proposed Timeline for Evaluation Tasks

Proposed Timeline	Remaining Tasks
November & December 2020	Complete Data Access for HARP Research Questions
January 2021	Data Analysis
February 2021	Data Interpretation
March 2021	Report Findings to DOH
April 2021	Summative Evaluation Report to CMS

This interim report describes RAND´s current understanding of the SDC pilot program and the questions the SDC pilot program evaluation aims to answer, and it outlines the methodology RAND proposed to conduct the evaluation. The final summative report, expected to be completed in Spring 2021, will provide a full discussion of the SDC pilot program evaluation findings and its implications for policy.

---

2. SELF-DIRECTED CARE AND THE LARGER DEMONSTRATION

2.1 INTRODUCTION TO THE SDC PILOT PROGRAM EVALUATION

The New York Medicaid Redesign Team Section 1115 Demonstration (hereafter, Section 1115 Demonstration) was originally approved in 1997 with the goal of improving access to and quality of health care for the Medicaid population through a managed care delivery system (New York State, 2020). The Section 1115 Demonstration included reforms specifically targeted to Medicaid beneficiaries with BH needs (hereafter, BH Demonstration), including the HARP program, which was phased in between 2015 and 2016. The SDC pilot program was implemented starting in 2018 as part of the BH Demonstration.

The RAND Corporation, a private non-profit research organization with a mission to provide policymakers with objective, rigorous, and credible research evidence to inform decision making, was selected to conduct an independent evaluation of the SDC pilot program (New York State Department of Health, 2019). The objective of this evaluation is to examine the implementation and impact of the SDC pilot program. This interim report describes the SDC pilot program and its policy background, the questions the independent evaluation aims to answer, and the proposed methodology to conduct the SDC evaluation. A Final Evaluation Report with a full discussion of the SDC pilot program evaluation findings will be submitted to CMS in 2021.

The SDC pilot program evaluation is designed to determine the extent to which three goals of the program were achieved during its first two years (January 1, 2018 to December 31, 2019). These goals are:

1. Implementation of a viable and effective SDC program for HARP enrolled/BH HCBS eligible individuals throughout NYS
2. Improvement in recovery, health, BH, social functioning, and satisfaction with care for SDC participants
3. Maintenance of Medicaid cost neutrality overall and reduction of BH inpatient and crisis service utilization and cost for SDC participants.

The SDC pilot program evaluation will use both primary (qualitative) and secondary (quantitative) data in a mixed methods empirical investigation of the program´s beneficiary- and system-level impacts. The evaluation seeks to examine SDC pilot program research questions related to implementation, intermediate outcomes, and long-term outcomes. Implementation and intermediate outcomes pertain to enrollment of eligible participants; access to outpatient services (primary and preventive services, BH services); utilization of acute care, namely, inpatient and emergency department (ED) services; and satisfaction with care, as well as a variety of qualitatively assessed outcomes. Long-term outcomes pertain to health and wellness, social outcomes (education, employment, community tenure), quality of life, social connectedness, Medicaid spending, and cost shifts from spending on acute care to community-based services.

2.2 THE SELF-DIRECTED CARE PILOT PROGRAM

The SDC program, grounded in the belief that greater autonomy and choice will permit a better match between individuals´ needs and health care and related services, aims to promote progress toward recovery goals, health, and stability in the community. An earlier version of the SDC program began to be offered in the 1990s by state Medicaid programs as part of the optional state plan personal care services benefit. With support from the Robert Wood Johnson Foundation, self-direction of Medicaid services has evolved over the years; currently, states have a number of mechanisms available to finance the self-direction option to Medicaid beneficiaries (Centers for Medicare & Medicaid Services).

In 2014, the NYS Office of Mental Health (OMH) was awarded a Substance Abuse and Mental Health Services Administration (SAMHSA) Transformation Transfer Initiative grant to fund the design of a self-directed care model to be pilot-tested and eventually scaled-up for delivery to eligible Medicaid beneficiaries with serious mental illnesses in a managed behavioral health delivery system (New York). At the time the BH benefit for most eligible beneficiaries was carved out of existing managed care arrangements, but that would soon change. In April 2015, NYS launched its Section 1115 Demonstration to improve access to and quality of health care delivered through managed care to Medicaid beneficiaries.

The Demonstration included several behavioral health components; this evaluation focuses on the BH Demonstration. In August 2015, NYS amended its Section 1115 Demonstration to enable qualified Managed Care Organizations (MCOs) to comprehensively manage BH care for SSI and non-SSI beneficiaries whose BH benefit was previously covered under a fee-for-service (FFS) payment arrangement. Additionally, the amendment provided for BH HCBS to be made available to eligible individuals meeting defined functional needs criteria.

The BH Demonstration sought not only to improve health care quality, costs, and outcomes for the New York´s Medicaid BH population but also to transform the BH system from an inpatient-focused system to a recovery-focused outpatient system. The BH benefits were made available through all mainstream Medicaid Managed Care (MMC) plans and through a separate coverage product, the HARPs, which are specialty lines of business operated by qualified mainstream MMC plans and available statewide. The HARP program was phased in, launched first in New York City (NYC) in October 2015 and the Rest of the State (ROS) in July 2016. BH HCBS were only available to qualified HARP and HIV SNP enrollees; the BH HCBS were offered beginning in January 2016 in NYC and in October 2016 for ROS.

Starting in September 2015, OMH began conducting preparatory activities to implement a BH SDC pilot program (e.g., selecting sites, creating a web-based portal) targeted to HARP enrollees. Under the demonstration extension approved December 7, 2016, a program making self-direction services available to eligible individuals was authorized as a pilot initiative with the goal of testing the viability and effectiveness of SDC prior to a statewide implementation. The effective dates of the pilot SDC program are January 1, 2017 through June 30, 2022.

The SDC pilot program allows individuals who are eligible for the HARP program benefit package and BH HCBS to use public dollars to purchase services and/or to employ service providers. SDC participants select a support broker with whom they work to identify recovery goals. The support broker then assists the participant with the creation and implementation of a budget to purchase the goods and services required to meet the recovery goals. SDC participation is voluntary, and participants may opt out at any time. Eligible enrollees wishing to participate after capacity has been exceeded are placed on a waiting list.

Two agencies, one in NYC and one in Newburgh (a small city close to Poughkeepsie), were chosen as SDC pilot sites. The agencies are responsible for recruiting and enrolling participants and for hiring, training, and supervising support brokers. (Support brokers work with a fiscal intermediary based at NYS OMH who provide training, support, and monitoring for the authorization and purchasing of goods and services.) Contracts between the agencies and NYS were finalized in July 2017, and the two-year SDC pilot program launched in January 2018 (Table 3.2 provides a timeline). NYS expected a total of 200 SDC participants at the two pilot sites.

2.3 SERVICES ELIGIBLE FOR SELF-DIRECTION

The services that the SDC pilot participants can purchase with their SDC funds include all BH HCBS services offered by the HARP program, as well as individual directed goods and services.

BH HCBS are delivered to HARP and HARP-eligible HIV SNP enrollees under a two-level tier structure determined by a person-centered plan of care developed by the Health Homes or other state-designated entities. Tier 1 services include Individual Employment Support, Education Support, and Peer Services. Tier 2 services include all Tier 1 services plus additional services for beneficiaries with a higher level of need. Eligibility for BH HCBS is assessed through the BH HCBS Eligibility Assessment, a standardized clinical and functional assessment tool also referred to as CMH Screen. Current eligibility threshold for Tier 2 services, higher relative to Tier 1 services, requires evidence of at least "moderate" level of need as indicated by a state-designated score on the CMH Screen (see Figure 2.1 for eligibility criteria). The original criteria were more stringent: until June 2018, eligibility for Tier 2 services required moderate need on at least four domains or extensive need on at least one domain. In addition, a third criterion permitting previously eligible BH HCBS users to continue receiving services was added in June 2019.

FIGURE 2.1. Determination of BH HCBS Service Eligibility

* Domains of Functional and Safety needs include employment/education, instrumental activities of daily living (IADLs), cognitive skills, social relations, stress and trauma, co-occurring conditions, engagement, substance use, and risk of harm.

The goods and services eligible for self-direction can be other services, equipment, or supplies that address an identified need in the service plan and are not otherwise available to the beneficiary (see Appendix A for a non-exhaustive list of non-treatment goods and services). These items or services must decrease the need for other Medicaid services, promote inclusion in the community and increase the participant´s safety in the home environment. Not all goods and services are eligible for self-direction. Ineligible items include experimental treatments, room and board in an assisted living or other residential facility, and services or goods that are recreational.

2.4 EVALUATION TIMELINE AND PROGRESS TO DATE

In early 2020, NYS DOH required a shift in priorities and resources to address the COVID-19 pandemic. This resulted in a delay executing data use agreements, applying for institutional review board (IRB) approval, and accessing data for analysis. At this time, this interim report only includes information pertaining to the design and implementation of the SDC pilot program evaluation. All findings and conclusions will be discussed in a final summative report, available in Spring 2021.

Revised Timeline

The original evaluation timeline was revised to allow for additional time for analysis. The timeline of activities to date are presented in Figure 2.2. As discussed in the methodology in Section 3, the ability to complete the analysis of the person-level data is integral to responding to the evaluation questions.

FIGURE 2.2. SDC Independent Evaluation Timeline of Activities to Date

Next Steps

All evaluation components will be completed per Table 1.1 and are expected to be published in a final summative report in Spring 2021.

---

3. EVALUATION DESIGN AND METHODS

The following sections provide an overview of the evaluation design and a description of the data sources and methods. Most of the methodology presented below is the design as planned; modifications have been made and more may be made during the analysis.

3.1 OVERVIEW

RAND is conducting an independent evaluation of the SDC pilot program that adheres to the evaluation standards set forth in the Special Terms and Conditions for the Section 1115 Demonstration (New York State, 2020, Section XI, Evaluation Requirements). The SDC pilot program evaluation employs a mixed method design and includes a process evaluation component and an outcome evaluation component.

Process Evaluation

The process evaluation will seek to understand how the SDC pilot program has been implemented, focusing on the elements that are critical to achieving program outcomes according to the logic model, with an eye toward informing broader scale-up of SDC. The evaluation will explore issues associated with barriers and facilitators to SDC implementation; clarity of roles and adequacy of training for key personnel (e.g., financial intermediary, support brokers); adequacy of policies, procedures, oversight, and monitoring from agency leadership and NYS; integration of SDC within agency services; coordination between pilot sites and the financial intermediary; recruitment and enrollment of SDC participants; and provision and receipt of SDC services, including experiences developing recovery plans and budgets.

This part of the evaluation will use a combination of quantitative and qualitative methods to address the three process-related research questions stated in Goal 1. The first question concerns enrollment in the SDC program and will be addressed through descriptive analyses of data from several administrative and survey sources (see Section 3.3, Quantitative Data Sources). The second and third questions of the process evaluation will be addressed using qualitative methods, i.e., a combination of focus groups, key informant interviews, site visits, and document reviews. Participants in the qualitative components of the process evaluation will include SDC participants, support brokers, pilot site agency leadership, Advisory Council members, fiscal intermediary staff, and OMH program staff, as well as any additional stakeholders identified as having relevant expertise and exposure to the SDC pilot program (e.g., policymakers, members of provider network).

Outcome Evaluation

The outcome evaluation will be used to address research questions related to recovery, health, functioning, and satisfaction outcomes (Goal 2) and to Medicaid service utilization and cost (Goal 3). The design of the outcome evaluation will be quasi-experimental. The outcome measures will be risk-adjusted to control for individuals´ characteristics such as sociodemographics, health status, clinical characteristics, and functional status. Risk adjustment will require multivariable analyses based on individual-level data. Causal models will permit appropriate adjustment for confounding factors, including the effects of other ongoing health care initiatives, time-varying covariates, and potential heterogeneity in program implementation effects. The evaluation team will use a difference-in-differences design (pre-post approach) and generalized linear mixed models (GLMM) with appropriate individual-level fixed and random effects to estimate post-policy individual level change in outcomes over time. The concurrent comparison groups for both approaches, when appropriate, will be constructed with a propensity score matching approach (see section 3.4, Analytic Approaches).

Concurrent policies as well as other unobserved factors could affect estimates of program effects if they are correlated with the BH Demonstration and specifically, the SDC pilot program. This possibility will be investigated by examining the relative timing of other key policies with the implementation of the SDC Pilot program, including controls for other policies in the causal models, and estimating models with time period indicators in difference-in-differences model settings to account for other time invariant unobserved policies or idiosyncratic effects.

The mixed methods approach planned for the SDC pilot program evaluation will provide a deeper and more nuanced understanding of client outcomes and implementation barriers and facilitators than would be possible using only one method. The mixed methods approach will primarily focus on seeking complementarity, wherein qualitative data will help inform interpretation of the quantitative results. In addition, qualitative data, in turn, will provide in-depth information on individual experiences of the pilot, the broader context, and other domains not covered by quantitative data, such as development of adequate policies (Tariq and Woodman, 2013).

The program goals, along with the associated research questions, data sources, and planned outcome measures, are illustrated in Table 3.1. Methods to address each of the research questions are discussed in further detail in section 3.6, Integration of Quantitative and Qualitative Methods.

TABLE 3.1 Outcome Measures by Goal and Research Question

Goals	Research Questions	Data Sources	Outcome Measures
1. Implementation of a viable and effective SDC program for HARP enrolled/BH HCBS eligible individuals throughout NYS	1. What are the characteristics of SDC participants and how do they compare to the HARP and BH HCBS eligible population?	Pilot Site Enrollment Data	Count of SDC participants stratified by sociodemographics, health status/clinical characteristics, and functional status
		Medicaid Data (Claims and Encounters)
		CMH Screen
		HARP PCS
	2. What was the experience of HARP enrolled/BH HCBS eligible individuals participating in the SDC Pilot program in relation to satisfaction with the SDC program and its impact on their recovery, quality of life, and benefit from health and BH services?	Transcripts of SDC participant focus groups	Description of participant perspectives on SDC program, staff, and process; impacts on their recovery, quality of life, health, and BH; satisfaction with services
	3. What was the experience of non-participant stakeholders in the SDC Pilot program (e.g., support brokers, pilot site agency staff, State program development/oversight staff, fiscal intermediary) in relation to SDC implementation including State oversight and contracting, fiscal policies and procedures, hiring of SDC staff, recruitment and work with participants, and coordination with the fiscal intermediary?	OMH administrative documentation	Description of program policies regarding selection, agreements, ongoing monitoring of SDC sites and fiscal intermediary, participant eligibility criteria, budgeting/use of funds, conflict of interest, and complaint/incident handling
		OMH administrative staff interviews
		Pilot site staff interviews
		Pilot site documentation on hiring, training, and supervising of support brokers	Description of support broker and supervisory staff demographics, credentials, training, supervision, and their perspectives on the pilot program and their relationship with participants and fiscal and State oversight
		Transcripts from interviews with support brokers, pilot site agency leadership/ supervisory, fiscal intermediary, and State oversight staff
		Pilot site administrative documents	Description of pilot site agencies´ process for recruiting participants, educating participants about
		Pilot site staff interviews	what SDC is and how they can participate, enrolling participants, and facilitating ongoing participation
		SDC participant focus groups
		Fiscal intermediary administrative and technical documents	Description of fiscal intermediary´s policy and infrastructure for providing payments, monitoring payments, and supporting customers
		Interviews with fiscal intermediary staff, pilot site staff, State oversight staff
	4. What were the facilitators and challenges to SDC Pilot implementation and how would they impact statewide roll-out?	Interviews with State oversight, fiscal intermediary, pilot site agency staff	Description of facilitators and challenges to the implementation of the SDC Pilot program
		Focus groups with participants
2. Improvement in recovery, health, BH, social functioning, and satisfaction with care for SDC participants between baseline and three (3) year and subsequent follow-up	1. Do HARP enrollees have improved quality of life after participating in SDC?	HARP PCS	Risk adjusted percentage of SDC participants whose quality of life is improved as a result of the program, by annual period when data are available
	2. Do HARP enrollees show improved indicators of health, BH, and wellness after participating in SDC?	HARP PCS	Risk adjusted percentage of SDC participants whose BH, overall health, and wellness is improved as a result of the program, by annual period when data are available (i.e., experience reduction in substance abuse/other harmful behaviors, misuse of prescription medications)
		CMH Screen
	3. Do HARP enrollees show improvement in education and employment after participating in SDC?	HARP PCS	Risk adjusted percentage of SDC participants whose employment status/hours worked in competitive employment and educational status/enrollment in educational programs is improved as a result of the program, by annual period when data are available
		CMH Screen
	4. Do HARP enrollees show improvement in community tenure (i.e., maintaining stable long-term independence in the community) after participating in SDC?	HARP PCS	Risk adjusted percentage of SDC participants whose community tenure is improved as a result of the program, by annual period when data are available (i.e., experience improved residential status/housing stability, reduced criminal justice system involvement, are under Assisted Outpatient Treatment order, achieve functional independence)
		CMH Screen
	5. Do HARP enrollees show improvement in social connectedness after participating in SDC?	CMH Screen	Risk adjusted percentage of SDC participants whose social connectedness is improved as a result of the program, as manifested by social relationship strengths and level of social activity, by annual period
	6. Do HARP enrollees report increased satisfaction with health and BH services after participating in SDC?	HARP PCS	Risk adjusted percentage of SDC participants who report that quality of care and helpfulness of services are improved as a result of the program, by annual period when data are available
3. Maintenance of Medicaid cost neutrality overall and reduction of BH inpatient and crisis service utilization and cost for SDC participants, between baseline and three (3) year and subsequent follow-up.	1. Does participation in SDC result in increased use (and cost) of outpatient BH services and primary care?	Medicaid Data (Claims and Encounters)	Risk adjusted percentage of SDC participants receiving BH services and primary care/preventive services, by annual period
	2. Does participation in SDC result in decreased use and cost of acute care services (BH inpatient, ED, and crisis services)?	Medicaid Data (Claims and Encounters)	Risk adjusted SDC participant rates of inpatient admissions and days for BH inpatient stays; rates of BH ED use; rates of non-BH ED use; and rates of BH crisis service use. By annual period.
		MHARS
	3. How does participation in SDC impact overall Medicaid spending?	Medicaid Data (Claims and Encounters)	Risk adjusted Medicaid PMPM costs, by annual period (PMPM/Y), for: BH outpatient services; primary care/preventive services; acute care services (ED use, BH inpatient use, and BH crisis services); overall.

3.2 DISCUSSIONS WITH EXPERTS TO REFINE APPROACH TO THE EVALUATION

To better understand the policy context, objectives, and challenges to the implementation of the SDC pilot program, the evaluation team held calls with SDC subject matter experts to discuss the background and implementation of the program. The evaluation team has been using the information gathered in these calls and the internal report on OMH´s preliminary evaluation of the SDC pilot program to inform the qualitative component of the evaluation and to revise and enhance the planned quantitative analyses (Chung, Elwyn and Radigan, 2019). In addition, the evaluation team held discussions with data experts within DOH, OMH, and the New York State Office of Addiction Services and Supports (OASAS) to review the feasibility of fully addressing the research questions, given the constraints on data availability. As a result, some of the planned analyses have been refined to better reflect the information available; subsequent changes may need to be made depending on data availability at the time analyses are conducted.

Using the information gathered in these calls along with publicly available NYS DOH documents, a timeline was developed to indicate key program-related events with the potential to impact the implementation and outcomes of the SDC pilot program. Table 3.2 presents these key events and associated dates.

TABLE 3.2. Timeline of SDC Implementation

Year	Date	Event
2014	February	SAMHSA awarded OMH a Transformation Transfer Initiative to fund the design of the SDC program for individuals with serious mental illness (SMI)
2015	March	New York State Health Foundation (NYSHF) provided start-up funding to OMH to conduct a preliminary evaluation of the SDC pilot program
	August	Amended Section 1115 Demonstration behavioral health reform initiatives include SDC
	September	OMH conducted preliminary activities for SDC (e.g., site selection, hiring an OMH fiscal intermediary, creating a web-based SDC portal)
2017	July	Contracts finalized with two SDC pilot site agencies
	October	Both sites began advertisement and outreach activities to recruit participants
2018	January	Start of 2-year SDC pilot
	March	Substantive pilot program enrollment begins
2019	May	219 participants enrolled (166 active)
	August	SDC Pilot Program Implementation Evaluation Report Released by OMH
2020	June	Contracts with site agencies are extended through June 30, 2022

3.3 QUANTITATIVE DATA SOURCES

The secondary data available for the evaluation of the SDC pilot program include data available within the NYSDOH and OMH from five main sources: pilot site enrollment data, Mental Health Automated Record System (MHARS) data, Community Mental Health (CMH) Screen data, HARP Perception of Care Survey (PCS) data, and Medicaid data.

Pilot Site Enrollment Data: Information on SDC enrollment information by site and recovery goal-related expenditures contained in a secure web application designed by OMH for use by SDC participants and support brokers. These data permit assessment of SDC pilot enrollment (outcome measure).

MHARS data: Information maintained by OMH on inpatient, residential, and outpatient utilization in NYS Psychiatric Centers, used to identify psychiatric inpatient utilization not captured in the Medicaid data. These data permit a complete assessment of number of inpatient admissions and inpatient days.

CMH Screen data: A mix of lifetime and current patient self-reported information and assessor-gathered information collected as part of the assessment of BH HCBS eligibility with the BH HCBS Eligibility Assessment, brief and full scales¹, a standardized clinical and functional assessment tool derived from the interRAI™ CMH Assessment (Hirdes et al., 2000). The CMH Screen is required annually for all HARP and HARP-eligible HIV SNP enrollees, including SDC pilot participants. Domains include sociodemographic characteristics (e.g., marital status, homelessness), health status (BH and chronic health conditions), functional status (independent living skills, cognitive skills, social relations, employment, education, and finances), BH service utilization, risky behaviors (substance use, harmful/self-injurious behaviors), traumatic events, and criminal justice system involvement. As such, the data may be used to describe program outcomes (e.g., health status, functional status), as well as risk factors (e.g., traumatic life events, homelessness, criminal justice involvement, substance use, chronic physical health conditions) and protective factors (e.g., social relations, education, employment, adequate finances). These data permit assessment of sociodemographic, clinical, and recovery-related outcomes for SDC participants (outcome measures), and they may also be used for risk adjustment in regression models.

HARP PCS data: Patient self-reported information on the HARP program, including perception of outcomes, access, and quality of care, appropriateness of services, social connectedness, wellness, and quality of life, that is collected through a survey of randomly selected HARP enrollees enrolled in HARPs or HIV SNPs. The survey was adapted from the Experience of Care and Health Outcomes (ECHO) Survey, the Mental Health Statistics Improvement Program (MHIP)/OMH Consumer Assessment of Care Survey, and others. All SDC participants are administered the HARP PCS survey. These data permit assessment of SDC participant experience and satisfaction with care; satisfaction with BH providers´ cultural sensitivity; and satisfaction with wellness, recovery, and degree of social connectedness.

Medicaid Data. Information maintained by the Medicaid Data Warehouse containing billing records for health care services, including pharmacy, for individuals enrolled in Medicaid in a given year, whether under FFS arrangements or MCOs (i.e., claims and encounters). Source of information on Medicaid enrollment status, plan membership, BH HCBS eligibility status, demographic, health status (diagnoses including BH and chronic physical health conditions; Clinical Risk Group categories), service utilization, provider associated with the billed services, and cost of health care for all Medicaid enrollees; available with a six-month lag. These data will permit assessment of SDC participants´ diagnostic characteristics, service utilization patterns, including BH HCBS, and cost of health care (outcome measures). May also be used for risk adjustment in regression models.

In addition to these NYS DOH/OMH data, the evaluation will incorporate contemporaneous data from Area Health Resource Files (ARF), a collection of publicly available data assembled by the Health Resources & Services Administration (HRSA) or PolicyMap, a web-based data warehouse. Both datasets aggregate information from multiple sources including the Centers for Disease Control and Prevention, HRSA, the U.S. Census, and other neighborhood-level datasets. Small area-level information being considered include sociodemographic characteristics (e.g., urbanicity, household income) and characteristics of the health care infrastructure (e.g., psychiatrists per 1,000 population, HRSA-designated health professional shortage area). This information is available at various geographic levels, including ZIP code and county.

3.4 ANALYTIC APPROACHES

The quantitative methods that will be employed in the evaluation of the SDC pilot program include descriptive statistics, difference-in-differences design, longitudinal mixed effect regression, and propensity score matching.

Descriptive Statistics (with corresponding graphical illustrations): This approach permits population-level, year-to-year comparisons during the evaluation period. For the SDC pilot program evaluation, this approach will be used to examine characteristics of SDC participants in each annual period since program implementation; that is, the outcome domain for Goal 1 of the evaluation. For categorical variables, this will consist of chi-square test and McNemar´s chi-square test (to compare binary outcomes between correlated groups for each region before and after implementation). For continuous variables, on the other hand, we will use the Analysis of Variance (ANOVA) test; paired t-test (to compare pairs of years); and across analyses, the Bonferroni adjustment for multiple pair comparisons. Whenever repeated ANOVA tests for yearly changes within each region may be desirable, the RAND team will evaluate whether the sphericity assumption is violated.

Difference-in-Differences: This design is a pre-post approach that may be employed when concurrent comparison groups are available, thus enabling a robust assessment of program outcomes. For the SDC pilot program evaluation, the outcome domains are those related to service utilization and cost (Goal 3). The treatment versus control groups are: HARP-enrolled and BH HCBS-eligible enrollees who participate in the SDC program versus HARP-enrolled and BH HCBS-eligible enrollees who do not participate in the SDC program and who reside in the same geographic areas as the pilot sites. An alternative control group will be HARP-enrolled and BH HCBS-eligible enrollees meeting SDC participation criteria residing in areas similar to the pilot locations.

Outcomes will be measured over two consecutive 18-month periods, prior to and following enrollment in the SDC pilot program. The measurement periods are approximate as the actual trends will be based on SDC participant enrollment:

Pre-Period: July 2016 to December 2017
Post-Period: January 2018 to June 2019

This quasi-experimental approach accounts for any secular trend/changes in the outcome metrics as it eliminates fixed differences not related to program implementation; thus, remaining significant differences may be validly attributable to the impact of program implementation. The difference-in-differences approach requires that pairs of treatment and control individuals comparable on key observed confounders be identified through Propensity Score Matching (discussed below).

Longitudinal Mixed Effect Regression: This approach employs a GLMM to estimate an average program effect while adjusting for key covariates when examining change trajectories. For the SDC pilot program evaluation, the outcome domains are quality of life; health status including physical health, BH, and wellness; functional status including education and employment, community tenure and social connectedness; and satisfaction with health and BH services (Goal 2). This quasi-experimental approach separates the effects of time from that of the SDC pilot implementation, accommodating the heterogeneity in the program implementation effect and accounting for serial correlations within individuals and variation of risk/protective factors and outcomes over time due to strong temporal trends. The multivariable mixed effects regressions will include fixed effects, namely demographics (age, gender, and race/ethnicity) and time, and random effects assessed at each annual time point, namely risk and protective factor levels as assessed with the CMH Screen. Random effects will be incorporated in the models on two levels: for changes over time nested within persons and persons nested within areas/site.

Propensity Score Matching: This approach controls for potential confounding by identifying individuals with similar characteristics belonging to the treatment and control groups, thus enabling the use of quasi-experimental causal models (such as the difference-in-differences design discussed above). In the SDC pilot program evaluation, propensity score matching (PSM) will be used in combination with difference-in-differences (for double robustness) to examine the impact of the program on the outcomes of interest. The comparison group will strengthen the planned analyses, as it will control for the effects of other policies and initiatives implemented concurrently with SDC. The method uses a logistic regression to estimate each individual´s conditional probability (or propensity score) of belonging to the treatment group (i.e., having the outcome of opting to enroll into SDC). Predictors will include variables related to sociodemographic, health status/clinical characteristics, functional status, and other variables such as service utilization variables assessed prior to program implementation. A greedy matching algorithm with an appropriate matching ratio of treatment to control individuals will be used to create a matched analytic cohort based on the estimated propensity score. RAND will a priori select the confounding variables for inclusion in the models using the team´s expertise but may also consult with additional subject matter experts. Balance in covariate distribution between treatment and control individuals in the matched analytic cohort will be assessed with standardized difference.

3.5 QUALITATIVE METHODS

The qualitative component of the SDC pilot program evaluation will consist of interviews with key informants and participants in the pilot program, and a review of program-related policy documents. The key informant interviews will be conducted with informants who represent diverse stakeholders in the SDC pilot program, including support brokers, agency leadership, clinical supervisors, fiscal intermediary, and NYS oversight staff. Informants will include representatives of advocacy organizations, plan administrators, and care providers, and they will be selected using a snowballing approach. An initial group of informants will be selected from a list provided by the DOH, and additional informants will be selected based on recommendations of individuals on the list. An effort will be made to ensure that a broad range of perspectives is represented in the study sample, including diverse advocacy groups and providers from New York City as well as both urban and rural regions upstate. The evaluation team anticipates conducting approximately 15 key informant interviews. In addition, SDC participant interviews will be conducted to understand perspectives on the pilot and to gauge satisfaction. The qualitative analysis will also be informed by review of documents that have been provided to the research team by DOH. The documents include policy documents, which describe how the program was designed.

The interviews and documents will be analyzed by the evaluation team to identify issues that have arisen in the course of the implementation of the SDC pilot. The interviews will also be used to understand staff perspectives on their relationships with participants, fiscal and state oversight, and the SDC program as a whole. For instance, the evaluation team will ask advocacy organizations whether the implementation has gone according to expectations, whether they have concerns about barriers to successful implementation, and whether there are aspects of the implementation that have been particularly promising. Issues raised by key informants will be summarized and compared across the categories of informants. While the key informant interviews cannot provide definitive information on the impact of the SDC pilot, they can be extremely helpful in identifying common areas of concern.

Protocol Development

A semi-structured interview guide for key informants representing a diversity of SDC pilot stakeholders was developed (Appendix B). It covers topics including barriers and facilitators to SDC pilot implementation; clarity of roles and adequacy of training for key personnel (e.g., financial intermediary, support brokers); adequacy of policies, procedures, oversight, and monitoring from agency leadership and NYS; integration of SDC within agency services; coordination between NYS, pilot sites, and the financial intermediary; recruitment and enrollment of SDC participants; provision and receipt of SDC services including experiences developing recovery plans and budgets; and participant outcomes.

A semi-structured interview guide for SDC participants is being developed. It will focus on topics including participant perceptions regarding enrollment, the process of developing recovery plans and budgets, relationships between participants and support brokers, satisfaction with health and BH services, and the impact of SDC on participant recovery and quality of life.

Respondent Selection

The evaluation team is using a purposive sampling approach to recruit key informants. To capture a range of perspectives, key informants representing various stakeholder organizations will be recruited, including the two pilot sites, the NYS Office of Mental Health, and provider/trade associations. Potential key informants will be identified through state and site-provided lists, as well as suggestions for additional informants from those who completed interviews. Key informants from the two pilot sites will include SDC direct provider staff (i.e., support brokers), other pilot site staff serving participants who are enrolled in SDC, and SDC program and agency leadership. Key informants from OMH will be recruited from several divisions/departments and generally represent leadership at the program or senior executive management level as well as staff directly involved in administering the program (e.g., fiscal intermediary functions). Key informants from the provider/trade associations will represent staff from the senior executive leadership level. The evaluation team anticipates conducting approximately 15 key informant interviews.

To identify SDC participants for interviews, evaluators will utilize purposive and convenience sampling strategies. To capture a range of perspectives, the evaluation will seek to maximize the diversity of SDC participants who participate, considering factors such as referring pilot site, length of time in SDC, SDC utilization patterns, and a range of demographic characteristics (e.g., gender, race, diagnosis). The evaluation team anticipates approximately ten interviews with SDC participants, with approximately five participants from each pilot site.

Respondent Recruitment

Potential key informants will receive an e-mail inviting them to participate in the evaluation interview and to contact the evaluators if they are interested in participating. An information sheet will be e-mailed to key informants in advance of scheduled interviews and reviewed prior to commencing the interview. SDC pilot site staff will identify potential SDC participants and provide them with information about the evaluation. SDC participants interested in participating can contact the evaluators directly or inform SDC staff that they consent to having the evaluators contact them.

Interviewer Training

Prior to conducting interviews, the qualitative team received training on the SDC pilot and the context of the state pilot implementation, including relevant Medicaid policies. The training included a review of documents provided by DOH, participation in discussions with DOH subject matter expert staff, and internal discussions with the project leads and technical advisors, who have experience with NYS Medicaid and the SDC program development. The training ensured that the interviewers were aware of issues relevant to the implementation when conducting interviews.

Conducting Interviews

Interviews with key informants representing SDC stakeholders will be conducted virtually and last one hour, on average. The majority of data collection will consist of individual interviews with one identified key informant; however, informants will be able to invite additional individuals to the interviews as needed to cover the relevant expertise and experience. Interviews with SDC pilot client participants will be conducted by phone or online.

Interviews will be conducted by one qualitative researcher, with an additional researcher taking notes concurrently that will inform a written interview summary. Interviewers will cover core topic areas but will flexibly maneuver through the interview guide and probe certain topics more in-depth as appropriate. Interviews will be audio-recorded and transcribed verbatim. The institutional review board of the NYS Psychiatric Institute determined that data collection with stakeholders who were not SDC pilot participants does not constitute human subjects research and was thus exempt from review. Review of data collection with SDC participants is pending.

Qualitative Data Analysis

Analytic methods, aligned with recommendations of Bradley, Curry, and Devers (2007), will follow a grounded theory approach to developing coding structures that emphasize inductive codes emerging directly from the data (Bradley, Curry and Devers, 2007). Consistent with grounded theory, qualitative analysis occurs concurrently with data collection, allowing interviews to be shaped by preliminary concepts and themes emerging from the data. The analysis will proceed in a series of steps: developing initial codes (open-coding), validating and using the codes (i.e., coding transcripts with a final code list), clustering and interpreting the codes, and developing broader findings and themes. Strategies for rigor include weekly data collection and analysis debrief meetings, development of interview summaries and memos, and the use of multiple coders.

3.6 INTEGRATION OF QUANTITATIVE AND QUALITATIVE METHODS

Findings from the quantitative and qualitative analyses will be integrated to refine and deepen the results from the different methods. Qualitative information from participant interviews will be combined with quantitative findings on change indicators (Goal 2) to gain a more nuanced understanding of participant outcomes. In addition, barriers and facilitators of SDC implementation identified through the qualitative data and methods of the process evaluation will be combined with quantitative findings derived from the two pilot sites to gain an understanding of whether there are elements critical to effective implementation.

3.7 DISCUSSION OF EVALUATION GOALS AND RESEARCH QUESTIONS

Goal 1. Implementation of a viable and effective SDC program for HARP enrolled/BH HCBS eligible individuals throughout NYS (Process Evaluation)

The evaluation team will develop a detailed design for the process evaluation through review of the SDC logic model; the literature on SDC programs; initial discussions with NYS DOH personnel; and review of documents describing the program developed by OMH, OASAS, the SDC Advisory Council, fiscal and administrative entities, and the pilot site agencies. The review will inform selection of respondents for the qualitative components of the process evaluation and the questions that will be included in the interview protocols. Descriptive analyses of the administrative and survey data on enrollment in the SDC programs, which will be conducted concurrently, will also inform the study design, guiding decisions regarding the diversity of participants.

Research Question 1.1: What are the characteristics of SDC participants and how do they compare to the larger HARP and BH HCBS eligible population?

Data from pilot site enrollment records and data from CMH Screens, HARP PCS, and Medicaid will be used to characterize the participants in the SDC programs. The enrolled population will be described with respect to basic sociodemographic characteristics (e.g., age, sex, race/ethnicity), prior behavioral and general medical health care utilization, behavioral and general medical diagnoses, and other characteristics of interest. In addition, the evaluation team will conduct comparisons of the SDC population with other HARP-and BH HCBS-eligible Medicaid beneficiaries from the same regions in which the SDC programs are located and statewide. The analyses will use basic descriptive statistics, with the possible addition of regression modeling to compare the SDC participants with other HARP-and BH HCBS-eligible Medicaid beneficiaries on multiple characteristics simultaneously. The comparative analyses will allow the evaluation team to observe whether the SDC participants are comparable to HARP and BH HCBS populations statewide. In addition, these analyses can help policymakers understand the potential scope of the SDC programs, were they to be expanded statewide using similar eligibility criteria and recruitment processes.

Research Question 1.2: What was the experience of HARP enrolled/BH HCBS eligible individuals participating in the SDC Pilot program in relation to satisfaction with the SDC program and its impact on their recovery, quality of life, and benefit from health and BH services?

Methods to address this question are designed to highlight the perspectives of SDC participants themselves. Interviews with SDC participants at both of the two SDC sites will be conducted with up to ten participants, recruited with the assistance of the site agencies. The evaluation team will work with each pilot site to identify and recruit individuals representing a diversity of SDC participants by individual characteristics such as race, gender, and diagnoses, as well as extent of SDC service use. The semi-structured discussion guide will focus on key aspects of the logic model as viewed by the participants. Topics will include participant perceptions about the process of developing recovery plans and budgets; relationships between participants and support brokers; satisfaction with health and BH services; and SDC impact on participant recovery and quality of life.

Research Question 1.3: What was the experience of non-participant stakeholders in the SDC Pilot program (e.g., Support Brokers, pilot site agency staff, State program development/ oversight staff, fiscal intermediary) in relation to SDC implementation including State oversight and contracting, fiscal policies and procedures, hiring of SDC staff, recruitment and work with participants, and coordination with the fiscal intermediary?

This question will be addressed through qualitative analysis of documents and interviews, focusing on identification of implementation barriers and facilitators, staff roles, SDC processes, and coordination among stakeholder organizations. Documents from NYS and the pilot sites will be analyzed, as will the interviews that are conducted with NYS agency officials/staff (e.g., OMH administrators, fiscal intermediary staff) and pilot site staff.

Research Question 1.4: What were the facilitators and challenges to SDC Pilot implementation and how would they impact statewide roll-out?

The final question of the process evaluation will draw on all the qualitative data described above, including interviews with pilot site agency staff (e.g., support brokers, leadership), state agency staff (leadership, financial/fiscal intermediary staff), and SDC participants to address the broad issues of facilitators and challenges that were faced during the pilot program and how these might impact a statewide roll-out of the program. Transcripts from qualitative data collection efforts will be analyzed with specific attention to codes related to barriers and facilitators and linkage of themes across the respondent types. This will allow the evaluation team to address issues from multiple perspectives. For instance, state officials may have concerns about enrollment based on the counts and characteristics of HARP-enrolled/ BH HCBS-eligible individuals who are successfully enrolled, whereas staff of the pilot sites have insights into the reasons that some HARP-enrolled/ BH HCBS-eligible individuals may or may not prefer to enroll in the program. Bringing these multiple perspectives together can provide useful lessons for the statewide rollout.

Goal 2. Improvement in Recovery, Health, BH, Social Functioning, and Satisfaction with Care for SDC Participants (Outcome Evaluation)

Research Question 2.1: Do HARP enrollees have improved quality of life after participating in SDC?

Research Question 2.2: Do HARP enrollees show improved indicators of health, BH, and wellness after participating in SDC?

Research Question 2.3: Do HARP enrollees show improvement in education and employment after participating in SDC?

Research Question 2.4: Do HARP enrollees show improvement in community tenure (i.e., maintaining stable long-term independence in the community) after participating in SDC?

Research Question 2.5: Do HARP enrollees show improvement in social connectedness after participating in SDC?

Research Question 2.6: Do HARP enrollees report increased satisfaction with health and BH services after participating in SDC?

To address the Goal 2 research questions, the evaluation team will use GLMM to assess changes in outcomes for SDC participants between baseline and multiple follow-up points over the first two years of the pilot program (January 1, 2018 to December 31, 2019), while controlling for variation in outcomes and risk factors over time and potential heterogeneity in program implementation. For the SDC pilot program evaluation, random effects will be incorporated in the models on two levels: for persons within areas/site and for change over time within persons. This approach will assess average trends on outcome measures derived from the CMH Screen and HARP PCS while controlling for possible confounding factors. Using data from the CMH screen, Research Questions 2.2 through 2.5 will be addressed with the additional benefit of an appropriate comparison group identified through PSM. It is not possible to rely solely on HARP PCS data for Research Questions 2.1 and 2.6, as the HARP PCS for non-SDC participants is based on annual random sampling. However, those data will be used to descriptively compare the larger HARP-enrolled population with SDC participants.

Goal 3. Maintenance of Medicaid Cost Neutrality Overall and Reduction of BH Inpatient and Crisis Service Utilization and Cost for SDC Participants

Research Question 3.1: Does participation in SDC result in increased use and cost of outpatient BH services and primary care?

Research Question 3.2: Does participation in SDC result in decreased use and cost of BH inpatient, ED, and crisis services?

For the Goal 3 Research Questions, the evaluation team will use difference-in-differences to assess the effect of the SDC pilot on rates of service utilization (BH outpatient, primary care, BH inpatient, and ED and crisis services) and Medicaid spending over a 36-month period. Outcomes over two consecutive 18-month periods will be measured, prior to and following enrollment in the SDC pilot program, and changes from the prior measurement period to the post measurement period will be compared between the SDC pilot participants and a comparison group identified through PSM. As mentioned above, the approximate measurement periods are July 1, 2016 to December 31, 2017 (pre-period) and January 1, 2018 to June 30, 2019 (post-period).

---

4. FINDINGS

Due to the Spring 2020 delays in initiating the SDC pilot program evaluation, no findings are yet available for discussion at this time. All findings will be reported in the final summative report in 2021. Proposed Timeline capturing the ongoing data access and analysis is presented above in Table 1.1.

---

5. POLICY IMPLICATIONS

Because there are no findings yet available, no policy implications can be provided at this time. A thorough discussion of the policy implications of the evaluation findings will be included in the 2021 final summative report.

---

6. INTERACTIONS WITH OTHER STATE INITIATIVES

An in-depth empirical investigation of the manner in which the implementation and effects of the SDC pilot were affected by other state initiatives is out of scope for the SDC pilot evaluation as proposed and executed in the RFP and RAND contract. As an alternative, information on other policy initiatives implemented by the state and potentially affecting the SDC pilot is being collected to assist with the design of the analyses and to interpret and provide context to the findings. Potential interactions with the SDC pilot will be discussed in the 2021 final summative report.

The state initiatives that will be reviewed for potential interactions with the implementation of the SDC pilot in the final summative report include:

• Other provisions of the Delivery System Reform Incentive Payment (DSRIP) Program, including payment reform in the form of a Value Based Payment (VBP) Roadmap
• Provisions of the Affordable Care Act (ACA), including the Medicaid Health Home program and Medicaid access expansion.

---

REFERENCES

Bradley, E. H., L. A. Curry, and K. J. Devers, "Qualitative Data Analysis for Health Services Research: Developing Taxonomy, Themes, and Theory," Health Serv Res, Vol. 42, No. 4, 2007, pp. 1758-1772.

Centers for Medicare & Medicaid Services, "Self-Directed Services." As of October 26

Chung, Chia-Ling, Laura Elwyn, and Marleen Radigan, Implementation Evaluation of the New York State Behavioral Health Self-Directed Care Pilot Program, New York State Office of Mental Health, August 2019.

Hirdes, John P, Mounir Marhaba, Trevor Frise Smith, Leah Clyburn, Lori Mitchell, Ra Lemick, N Curtin Telegdi, Edgardo Perez, Peter Prendergast, and Terry Rabinowitz, "Development of the resident assessment instrument–mental health (RAI-MH)," Hosp Q, Vol. 4, No. 2, 2000, pp. 44-51.

New York, FY2014 TTI Project: Design a model Self-Directed Care (SDC) for individuals with SMI under the Medicaid authority of NY´s 1115 waiver, Albany, NY: New York State. https://nasmhpd.org/sites/default/files/New%20York14.pdf

New York State, "MRT Plan Current STCs - August 10, 2020," September 2020. As of October 29.

New York State Department of Health, Office of Quality and Patient Safety, "Independent Evaluation of the New York State (NYS) Health and Recovery Plans (HARP) Program and Self-Directed Care (SDC) Pilot Program: Request for Proposal #20024," February, 2019. https://health.ny.gov/funding/rfp/20024/index.htm

Tariq, Shema, and Jenny Woodman, "Using Mixed Methods in Health Research," JRSM short reports, Vol. 4, No. 6, 2013, p. 2042533313479197.

---

APPENDIX A. LIST OF SDC GOODS AND SERVICES

Non-treatment goods and services that support treatment goals in a Participant´s Action Plan may include, but are not limited to (Chung, Elwyn and Radigan, 2019):

• Wellness activities
  • Gym/ health club membership
  • Wellness coaching
  • Smoking cessation tools/ education
  • Dental care
  • Eyeglasses/care
  • Out of network health/BH/specialty services
  • Family planning and sexual health education/ services
  • Acupuncture/pressure
  • Yoga classes/meditation guidance
  • Massage/reiki/ shiatsu/tai chi instruction
  • Pet adoption funds, including appointments/resources related to pet health and maintenance
  • Workout equipment and clothing
  • Nutritional supplements and vitamins
• Occupational/skills development
  • Computer literacy
  • Resume development
  • Interview preparation
  • PC/communication technology
  • Personal preparation/resources to prepare for interviews or to enhance confidence during employment, including purchase of a wardrobe or maintenance of personal hygiene (including but not limited to skin and hair care)
  • Resources for entrepreneurial development, including business cards, website development
  • Educational course fees and materials
• Transportation
  • Public transportation costs
  • Car repair/maintenance
  • Bicycle and related costs
• In-home/social/community supports
  • Training and supports for daily living including cooking and nutrition classes, sequencing, time management, etc.
  • Housing start-up (down payments), non-recurring housing bills or costs related to home maintenance, including furniture or air conditioner
  • Groceries
  • Travel to and from family or social functions, including special trips to visit family members or friends
  • Meetings in the community with friends or family members at restaurants, coffee houses, or other social venues, that promote the social inclusion of the participant
  • Financial contributions at social activities including church services
  • Registration fees for conferences, trainings, community activities
  • Membership dues in groups, societies, guilds, leagues

---

APPENDIX B. KEY STAKEHOLDER INTERVIEW PROTOCOL

Interview Guide: Non-Client Agency Leadership Stakeholder

Participant ID: ___________________________________ Interview Date: _______________________________

Region: NYC ________ Beacon ________

Stakeholder Type: _____________________________

Agency Type: _______________________________________________________

Interviewer: ____________________________________

The purpose of this interview is to explore your perspective and experience with the Self-Directed Care pilot program. The Self-Directed Care program allows individuals with behavioral health needs who are participating in the pilot program to use State funds to purchase goods and services and/or to hire service providers that can facilitate the person´s recovery. The SDC pilot seeks to increase autonomy and choice over benefits in order to enhance participants´ progress toward recovery goals and improve health for individuals with behavioral health needs. The SDC pilot is being implemented at two behavioral health agencies in New York State.

Before we begin, I want to discuss the process of this interview. The interview will take approximately 60 minutes to complete. Again, the goal of this interview is to learn about your views and experiences regarding the implementation of the SDC Pilot Program. There are no right or wrong answers to these questions. We are only interested in your honest opinion. Any questions before we begin?

<< BEGIN RECORDING >> << BEGIN RECORDING >> << BEGIN RECORDING >>

Role:

1. What is your current role at [organization]?
   Probe: How do your responsibilities relate to the SDC pilot?

SDC Pilot

2. How would you describe the mission and goals of the SDC pilot?
3. What has been your experience with the SDC program?
4. How were participants enrolled in the program?
   1. How was eligibility assessed? Were there any challenges?
   2. To what degree is it reaching the target population?
   3. What were the most common reasons that participants were not eligible? Would this need to be changed if the program were to scale-up?
   4. What motivated participants to join the SDC program?
   5. How many participants were eligible but did not enroll? Why?
5. What have been some of the benefits of implementing SDC?
   1. What has gone well with SDC? For participants? For the organizations? For the overall system of care?
   2. How would you define success for SDC?
6. How has the SDC program impacted SDC participants?
   1. How has it impacted the paperwork they have to do (e.g., purchase requests) regarding managing their benefits?
   2. How has it impacted their access to services?
   3. How has it impacted their access to goods?
   4. How has it impacted participants´ sense of autonomy and choice?
   5. How has it impacted participant outcomes (e.g., recovery, quality of life, health/wellness, community integration, functioning)?
   6. For whom does the program work well?
   7. For whom does it not work as well? Can you give an example?
7. What services or goods has SDC increased access to the most?
   1. How do these services or goods meet participants´ needs?
8. What services or goods have been more challenging for SDC participants to utilize?
   1. What has been challenging about accessing these services or goods?
9. How does access to goods and use of services differ between SDC participants and other people with behavioral health needs served by [organization(s)]?
   1. What goods/services are SDC participants more likely to use/access?
   2. What goods/services are SDC participants less likely to use/access?
10. How well has the process of SDC participants identifying goals and needs, requesting funds, and having them reviewed been going?
    1. Developing participant goals? Developing budgets?
    2. Participants identifying goods/services needed?
    3. Participants requesting funds?
    4. Review/approval of funding requests?
    5. Placing funds on participants´ cards?
    6. Which parts of the process do participants need the most support with?
11. What are some of the most common reasons that participants´ purchase requests are denied?
    1. How is it determined whether requests are an appropriate use of SDC funds?
    2. How is it determined whether requests are related to goals?
    3. Can participants appeal request denials?
12. What is the process for identifying misuse of funds?
    1. What are the most common ways in which funds have been misused?
    2. Do any changes need to be made to the types of oversight that are now in place?
13. What have been some of the challenges of providing SDC services?
    1. Engaging participants?
    2. Staff delivering the services? Staff retention?
    3. Communicating/coordinating across staff/agencies?
    4. Reviewing/approving purchases?
    5. Timeliness with which requests/purchases are completed?
    6. Funding for SDC?
    7. Administrative burden for organizations/agencies?
    8. Any dilemmas or ethical issues that arise?
    9. What could be improved? What would help address some of these challenges?
14. What changes would you suggest to the program?
    1. What changes would be needed to help scale-up the program to other organizations and participants throughout the state?

Support Brokers

15. What is the role of the support broker within the organization?
    1. To what degree does the work of the support broker match how the role was planned?
    2. What aspects of the role have had to be clarified or negotiated over time?
    3. What changes might need to be made to the role of the support broker?
16. How did the organization select a support broker to work with participants?
    1. Were there any challenges to hiring the support broker?
    2. Any challenges to integrating this role into the agency?
    3. To what extent do support brokers work with other staff at the organization?
17. How were support brokers oriented and trained in the SDC program?
    1. How are they introduced to participants?
    2. What additional training might be needed for support brokers?
18. How are support brokers supervised?
    1. Who provides supervision?
    2. Do supervisors receive any specialized SDC training?
    3. What type of issues are discussed in supervision/with supervisors?
19. What are the benefits of having the support broker role compared to folding this into other staff roles?
20. What are the challenges of having the support broker role?
21. How does the SDC pilot fit in with other types of behavioral health services that are delivered by the [organization(s)]?

Fiscal Intermediary Role:

22. What is the role of the fiscal intermediary?
    1. To what degree does the work of the fiscal intermediary match how the role was planned?
    2. What aspects of the role have had to be clarified or negotiated over time?
    3. What changes might need to be made to the role of the fiscal intermediary?
23. What is communication/coordination like between the fiscal intermediary as part of SDC?
24. What are the benefits specifically of having the fiscal intermediary role?
25. What are the challenges of having the fiscal intermediary role?

Overall Program Evaluation

26. How would you evaluate the overall success of the program?
27. Do you believe the program should be expanded?
    1. Probe: Why or why not?
28. Any thoughts on how to improve the program?
29. What are the next steps for SDC?
    1. Probe: Do you believe that SDC is an effective and viable program for HARP enrollees across NYS?
    2. Long-term sustainability?
30. Is there something we didn´t ask that you would like to add?

___________________________________

^{1. The BH HCBS Full Assessment ceased to be required in March 2017. 1}