Summary of Public Comments for the DSRIP Mid-Point Assessment Overview

• Summary is also available in Portable Document Format (PDF, 171KB)

On February 17, 2016, the Independent Assessor (IA) released the DSRIP Mid-Point Assessment Overview document for a 30 day public comment period. During that time, the IA received a total of 60 comments from 14 different entities covering providers, individuals, trade associations and advocacy groups. This document contains a summary of the comments received along with a response from the IA on how the comment was incorporated in to the revised Mid-Point Assessment Overview document or an explanation for why a revision was not made. The comments below have been grouped in to topic areas to highlight the various comments received around common themes.

Safety Net Providers
Comment	Independent Assessor Response
Could you please provide such a mechanism to evaluate the transparency of the process for the selection of awardees and the flow of funds disbursement from the lead organization in the PPS to safety net providers?	The public comment period will allow for the public and providers to comment on items such as the transparency for the selection of awardees and the funds flow disbursements from the PPS lead organization to safety net providers.
What is the rationale for waiting until mid-point assessment process to open up the Safety Net appeals process? Can the appeals process be opened up earlier?	The Safety Net appeals process has been planned for Mid-Point Assessment to align with PPS efforts to
Please clarify the specific timeline for Safety Net appeals. The guidance says "At the Mid-Point Assessment," a process which spans from August 1, 2016–March 31, 2017. Will the Safety Net appeal be at the beginning, middle, or end of this period?	The Safety Net appeals process is tentatively scheduled to begin in September 2016. A formal timeline will be communicated once it is finalized.

Lack of Project Specific Reporting Contributing to the Midpoint Assessment
Comment	Independent Assessor Response
At the time of the midpoint assessment (DY2Q1), many PPSs (NYP PPS included) will not have provided any project-specific reporting (e.g. Domain 1 requirements) besides patient engagement scale and speed. The reporting to-date will have focused mostly on the Domain 1 Organizational Milestones which have little relation to the project requirements or meaningful improvements to the healthcare provided to beneficiaries. This limitation would seem to prohibit the IA from objectively using "the available data to determine if the PPS are implementing the projects in accordance with their approved DSRIP Project Plans and to assess PPS progress towards meeting project implementation milestones." Recommendation: Allow the PPS to contribute a non-quarterly reporting narrative to accompany their individual assessment to contextualize any IA- provided scores when released to the public.	This comment has been received and adopted into the redlined Mid-Point Assessment. PPS will have the opportunity to submit a narrative to provide context around project progression. The Independent Assessor will consider this when evaluating projects. Details on the content of the narrative have been added to the Mid-Point Assessment Overview document.

Risk Scores
Comment	Independent Assessor Response
The majority of the work completed by PPSs by DY2Q1 will have been focused on the Domain 1 Organizational Milestones; many of these build the building blocks of a well-established collaborator network. The risk scores are only assigned to the projects, not the Organizational Milestones (which drive a significant portion of funds). Recommendation: Recognize the efforts the PPS efforts to-date by also providing a score on the various organizational domains.	Pursuant to Section VI.d. of Attachment I of the STCs the Independent Assessor is to conduct a focused review of projects during the Mid-Point Assessment. While it is noted that the PPS made significant efforts towards completing the Organizational Milestones it must be noted that the Mid-Point Assessment is a review towards progress made toward project completion. As articulated in the redline, PPS will have the opportunity to submit a narrative to provide context around project progression. The Independent Assessor will consider this when evaluating projects.
It is not clear if the Project Risk Scores will be weighted in any way when determining a combination of the overall DSRIP Project Plan compliance. Recommend giving an opportunity for PPSs to comment on the process to determine Project Risk Score once it is released.	The Independent Assessor will review all projects consistently, however an emphasis will be placed on those project milestones that impact multiple project efforts. For example, the project requirement for Primary Care Physicians to reach PCMH Level 3 spans multiple projects and will therefore be weighted in a way to emphasize its importance across DSRIP projects.
With regards to the PPS Project Risk Scores, the document indicates that "While it is not expected that PPS will have completed all of the project requirements at the Mid-Point Assessment, it is important that the PPS demonstrate progress towards the completion of the project requirements associated with each project, specifically those with a required completion date in DY2." Please provide more detail on how progress will be evaluated in instances where project requirements are not due until after DY2Q1. For example, will progress evaluation be linked to the proportion of tasks completed?	The Independent Assessor's review will focus on the tasks the PPS has documented in the Implementation Plan and the PPS progress towards completing those milestones. The Independent Assessor will be evaluating what milestones the PPS has completed through DY2, Q1 and that the PPS is on track for completing all milestones by the required completion dates. PPS will also be able to submit narratives for each project to provide further details on the PPS project implementation efforts to ensure the Independent Assessor has a comprehensive view of all PPS activities completed through DY2, Q1.

Limit on removing 5% of network
Comment	Independent Assessor Response
What if we need to remove more than 5% of the network?	The cap on the number of network partners that may be removed at the Mid-Point Assessment has been raised to 10%. A PPS will not be able to remove any more than 10% of their total partner network.
The document references "removal" due to non- performance. Please consider providing specific language on voluntary withdrawal of a PPS Network Partner. In addition, please clearly state whether or not PPS will have the opportunity to update speed and scale commitments given changes in Network membership (adding, or removing, partners).	The Mid-Point Assessment Overview has been updated to note that voluntary withdrawals from the PPS network will be allowed during the Mid-Point Assessment. The voluntary withdrawals will be counted towards the 10% cap on network partner removals. The removal of network partners does not allow the PPS to modify the Speed & Scale commitments made during the Project Plan Application. PPS should consider these commitments when making network modifications, removals and additions, at the Mid-Point Assessment.
The document states "In requesting the removal of PPS network partners, the PPS will not be able to remove more than 5% of their Network Partners." Please provide clarification: Is this referring to 5% of the overall number of Partner Organizations, or is the intention that changes in Network partners result in a maximum loss of 5% of the total number of attributed lives?	The determination of the now 10% cap on the removal of network partners is based on total number of partners in the PPS network. For example, a PPS with 100 network partners would be able to remove no more than 10% of its network partners through the Mid-Point Assessment.
If a partner that belongs to multiple PPSs is requested to be removed from a PPS network, what impact will this have on any other PPSs that the partner may belong to? Recommend revising the rule on removing 5% of the network partners to say "PPSs will not be able to drop network partners constituting up to a loss of 5% of attributed lives for those partners that bill for Medicaid services. For those organizations that don't bill Medicaid (i.e. social services organizations, food services, etc.), the PPS will follow the guidance outlined in Section X of Attachment I as described above". - The reasoning behind this is that there are multiple providers, especially in the downstate/New York City area, that are members of multiple PPSs. If they only have a small number of lives with one PPS but a large number with another, it is administratively more beneficial for that provider to be in the network where they have the majority of attributed lives.	A network partner from one PPS would only be removed from the PPS removing them. This would not impact that provider's status in another PPS network. The determination of the now 10% cap on the removal of network partners is based on total number of partners in the PPS network. For example, a PPS with 100 network partners would be able to remove no more than 10% of its network partners through the Mid-Point Assessment. For example, Mental Health Provider 1 is in the network of both PPS A and PPS B. It has been removed from the network of PPS A, either by PPS A or through its own voluntary removal. Mental Health Provider 1 would still remain a network partner of PPS B.
Lastly, there is some confusion regarding the potential removal, if necessary, of 5% of network providers, and how that is determined in the case of organizations/practices that employ multiple providers (e.g. group practices).	The determination for the removal of network providers is based on how the provider is identified within the PPS network tool. If a practice employing multiple providers is listed as a single entity in the PPS network, i.e. a group practice, then the removal of the group would count as a single removal. If the individual providers are listed separately, each provider removed would count individually towards the cap of removed providers.

Implementation of Modifications
Comment	Independent Assessor Response
The PPS will only have two months to implement changes – this seems like a very difficult/unrealistic timeline to make changes to governance, PPS leads, and/or project focus. At this point, PPSs will have had nearly two years to stand up their networks and projects, but will only be given two months (January 31 to March 31, 2017) to quickly pivot to any recommendations. In addition, the PPSs' P4P measurement will start at the exact same time, with no significant time to have an impact on the measurement year that would conclude in June 2017. Recommendation: If there are significant changes to a PPS lead, governance structure, and/or project plan – the State and Independent Assessor should consider a delay in the transition from P4R to P4P for those metrics affected.	The Mid-Point Assessment timeline has been revised to allow for the PPS to develop a Mid- Point Assessment Action Plan (Action Plan) upon the submission of the final recommendations from the Commissioner of Health to CMS. The Action Plan will require the PPS to define all of the tasks and timelines necessary for the PPS to implement the recommended modifications. The PPS will need to submit the Action Plan to the Independent Assessor within 30 days from the date the recommendations are submitted by the Commissioner of Health to CMS. The Independent Assessor will have 30 days to review and approve the Action Plan for the start of DY3. At this time, the timelines set forth for the transition of Domain 2 and Domain 3 performance measures from P4R to P4P are not subject to change form those timelines established in the STCs of the waiver and Attachments I and J.
The timeline indicates the PPS is required to complete modifications to DSRIP Project Plans by March 31, 2017. We are concerned this may be a very short timeframe if modifications are substantial, such as merger with another PPS. As such, please consider language indicating that the due date for the PPS to complete modifications to DSRIP Project Plans will be set by the IA, in accordance with the magnitude of the modifications. In most cases, modifications will be due. Another option is to indicate a due date for the PPS to submit a proposed timeline for modifications that the IA would then need to approve.	The Mid-Point Assessment timeline has been revised to allow for the PPS to develop a Mid- Point Assessment Action Plan (Action Plan) upon the submission of the final recommendations from the Commissioner of Health to CMS. The Action Plan will require the PPS to define all of the tasks and timelines necessary for the PPS to implement the recommended modifications. The PPS will need to submit the Action Plan to the Independent Assessor within 30 days from the date the recommendations are submitted by the Commissioner of Health to CMS. The Independent Assessor will have 30 days to review and approve the Action Plan for the start of DY3.
In general, the proposed timeline of approximately two months, does not build in sufficient time to implement recommendations, modifications, and other changes. DSRIP is an inherently complex program with complicated governance and many thousands of participants. Any significant shifts will likely take more than 8 weeks, especially if it involves governance changes.	The Mid-Point Assessment timeline has been revised to allow for the PPS to develop a Mid- Point Assessment Action Plan (Action Plan) upon the submission of the final recommendations from the Commissioner of Health to CMS. The Action Plan will require the PPS to define all of the tasks and timelines necessary for the PPS to implement the recommended modifications. The PPS will need to submit the Action Plan to the Independent Assessor within 30 days from the date the recommendations are submitted by the Commissioner of Health to CMS. The Independent Assessor will have 30 days to review and approve the Action Plan for the start of DY3.

Release of Recommendations
Comment	Independent Assessor Response
Simultaneous Release of IA assessment to PPS and Public for Review. The currently proposed approach releases the initial IA midpoint assessment to the PPS and public for review simultaneously. The PPSs will be uniquely positioned to understand the specific guidance provided by the Independent Assessor when it comes to changes to PPS leadership, governance, or project plans; the public will have not seen this level of detail prior to the midpoint review. The PPS will also need to review, digest, and prepare a dissemination plan to its collaborators before it is released to the public. Recommendation: Release the initial midpoint assessment recommendations (October 28, 2016) to the PPS Lead organizations with a two-week initial window for review prior to releasing to the public.	The Mid-Point Assessment Overview has been revised to note that the Independent Assessor recommendations will be provided to the PPS one week prior to the release of the recommendations to the public for the public comment period.
The Independent Assessor Recommendations are set to be released on October 31, 2016, for PPS and Public comment in parallel. Given the importance of the Mid-Point Assessment process and the significant potential outcomes of the process, we strongly advocate that IA be required to provide each PPS with an in-person, individual, opportunity for direct dialogue in response to the IA's draft recommendations, in advance of such recommendations being made public.	The Mid-Point Assessment Overview has been revised to note that the Independent Assessor recommendations will be provided to the PPS one week prior to the release of the recommendations to the public for the public comment period. Upon receipt of the Independent Assessor recommendations PPS will have the opportunity to request a meeting with the Independent Assessor to discuss the recommendations.
What is the forum for submitting these recommendations? In writing or will there be an opportunity to have in-person meetings/discussions?	The comments/recommendations must be submitted in writing to the Independent Assessor. The Mid-Point Assessment Overview has been updated to note that PPS will have the ability to request a meeting with the Independent Assessor upon receipt of the Independent Assessor's recommendations.
Mid-point assessment IA recommendations should not be released to the public prior to the PPS having time to respond / remediate	The PPS will receive the Independent Assessor's recommendations for one week prior to the release of the recommendations to the public for the public comment period.

Behavioral Health
Comment	Independent Assessor Response
The assessment of the role of Behavioral Health (BH) in DSRIP may be supported by considering the following comments: 1. To date, milestones have focused on setting up governance, MOUs, and engaging providers so it's unlikely there will be any data about how many individuals are getting screened for BH conditions and/or getting access to BH services. 2. It is not known whether the BH role has been adequately examined until the Independent Assessor (IA) does the actual reviews. Is it possible for this document to include specific provisions on behavioral health?	The PPS Quarterly Reports currently capture the number of Medicaid members that have been engaged by each PPS for each project the PPS is implementing. Each project has a specific definition for what services, screens, activities must be performed for a Medicaid member to be considered Actively Engaged by the PPS. From this data it is possible to get a sense of the number of individuals getting access to BH services and/or getting BH screens. The Mid-Point Assessment will include a review of all PPS projects including those that include BH components and require the participation of BH providers.

Miscellaneous
Comment	Independent Assessor Response
In review of the DSRIP Mid-Point Assessment Overview, it is mentioned that an Independent Assessor Recommendation could be "consolidation of multiple PPS in to a single PPS." Could you please provide detail as to what would qualify a PPS for the IA to make this recommendation? For example, if a PPS is performing with deliverables being met and progress being made, would there be any chance of this recommendation being made?	The Independent Assessor's recommendations, including any recommendation for the possible consolidation of PPS, would be based on the data available to the Assessor at the time of the Mid-Point Assessment.
Limited PPS-to-NYS DOH Communication. The Independent Assessor has been positioned as the collector of PPS responses/recommendations, "Recommendations from the PPS will be reviewed by the Independent Assessor and considered in the development of the final recommendations to be submitted to DOH. The PPS will receive a standard format to be used in submitting recommendations for the Independent Assessor's consideration." This removes PPS and public recommendations from the public record. Recommendation: Allow PPSs to directly submit recommendations to New York State Department of Health.	As part of the Mid-Point Assessment, any comments or recommendations received by the Independent Assessor through the public comment periods will be documented and made available through the DSRIP website, consistent with the comments received on other DSRIP efforts such as the Project Plan Application. Further, consistent with the DSRIP Project Plan Application process, PPS will have the ability to submit comments or recommendations to the Commissioner of Health following the Project Approval and Oversight Panel (PAOP).
There is no mention of an appeals process.	The PPS will have the ability to appeal the DY2, Q1 determinations of the Independent Assessor, consistent with the appeals process in place for the PPS Quarterly Reports. While there is no formal appeals process during the Mid-Point Assessment, the PPS will have the ability to review the recommendations of the Independent Assessor and respond with their own recommendations or comments during a 30 day comment period. The PPS will also have the ability to request a meeting with the Independent Assessor to discuss the recommendations. PPS will also have the opportunity to present to the PAOP before the final recommendations are sent to the Commissioner of Health. Finally, the PPS will be able to submit comments or recommendations directly to the Commissioner of Health for his consideration prior to the submission of the final recommendations to CMS.
Moving the date of the mid-point assessment up is unfair to start-up PPSs that are still being established. Pre-existing networks will be much more prepared to participate in this process earlier.	The Independent Assessor understands this concern, however, it was determined that engaging in the Mid-Point Assessment process at an earlier date would help to provide all PPS with earlier feedback on the project implementation process. The earlier feedback is intended to allow PPS sufficient time to make the necessary modifications to promote improved performance outcomes once funding transitions to P4P.
What happens to our patients when we remove a provider and they are still caring for our patients?	Patient attribution is driven by the patient/provider relationship. Therefore if a PPS removes a provider from their network, the patients attributed to that provider and by extension the PPS would not be part of the PPS' attribution.
Some of these suggestions may not be feasible due to legal concerns. For example, can PPSs actually be forced to consolidate? Can providers be forced to join a network? Does the state or the PPS have this authority?	While the State does not have the authority under the STCs to force PPS to consolidate or to force providers to join a network, the STCs do allow the State to discontinue funding to a PPS as a result of the Mid-Point Assessment. In the event that a recommendation is made to discontinue funding to a PPS, that PPS lead organization and its network partners would have the ability to join another PPS network. The Independent Assessor may recommend a specific PPS to that PPS and its network partners based on its assessment of common projects, service areas, and other commonalities across the two entities.
On Page 4, there is a reference to Claims and Non- Claims data; what reporting period will be reviewed?	As articulated in the Mid-Point assessment, the Independent Assessor will focus on the progress made by each PPS through the end of the first quarter of DSRIP year 2. For claims and non-claims based data, the Independent Assessor will use data from the most recently completed measurement year
If the "DSRIP Project Plan Application" referenced is the Project Plan submitted in December of 2014, we would advocate that the reference point be the Implementation Plan submitted in June 2015, as the June submission of the Implementation Plan is what the PPS has been monitored against, and has received IA feedback on each quarter. In some cases, language in the narrative in the December 2014 Project Plan Application may no longer be accurate, and the more recent version is the Implementation Plan.	The DSRIP Project Plan referenced in the Mid- Point Assessment Overview is inclusive of the DSRIP Project Plan Application submitted in December 2014, the Speed & Scale commitments submitted in January 2015, and the Organizational and Project Implementation Plans submitted as part of the DY1, Q1 PPS Quarterly Report.
Will these recommendations be PPS specific or aggregated for the state as a whole? It is not clear in the document.	The recommendations will be PPS specific.
It is not clear if the IA team conducting the mid- point assessment will be different from the team completing the quarterly reports thus far.	The Independent Assessor team conducting the PPS Quarterly Report reviews is the same team conducting the Mid-Point Assessment.
It is vital that appropriate context is provided when examining the various milestones and metrics within the mid-point assessment. For example, while all components of DSRIP are important the assessment report should consider the value of the milestone with respect to the overall DSRIP program.	The Independent Assessor will review all projects consistently, however an emphasis will be placed on those project milestones that impact multiple project efforts. For example, the project requirement for Primary Care Physicians to reach PCMH Level 3 spans multiple projects and will therefore be weighted in a way to emphasize its importance across DSRIP projects.
Timeline for Mid-Point Assessment The proposed timeline needs to consider/account for the new appeals process for DY2, Q1 Quarterly Report. We believe the approximately 10-day appeals process conflicts with the September 29, 2016 date noted for the IA to finalize DY2, Q1 Quarterly Report and may have an effect on IA's ability to finalize the Mid-Point Assessment Recommendations by October 28, 2016. The proposed timeline should be modified to recognize the new appeals process and IA resources needed to meet the October recommendations deadline.	The Independent Assessor appreciates the concerns over the aggressive timelines set forth in the Mid-Point Assessment Overview, however these timelines are important to ensure that all recommendations are finalized and that PPS have plans in place to implement the recommended modifications for the start of DY3. It is anticipated that the PPS will have the ability to appeal any determinations made by the Independent Assessor for the DY2, Q1 PPS Quarterly Reports. The Independent Assessor will complete their reviews of the DY2, Q1 PPS Quarterly Reports and provide the results to the PPS by September 29, 2016. PPS will have until October 6, 2016 to submit an appeal of the Independent Assessor's determination. The Independent Assessor will have until October 13, 2016 to make a final determination on the appeal. Any determinations resulting from this appeals process will be considered when the Independent Assessor finalizes its Mid-Point Assessment recommendations by October 28, 2016.
To incorporate individual consumer feedback and assessment of social determinants, we suggests that surveys of consumers attributed to the PPS be incorporated into the Mid-Point Assessment process. A sample of consumer responses could be reviewed, scored and incorporated into the overall assessment scoring.	As noted in the Mid-Point Assessment Overview, the Independent Assessor will include claims and non-claims based data, including CAHPS data, as part of the review process. The CAHPS data will have the Independent Assessor to see some individual consumer feedback as part of this process.
As the scoring procedures of the Mid-Point Assessment are finalized, we urge the IA to include identification of barriers to accomplishing goals overall. In addition to measuring how the PPSs are doing at meeting expectations and making progress on reaching their intended outcomes, PPSs should be asked whether they have encountered any barriers as they carry out their projects. The PPSs should also be asked to indicate how these barriers were addressed and/or what plans they have to address potential barriers in the future.	The Mid-Point Assessment Overview has been updated to include a narrative for each project. As part of the narrative, the PPS will be required to document any challenges/barriers to project implementation and how the PPS has worked or will work to overcome the challenges/barriers. The narrative will also require the PPS to identify any best practices or innovative approaches the PPS has implemented.
For the public to understand the process and what the IA is recommending, information must be made as accessible as possible so the public comments are as robust and meaningful as possible. The IA should use summaries, matrixes, charts, and whatever other tools it can to explain the process and provide justification for their recommendations. We urge the IA to provide such summaries and other tools at least one week prior to the public comment period so there is adequate time for review of the materials before commenting.	The Independent Assessor will consider this comment in the preparation of materials during the Mid-Point Assessment process.
The Mid-Point Assessment should include distinct questions designed to capture information about the types of CBOs engaged in the PPS networks, how they are involved in project implementation, whether or not they are receiving DSRIP funding from the PPS, and more. This will allow the IA to see where each PPS could be doing better at reaching historically-underserved populations and addressing social determinants of health by engaging CBOs and involving them in their projects.	The Independent Assessor will have access to data from the PPS indicating the engagement of CBOs in project implementation efforts as well as on the flow of funds to CBOs from the PPS. This information will be used, in conjunction with the data collected through organizational milestones focused on CBO engagement to assess PPS efforts in engaging CBOs as part of their DSRIP efforts.
Additionally, we suggest integrating a "360 evaluation" type component in the Independent Assessor's review of DSRIP projects in which the non-lead partners are interviewed and asked about their experience with substantive participation in governance, project development, project execution, communication with and between the lead and other partners, conflict resolution, dollar flow, and general satisfaction with the lead PPS.	The Independent Assessor has revised the Mid- Point Assessment Overview to include a "360- like" evaluation of the PPS. The Independent Assessor will be creating a survey to capture feedback from a sample of partners from each PPS network. The sample will cover a cross- section of all partners in the network to capture diverse perspectives on the PPS. The survey will focus on items such as network partner experience with the PPS on participation in governance, project development and project participation. The survey will also capture feedback on PPS communication with network partners, funds flow, and general satisfaction with the PPS.
We recommend that the Independent Assessor uses the Primary Care Plans submitted by the PPS leads as a data source to determine if the PPS lead is meeting goals related to its identified plan for developing primary care capacity and access.	The Independent Assessor will consider this recommendation however, at this time no modifications have been made to the Mid- Point Assessment Overview.