Behavioral healthcare providers are eager to create industry-wide core performance measures. Individual facilities collect voluminous data within their own organizations. However, a pilot test of key benchmarking indicators found that substantial challenges remain in gathering, reporting, and comparing data across systems. The National Association of Psychiatric Health Systems (NAPHS) White Paper: Lessons Learned from Pilot Testing of the NAPHS Benchmarking Indicators identified measures that hold great promise for the field. More importantly, the participants in this 2000 pilot test demonstrated their willingness to share this critical data. The project was coordinated by the NAPHS Benchmarking Committee, chaired by NAPHS Immediate Past President Peter Panzarino, M.D., chairman of the Department of Psychiatry at Cedars Sinai Medical Center in Los Angeles.

The pilot test, which focused on nine indicators chosen from an earlier consensus-driven process of the NAPHS Benchmarking Committee, analyzed data from 48 organizations offering a total of 637,241 days of inpatient care; 601,595 days of residential care; and 161,993 days of partial hospital care. The facilities represented all levels of care (including inpatient, residential, partial hospitalization, and outpatient services) and all populations (including child, adolescent, and adult). The indicators reviewed in the pilot test were:

-- adverse drug reactions
-- completed suicide
-- attempted suicide
-- restraint
-- seclusion
-- symptom/function measure
-- readmission
-- patient satisfaction
-- peer review

While no specific data is reported in WHITE PAPER: Lessons Learned from Pilot Testing of the NAPHS Benchmarking Indicators, the document outlines challenges identified in the testing phase. Commentary is provided on each of the indicators in the pilot test.

CHALLENGES FOR THE FIELD

According to the White Paper, any data-collection effort -- and any national core performance measures in particular -- must:

-- provide value in the data generated that is in proportion to the intensity of the data collection effort. Allocation of limited resources needs to be directed to the collection of the most important data.

-- focus on indicators that provide the most useful clinical and operational data possible within the scope of the data currently available within organizations.

-- recognize that definitional decisions have significant policy implications. Widely diverse state, local, and national standards often lead organizations to adopt similar -- but not identical -- definitions. This makes meaningful data-collection extremely difficult -- and potentially costly. If organizations use operational definitions that are different from the question asked, they are not able (short of going to primary sources such as the patient record or reviewing their entire data base and applying the study definitions) to report the data.

An example of the challenges of definitional variations became clear in looking at restraint/seclusion indicators. In this pilot test, there was great variation in the definitions of restraint and seclusion, particularly as they relate to children and adolescents. Reporting mechanisms appear to be in place in all organizations for collecting data about restraint and seclusion. If the field (in conjunction with regulatory and accrediting organizations) can agree on consistent definitions, the possibility for developing meaningful benchmarks seems to be very strong.

Similarly, the definition of attempted suicide used in this pilot test shows how difficult it is to standardize definitions. Because of the wide variation in incidences reported, we suspect that some organizations reported incidences that were not of the severity of the operational definition used in the pilot (actually or potentially life-threatening or resulted in the need for urgent intervention).

According to the report, indicators with the most promise for benchmarking across systems are those that have generally similar definitions, are collected by a majority of sites, are retrievable, and do not require extensive re-collection of data.

"As this pilot test demonstrates, there is high interest and commitment by behavioral health providers to work towards core performance measures that will help organizations improve the quality of care and be responsive to the needs of those who seek mental health care," said NAPHS Executive Director Mark Covall. "We have learned a great deal about the importance of focusing on data that is relevant to clinical operations and collected in ways that conserve limited resources. We look forward to sharing the knowledge we have gained about what works and the challenges ahead. Our experience can help to inform other national efforts to identify core performance measures and will help us to improve our own association's data collection efforts," he said.

COMMITTEE MEMBERS
NAPHS Immediate Past President Peter Panzarino, M.D., of Cedars Sinai Medical Center, CA was the chair of the Benchmarking Committee which oversaw the pilot test. Other committee members included James M. Cole, Devereux Foundation, PA; Allen S. Daniels, Ed.D., University Managed Care/University Psychiatric Services, OH; Naakesh A. Dewan, M.D., University Managed Care/University Psychiatric Services, OH; Susan Eisen, Ph.D., McLean Hospital, MA; Frank Ghinassi, Ph.D., Western Psychiatric Institute & Clinic/University of Pittsburgh Medical Center, PA; Leonard S. Goldstein, M.D., Integrated Behavioral Care, VA; Gay C. Hartigan, Liberty Management Group, NJ; John Lehnhoff, Ph.D., Richard Young Center, NE; Robert Mansfield, Carrier Foundation, NJ; William Nolan, Ph.D., consultant, TN; Richard T. Palmisano, Brattleboro Retreat, VT; Howard Waxman, Ph.D., Belmont Center for Comprehensive Treatment, PA; Martin Schappell, Universal Health Services, PA; Kathleen McCann, R.N., D.N.Sc., and Carole Szpak, NAPHS Staff Liaisons.

ORDERING INFORMATION

The WHITE PAPER: Lessons Learned from Pilot Testing of the NAPHS Benchmarking Indicators, is $40"prepaid"from the National Association of Psychiatric Health Systems, 325 Seventh Street, NW, Suite 625, Washington, DC, 20004. Call 202/393-6700, Ext. 15, for ordering information (Mastercard and Visa accepted

---National Association of Psychiatric Health Systems

Back to The Science of Mental Health