Evaluation

 
Introduction to Realist Evaluation, by Mansoor A. F. Kazi, PhD, Lead Evaluator for Chautauqua Tapestry

 

Jamestown Public Schools, Chautauqua County Mental Heath Clinic, Family Services of Chautauqua Region and Boys and Girls Club are among the 28 agencies that participate in the SAMHSA-funded Chautauqua Tapestry’s local evaluation. The local evaluation strategy utilizes a new approach to evidence-based practice based on realist evaluation, with the central aim of investigating what interventions work and in what circumstances (Kazi, 2003). This approach essentially involves the systematic analysis of data on 1) the client circumstances (e.g. demographic characteristics, cultural differences and needs, environments in which people live and function, and the nature of baseline target problems); 2) the dosage, duration and frequency of each intervention in relation to each client; and 3) the changes in the outcomes as observed through the repeated use of reliable outcome measures with each client. This is a mixed methods approach, combining the traditions of epidemiology and effectiveness research in social work. As the research designs unfold naturally, data analysis methods are applied to investigate the patterns between the client-specific factors, the intervention variables, and the outcomes. This analysis can be repeated at regular intervals and helps agencies to better target their interventions, and to develop new strategies for users in the circumstances where the interventions are less successful. The Tapestry evaluators assist the partner agencies in the repeated use of a reliable outcome measure (e.g. school outcomes) and the regular updating of information on their existing management information systems. This approach is based on the realist evaluation process with the merger of the school district data to help with the investigation of what interventions work and with whom.

 

This evaluation strategy includes the application of robust research methods drawn from both effectiveness research and epidemiology traditions on management information system data routinely collected by humans service agencies in Chautauqua County in New York State, all partners in the Chautauqua Tapestry System of Care funded by the federal Substance Abuse and Mental Health Services Administration ($9,000,000; 2008 to 2015). As part of this grant, this evaluation strategy was offered to all human service agencies in the county, and to date 28 have taken part, including the county’s mental hygiene and social services departments, not-for-profit agencies, and 7 school districts. All services (social services, mental health, schools, and probation) are evaluated in all human service agencies taking part.

 

Brief description of the Realist evaluation strategy

 

The main purpose of the evaluation is to partner with human service agencies and school districts to utilize the data that is already collected by the agencies, to investigate what programs work and for whom, enabling the development of effective programs and/or the linkage of existing effective programs to the schools. This approach was first developed in United Kingdom and Finland, and also applied in Chautauqua County, NY; and in 2010, Chautauqua Tapestry (NY) received SAMHSA’s Gold Award for Outstanding Local Evaluation, having used realist evaluation strategies in schools, mental health, youth justice and other human service agencies. These agencies routinely collect data that is typically not used for evaluation purposes. The 100% evaluation strategy utilizes a new approach to evidence-based practice based on the realist evaluation paradigm, with the central aim of investigating what interventions work and in what circumstances (Kazi, 2003). This approach essentially involves the systematic analysis of data on 1) the service users' circumstances (e.g. demographic characteristics); 2) the duration and frequency of each intervention in relation to each service user; and 3) the repeated use of reliable outcome measures with each service user e.g. academic outcomes already recorded by schools. This is a mixed methods approach, combining the traditions of epidemiology and effectiveness research in human services (Kazi, 2003; Videka, 2003), helping participating agencies to use the data they have already collected at any given time. The main strength of this approach is that the data is already collected by the participating agencies and routinely entered in their management information systems. Each participating agency continues to manage and hold their own data, and uses a de-identified copy of the data which is analyzed with and for each agency for evaluation purposes. Also, the data analysis is done three times a year in real-time. The findings are for the agency, and each agency decides if and when to share these findings with others.

 

What difference did the evaluation make?

 

The strategy that is used in Chautauqua is Realist Evaluation, as applied in schools, mental health, youth justice and other human service agencies.  One of the first adopters was Jamestown Public Schools (JPS), the largest school district in the county. Studies have indicated that school interventions can be utilized to make numerous improvements in the overall well being of students (e.g. Goldenson, 2011). In order to evaluate interventions, most researchers utilize pre/posttest designs, looking at the initial baseline levels of the targeted behavior before and after the intervention is implemented. Chautauqua Tapestry’s evaluation team has been utilizing whole-school data for evaluation. Kazi (2014) evaluated school-based programs in Jamestown, NY using whole-school data, comparing academic outcomes of those receiving interventions with those who were not. It was found that, although Caucasian children and youth achieved better academic grades compared to Hispanic and African Americans, the binary logistic regression indicated that the actual predictors for improved academic achievement were the tutoring and mental health services as well as individualized education programs. It was found that those receiving these interventions had significantly improved their Grade Point Average (GPA) scores when compared to those that had not received these interventions. The data analysis helps school administration to identify gaps in care, and helped to build partnerships for additional mental health counseling on-site for the school district.  At this time, 7 out of 18 school districts have participated in local evaluation.  A very important additional method has come out of the work that would be carried out through the next four years, namely  “Innovate Collaborate Educate ICE-8”, enabling the participating agencies and schools to coordinate and maximize public, non-profit and private resources to deliver critical services to students and their families, thereby increasing student achievement and generating other positive health and wellbeing outcomes. This new all County program has been signed on by all the 18 school districts in Chautauqua County under the joint direction and support of Tapestry and of Erie 2 BOCES. Strategy for expansion is to build the necessary champions within each organization and to develop learning communities to support specific learning goals, based on findings from the data regularly analyzed with each agency.

 

 

Why did this evaluation have the impact that it did?

 

Tapestry’s local evaluation demonstrates that human service agencies can use their data for evaluation and apply robust research methods to investigate their effectiveness in real-time (Kazi, Coombes, Krause & Anderson, 2015). This realist evaluation in both United Kingdom and USA shows how the actual data routinely collected by human service agencies can be used for evaluation. Findings at regular intervals from a 100% evaluation with the entire population helps to better target and develop the interventions in a longitudinal, continuous evaluation. However, it is important to recognize that the impact of this evaluation is high because this is a partnership between academics and the human services, helping these services to utilize the data in a suitably de-identified form, and not the usual approach of the academics taking the data away and producing reports after practice has moved on.

 

For example, school based interventions are often used to target specific student behavior, including socio-emotional problems and dealing with stress. Many studies in both USA and United Kingdom have indicated that school interventions can be utilized to make numerous improvements in the overall well being of students. However, most evaluations of interventions targeting school students do not use academic outcomes routinely collected on all students and instead concentrate on the use of standardized measures or surveys mainly with smaller samples. On the other hand, as part of the strategy demonstrated here, Kazi (2014) evaluated school-based programs in Jamestown, NY using whole-school data, comparing academic outcomes of those receiving interventions with those who were not. A 100% evaluation with the entire population helps to better target and develop the interventions in a longitudinal, continuous evaluation.

 

Human service agencies routinely collect data, but this data is not typically used for evaluation. Tapestry’s local evaluation demonstrates how to make the best use of this available data from the management information systems of schools, mental health and other services to continuously evaluate alongside the repeated universal screening tools and to develop effective services. Evaluation can have a maximum impact when used effectively to expand collaboration between human service systems. The agencies are able to develop their services and to make informed decisions based on the evaluation findings at regular intervals, helping to enhance practice by utilizing findings from data analysis methods drawn from a combination of epidemiology and effectiveness research to investigate if the comprehensive array of services and supports are working to provide better outcomes for youth and families.  This evaluation strategy has helped to enhance the internal evaluation capacities within each of the agencies with regard to improved data collection within their management information systems, and the regular use of this data to investigate what interventions work and in what circumstances.

 

Tapestry Partners’ Services are Predictors for Improved Behavior

 

This evaluation is based on findings from the universal screening (SDQ) with grade levels K to 4, and the changes in the incidents of behavior between the school years 2013-14 and 2014-15. It was found that there was some evidence of effectiveness in improving the SDQ abnormal total difficulties score with the mental health services in relation to both county mental health and FSCR. With regard to changes in reported incidents of behavior, all three services were predictors for improvement from one school year to the next. In relation to each of the three services—county mental health, Family Services of Chautauqua Region and the Boys & Girls Club—it was found that those receiving these services were almost twice as likely to improve on the incidents of recorded behavior than those not receiving these services.

 

The examples of evaluation described here provide the evidence of how Chautauqua County is a world leader in the regular analysis of data already collected by human services, and combined with outcome measures and school outcomes from the whole school data, are helping to improve and develop more effective services. That is why we received SAMHSA’s first Gold Award for Outstanding Local Evaluation in 2010. Since 2008, the Evaluation Team (with Dr. Kazi as the lead evaluator) has been committed to developing a rigorous evaluation of the Tapestry Expansion Initiative that is consistent with the principles of ethical and culturally sensitive human service research. This evaluation strategy is based on Tapestry 2.0’s overall aim of moving the system of care towards wide scale adoption, and as one of the 27 rural counties in New York State, Tapestry 2.0 will demonstrate how the system of care can be developed from innovation through to wide-scale adoption.

 

The purpose of this program is to improve mental health outcomes for children and youth (4 to 21 years of age) with serious emotional disturbances (SED) and their families.  The program will support the wide scale operation, expansion and integration of the system of care. SAMHSA’s Theory of Change framework is organized into five phases: Innovation, Translation, Dissemination, Implementation, and Wide-Scale Adoption. Each phase encompasses a range of strategies, activities, programs, and tasks, which pave the way toward strategic and evidence-based behavioral health system change. Surveillance and evaluation underpin each element of this process, and in Chautauqua County through the work of Chautauqua Tapestry since 2008, there has been a strong emphasis on evaluation, helping participating agencies to use the data they already collect in their management information systems to investigate what works and in what circumstances at regular intervals during the year, and to use the findings from evaluation to inform the development and expansion of the system of care.  This evaluation strategy has become a component part of Tapestry 2.0’s system of care, and wide scale adoption also means the wide scale adoption of this evaluation strategy. At present, Chautauqua County is the only one in the world that combines all human service data regularly with all school district data, using robust evaluation research methods in partnership with each agency to investigate what works and for whom at quarterly intervals in each year.

 

Evaluation has been a stand out point for Tapestry, and will continue to do so for Tapestry 2.0.  The strategy that is used in Chautauqua is Realist Evaluation, as applied in schools, mental health, youth justice and other human service agencies.  One of the first adopters was Jamestown Public Schools (JPS), the largest school district in the county. Studies have indicated that school interventions can be utilized to make numerous improvements in the overall well being of students (e.g. Goldenson, 2011). In order to evaluate interventions, most researchers utilize pre/posttest designs, looking at the initial baseline levels of the targeted behavior before and after the intervention is implemented. Chautauqua Tapestry’s evaluation team has been utilizing whole-school data for evaluation. Kazi (2014) evaluated school-based programs in Jamestown, NY using whole-school data, comparing academic outcomes of those receiving interventions with those who were not. It was found that, although Caucasian children and youth achieved better academic grades compared to Hispanic and African Americans, the binary logistic regression indicated that the actual predictors for improved academic achievement were the tutoring and mental health services as well as individualized education programs. It was found that those receiving these interventions had significantly improved their Grade Point Average (GPA) scores when compared to those that had not received these interventions. The data analysis helps school administration to identify gaps in care, and helped to build partnerships for additional mental health counseling on-site for the school district (Kazi & Kim, 2015).

 

At this time, 7 out of 18 school districts have participated in local evaluation.  A very important additional method has come out of the work that would be carried out through the four year Expansion, namely ICE-8. This new all County program has been signed on by all the school districts in Chautauqua County under the joint direction and support of Tapestry and of Erie 2 BOCES.  Evaluation is an integral component of the ICE-8 initiative, and as a result of this SOC expansion grant, all 18 school districts will be invited to take part in this evaluation in the same way as JPS has been. Strategy for expansion will be to utilize the newly developed Chautauqua Evaluation and Translational Research innovation Center (CETRiC) to build the necessary champions within each organization and to develop learning communities to support specific learning goals.

 

Plan for Chautauqua Tapestry X 2 (2015-2019)

 

  1. To date, Chautauqua County is the only one to receive SAMHSA’s Gold Award for Outstanding Local Evaluation. An integral part of the plan is to expand this work to include all school districts and human service agencies in the county. Chautauqua County is a world-leader in realist evaluation, and to date 28 agencies (including 7 school districts) have collaborated in applying robust research methods to the data they already collect to investigate what works and for whom.

  2. The evaluation team will work in partnership with each participating agency to undertake regular analysis of their data every three months, and to prepare reports on the findings to help better target and to develop the human services.

  3. Data from the participating agencies’ management recording systems, including all 18 school districts, will be accessed to analyze what interventions work and in what circumstances, every three months. The outcomes would include all school outcomes (academic outcomes, attendance and reported incidents of behavior), the Strengths and Difficulties Questionnaire (Goodman, 2001) universal screening in schools, and other standardized measures selected by the agencies and/or required for the SAMHSA grant. Data analysis will be carried out by the evaluation team with each agency, and then reported by the agencies with the participation of the evaluation team to the governance body and the advisory committee in each three-month period. In this way, all the data will become an important part of the Continuous Quality Improvement process.

  4. The Evaluation Team is committed to utilizing a collaborative, empowerment approach with an advisory committee comprised of families and youth from within the County system of care.  Through proactive collaboration with Tapestry's cultural competence consultant, as well as members of the evaluation team serving on the cultural competence committee, it is intended that representatives on the evaluation advisory committee will represent all ethnic groups served, as well as be inclusive of other groups served (e.g., sexual orientation, gender, rural). The evaluation will be based on consumer perspectives as well as evidence of the impact of the services on the needs and aspirations of the service recipients.

  5. Sustainability is an important goal for the Evaluation team. This will be achieved through Chautauqua Evaluation and Translational Research innovation Center (CETRiC) which will strengthen the partnership of all Tapestry partners (including United Way and Chamber of Commerce) in integrating this evaluation strategy into the daily practice of participating human service agencies, as well as enabling agency partners to apply for other sources of funding. The Evaluation team will assist each agency to enhance its own capacity to sustain evaluation, undertaking regular analyses of the data from their management information systems, to investigate what works and for whom in real time.

 

 

References

 

 

Goldenson, J. (2011). When there is no blueprint: The provision of mental health services in alternative school programs for suspended and expedited youth. Child & Youth Services, 32(2): 108-123.

 

Goodman, R. (2001) ‘Psychometric Properties of the Strengths and Difficulties Questionnaire (SDQ)’, Journal of the American Academy of Child and Adolescent Psychiatry, 40, pp. 1337-1345.

 

Kazi, M.A. F. (2003). Realist Evaluation in Practice, London: Sage Publications.

 

Kazi, M. A. F. (2014) "Realist Evaluation and Effectiveness Research: An Example from School-based Interventions" in Kalliola, S. (ed.) 'Evaluation as a Tool for Research, Learning and Making Things Better', Newcastle-Upon-Tyne, UK: Cambridge Scholars Publishing, pp. 31-44. 

 

Kazi, M. A. F., Coombes, M., Krause, D., & Anderson, J. (2015). Effects of Solution-Focused Training on the Permanency Outcome in Child Welfare: A Realist Evaluation. International Journal of Interdisciplinary Social and Community Studies, 10(4), pp. 27-39.

 

Kazi, M. A. F. & Kim, Y. (2015). Realist Evaluation of What Programs Work, for Whom and in What Circumstances: Utilizing Whole School Data to Improve Student Outcomes. Journal of Academic Perspectives, vol. 2.

 

Videka, L. (2003). ‘Accounting for variability in client, population, and setting characteristics: moderators of intervention effectiveness’, in Rosen, A. & Proctor, E. K. (eds.) Developing Practice Guidelines for Social Work Intervention, New York: Columbia University Press, p. 169-192.

Call us:

(716) 753-4507

Find us: 

2 Academy Street, Suite 202

Mayville, NY 14757

© 2015 Chautauqua Tapestry

 

Contact webmaster at KnappK@co.chautauqua.ny.us