Circles of Care Evaluation Technical Assistance Center

It was out of the growing appreciation for the high prevalence of mental health problems among American Indian and Alaska Native children and adolescents, the inadequacy of existing services for these children, adolescents, and their families, and the need for community-based strategic planning to improve these services that Circles of Care was born. CMHS initially set the following goals for the initiative: 

• to support the development of mental health service delivery models that are designed by AI/AN communities to achieve outcomes for their children that they chose for themselves;
• to position tribal and urban Indian organizations advantageously for future service system implementation and development;
• to strengthen tribal and urban Indian organizations’ capacity to evaluate their own service system’s effectiveness; and
• to develop a body of knowledge to assist tribal and urban Indian organizations, and other policy makers and program planners from all child-serving systems in improving systems of care for the AI/AN population overall.

As a part of this effort, from 1998-2014 CMHS funded two organizations to provide technical assistance to the Circles of Care grantees. The National Indian Child Welfare Association (NICWA) was charged with providing technical assistance in regard to the program development activities of the grantees. CAIANH was charged with providing technical assistance in regard to the program evaluation activities of the grantees. The description of the evaluation provided here applies to the period of time that the Circles of Care Evaluation Technical Assistance Center (CoCETAC) supported grantees’ program evaluations (1998-2014).

The Circles of Care initiative required an Evaluation Plan that both informs and assesses the strategic planning process. The overarching goals of the Circles of Care Evaluation Plan were as follows: 

• to provide a knowledge base for the planning effort;
• to facilitate the process for developing the capacity for ongoing evaluation efforts;
• to examine the feasibility of the service system models; and
• to document and disseminate the results of the initiative.

Thus, evaluation activities are designed to assure that the final service delivery models developed through the Circles of Care initiative are consistent with community needs, developed through community consensus building, and practical and feasible given the resources available.

Evaluation components

The Circles of Care evaluation consisted of six specific components designed to achieve the four overarching goals noted above: 1) to describe the need for services within each participating community; 2) to develop a locally-relevant definition of SED; 3) to describe the current services available in these same communities; 4) to design a plan for measuring outcomes for the new service delivery model; 5) to assess the feasibility of this model system of care; and 6) to describe and assess the strategic planning process. 

Technical assistance activities

CoCETAC utilized a variety of technical assistance activities that are aimed at achieving the goals outlined above. The complex nature of the evaluation effort requires careful orientation to the evaluation goals and components, attention to each grantee’s priorities, coordination of efforts across grantee communities, and ongoing feedback and troubleshooting. Rather than rely on a single activity to achieve the goals for the Circles of Care evaluation, CoCETAC employed the following menu of activities:

1. Grantee Meetings. In many ways, the Grantee Meetings were the core of CoCETAC’s technical assistance activities. CoCETAC used didactic sessions to orient the grantees to each of the evaluation components as well as evaluation-related tasks (such as working with Institutional Review Boards). These didactic sessions were invariably followed by a series of facilitated large and small group discussions (which include program staff, evaluators, and family representatives) aimed at increasing the grantees’ appreciation of the nature of these evaluation efforts, including both the specific expectations and individual flexibility embedded in these activities. CoCETAC also facilitated special sessions for evaluators that allow the grantees to share their experiences with each other and problem solve as a group around difficult issues that arise during their work. The Grantee Meetings were convened jointly by CoCETAC and NICWA, the program development technical assistance provider, with the goal of demonstrating and conveying the important relationships between evaluation and program development for this initiative.
2. Evaluator Meetings. While not envisioned in CoCETAC’s Scope of Work, the need for meetings focusing specifically on evaluation became quite apparent. These meetings consisted mostly of large-group discussions that focus on developing a common set of domains that each of the grantees is expected to address in their evaluation efforts. These meetings also provided additional opportunities for discussion of exemplary evaluation practices and group problem solving. While intended primarily as an opportunity for the evaluators to meet as a group, the vast majority of grantees send at least one representative from their program staff to participate. Representatives from CMHS, IHS, and NICWA participate in these meetings as well.
3. Site Visits. While the Grantee and Evaluator meetings were the primary avenues for group-oriented technical assistance to the grantees, the annual site visits to the grantees is the key activity for engaging the grantees in intensive, individualized discussions. Topics covered in these 1½ day visits include survey development, conducting focus groups and case studies, database development, data analysis and interpretation, as well as strategies for building support for the initiative and its evaluation effort among families, youth, service providers, and tribal leadership.
4. Scheduled Technical Assistance Conference Calls. Every month, CoCETAC’s Associate Director conducted scheduled conference calls with each grantee’s Program Director and Evaluator. These calls were another opportunity for intensive, individualized discussions with grantees, and assisted the grantees in addressing specific evaluation issues as they arise, tracking progress, and identifying issues requiring further discussion.
5. Ad Hoc Conference Calls and Email Exchanges. In addition to the scheduled conference calls, CoCETAC staff conducted a number of ad hoc conference calls and email exchanges to address a variety of issues that arise during the grantees’ evaluation activities. 
6. Detailed Reviews and Critiques of Evaluation Reports. During the course of their programs, all of the grantees submitted draft reports that were reviewed in detail by CoCETAC staff. This provided the opportunity for CoCETAC to help the grantees further refine their evaluations and tailor their reports for both local and national use.
7. Referral to and Provision of Key Reference Materials. CoCETAC assisted the grantees in identifying reference materials that support their evaluation activities. In addition, CoCETAC provides copies of many key reference materials that are highly relevant to all grantees.

This wide variety of evaluation activities allows CoCETAC to meet the diverse evaluation technical assistance needs of the Circles of Care grantees and supports their efforts to produce results relevant to their strategic planning efforts.

Publications