How a Research Process Affected Practitioners

The research process brought frustrations, but mostly benefits, for the Head Start program.
by Corinne Lewkowicz and Stacy Dimino

Research is a dynamic process, affecting both researchers and participants (Wagner 1997; Howard, Lewkowicz, & Dickinson in press). Issues affecting the relationship are both internally and externally imposed. Internally, agencies have other bodies to which they must report (e.g., internal review boards, boards of directors); externally, agencies must offer proof of efficacy (e.g., research results or child outcome measures) to funding agencies or peer-reviewed journals. External constraints are particularly important for agencies that provide Head Start services because they use government resources. Therefore, accountability is critical.

First, we will describe the research process—including its frustrations—that brought us together: a nonprofit agency that offers Head Start services and the research consortium whose specific goal was to study Head Start quality. Then, we will focus on how we problem-solved and what we learned from our collaboration.

The Research Partners

Communities United, Inc. (CUI) provides child care services, including Head Start, to 11 communities in the greater Boston area. Founded in 1970 by a group of parents, volunteers, and town officials, CUI’s expressed goal is to provide comprehensive education, health, and social services to economically disadvantaged children and their families. Additionally, CUI is the lead agency for five Community Partnerships for Children (CPCs), a statewide effort to provide child care coverage for working parents whose incomes are too high to qualify for Head Start, yet too low to cover child care costs.

In 1995, CUI and the New England Quality Research Center (NEQRC) began a five-year collaboration examining the efficacy of Head Start programs. The NEQRC was part of a larger Head Start Quality Research Center (QRC) Consortium that included the High Scope Educational Foundation (Ypsilanti, Michigan), Georgia State University, and the Frank Porter Graham Child Study Center at the University of North Carolina at Chapel Hill. QRCs were funded by the Administration on Children, Youth and Families (ACYF) to respond to questions about the quality of Head Start programs nationwide.

The NEQRC itself was part of the Education Development Center’s (EDC) Center for Children and Families. In addition to EDC, its local research partners included the Massachusetts Society for the Prevention of Cruelty to Children, Harvard University, and Boston College. The primary goal of the NEQRC was to examine children’s language, literacy, and social development related to Head Start services. Not surprisingly, this overarching research agenda corresponded directly to the interests of the individual members of the NEQRC.

The Stated Goals

As a direct service agency, CUI was primarily interested in the provision of high quality services to children and families in their programs. While CUI staff appreciated the need for research that illuminated the issues affecting the national Head Start population, their primary concern was the children in their care. CUI staff were willing to contribute to generalizable research findings, but a more pressing need was to better understand their classrooms and children. They wanted to learn specific information that could substantially improve the lives of the children in their programs. CUI staff are direct-care providers, a role that is necessarily time- and labor-intensive and, therefore, leaves little time to participate in research that is not directly connected to their immediate work.

CUI senior management articulated their goals for quick data turnaround and classroom- and child-specific data. There were two compelling reasons: CUI faced both internal (i.e., the Board of Directors) and external (i.e., the Head Start Bureau) pressures to document the efficacy of its programs, and staff members wanted to respond to situations and/or needs that became evident during data collection. However, CUI’s goals were incompatible with the nature of the research process. Data turnaround is necessarily slow. Plus, the requirements of a large sample for statistical analyses and of participant confidentiality preclude the sharing of site- or child-specific information.

These disparate goals, not surprisingly, led to frustrations for both sides (Howard, Lewkowicz, & Dickinson in press). CUI was frustrated with the length of time required for data analyses and with the lack of program-specific findings. Furthermore, as the research process evolved, and in response to questions posed by various agencies (e.g., funding agencies, the Department of Education), CUI management had an even greater need for tools that could answer program-specific questions. As one manager said, “Sometimes it felt as if we were just having data extracted with nothing gained except broadly helping Head Start.” EDC, in turn, was frustrated by the research constraints that made it impossible to respond to CUI’s pressing needs.

CUI Takes the Initiative

In response to these frustrations, CUI made several adaptations in the research process. The agency identified methods and tools that allowed it to collect and analyze data for internal program evaluation and for reports to external agencies. In short, CUI became more independent in terms of defining and executing an internal research process that would address its needs.

One of CUI’s first steps was to adopt The Early Childhood Environment Rating Scale-Revised (ECERS-R) for program-wide assessment (Harms, Clifford, & Cryer 1998). The NEQRC had introduced this Scale as an overall measure of classroom quality. But concerns arose when CUI management was not able to identify individual classroom performance on the ECERS-R. When one classroom scored significantly lower than others, its identity could not be revealed by EDC due to confidentiality restrictions. Therefore, CUI could not target support to a classroom in great need and improve its service to children and families. CUI concluded that the Scale, when used by EDC for research purposes, was only useful as a very general measure of overall program quality.

CUI management was also concerned about the “snapshot” nature of the Scale and its scoring system. EDC used only the language and literacy subscale of the ECERS-R; other aspects of the classroom environment were not scored. However, CUI thought the “full classroom picture” was needed to assess program quality.

EDC staff responded to CUI’s concerns by training CUI staff in administering the ECERS-R in its entirety. The Scale is now used by CUI as part of its annual program self-evaluation. All aspects of the classroom environment are assessed, and data on individual classrooms are available to CUI management. It is now possible to track classroom and program quality from year to year and to point to areas of strengths as well as priority areas for continuous improvement.

CUI also adopted another research tool first introduced by NEQRC. The Head Start program decided to use the Self-Regulation Scale (SRS), a 10-item measure, to rate children’s social competence and task mastery (Bronson 1999). It was developed in response to a desire for a brief, easily completed scale that measures some of the same constructs assessed by NEQRC with the much longer Bronson Social and Task Skill Profile: Teacher Version (Bronson 1996).

At present, the shorter Scale (SRS) is completed for each child as part of the initial screening and again, at the end of the year. It is used by CUI for two main reasons: 1) to identify potential themes for classroom curricula, and 2) to identify specific children needing further intervention. The developer of the scale, Dr. Martha Bronson, has been and is actively involved in discussions with CUI staff regarding the best use of the tool.

These two instruments, adopted from the original versions used in the research design, provide CUI with useful information. The results are used internally to identify needs and areas of strength and to suggest topics for in-service trainings. The results are also used externally. For funding agencies, CUI has proof of self-evaluation with widely accepted tools. Furthermore, funders increasingly rely on the ECERS-R to examine program quality. By using this Scale on a yearly basis, CUI ensures program quality in a manner directly comparable to other agencies.

Lessons Learned

CUI’s partnership with EDC has led to changes within the agency. A copy of all data collected by CUI staff is kept on-site in order to answer agency-specific questions in a timely manner. Additionally, CUI has access to a research consultant, as well as appropriate statistical packages, to assist in data analysis. CUI now requests feedback from studies in which it participates and uses this information to improve the agency’s services. For example, CUI recently participated in a large-scale pilot study conducted by the Massachusetts Department of Education (DOE). School systems, child care agencies, aftercare programs, family child care providers, and parents answered questions regarding the availability of and need for child care services. CUI requested and received a copy of their data from the DOE and has used them to obtain needs-related information for grant applications.

In sum, what are the lessons CUI staff learned from their research collaboration with the NEQRC?

• They learned the necessity of clearly identifying each organization’s research goals and methods.
• They learned the importance of identifying benefits of participating in the research process, both general and specific.
• They learned the necessity of identifying demands that will be made on staff and how best to assist staff with these demands (including, but not limited to, the use of tools added after the research had begun).
• They gained a greater sense of the time required for data collection and analysis and what expectations are realistic.
• They learned the difference between research purposes: that is, national research that feeds into policy analysis and national program evaluation versus in-house research that guides internal program improvement and informs practice.

The collaboration between these two partners, while at times difficult, was generally successful. In fact, the CUI managers think the research process was a “great learning experience and now we feel quite savvy about research.” The Policy Council is eager for reports on how the program is doing and for evidence of positive change. They have come to appreciate how research investigation and data collection can lead to program improvement.

Recently, CUI identified an important research agenda around language and literacy. Although the ECERS-R scores were showing overall improvement in the classroom environments, the teaching of language and literacy needed to be strengthened in the program. CUI agreed to participate in another national Head Start research investigation with EDC that addressed these educational concerns. A classroom observation tool (ELLCO) developed by EDC was used to collect baseline data. EDC and CUI co-wrote a teacher training program. After one year of in-service training, changes are evident in the classrooms. According to a CUI administrator, “The teachers are doing more intentional teaching around language and literacy.” Post-training results from the ELLCO and teacher and parent surveys show positive gains. EDC will no longer be formally involved in CUI’s literacy initiative, but CUI is prepared to continue on its own to provide training and evaluate change in its educational program.

EDC is now taking the lessons it learned in the research collaboration with CUI and applying them to its work with another Head Start agency, Action for Boston Community Development (ABCD). EDC’s work with CUI in the area of language and literacy will be expanded in ABCD to include more classrooms and more instruments and will be conducted over a longer time period. There is no doubt that both CUI, the Head Start agency, and EDC, the research organization, benefited in ways that ultimately improved their organizational effectiveness and translated into improved services for Head Start children and their families.

References

Bronson, M. B. 1996. The Bronson Social and Task Skill Profile: Teacher version. Chestnut Hill, MA: Boston College.

Bronson, M. B. 1999. Self-Regulation Scale. Chestnut Hill, MA: Boston College.

Harms, T., R. M. Clifford, & D. Cryer. 1998 . Early Childhood Environment Rating Scale. Revised. New York: Teacher’s College Press.

Howard, C., C. J. Lewkowicz, & D.K. Dickinson. In press. Both sides now: An examination of the relationship between researchers and Head-Start practitioners. NHSA Dialog: A Research to Practice Journal for the Early Intervention Field.

Wagner, J. 1997. The unavoidable intervention of educational research: A framework for reconsidering researcher-practitioner cooperation. Educational Researcher 26 (7): 13-22.

Corinne Lewkowicz is a Research Consultant for Communities United, Inc. T: 781-736-7890; E: lewkowitz@aol.com. Stacy Dimino is Executive Director of Communities United. T: 781-736-7890; E: sdimino@communitiesunitedinc.org.