DAVID FETTERMAN: RESPONSE TO LEE SECHREST'S BOOK REVIEW
ENVIRONMENT AND BEHAVIOR
Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Edited by David M. Fetterman, Shakeh J. Kaftarian, & Abraham Wandersman. Thousand Oaks, CA: Sage, 1996, pp. xii + 411.
By way of background for the reader and preface to this response, Sechrest and I have been colleagues for many years. We have served on evaluation boards together and engaged in public dialogue and dispute (see Sechrest, 1991, and then Fetterman, 1992). We have also published together, highlighting the synergy associated with integrating qualitative and quantitative methods (Denis, Fetterman, & Sechrest 1994). We are familiar with each other's style and approach, and respectful enough to weather our methodological and theoretical disagreements.
In this vein, I'd like to start out by thanking the reviewer for stating that Empowerment Evaluation is of considerable merit and is "replete with excellent ideas about evaluation that should be of interest to, and taken seriously by, almost any program evaluator...." (p. 422). I particularly appreciate the statement that "Any reasonably thoughtful evaluator of whatever persuasion who reads Empowerment Evaluation will come away from it with increased awareness of the need to get program recipients directly involved in program planning and evaluation" (p. 422). If this message was clear even to critics of this process, I consider the work to be successful.
Numerous errors and misperceptions in Sechrest's (1997) review, however, need to be corrected to inform a meaningful scholarly inquiry and dialogue. This discussion highlights a few of these errors, in some cases providing verbatim quotations from the book as documentation.
EMPOWERMENT EVALUATION
AS A MOVEMENT Sechrest's (1997) comment that "Empowerment evaluation is a movement, not simply an 'approach:' or strategy," although based on his belief that empowerment evaluation is an ideology rather than a form of evaluation, does pay indirect tribute to the widespread interest in this new evaluation approach. Empowerment evaluation has been adopted by colleagues throughout the United States in foundations, such as the Kellogg Foundation; academe,
427
428 ENVIRONMENT AND BEHAVIOR /May 1997
including accreditation self-studies in higher education (Fetterman 1997); and government agencies, including the Environmental Protection Agency and the Health and Human Services' Center for Substance Abuse Prevention. It has also been adopted by colleagues in countries ranging from Brazil to South Africa.
It has gained, over the past several years, a solid presence in professional associations as well. The American Evaluation Association has a Collaborative, Participatory, and Empowerment Evaluation topical interest group with its own newsletter and Internet home page (http://www.stanford.edu/~davidf/empowermentevaluation.html) and mailing listserve. Empowerment evaluations have been well represented in evaluation, education, health, and anthropological association meetings.
Although I do not consider it a movement in the sense Sechrest suggests (1997), I do recognize the dedication and commitment of those colleagues conducting quality work in this area, and the intellectual excitement we share in pursuing this path together. This approach is not a complete product of an individual expression or conception; it is an ongoing collective effort to refine and further develop a form of self-evaluation in which traditional evaluation concepts and techniques are used to foster self-determination and program improvement, building capacity in the process. The frank assertion that empowerment evaluation was designed with the disenfranchised in mind seems particularly worrisome to Sechrest, who sees in this a rigid ideological agenda that establishes worthy and unworthy clients. At the same time, he notes that empowerment evaluation is open to abuse of all kinds. In the interests of rhetoric, Sechrest cites extreme examples from his own or his colleagues' experience, such as the business executive in a downsizing corporation, who wanted "to be empowered to fire [people] without remorse despite any of their demands for decency for pleas for mercy" (p. 422). All science is open to abuse--as are laws, religion, government, technology, medicine, education, and so on ad infinitum. Empowerment evaluation is no more -- and no less -- open to abuse than traditional evaluation approaches. As with all other advances, empowerment evaluation must apply appropriate standards of ethical conduct, such as the Guiding Principles for Evaluators (1994) and the Joint Committee's Program Evaluation Standards (see Fetterman 1995 for an application of the standards to empowerment evaluation), and work toward ensuring that these standards are maintained.
STAKEHOLDERS A particular concern emerging from Sechrest's review (1997) is that in addressing the needs of the disenfranchised, empowement evaluation ignores the needs of important stakeholders. He writes, "The book is written as if it were sufficient that program recipients be satisfied with what is happening
RESPONSE 429
to them whether those who provide the funds to make things happy are satisfied or not. No mention is made of what action is required if a program is satisfying to recipients because they like the immediate benefits it brings but is dissatisfying to sponsors because there is no evidence of any longer range benefit" (Sechrest, p. 423).
In fact, the chapters in this book present a clear model of stakeholder involvement throughout an evaluation and highlight the value of having sponsors and other external agencies involved at every stage. As the book notes, "The process of conducting an empowerment evaluation requires the appropriate involvement of stakeholders. The entire group -- not a single individual, not the external evaluator or an internal manager -- is responsible for conducting the evaluation." (Fetterman 1996, p. 22-23).
Empowerment evaluation is most meaningful and powerful when it embraces real-world constraints and realities. It often provides a bridge between the program and the sponsor or external agency. Although it is unfortunately more common to see program recipients ignored than program funders, any evaluation that ignores important stakeholders undermines its own credibility. Working in collaboration is richer and more rewarding than working in isolation. Empowerment evaluation attempts to involve all parties at the same table. Program participants expand their horizon concerning critical policy and accountability issues, and sponsors gain greater involvement and understanding from being part of the communication and interaction process in the project and/or program.
It is not uncommon in empowerment evaluation to find the funder literally sitting at the same table as program staff members (and in many cases program participants) when setting goals and objectives, discussing program processes, formulating the evaluation design, including credible documentation to test specific program goals, making mid-course corrections, and constructing and reviewing report drafts. This is true of my own work with Margret Dugan (1996), from the Empowerment Evaluation Institute, and the Marin Community Foundation. Similarly, Andrews provides an inclusive list of stakeholders, "the stakeholders in this effort include the service recipients, workers, and managers in the agencies, board members, and donors, including various sponsors such as churches, the United Way, and county, state, and federal government sources" (p. 151).
PROGRAM OUTCOME OR IMPACT Sechrest states erroneously that the book "makes virtually no mention of what 'traditional' evaluators would think of as program outcomes or impact" (Sechrest, p. 423). To the contrary, the book begins and ends with discussions of these topics. A brief list of examples is provided below to assure evaluators unfamiliar with empowerment evaluation that these issues
430 ENVIRONMENT AND BEHAVIOR / May 1997
are at the heart of many deliberations. I hope these examples will also entice readers to join the exciting and thoughtful exchange in this area.
Zimmerman (in press): "Empowered outcomes refer to operationalization of empowerment so we can study the consequences of citizen attempts to gain greater control in their community or the effects of interventions designed to empower participants" (p. 4).
Fetterman (1996): "Goals and outcomes are geared toward the appropriate developmental level of implementation. Extraordinary improvements are not expected of a project that will not be fully implemented until the following year. Similarly, seemingly small gains or improvements in programs at an embryonic stage are recognized and appreciated in relation to their stage of development. In a fully operational and mature program, moderate improvements or declining outcomes are viewed more critically" (p. 6).
Henry Levin (1996): "An Accelerated School requires that school staff, parents, and students take responsibility for the major decisions that will determine educational outcomes" (p 53); he continues to discuss governance outcomes later in his chapter (p. 61).
Ricardo Millett (1996) discusses "the need to reconcile evaluation research methodology with the need of the grantee to use outcome data to improve program implementation" (p. 70).
Gómez and Goldstein (1996): "[The] objectives of the initiative were to fund 6 to 10 HIV prevention programs for 18 months of operation, work with funded agencies over a 24-month period to evaluate the effectiveness of those programs in changing high-risk behaviors, and help disseminate the findings of those evaluations to providers and policymakers" (p. 105).
Grills and colleagues (1996) explain how "existing traditions and strategies become a foundation for current efforts in process, outcome, and impact evaluation (Watts, 1993)" (p. 130).
Andrews (1996): "Simply describing the processes through which services recipients and workers go and the expected outcomes may be an essential first step in promoting the capacity for evaluation. Several of the participating organizations have been unable to produce routinely such descriptive information. The monitoring of longitudinal change, ideally with comparative or control data, is desperately needed to shed light on how people in transitional programs achieve stability" (p. 156).
In addition, several contributors use the Prevention Plus III Model described by Linney and Wandersman in chapter 12, which explicitly discusses outcomes and impacts in very conventional terms (Step 1. Identify
RESPONSE 431
Goals and Desired Outcomes, Step 2. Process Assessment Worksheet , Step 3. Outcome Assessment Worksheet; p 266-271. (A long list of additional examples is available. )1
In fact, each contributor (all seasoned evaluators) grapples with these issues, in projects ranging from audit work with the Texas Office of the State Auditor to federal and local level empowerment evaluation with the Center for Substance Abuse Prevention.
Moreover, it is important to recognize that empowerment evaluation views the evaluation process differently than traditional evaluation.
The assessment of a program's value and worth is not the end point of the evaluation -- as it often is in traditional evaluation -- but part of an ongoing process of program improvement. This new context acknowledges a simple but often overlooked truth: that merit and worth are not static values. Populations shift, goals shift, knowledge about program practices and their value changes, and external forces are highly unstable. By internalizing and institutionalizing self-evaluation processes and practices, a dynamic and responsive approach to evaluation can be developed to accommodate these shifts. Both value assessments and corresponding plans for program improvement -- developed by the group with the assistance of a trained evaluator -- are subject to a cyclical process of reflection and self-evaluation. Program participants learn to continually assess their progress toward self-determined goals and to reshape their plans and strategies according to this assessment. In the process, self-determination is fostered, illumination generated, and liberation actualized." (p. 5-6).
TRADITIONAL EVALUATION
AND THE ACCELERATED SCHOOLS PROJECT Sechrest (1997) argues that empowerment evaluation "appears to be at least disdainful, if not actively hostile to, traditional evaluation," which he defines as "science and the scientific method" (p. 424). It is true that empowerment evaluation values and relationships differ significantly from many forms of traditional evaluation. Given the book's open statements that empowerment and traditional evaluation are not mutually exclusive (p. 6), and that many of the same tools used in traditional evaluation -- both qualitative and quantitative -- are transferable to empowerment evaluation, this is a puzzling stance.
In practice as well as in theory, empowerment evaluations often include traditional elements. In an empowerment evaluation as part of an accreditation self-study, external reviews were required by the group as part of the
432 ENVIRONMENT AND BEHAVIOR /May 1997
self-evaluation process. Another set of eyes were valued and specified in the design (Fetterman 1997). Levin's Accelerated Schools Project is another example combining empowerment and traditional elements, although Sechrest notes this as an oddity, and explains it by determining that "Fetterman appears to have designated the Accelerated Schools program as a model of empowerment evaluation even though it was not in any explicit way guided by the philosophy of empowerment evaluation" (Sechrest, p. 423-424). In fact, however, Levin (rather than Fetterman) states that "I believe that there is a high congruity between empowerment and empowerment evaluation in Accelerated Schools and the overall constructs and concepts of community empowerment developed by Fetterman, Rappaport, and Zimmerman" (p. 50). He notes in addition that
Evaluation is embedded in virtually every part of the Accelerated Schools process. By combining systematically the theme of school empowerment with a reflective and continuous process in which assessment and evaluation are firmly embedded, the Accelerated School appears to embody rather fully the concept of empowerment evaluation" (p. 63).
For specific examples of Levin's use of empowerment concepts, see Note 2.
INTELLECTUAL ISOLATION I have always enjoyed Sechrest's ability to spark controversy. However, certain of his phrases in this review serve instead to emphasize the dangers of intellectual isolation. Statements like "Nowhere is there even a suggestion that people might be 'empowered' to endure hardship" (Sechrest, 1997, p. 425) remind us just how out of touch we can become. More important than the obvious logical fallacy is the lack of real understanding about the problems inherent in many programs.
In this regard, Sechrest (1997) bemoans the use of pejorative modifiers--such as "rigid," "heavy-handed," "prescriptive," and "inflexible" -- associated with traditional evaluation, and states that it has been a long time since he has seen evaluations conducted in this fashion. In fact, these phrases have emerged from current or recent practice, from observations that participants have been largely ignored, told what to do or what is best for them, and made to feel that they are guinea pigs in social science. Although most evaluators responsibly work to understand and incorporate participants' perspectives into their findings, many participants in social service programs continue to see and experience these problems. Conrad (1994) provides a critical exami-
RESPONSE 433
nation of the role of experiments in field settings. He offers an insightful perspective from colleagues who use experimental design in their work, which makes the modifiers used in some Empowerment Evaluation chapters pale by comparison.
Science and evaluation do not have an unblemished reputation in many communities, particularly in traditionally disenfranchised communities. We would do well to hear the message underlying these modifiers, but we should not recast that message into a blanket statement of disdain for traditional evaluation. Let us focus our concern on the message, rather than targeting the messenger.
CONCLUSION Sechrest (1997) states that "Empowerment Evaluation has some work ahead of it" (Sechrest, p. 426). I agree wholeheartedly: It is an evolving entity. We have learned a great deal about what works and what does not work, and we are beginning to make finer distinctions between types of approaches within this domain. Discussion, debate, and critique move the process of understanding forward.
-- David Fetterman
Stanford University
NOTES 1. See also Fawcett and colleagues comments concerning monitoring process and intermediate and ultimate outcomes (p. 174); Yin, Kaftarian, and Jacobs' discussion about empowerment outcomes including action outcomes, validation outcomes, and empowerment impact (p. 201-203); Dugan's discussion concerning how the process and outcome data were analyzed collectively (p. 285); Mayer's comments concerning outlines for reporting outcomes (p. 338), and a list of worksheets focusing on outcomes (p. 340-370).
2. "All staff members, parent representatives, and student representatives are expected to participate in decisions. The school initiates its empowerment by taking stock of its resources, activities, students, community, and other dimensions (Hopfenberg et al., 1993, pp. 60-73). All members of the school community are involved in taking-stock activities, from identifying which dimensions of the school to investigate to setting out the questions that must be answered and the methods for answering them. Data are gathered using these methods and analyzed by taking-stock committees that are devoted to particular school dimensions. As the "school is the center of expertise," it must immediately engage itself in developing expertise about itself through setting out research questions and answering them through systematic inquiry. Analysis of the data
434 ENVIRONMENT AND BEHAVIOR / May 1997
and reflection provide contributions to a schoolwide report to which everyone contributes. Especially central is the search for school and community strengths and resources as well as challenges. The school will ultimately build on these strengths to overcome its challenges" (p. 57).
"The very launch of an Accelerated School immerses the school in evaluation through taking stock. Taking stock is largely a summative evaluation activity with its emphasis on working collectively to establish a baseline for the school. Much of taking stock is descriptive, summarizing the community, programs, facilities, and achievements and seeking to find both strengths and challenges. The empowering aspect of this activity consists of the inclusion of all staff, and parent and student participation, in this initial evaluation as well as its emphasis on the participants constructing the questions that they will ask and the methods by which they will answer the questions. Although guidance is provided by a coach through questioning strategies, the actual evaluative decisions are made by the participants and are summarized for the benefit of the entire school community" (p. 59-60).
"Constructing a collective dream for the school is an evaluation activity that requires drawing upon and combining the perspectives of students, staff, and parents as well as setting out criteria for assessing whether the dream is becoming a reality" (p. 60).
"Long before the school reaches its vision, it must participate in the Accelerated School transformation process. Periods for critical reflection are built into the Accelerated School process in which each school must evaluate whether it is following the process by asking crucial questions. For example, has the process of taking stock engaged all participants?" (p. 61).
"An Accelerated School strives to achieve powerful learning and accelerated outcomes for all of its students by creating a school that makes and implements decisions toward that end" (pg. 62).
"All of these activities require assessment and evaluation. In short, evaluation is embedded in virtually every part of the Accelerated Schools process. By combining systematically the theme of school empowerment with a reflective and continuous process in which assessment and evaluation are firmly embedded, the Accelerated School appears to embody rather fully the concept of empowerment evaluation" (p.63).
REFERENCES Andrews, A.B. (1996). Realizing Participant Empowerment in the Evaluation of Nonprofit Women's Services Organizations: Notes From the Front Line. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Conrad, K.J. (1994). Critically Evaluating the Role of Experiments. New Directions for Program Evaluation. No. 63. San Francisco, CA: Jossey-Bass.
Denis, M.L., Fetterman, D.M., and Sechrest, L. (1994). Integrating Qualitative and Quantitative Evaluation Methods in Substance Abuse Research. Evaluation and Program Planning, 17(4):419-427.
RESPONSE 435
Dugan, M.A. (1996). Participatory and Empowerment Evaluation: Lessons Learned in Training and Technical Assistance. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J.A., Richter, K.P., Lweis, R.K., Harris, K.J., Williams, E.L., Berkley, J.Y., Lopez, C.M., and Fisher, J.L. (1996). Empowering Community Health Initiatives Through Empowerment. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Fetterman, D.M. (1992). In Response to Sechrest Sechrest's 1991 AEA Presidential Address: "Roots: Back to our First Generations," Evaluation Practice, 13(3):171-172.
Fetterman, D.M. (1995). In Response to Dr. Daniel Stufflebeam's: Empowerment Evaluation, Objectivist Evaluation, and Evaluation Standards: Where the Future of Evaluation Should Not Go and Where It Needs to Go. Evaluation Practice, 16(2):179-199.
Fetterman, D.M. (1996). Empowerment Evaluation: An Introduction to Theory and Practice. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Fetterman, D.M. (1997). Empowerment Evaluation and Accreditation in Higher Education. In Chelimsky, E. and Shadish, W.R. (eds.) Evaluation for the 21st Century: A Resource Book. Thousand Oaks, CA: Sage.
Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Gómez, C.A. and Goldstein, E. (1996). The HIV Prevention Evaluation Initiative: A Model for Collaborative and Empowerment Evaluation. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Grills, C.N., Bass, K. Brown, D.L., and Akers, A. (1996). Empowerment Evaluation: Building Upon a Tradition of Activism in the African American Community. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Guiding Principles for Evaluators. (1994). American Evaluation Association. As reprinted in Shadish, W.R., Newman, D.L., Scheirer, M.A., and Wye, C. (1995). Guiding Principles for Evaluators. New Directions for Program Evaluation. No. 66. San Francisco, CA: Jossey-Bass.
Hopfenberg, W., Levin, H.M., Chase, C. Christensen, S.G., Moore, M., Soler, P., Brunner, I., Keller, BN., and Rodriguez, G. (1993). The Accelerated Schools resource guide. San Francisco, CA: Jossey-Bass.
Keller, J. (1996). Empowerment Evaluation and State Government: Moving From Resistance to Adoption. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
436 ENVIRONMENT AND BEHAVIOR / May 1997
Levin, H.M. (1996). Empowerment Evaluation and Accelerated Schools. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Linney, J.A. and Wandersman, A. (1996). Empowering Community Groups With Evaluation Skills: The Prevention Plus III Model. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Mayer, S.E. (1996). Building Community Capacity With Evaluation Activities That Empower. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Millett, R.A. (1996). Empowerment Evaluation and the W.K. Kellogg Foundation. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Sechrest, L. (1991). Roots: Back to our First Generations," Evaluation Practice, 13(1):1-7.
Watts, R.J. (1993). "Resident research" and community psychology. American Journal of Community Psychology, 21(4), 483-486.
Yin, R.K., Kaftarian, S.J., and Jacobs, N.F. (1996). Empowerment Evaluation at Federal and Local Levels: Dealing With Quality. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
Zimmerman, M.A. (in press) Empowerment theory: Psychological, organizational, and community levels of analysis. In J. Rappaport & E. Seldman (Eds.), Handbook of community psychology. New York: Plenum. As cited in Fetterman, D.M. Empowerment Evaluation: An Introduction to Theory and Practice. In Fetterman, D.M., Kaftarian, S.J., Wandersman, A. (1996). Empowerment Evaluation: Knowledge and Tools for Self-assessment and Accountability. Thousand Oaks, CA: Sage.
ENVIRONMENT AND BEHAVIOR, Vol. 29. No. 3, May 1997 427-436
© 1997 Sage Publications, Inc.