Catálogo de publicaciones - revistas

<jats:p> The evaluation field has advanced sufficiently in its methodology and public service that evaluators can and should subject their evaluations to systematic metaevaluation. Metaevaluation is the process of delineating, obtaining, and applying descriptive information and judgmental information about an evaluation’s utility, feasibility, propriety, and accuracy and its systematic nature, competence, integrity/honesty, respectfulness, and social responsibility to guide the evaluation and publicly report its strengths and weaknesses. Formative metaevaluations—employed in undertaking and conducting evaluations—assist evaluators to plan, conduct, improve, interpret, and report their evaluation studies. Summative metaevaluations—conducted following an evaluation—help audiences see an evaluation’s strengths and weaknesses, and judge its merit and worth. Metaevaluations are in public, professional, and institutional interests to assure that evaluations provide sound findings and conclusions; that evaluation practices continue to improve; and that institutions administer efficient, effective evaluation systems. Professional evaluators are increasingly taking their metaevaluation responsibilities seriously but need additional tools and procedures to apply their standards and principles of good evaluation practice. </jats:p>

<jats:p> Formative evaluation provides stakeholders with timely feedback to support an intervention's improvement during implementation to maximize its effectiveness. We describe the qualitative methods that guided one study within a formative evaluation of a multicomponent care delivery intervention. We then describe the challenges and lessons learned that emerged from this study, organizing them by the study's four overarching challenges: (1) addressing multiple research questions, (2) working with a large interdisciplinary team, (3) triangulating qualitative results with quantitative results, and (4) studying implementation in real-world delivery settings. Overall, the evaluation generated important findings to support improvement of the intervention during implementation. We hope that sharing the lessons learned will increase the rigor and efficiency with which formative evaluations of complex care delivery interventions are conducted and the likelihood that they will improve implementation in real time. We also hope the lessons learned will enhance the satisfaction of the researchers working on these evaluations. </jats:p>

<jats:p> As part of its Guiding Principles for Evaluators, the American Evaluation Association (AEA) requires that evaluators develop cultural competencies. Using a successive-independent-samples design, the researchers sought to compare perceptions of cultural competence across a duration of 10 years. Qualitative data were collected via online surveying, which included 168 program evaluators in 2009 and 110 program evaluators in 2019. Content analysis was utilized, and content categories were identified and quantified for both data collections. The data reflect that, from 2009 to 2019, there has been an increased recognition of what cultural competence entails and a closer alignment between what the Guiding Principles for Evaluators promotes and what evaluators demonstrate. However, the data also indicate that perhaps preferences have evolved past the current cultural competence paradigm as well as the term “cultural competence” itself. These findings and implications are discussed in further detail. </jats:p>

<jats:p> This paper unpacks our efforts as external evaluators to work toward decolonizing our evaluation practice. Undertaking this writing exercise as a form of reflective practice demonstrated that decolonization is much more complex than simply translating materials, organizing locals to collect data, and building participants’ capacity around Western modalities. While this complexity is clear in the decolonization literature, practice-based examples that depict barriers and thought processes are rarely presented. Through this paper, we deconstruct our deeply held beliefs around what constitutes good evaluation to assess the effectiveness of our decolonizing approach. Through sharing our critical consciousness-raising dialoguing, this paper reports our progress thus far and provides information and provocations to support others attempting to decolonize their practice. </jats:p>

<jats:p> The merit of narrative film methods to support participatory approaches and professional development has been increasingly demonstrated by research in several fields and education. However, the use of digital storytelling and other film methods in evaluation remains largely unchartered territory. This article provides a case study of a digital storytelling evaluation initiative in monitoring and evaluation (M&amp;E) in an Australian community not-for-profit. The aim is to offer practical insights for evaluators and organizations considering digital storytelling and other film narrative methods for participant-centered evaluation. Embedding digital evaluation stories into M&amp;E evolved through collaboration between the external evaluation team and organizational leadership, requiring capacity building in evaluation, digital and qualitative methods, and new systems and processes. Benefits include transformation into a participant-centered evaluation and learning culture. Several challenges are discussed, including the extent of organizational change required, the associated time, energy, and cost, and the positive bias of visual narratives. </jats:p>

<jats:p> There is a lack of instructional literature on how to conduct a feminist evaluation to highlight and transform systemic issues in gendered and intersecting power relations. Feminist Participatory Action Research (FPAR) enables a process for conducting community-driven, -led and -owned feminist evaluations that drive social justice actions. By undertaking a critical review of existing literature, this article presents guiding principles and practices in how to conduct a feminist evaluation using FPAR. These principles and practices provide a framework for those who are seeking an evidence base for transformative social justice action in communities, particularly those who are working with complexity in systems-change interventions with multiple stakeholders. </jats:p>

<jats:p> According to Jennifer Billman, western evaluation bias against indigenous thinking is due to ontological incompetence. If so, the solution she offers (a highly abstract list of criteria) is inadequate since it fails to address let alone resolve a wide range of philosophical dilemmas at the intersection of logic and ontology. Furthermore, it fails to “frame evaluation in reality” since it ignores the patent fact that, in the market society, positivist evaluators dominate. They are value free, embrace a “clockwork” conception of the natural and social world, and do not question decision makers' goals. By contrast, constructivist evaluators recognize that social facts differ from natural facts since they are socially constructed and clustered within institutions that define roles, norms and expectations. It follows that constructivist evaluation holds the key to the problem identified by Billman since it resists capture by vested interests, gives pride of place to the relational context and embraces the validity of indigenous thinking. </jats:p>

<jats:p> This study challenges persistent misrepresentations of evaluation as a value-neutral inquiry process by presenting an empirical study that deepens understanding of evaluators’ values and how they “show up” in evaluation practice. Through semistructured interviews and inductive analysis, we examined the values advanced by a sample of eight experienced evaluators. We surfaced and examined 12 values, which we organized into five clusters, that shaped the constitutive elements of the studies these evaluators conducted and guided how the evaluators positioned their work. Our findings provide empirical evidence about the role of values in evaluation practice and can support evaluators in reflecting on their own values and enacting their professional and ethical responsibilities to identify and articulate their values in the context of evaluation practice. </jats:p>