I believe strongly that the systems field can contribute strongly to the development of the evaluation field.
How so ?
For me, systems concepts provide me with very powerful ways of exploring inter-relationships, perspectives and boundaries. These are important issues within evaluation.
Inter-relationships are the key to understanding how programs behave.
Perspectives provide insight into motivations and thus how people behave.
Boundaries determine who wins and who loses from an intervention, or what is "in" and what is "out" of an assessment of that intervention.
In other words boundaries indicate judgements of value or worth.
More than evaluation, the systems field has thought deeply about these three concepts and come up with approaches that can transform the way in which evaluation does its job.
The material in this section indicates what is possible and how people have used systems concepts in evaluation.
I'm indebted to many people in developing this work. They are far too many to list here and all played vital roles. I stand on the shoulder of giants. What follows is as much to do with them as me - although all the mistakes are mine. Much of the other material emerged from workshops for local and national evaluation associations, various multilateral agencies and way way back the W.K. Kellogg Foundation and I thank them for their support.
SYSTEMS DIAGRAMS - A PRACTICAL GUIDE
The past few years has seen increased use of system diagrams in evaluation and by evaluation clients. System diagrams provide an excellent way of clarifying complex situations and help to address identified problems. Potential users are not spoilt for choice - there are literally dozens of different diagram-based approaches.
On the other hand, there are not many easily accessible publications that describe how to match and show how to use appropriate diagrams to particular tasks. That's why System Diagrams; A Practical Guide came to be written.
SYSTEM DIAGRAMS; A PRACTICAL GUIDE ebook covers six widely used diagram approaches that between them cover a range of potential uses by evaluators;
Rich Picturing. Useful when trying to get to grips with the complexity of a situation and work out which aspects you should be focusing on.
Influence Diagrams. Useful when you want to have a snapshot of what is influencing a particular issue or topic at a particular moment in time.
Causal Diagrams. Useful when you want to explore the possible consequences of specific causal relationships within a particular issue or topic over a period of time.
Cynefin. A diagram based approach that helps you to think about and respond to a particular situation, or intervention, through five different framings; simple, complicated, complex, chaotic and confused.
Viable System Model (VSM). A diagram based approach that helps to assess and ensure the viability of a purposeful activity, given the range of
competing demands from different components of that endeavour.
Cultural Historical Activity System Model (CHAT). A diagram based approach that explores a series of factors that enable people with different motivations to work together creatively and effectively, especially when responding to changing and challenging events.
The discussion of each approach is accompanied by a worked example and a set of questions that that the diagram addresses. There are also chapters on the role of diagrams in general; what constitutes a systemic inquiry and the overall nature of system diagrams.
The ebook is free with an optional donation to cover production costs.
It can be downloaded in pdf and ebook formats from http://gum.co/systemdiagrams
WICKED SOLUTIONS : A SYSTEMS APPROACH TO COMPLEX PROBLEMS
ThisERRbook, written with Sjon van ‘t Hof, is about the use of core systems ideas in dealing with wicked situations. Wicked situations are those where identifying problems is not easy and selecting good solutions is even more difficult. Many societal, business and development challenges are in fact wicked problems.
Using three basic systems concepts – inter-relationships, perspectives, and boundaries - this book will help you :
■unpick the tangle of issues that need addressing
■design suitable ways of tackling those issues
■deal with some tricky aspects of working in wicked situations
■find more information about systems methods and managing interventions systemically.
A free, 24-page partial preview can be downloaded from here
You can get an ebook version here
And a physical book here
OBTAIN A COPY HERE
A video introduction can be viewed here
Evaluation is about making judgments of worth.
Design is a process that narrows things down from an infinite number of judgments of worth, to those that really matter. It decides 'matters to whom' and in what circumstances. It decides what methods we have to use to know who benefits and who doesn't.
The interventions that evaluation investigates are complex, with many different perspectives on what is worthwhile and what is not. Evaluations are about values. Evaluation treads on people's dreams. So these are not easy decisions.
Evaluators have an obligation to be well informed; to understand the situation, the intervention, its context and to understand the evaluation and the likely consequences. This is both an evaluation design issue and also an evaluation management issue.
Systems thinking has a long history in design and management of interventions like evaluation.
In particular systems concepts help you;
· determine the purposes of both the intervention to be evaluated and the evaluation
· understand the potential and desirable uses, influence and consequences of the evaluation
· elicit the criteria by which evaluative judgements of merit, value or worth
· identify the key evaluation questions that will enable these judgments
The workbook Systemic Evaluation Design brings that experience to evaluation.
Systemic Evaluation Design is a workbook that takes you through the design process from soup to nuts. It helps you understand more deeply both the intervention and the evaluation. It also recognises that 'design' is a process that occurs throughout the evaluation and beyond. It's not just something you do at the beginning. So it is about effective evaluation management as much as one-off design
It's a workbook not a text book, so specifically designed to help you design your evaluation.
In English it is available as an ebook here
In Spanish, thanks to the work of Alain Santandreu of ECOSAD, and Brenda Bucheli of Pontificia Universidad Católica del Perú (PUCP) and EvalPeru, you can get a free copy of the first edition from here. ECOSAD also sell a beautifully produced hardcopy version.
The Research Institute for Humanity and Nature (RIHN) in Kyoto, Japan is a remarkable institution dedicated to the transdisciplinarly study and resolution of problems resulting from humanity's interaction with nature. On the ground floor it has all the fancy scientific equipment you could imagine, and on the floor above multi-disciplinary teams working on complex socio-political-scientific issues.
For each project, the teams have a year to come up with a design. If that design is considered adequate then they have three years to implement that design. So clearly design is an important element of their work - but how to you design an intervention in highly complex environments using multiple knowledge systems (scientific, indigenous, political)?
I've had a working relationship with RIHN for a while and given my interest in design ran a two day workshop based on Wicked Solutions and emerging ideas that became Using Systems Concepts in Evaluation Design (although in this case oriented towards research design). Participants worked on their own projects throughout the two days.
The workshop was recorded and brilliantly edited into two separate videos; one exploring the basic principles and the other covering the more detailed aspects of research design using systems ideas. The two videos ended up as one of iTunes top ten scientific videos for 2016, which is kinda cool.
You can view the two videos on the following YouTube channels
https://www.youtube.com/watch?v=lFcWhGE7moQ
https://www.youtube.com/watch?v=5RRHpXl2hrw
SYSTEMS CONCEPTS IN ACTION : A PRACTITIONER'S TOOLKIT
The book by Richard Hummelbrunner and myself is focused on the practical use of systems ideas. It describes 19 commonly used systems approaches, complete with case studies, variations and discussion of each approach's pros and cons. Each chapter begins with a set of questions that the particular method addresses.
To the best of our knowledge no other book comprehensively explores the practical side of such a large range of systems methods.
You can find it on Amazon, or get it direction in various formats from Stanford University Press here
The methods are :
Causal Loop Diagrams
System Dynamics
Social Network Analysis
Outcome Mapping
Process Monitoring of Impacts
Strategic Assumption Surfacing and Testing
Strategic Area Assessment
The CDE Model
Assumption-Based Planning
Cynefin
Solution Focus
Viable System Model
Cultural Historical Activity Theory
Soft Systems Methodology
Dialectical Methods of Inquiry
Scenario Technique
Systemic Questioning
Circular Dialogues
Critical Systems Heuristics
I provide training and consultancy support in the use of systems concepts in evaluation. This includes the material below, workshops, evaluation design, and one-on-one mentoring and advice. Contact me for more details about what I can do for you and your work.
