Systems Concepts and Tools

For me, systems concepts provide me with very powerful ways of exploring inter-relationships, perspectives and boundaries. These are important issues within evaluation.

Inter-relationships are the key to understanding how programs behave.

Perspectives provide insight into motivations and thus how people behave.

Boundaries determine who wins and who loses from an intervention, or what is "in" and what is "out" of an assessment of that intervention.

In other words boundaries indicate judgements of value or worth.

More than evaluation, the systems field has thought deeply about these three concepts and come up with approaches that can transform the way in which evaluation does its job.

The material in this section indicates what is possible and how people have used systems concepts in evaluation.

I'm indebted to many people in developing this work. They are far too many to list here and all played vital roles. I stand on the shoulder of giants. What follows is as much to do with them as me - although all the mistakes are mine. Much of the other material emerged from workshops for local and national evaluation associations, various multilateral agencies and way way back the W.K. Kellogg Foundation and I thank them for their support.

Values in Action: A Critical Systems Heuristics Workbook

Values in Action workbook cover

Critical Systems Heuristics is one of the most useful systems approaches for any intervention, strategy, plan, policy or evaluation. It is by far the most influential systems approach on my own practice. It is based on a series of premises:

That all endeavours set boundaries, because you cannot do everything. Whatever you do excludes or marginalises something or somebody.

These boundary decisions have both ethical and practical implications. Because of this, you have a duty to consider the impact of those decisions.

Those impacts include those involved in whatever you are doing and those affected but not involved. Especially those who feel marginalised or negatively affected by your boundary decisions.

There are four main boundary decision areas:

Purpose: Who or what benefits

Control: Who or what decides

Expertise: Who or what informs

Legitimacy: Who or what allows your actions

Each of these four areas contains three heuristics: generic learning questions that are adapted to the situation.

These twelve heuristics are asked in two modes: an observation mode (is, was, will be) and a normative mode (should, ought, could).

These basic components appear relatively simple and obvious, but as anyone well versed in Critical Systems Heuristics will tell you, they collectively and progressively unlock profound discussions and deep insights into your endeavour. They reveal and challenge your and others' values, beliefs and assumptions, and challenge you to explore them critically.

By far the best way to learn how to use Critical Systems Heuristics is to use it. And that is why I wrote Values in Action as a workbook and not a textbook. It contains exercises, case examples and tasks that guide you through a learning journey. The workbook is donationware; pay nothing or what it is worth to you. Your choice. There is also a 30 minute video that explains the approach.

https://gum.co/valuesinaction

Video https://youtu.be/87pD2gjrcUs

Systems Diagrams: A Practical Guide

system diagrams

The past few years has seen increased use of system diagrams in evaluation and by evaluation clients. System diagrams provide an excellent way of clarifying complex situations and help to address identified problems. Potential users are not spoilt for choice - there are literally dozens of different diagram-based approaches.

On the other hand, there are not many easily accessible publications that describe how to match and show how to use appropriate diagrams to particular tasks. That's why System Diagrams; A Practical Guide came to be written.

SYSTEM DIAGRAMS; A PRACTICAL GUIDE ebook covers six widely used diagram approaches that between them cover a range of potential uses by evaluators.

Rich Picturing. Useful when trying to get to grips with the complexity of a situation and work out which aspects you should be focusing on.

Influence Diagrams. Useful when you want to have a snapshot of what is influencing a particular issue or topic at a particular moment in time.

Causal Diagrams. Useful when you want to explore the possible consequences of specific causal relationships within a particular issue or topic over a period of time.

Cynefin. A diagram based approach that helps you to think about and respond to a particular situation, or intervention, through five different framings: simple, complicated, complex, chaotic and confused.

Viable System Model (VSM). A diagram based approach that helps to assess and ensure the viability of a purposeful activity, given the range of competing demands from different components of that endeavour.

Cultural Historical Activity System Model (CHAT). A diagram based approach that explores a series of factors that enable people with different motivations to work together creatively and effectively, especially when responding to changing and challenging events.

The discussion of each approach is accompanied by a worked example and a set of questions that the diagram addresses. There are also chapters on the role of diagrams in general, what constitutes a systemic inquiry and the overall nature of system diagrams.

The ebook is free with an optional donation to cover production costs.

It can be downloaded from here https://gum.co/systemdiagrams

Wicked Solutions: A Systems Approach to Complex Problems

Wicked solutions

This book, written with Sjon van ‘t Hof, is about the use of core systems ideas in dealing with wicked situations. Wicked situations are those where identifying problems is not easy and selecting good solutions is even more difficult. Many societal, business and development challenges are in fact wicked problems.

Using three basic systems concepts – inter-relationships, perspectives, and boundaries – this book will help you:

  • unpick the tangle of issues that need addressing
  • design suitable ways of tackling those issues
  • deal with some tricky aspects of working in wicked situations
  • find more information about systems methods and managing interventions systemically.

Get a Copy

A free, 24-page partial preview can be downloaded from ewExternalFiles/Wicked.pdf

You can get an ebook version at https://bobwilliams.gumroad.com/l/wicked

And a physical book at http://bit.ly/1SVoOH3

Systems workshop videos

Systems workshops

The Research Institute for Humanity and Nature (RIHN) in Kyoto, Japan is a remarkable institution dedicated to the transdisciplinarly study and resolution of problems resulting from humanity's interaction with nature. On the ground floor it has all the fancy scientific equipment you could imagine, and on the floor above multi-disciplinary teams working on complex socio-political-scientific issues.

For each project, the teams have a year to come up with a design. If that design is considered adequate then they have three years to implement that design. So clearly design is an important element of their work - but how do you design an intervention in highly complex environments using multiple knowledge systems (scientific, indigenous, political)?

I've had a working relationship with RIHN for a while and given my interest in design ran a two day workshop based on Wicked Solutions and emerging ideas that became Using Systems Concepts in Evaluation Design (although in this case oriented towards research design). Participants worked on their own projects throughout the two days.

The workshop was recorded and brilliantly edited into two separate videos; one exploring the basic principles and the other covering the more detailed aspects of research design using systems ideas. The two videos ended up as one of iTunes top ten scientific videos for 2016, which is kinda cool.

Watch the Videos

You can view the two videos on the following YouTube channels:

https://www.youtube.com/watch?v=lFcWhGE7moQ
https://www.youtube.com/watch?v=5RRHpXl2hrw

Systemic Evaluation Design

Systemic Evaluation Design

Evaluation is about making judgments of worth.

Design is a process that narrows things down from an infinite number of judgments of worth, to those that really matter. It decides “matters to whom” and in what circumstances. It decides what methods we have to use to know who benefits and who doesn't.

The interventions that evaluation investigates are complex, with many different perspectives on what is worthwhile and what is not. Evaluations are about values. Evaluation treads on people's dreams. So these are not easy decisions.

Evaluators have an obligation to be well informed; to understand the situation, the intervention, its context and to understand the evaluation and the likely consequences. This is both an evaluation design issue and also an evaluation management issue.

Systems thinking has a long history in design and management of interventions like evaluation.

In particular, systems concepts help you:

  • determine the purposes of both the intervention to be evaluated and the evaluation
  • understand the potential and desirable uses, influence and consequences of the evaluation
  • elicit the criteria by which evaluative judgements of merit, value or worth
  • identify the key evaluation questions that will enable these judgments

The workbook Systemic Evaluation Design brings that experience to evaluation.

Systemic Evaluation Design is a workbook that takes you through the design process from soup to nuts. It helps you understand more deeply both the intervention and the evaluation. It also recognises that “design” is a process that occurs throughout the evaluation and beyond. It's not just something you do at the beginning. So it is about effective evaluation management as much as one-off design.

It's a workbook not a text book, so specifically designed to help you design your evaluation.

Get a Copy

In English you can download the ebook at https://gumroad.com/l/evaldesign

In Spanish, thanks to the work of Alain Santandreu of ECOSAD, and Brenda Bucheli of Pontificia Universidad Católica del Perú (PUCP) and EvalPeru, you can get a free copy of the first edition at:

http://www.ecosad.org/images/Libro-Bob-Williams.pdf

ECOSAD also sell a beautifully produced hardcopy version.

Systems Concepts in Action: A Practitioner's Handbook

Systemic Evaluation Design

The book by Richard Hummelbrunner and myself is focused on the practical use of systems ideas. It describes 19 commonly used systems approaches, complete with case studies, variations and discussion of each approach's pros and cons. Each chapter begins with a set of questions that the particular method addresses.

To the best of our knowledge no other book comprehensively explores the practical side of such a large range of systems methods.

You can find it on Amazon, or get it directly in various formats from Stanford University Press:

https://www.sup.org/books/business/systems-concepts-action

The 19 Methods Covered

  • Causal Loop Diagrams
  • System Dynamics
  • Social Network Analysis
  • Outcome Mapping
  • Process Monitoring of Impacts
  • Strategic Assumption Surfacing and Testing
  • Strategic Area Assessment
  • The CDE Model
  • Assumption-Based Planning
  • Cynefin
  • Solution Focus
  • Viable System Model
  • Cultural Historical Activity Theory
  • Soft Systems Methodology
  • Dialectical Methods of Inquiry
  • Scenario Techniques
  • Systemic Questioning
  • Circular Dialogues
  • Critical Systems Heuristics

Systems Concepts in Evaluation

Systems Concepts in Evaluation

A special monograph published by the American Evaluation Association and funded by the Kellogg Foundation. It was one of the first publications to explore the relationship between the two fields. After a key introduction by Professor Gerald Midgley, each chapter focuses on the use of a particular systems approach to a described evaluation example.

This issue explores how systems thinking can enhance evaluation practice and includes contributions from leading practitioners in the field.

The articles examine the application of various systems approaches to evaluation challenges, demonstrating practical applications of systems concepts in real-world evaluation contexts.

You can download the PDF of this monograph here:

https://www.bobwilliams.co.nz/ewExternalFiles/system concepts in evaluation.pdf

Beyond Logframe: Using Systems Concepts in Evaluation

Systems Concepts in Evaluation

In December 2009, Patricia Rogers and I held a series of workshops for the Foundation for Advanced Studies in International Development (FASID) in Japan. FASID researches the effectiveness of Japanese international development projects. This publication emerged out of these workshops and reflects the state of my current thinking about the role of systems ideas in evaluation. The publication contains three key articles.

The first is a paper by Richard Hummelbrunner that looks at systemic alternatives to LogFrame. Richard is currently helping the German international development agency (GTZ) replace LogFrame as their key management and evaluation tool.

The second is an expanded version of the Capacity.org paper mentioned above. It goes into much greater detail about the specific contribution of systems methods to capacity development.

The third is a "constructed" conversation between FASID staff, Patricia Rogers and myself about the potential for systems ideas in evaluation, and some of our experience (both good and bad) in applying them. I especially like this piece because it is more reflective than promotional.

You can download the PDF here:

https://www.bobwilliams.co.nz/ewExternalFiles/Beyond%20Logframe.pdf

Prosaic or Profound: The Adoption of Systems Ideas by Impact Evaluation

One thing that frustrates me above everything else is when people use words from the systems field without actually using the ideas that underpin those words. When you start getting in touch with the ideas, everything changes. You just cannot deal with the world in the same way; the consequences are profound rather than prosaic. On the whole, evaluation has grabbed the words and not really confronted the reality of applying the ideas. Partly this is because the use of systems ideas challenge and undermine the way in which evaluation is commonly practiced. A few years ago I was asked to give a keynote speech at an evaluation conference where I explored what I felt were the potential consequences for evaluation practice of the application of systems ideas. The presentation was later published in a modified form by the Bulletin of IDS (the Institute for Development Studies at the University of Sussex in the UK).

I need to say that the paper was hated by the Bulletin reviewers. To their credit the editors allowed the paper to be published with relatively minor changes. Ironically, it is one of the most downloaded files on this website.

https://www.bobwilliams.co.nz/ewExternalFiles/Prosaic or Profound.pdf

Using Systems Concepts to Navigate Complexity

During the development of the Systems Concepts in Evaluation publication, most of the authors got together in Berkeley, California to discuss what everyone had in common. The idea was to write the opening chapter, but things got a bit bigger than that. In the end we came to the conclusion that despite the huge diversity of methods, methodologies and notions floating around the systems field, three things glued us all together: a deep interest and orientation around inter-relationships, perspectives and boundaries.

This document provides an introduction to those ideas and links them to various systems methods. Since evaluators are primarily interested in questions, I've linked each element and method to a set of evaluation style questions. So if you are attracted to Question X, use systems approach Y.

This particular document was prepared for a symposium – Navigating Complexity – organised by Wageningen University in the Netherlands. My thanks to Jim Woodhill and Irene Guijt for making that possible.

Various writings spun off this text including an article in Capacity.org called "Thinking Systemically" that is, in my view, a useful companion piece.

You can download the PDF here:

https://www.bobwilliams.co.nz/ewExternalFiles/Systems_Intro.pdf

SYSTEMS METHODOLOGIES, METHODS AND TOOLS

The following table provides a quick reference to various systems methodologies, methods, and tools. Each has its particular strengths and is suited to different types of evaluation and inquiry challenges.

Approach Best Used For Key Features
Soft Systems Methodology (SSM) Understanding complex problem situations with multiple stakeholders Rich pictures, root definitions, conceptual models, comparison with reality
Critical Systems Heuristics Surfacing and challenging boundary judgments; examining whose interests are served Boundary questions, stakeholder analysis, examining assumptions
Viable System Model Assessing organizational viability and governance Five systems framework, recursive structure, variety engineering
CHAT (Cultural-Historical Activity Theory) Understanding activity systems and contradictions driving change Activity triangles, contradictions, expansive learning
System Dynamics Understanding feedback loops and dynamic behavior over time Stock and flow diagrams, causal loop diagrams, simulation modeling
Complex Adaptive Systems Modelling emergent behaviours Agent Based Modelling, CDE Model
Cynefin Framework Sense-making and determining appropriate responses to situations Five domains: simple, complicated, complex, chaotic, confused

SYSTEMS METHODS AND APPROACHES DOWNLOADS

Soft Systems Methodology

Systems Concepts in Evaluation

Peter Checkland's Soft Systems Methodology (SSM) is one of the most important (and possibly most misunderstood) systems approaches. Even Checkland himself acknowledged that it took him the best part of twenty years to discover its most powerful role as a learning tool rather than a modelling or planning tool. That learning primarily depends on a core underlying proposition; in order to improve the real world, you have to forget about it for a while.

It turns out that this is easier said that done. However, the seven steps of the classic SSM smooths the path greatly. It does it so well that it is easy to forget the proposition along the way. Don't, just don't, because you really miss the benefits of SSM.

Soft Systems Methodology was my introduction to the systems field and is still very influential in my practice. Over they years I've written extensively about SSM, but until recently had never developed a specific document that explores it in depth. A few months ago, a colleague who now runs a permaculture farm floated the idea of producing a guide to SSM for those in the permaculture field, This is the result and includes both a description of the methodology, and an excellent case example that really demontrates the usefulness of SSM.

I've also posted an alternative workshop form developed by Bob Dick, that provides an insightful and refreshing way of understanding SSM. I've altered it based on my own experience. This workshop is a lot of fun and a very useful way for people to learn about Soft Systems Methodology. It is best used on a real life issues, which further adds to the pleasure.

Critical Systems Heuristics

The basics of Critical Systems Heursitics are described at the top of this webpage describing my workbook Values In Action. Here is a brief recap plus some historical context.

The idea of holism in systems approaches misleads us into thinking we can look at everything. But that is an absurd notion - we can't consider everything, neither in theory nor in practice. So we have to draw boundaries. Important boundary decisions are based on ethical as well as technical criteria. So it behoves us to assess them with a critical eye.

The Critical Systems Heuristics approach was developed in the early 1980s. During that period, questions were being raised about the ethical dimension of drawing boundaries within systemic practice, in particular by C. West Churchman. He argued that at a practical and theoretical level we had to think of the consequences of the boundaries we draw. Somewhat later, the German planner Werner Ulrich produced a heuristic that provides a practical way of implementing Churchman's ideas. Critical Systems concepts have had a profound impact on my thinking and practice as an evaluator, since criteria and values both delineate boundaries.

So, check out the workbook: https://gum.co/valuesinaction

And the 30-minute video of a talk I gave to the Centre for Evaluation of Complexity across the Nexus (CECAN) network: https://youtu.be/87pD2gjrcUs

Viable System Model

Systems Concepts in Evaluation

The Viable System Model (VSM) is a systems approach that identifies the minimum requirements that must be placed on collective human endeavours if they are to prove enduring and capable of development. The aim is to achieve optimum performance in terms of the systems’ purposes and to adapt and survive in a complex and dynamic environment.

The VSM diagram highlights five sub-systems that can be observed in every human activity system: operational activities, coordination and information, management and control, development and interaction with the environment, and strategy, identity and sustainability.

The core task of managing these tensions generally boils down to managing the tension between autonomy and control. Particular parts of the system need to have freedom to respond to their own needs for variety, but too much autonomy can create more variety than the rest of the system can handle. On the other hand, too much control can limit the ability to respond effectively to change.

There is a description of VSM in my System Diagrams book: https://gum.co/systemdiagrams

Cultural-Historical Activity Systems (CHAT)

Systems Concepts in Evaluation

Despite a rather off-putting name, CHAT is conceptually relatively simple; our actions are affected by our environment (which includes our histories and cultures). It identifies three core affecting factors: the tools we use (including language), the roles we play, and the rules that our work, home or cultural operate under. When we work collectively on a shared activity, these factors will affect how we engage in that activity. Conflicts, contradictions and sudden changes in these factors can prevent or derail how we work on that joint activity. On the other hand, if we can manage them well then those conflicts, contradictions and changes can enable us to work together more creatively and effectively. CHAT provides a way to identify these dynamics and create more flexible and safer spaces for our activities. Alongside Argyris and Schon's Action Science, Churchman’s Critical Systems and Checkland's Soft Systems, CHAT probably forms the foundations of my professional practice.

CHAT is really an attempt to merge systems, Vygotskyian-based learning and action research theories. It was developed primarily by Yrjo Engestrom at the Centre for Research on Activity, Developmental and Learning (CRADLE) and adopted by colleagues at WEB Research here in New Zealand.

If this interests you, and it really should, then there is a chapter on CHAT in my System Diagrams book.

System Dynamics and Causal Loop Diagrams

System Dynamics is probably the most familiar of all the systems-based approaches. Because it has been marketed so successfully through books such as Peter Senge's Fifth Discipline and associated with prestigious institutes such as Harvard and MIT, many people think it is the only systems approach.

People criticise system dynamics for being mechanistic, over-simplifying complex situations and being unreliable when it comes to predicting real life. Much of that says more about unrealistic expectations than what it is really about. It is primarily about generating insights into messy problems.

Another problem is that while system dynamics is an approach and not a method, it often tends to be conflated with a particular method known as causal loop diagrams. Whatever the arguments, CLDs are popular, if frequently misapplied and misused. You can find out more about good use of CLDs in my System Diagrams book.

Complex Adaptive Systems and Human Systems Dynamics

Complexity or complex adaptive systems (CAS) in its early days came close to being promoted as the answer to everything. A few decades on, people tend to be breathing through their noses a bit more. However, CAS still has to demonstrate to many people that it is more than pretty pictures of fractals on the screen, or obtuse mathematical expressions and dynamics.

On the other hand, within management academia, especially in the UK, it has developed considerable grunt and a large following. Personally I find a lot of the concepts make sense, but I was always at a loss to work out how to put them into practice. That is pretty much how I met up with Glenda Eoyang from the Human Systems Dynamics Institute and her grounded, innovative and practical approach to such an esoteric field.

I've been enormously fortunate to work with Glenda on a range of workshops for the W.K. Kellogg Foundation and the American Evaluation Association. Here's an excellent piece she prepared for a workshop she, Bill Harris and I did for Kellogg. In addition to the usual butterfly wings and fractals, it describes unique and powerful tools developed by Glenda and her colleagues over the past few years. I use them all the time.

Cynefin Framework

Systems Concepts in Evaluation

There's an ironic tendency towards absolutism in the systems and complexity field. Systems are either "simple", "complicated" or "complex". I've always felt uneasy about such definitive statements, they didn't match my own way of understanding things or indeed what I saw on the ground. The Cynefin framework provides a very useful means to navigate these distinctions, plus provides a base to do many other useful things.

The Cynefin approach is a decision support approach that allows you to explore, appraise responses to different real and perceived dynamics of a particular situation. Developed by David Snowden, Cynthia Kurtz and others at IBM’s Institute of Knowledge Management, the approach identifies five different ways humans understand a situation and act on that basis: Simple, Complicated, Complex, Chaotic and Confused. It draws not only on systems and complexity concepts, but also on network theory, learning theories, and third-generation knowledge management.

I was initially highly sceptical about it and some within the knowledge management field still are. However, the more I used it the more useful I found it in all manner of ways. Whatever the views on the theoretical basis for Cynefin, it is a very smart piece of work and a useful one too. Just remember that maps are not territories and that this is both an ontological tool to describe aspects of the real world and an epistemological tool to assist how you think about the real world. Here is my attempt to explain the framework and how it can be used. More comprehensive versions are also in the Systems Concepts in Action book and the System Diagrams book.