Committee for Conflict Transformation Support

CCTS
Newsletter 16

Evaluation: Reflections on experience

In this article Diana Francis reflects on some of the problems and possibilities of evaluation work, prompted by her recent involvement in a large evaluation process.

In January this year I was invited to be part of an evaluation team reviewing a large government programme in support of peace and stability in the Balkans. I will not name the government or describe the programme further. Suffice it to say that it was substantial and that the civil servants involved were passionately committed to it. They also had a commitment to open-ended process and to giving as much power as possible to their partners in the region.

For several reasons I was reluctant to join the team. I had relished the idea of a quiet start to the year. I am sceptical about evaluation in some respects and have reservations about the capacity of outsiders to contribute to it, other than as facilitators. I have seen many instances where external consultants who made judgements and recommendations got things wrong because they had missed crucial bits of understanding or information. I was persuaded to accept involvement by the plight of the civil servants responsible who urgently needed to find people to do this external evaluation, which was built into the design and timeframe of the programme. (Probably not a good reason for accepting an assignment! And in this case time pressure was one of the worst aspects.) The work would give me an opportunity to visit old friends in the region and to get a feel for the current situation in the places I visited. And I would have a chance to learn more about evaluation, how it can work and what it can and cannot accomplish.

The need for preliminary grounding

As indicated, the pressure of time was there from the beginning. Those who had been responsible for setting the wheels in motion had clearly left it too late for a steady, strategic beginning. They had not counted on the busyness of the people they wanted to invite, or on the time it would take to bring them together. They had probably not visualised the different steps that would be needed to bring a team together and brief them; to give them time to get to know each other and to find available time in common; to allow them to share assumptions and understandings about peace and stability, explore their respective approaches to evaluation, agree purposes, devise a strategy and define tasks, identify strengths and weaknesses in relation to what was to be done, and to clarify roles, assign tasks and agree procedures. Nor, I think, can they have thought how much time would be needed to make the necessary plans for travel in the region, including the purchase of air tickets, obtaining of visas and making appointments for visits to different organisations. Time needed to be allowed also for the digestion (individual and collective) of information and the writing of the report itself, especially a team's findings and the ways they are represented have to be agreed by several people, working at a distance.

In practice little of the preparatory thinking outlined above was done by the team, except on the hoof. The three original team members, whose diaries were already complicated, came from the US, South Africa and the UK and had never met each other before. They were chosen on the basis of existing connections with those who were commissioning the evaluation, but the fact that they did not know each other in advance, and their distance from each other and from the region, clearly did not help to expedite the process or reduce its complexity. Two further members were added after the process had begun, by the decision of the 'lead consultant' whose colleagues they were. I have reflected, frequently, during the process, on the merits of composing a team of very diverse people - one from the field of development, one a journalist specialising in journalism and human rights, and me (plus, subsequently, two more on the development side) - with very different experiences, perspectives and personalities and regional knowledge ranging from substantial to non-existent. The different fields of expertise were arguably important and contributed to a collective knowledge base much larger than could have been provided by one person - or, perhaps, by a team from one field alone. But since in practice we had to divide the work of visits and interviews, and not always in line with our particular knowledge and experience, what this meant was that different parts of the elephant were viewed through very different spectacles and described in very different voices. (Had we had adequate time for the kind of team building suggested above, this difficulty might have been greatly reduced.)

We did agree, from the start, that the process we wished to initiate would be participatory, facilitated by us but done in conjunction with the government's 'implementing agencies' and with their local partners, contributing to their own review of their work and achievements and their planning of the programme's next phase. What I think we did not adequately acknowledge was that the broad overview which we were supposed to reach might need a perspective which could not be found within the programme. A huge amount of paper - minutes, reports, interim evaluations, internal and external - was loaded onto us: almost too much to carry from the taxi into the airport. I digested as much as I could through erratically reading and skimming, but at a certain point realised that I could absorb no more through print and that my efforts were becoming counter-productive. I realise I need a more strategic approach to reading - or someone to do some judicious pre-selection for me.

We never had time to make an overall plan for our travel and meetings, staggering from one phase to the next. Nor did we approach those we met with an agreed list of questions. This made conversations more fluid and interesting, but meant that we tended to focus on different things, so that it was hard to reach a shared overview of what was important to those we had met.

Conflicting voices and other grounds for uncertainty

To begin our conversations in the region, we agreed to invite different partners to a conference to discuss their experiences and views. This, like the subsequent visits to a great number of individual organisations, was difficult to organise at short notice. The whole point about these organisations was that they had work to do! The idea was mooted that our draft report should be shared with them at a final conference, so that they had a chance to propose adjustments to it and also to use it in their thinking about the future. In fact that will not happen. There was not sufficient clarity about it in time for dates to be found which were convenient to all or most of those who would have liked to participate, or for general enthusiasm for such an event to be generated. Instead there were separate meetings with different categories of people after the draft report had been circulated, so that they could respond to it and influence the final version. This proved an important, useful part of the process.

While we chose a participatory approach and had a clear task of representing in a faithful way the voices of those we met, listened to and interacted with, those voices were at times conflicting and always different in particulars or emphasis, making generalisations difficult, if necessary. Furthermore, meeting representatives of the organisations concerned is not the same as seeing what they are doing (though in a few cases we did that too). A great deal has to be taken as given. Yet one of the points of difference between the accounts we heard lay in the assessment made by those interviewed of the real substance of some of the work which appeared on paper to be being done.

Not only did it prove difficult to construct what felt like a reliable overall picture or detect a common voice (though distinct voices were to be heard) to represent the views of those involved in the programme; we were also expected to add our own commentary and opinions and, as already indicated, an overview which would be strategically useful. This felt to me like a heavy responsibility. What if people lost funds because of what we said and we were wrong? What if the programme were adjusted on the basis of our recommendations and those recommendations were not well founded? How seriously would our conclusions be taken and how could we handle the power we had been given responsibly, especially when time was so limited?

The cost of the exercise, in terms both of the time of all concerned and in the money spent, has been a matter of concern to me. I can see that if a great deal of money is being allocated to a programme it is worth trying to ensure that it is well spent. And there are aspects of any programme which, with all the caveats above, can be assessed in relation to their own explicit aims - in this case, for instance, the goal of building practical co-operation. It is also possible to see where relationships and organisations are working well, where proper analysis is being done and applied, and whether objectives set are being met.

But the impact of a programme in terms of its wider aims, in this instance of contributing to peace and stability, and its assumptions (that establishing democracy is a guarantee against civil or regional war, for example, in this case) - these are impossible to determine by any empirical means - certainly within the confines of such an evaluation process. Shifts in the big political scene, when they can be identified, are not often capable of attribution to any one source. The relationship between grass roots work and high politics is notoriously difficult to delineate. And who can demonstrate beyond reasonable doubt the importance of such things as refugee return, human rights advocacy or media development in contributing to peace and stability? Or their relative value compared with each other, or with economic development, or with work to address ongoing conflicts, and old resentments and hostilities, in more direct ways? These things can, as far as I can see, be assessed only theoretically and philosophically, and evaluators will have differing viewpoints. So it was with us, causing us considerable difficulties in reaching agreement on our assessment of the situation(s) in the region and consequent recommendations, all having to compromise. This in itself must relativise the reliability of evaluators. I know I would have made different recommendations if I had been left to my own devices. Whether they would have been better I cannot say.

Maybe the best one can do with all this is to encourage the clarification of at least working definitions of aims, an explicit rationale to relate activities to those aims, agreement on criteria for success in particular endeavours which can be measured, and 'indicators' of whether these criteria have been met. An explicit rationale will involve a number of fundamental assumptions and values, which can, perhaps, be brought into awareness, reviewed, challenged and affirmed or modified.

Challenging assumptions

In this particular evaluation process, some of my own assumptions and values were brought into relief and challenged. For instance, while we were in Sarajevo, one of our team met an old man in a meeting who came back to see her later on his own, to tell his own story and press his concern. He had been 'somebody' in his younger life - had worked for the government in Africa and elsewhere, owned a nice apartment and was widely known and respected. Then in the war he had been forced to flee. When he returned, he found his flat occupied by strangers. They would not even allow him to collect his personal papers, let alone his other possessions. He was now without property or status, floating on the edge of a society of which he had once been an integral and respected part. He argued that if human rights were important to peace, then the plight of old people like him was important and should receive concerted attention.

In any general and generous view of peace and stability, this man matters. The sad truth is that at the strategic level, in terms of Peace and Stability with capital letters, his plight is unimportant. It is a truism that young people 'are the future'. Top politicians or the economically powerful are seen as important because they are the ones who wield power on the large scale. An old man without power is, by definition, unimportant. Many of the refugees who have not been able to integrate themselves into new contexts or to return to their original homes are not 'important' people, except in so far as they provide grist for the political mill or can be manipulated to fuel resentment. Yet I would argue that the foundations of peace are laid in respect for the weak and that, without that respect, what I want to call peace cannot exist. Without the values that would give such members of society a dignified place within it, the only stability that can be achieved is the stability of control - a contradiction in terms. But in the short term, which must be secured if the long term is to be reached, do there have to be other priorities? That would be hard to accept, and maybe we should resist the idea that a choice has to - or can - be made. Can peace exist within a society without compassion at the individual level?

It is by no means clear that such philosophical discussions have been held by the government and civil servants in question. The programme is doing 'good things', worthy in themselves. How the Peace and Stability to which they are meant to contribute are understood is not apparent, and whether those good things which are being done are the most cost-effective contribution to its potential establishment was therefore not only hard but impossible to assess. However, there was agreement in the team that, given the prominent use of the words 'reconciliation' and 'democracy' in the programme's literature, and in the light of the overarching goal of Peace and Stability, those activities supported should be reviewed by the programme's participants, to explore whether their design and implementation could be more strategically related to that purpose.

In addition I proposed the inclusion in the programme of organisations and projects whose purpose was to address conflict as such and of organisations directly concerned with democracy, to promote activities specifically focused on reconciliation and education for pluralism and the constructive, democratic handling of conflict. My proposal was not, however, included as a recommendation in the report.

One of my personal assumptions is that conflicts are better addressed explicitly and directly rather than avoided or addressed obliquely. I realise that this is a temperamental as well as theoretical and sub-cultural viewpoint. I know it is open to challenge and perhaps deeply flawed. Nonetheless, in the programme in question there is what I see as a lack of any overt discussion of underlying conflicts within the regional networks whose work and impact are in the process of evaluation. Issues which divide are ignored, in the hope that contact and co-operation will overcome the divisions. This is not an unusual approach, but it is not mine, and I see that some tensions are exacerbated by clumsy choices which could have been avoided if the relevant issues had been 'out there'. I believe that addressing conflict constructively and directly, and learning to understand and talk about its elements and dynamics, is like learning grammar as a route to competence in a language, saving a lot of time and mistakes and providing firm foundations on which to build. (Maybe deeper exploration of this metaphor would reveal more of the pros and cons of my approach.)

The role of the outsider

Given that those organisations involved in this programme had their own raison d'etre before the programme began, it is maybe inappropriate for an external donor, avowedly wishing to support local endeavours, to ask them to refocus their efforts in line with external goals. The relationship between 'outsiders' and 'insiders' was a strong theme in our evaluation, but one which I do not have room to explore here. I should note, however, that it was not unrelated to my doubts about our role - as an external team of evaluators working for an external donor. The potential importance of our coming from outside the region was perhaps to give a strategic overview and critique of the programme's structure, rationale and outworking. This we largely failed, I believe, to do.

Not surprisingly, perhaps, given so many doubts and questions, I found myself by far the most reluctant in the team to reach conclusions on anything. This can be seen as a disability in an undertaking looking for clear assessments and recommendations. Working in a team was reassuring when we were substantially in agreement amongst ourselves (though I am not convinced that it should necessarily be so - it would be possible to generate some kind of suspect dynamic together). When we were in disagreement, I found myself becoming, paradoxically, more sure of myself - perhaps simply more stubborn.

My greatest comfort was the fact that our power was far from absolute. Those affected, in whatever way, by our findings could argue with them, take them with a pinch of salt or reject them - as they did with some of the views and recommendations of past evaluators. The fact that we were hired did not mean that those who hired us gave away their own power and responsibility. And those whose activities are funded (in part) by them also have the capacity to evaluate our report and make their own choices.

In spite of all my misgivings, I do believe that, thanks to all our hard work, and to the tenacity, sang-froid and good humour of our highly skilled team leader, we handed in, at last, a substantial account and appraisal of what we had seen and observed, and recommendations for the future which will provide, I think, at the very least, useful food for thought. It was important that we took the time and trouble to receive feedback from all concerned on the first draft of the report, which enabled us to check our perceptions and resulted in considerable improvement to it in terms of both accuracy and cogency.

As time passes I shall further digest this experience and hope that it will indeed have added to my personal understanding of the work we in CCTS are engaged in and the ways we can think honestly and constructively about its effectiveness. In the meantime I have been heartened both by the commitment and professionalism of the people we met in our travels and by the vision and enthusiasm of the government employees who are seeking to support them and stimulate co-operation and shared agendas. The ethical issues surrounding the role of outsiders in stimulating anything in other people's countries would be the subject for another long article!

Diana Francis

Next Article

newsletter  |  ccts