|
Social Change Training Manual
BUILDING LEARNING ORGANISATIONS
Presenter: Garth Luke
- documentation, monitoring and evaluation.
Increasing effectiveness and reducing stress through documentation, monitoring and evaluation.
This paper is intended to allow readers to:
- Develop an understanding of the importance of DME in building an effective organisation;
- Identify the key DME methods and how to apply them;
- Consider some useful applications of DME in their organisation.
- Training can't teach you everything you need to know
Staff and volunteer training by outside 'experts' and trainers can help to build more effective organisations but there are lots of things that you need to know that can't be provided by outsiders.
This is the specific knowledge base that an organisation builds up as it carries out its work. Each event, each activity provides important knowledge. This includes information about:
- Who is willing to support the organisation,
- How to effectively run a particular event,
- How best to explain key information to new volunteers,
- How to make the photocopier duplex without eating the paper,
- What members of parliament are supportive,
- What went wrong at the AGM last year etc.
An organisation that forgets these lessons is doomed to keep wasting time, frustrate staff and reinvent the wheel. An organisation that can learn and remember lessons, question its approach, develop more effective strategies and continue to build upon this knowledge is a learning organisation. Such an organisation makes much better use of the time of its staff and supporters, achieves more and reduces frustration for everyone.
Documentation, Monitoring and Evaluation (DME) is - the core of learning
A lot has been written about learning organisations, much in the general area of quality management. In this workshop I would like to focus on three key tasks to achieve greater organisational learning. These are:
Documentation, monitoring and evaluation.
- DOCUMENTATION is the recording and sharing of useful information. It can involve evaluation reports, procedure manuals, notes stuck on walls and machines, audio and video tapes, even songs. Documentation seeks to share valuable knowledge to help people avoid pitfalls and utilise the most effective strategies. It contains the fruit of our and others' mistakes. Good documentation is accessible, relevant and utilised in planning.
- MONITORING is the regular collection of information in order to test the progress of activities and make corrections as work progresses.
- EVALUATION is the review of activities against certain goals, standards or values.
Many organisations state that they are committed to DME but in practice they often give them a very low priority - preferring instead to concentrate on planning and action. Unfortunately their plans are unlikely to be very effective because they are based largely on hunches.
Vision, leadership, team work, decision making, skilled staff, effective conflict resolution, thorough planning, good negotiation are all necessary for an effective and positive working environment - but they are not sufficient. The best run team also needs relevant information to carry out its role.
Documentation, monitoring and evaluation can provide this information. It is especially important for organisations trying to do complex jobs (eg creating social change) with few resources, high staff turnover and a heavy reliance on volunteers.
This workshop is not about starting DME in your organisation as all people and organisations do evaluate and document things each day. It is about doing it as effectively as possible and helping to build a culture in which its effective use is encouraged.
My own experience of community organisations and the results of many quality management studies indicate that 40% or more of the resources of community organisations are often used reinventing the wheel and correcting mistakes. Not only is this a terrible waste of limited resources but it also saps the morale of those involved and increases stress in the workplace. In this context one of the key issues to think about is:
Continuous improvement
A useful way of thinking about organisational learning is described by the continuous improvement cycle described in total quality management. This can be represented graphically as:
- plan do
- act check
- documentation
Documentation, learning & quality improvement
The four parts of the cycle are Plan ---> Do ---> Check ---> Act. Checking is achieved by monitoring and evaluation while the check represents documentation of the lessons learned.
Documentation stops the organisation slipping backwards as it learns to improve its performance.
While evaluation and documentation are generally seen as important there usually seems to be more interest in planning and doing - with the result that not enough effort and time goes into ensuring that lessons are learnt.
Often people put these things last and never get to them, or they think they know all they need to know or that they are not skilled enough to carry out evaluation.
In reality most organisations can become much more effective at learning just using their existing resources. While some techniques and situations require significant expertise very useful information can be obtained using fairly simple methods. This does however require a commitment by the people in an organisation.
Those organisations which are effective learners tend to allocate about 5% or more of their resources to monitoring and evaluation. Useful documentation and sharing of information can be achieved with a similar commitment of about 5% of resources.
In return people can expect significant improvements in effectiveness, reduction of stress, improved morale and large reductions in the amount of time and resources correcting problems. In other words improved documentation, monitoring and evaluation usually pays for itself several times over.
Too often organisations make decisions purely on hunches, reinvent the wheel, forget important lessons learned. It is often as if they have no memory and are destined to repeat the same mistakes and learning cycles over and over again. The key to reducing this wastage is for the organisation to learn how to do things better and to remember this.
How much time is wasted showing each new person how to use the photocopier, how many supporters are lost when their request for less mail is forgotten, how many times are the same mistakes made in organising an annual event, how much energy goes into inventing and reinventing procedures that are not written down clearly, how many times does one group try an approach which has failed in every other state?
Methods of monitoring
In some ways monitoring could be considered simply as a form of evaluation but there are advantages in seeing it as a separate function. Evaluation often conveys ideas of thorough investigation after the event with comparison against objectives or values. Monitoring on the other hand emphasises regular checking of an activity as it progresses.
In this way it assists project implementation and allows fine (and sometimes not so fine) tuning of a project as it proceeds. Used effectively it can prevent you making expensive mistakes where you end up saying 'Oh well, maybe we'll do it better next time'. It is also a useful part of staff supervision.
The sort of questions monitoring asks are:
- Have we done what we planned to do?
- Are we working to the planned timetable?
- Has it had the effect that we planned?
- Are there instances of success and failure in different areas?
- Does anyone need assistance in carrying out the tasks?
Information used for monitoring does not have to be numeric - it can be opinions or a simple set of Yes/No answers (eg Did you check the doors were locked before launching the space shuttle?). It can be recorded on paper or on computer or on a white board in the office.
Examples of monitoring tools are:
- Performance indicators which summarise key measures;
- Checklists of activities and problems, regular personal and group reflection;
- Asking others to rate performance; financial and other targets.
It does need to be:
- Relevant - the information has to answer useful questions
- Accurate
- Timely - its not much good if the horse has already bolted
- Understandable
- Objective - at least not self interested
- As simple as possible - don't waste people's time collecting unnecessarily complex information
- Well defined - people need to have a shared understanding about what the information means - this is particularly important for historical comparisons
- Consistent - to allow comparisons over time and across areas
- Relatively easy to collect - don't want staff spending all their time collecting information rather than doing the task
- Used - there is no point in collecting information if it is not used
- Operationally focussed - the level of data collected and summarised should focus on the management unit
- Reaching the right person - this may be a manager or committee or it may be the worker carrying out the task.
- It is also handy if it can meet a number of needs at once eg external reporting requirements as well as internal management needs.
Most of the above characteristics may seem self-evident but it is amazing just how often they are not achieved and how often people seem to collect information that is not used or useful. This just wastes trees and everyone's time.
When monitoring a program it is important to identify the minimum key information required to manage the activity.
One useful way to identify this is to write down the objectives of the project and then to sort them into a hierarchy of outcomes. For example a program to increase home composting may have the following hierarchy of objectives:
- Gather the appropriate information
- Develop TV, radio ads and newspaper articles on home composting
- Get agreement from media outlets to publicise the information
- Increase the public's knowledge of home composting
- Change public attitudes to home composting and the tipping of organic matter
- Increase the level of home composting
- Decrease the pollution caused by tipping organic matter.
- Once such a hierarchy is established it helps in checking the logic of the project, issues that need to be addressed to achieve the goals, and suggests the key points that need to be monitored to adequately manage the project.
It is important that people can make use of the monitoring information. Simple, easy to understand presentation (graphics often help) should be presented to people at relevant times.
The best information is useless if it arrives too late.
One should not let a formal monitoring process replace more subjective and intuitive information. You should still keep your eyes and ears open and discuss how things are going. The formal monitoring adds to this information and helps to check its validity.
Some possible monitoring activities for a conservation group:
- Weekly meetings with staff and volunteers on each project
- Specific project indicators
- Regular comparisons with the budget
- Staff workloads
- Revue of press clippings
- Testing the effectiveness of mailings
- Indicators of response to clients/supporters - eg time to send receipts for donations
- Checking that decisions have been implemented
- Checking how staff and volunteers are feeling
- Auditing procedures
- Any other suggestions?
Methods of evaluation
Most of the rules that apply to monitoring information also apply to evaluation and many of the methods are shared. In practice the two overlap while probably the most important difference is that monitoring is seeking a fairly shallow picture of events to see if things are going according to plan.
Evaluation on the other hand looks more deeply to try to answer the question 'Why?' and to determine whether the plan itself should be changed. Evaluation looks at such things as effectiveness, efficiency, relevance and impact.
It is important that evaluation should not just focus on the formal objectives of the project. These are important but there may be other positive and negative effects not thought of. For this reason it is important to encourage other relevant information through observation, discussion and open ended prompts in interviews and surveys (eg questions like 'What did you want to get from the event?' or 'Do you have any other comments that you would like to make?')
It is also important to identify early on just what information you will need for an evaluation. It may be impossible or very expensive to collect relevant information after the completion of a project. If possible use existing data collection to reduce extra work for staff and always keep accurate and well defined historical data to allow comparisons over time.
With both monitoring and evaluation it is valuable to have as much participation by all the stakeholders as possible. There are usually four main parties - the evaluators, the evaluated, the clients of the program and those the evaluation is for. It is useful to try to maximise the overlap of these four groups. This helps to build skills in the organisation, ensures that the information collected is relevant and helps to build commitment to the process and results.
Techniques commonly used in evaluation include surveys, feedback sheets, focus groups, benchmarking, quality circles, observation, discussions with participants, numeric analysis, interregional and historical comparisons.
While there are risks in incorrect use of methods a simple formal evaluation can usually provide a valuable supplement to more subjective assessments. The main rules are to keep measurements simple, to compare like with like and not to jump to conclusions. Checking information by using more than one perspective or source of information is important.
The evaluation focus should not just be on work outputs (eg the number of appeal letters sent out) but on the outcomes (eg the number who responded to those letters by giving a donation). The focus on program outcomes should however not preclude checking the feelings, workloads and needs of staff and volunteers - unfortunately these are often overlooked in evaluations.
When writing up evaluation reports it is useful to have a consistent reporting style in the organisation with well defined information that is authored and dated for future reference. The writing should be sufficiently detailed to allow an intelligent outsider to understand just what is being discussed.
The job is not over once the report is written. If valuable information has been identified in the evaluation then it is important that it is used in future planning. This may require the 'selling' of the report and follow up to ensure that the results are taken into account.
Finally, when looking at any activity it is important to highlight the successes as well as the failures. While it is important to understand what can be improved it is also important for staff and volunteers to receive positive feedback.
Some possible evaluation activities for a conservation group
- Brainstorms and feedback after significant events and meetings - ask everyone involved to write down what went well and what could be improved
- Testing the effectiveness of mailings
- Quality circles to resolve continuing problems
- Six-monthly or yearly reviews of activities (perhaps linked to staff objectives or key result areas)
- Survey of staff and volunteer satisfaction
- Focus groups made up of supporters or the public to identify how issues are perceived
- Suggestions process
- Surveys of customer/member satisfaction
- Evaluation of relationship with other environmental organisations
- Assessment of effectiveness of newsletter - what are people saying, focus group, survey or responses to requests for action
- Campaign evaluations
- Fundraising evaluations - including cost benefit analysis
- Staff workload analysis
- Exit interviews
- Any other suggestions?
Documentation
Documentation can mean anything from a procedures manual, to a newsletter which shares fundraising and campaigning ideas, from a volunteer induction video or computer based training to a sign on the door which says 'Please close after you'.
It's all about sharing information so that people don't have to relearn things from scratch. Documentation also helps to provide consistency
The key factor to think of when producing documentation is who the audience is. The material has to be used and understood by the target audience.
It is true that many organisations have too much paper floating around but good documentation can actually reduce the volume of paper. Effective procedure manuals usually reduce unnecessary communication because people know what they need to do. Well documented and implemented communications and decision making procedures also allow people to feel confident that they will get the information that they need and that they will be listened to.
Summarising learning and removing out of date material can also reduce the volume of paper considerably.
It is common for people to join long established organisations only to find that virtually no procedures are written down or that no one is quite sure where to find them. Its very hard to improve procedures if you don't know what they are.
Documentation does take time and patience but as with all appropriate systems it can save much more time than it takes. It also frees people up to do something new rather than answer the same old questions from new staff and volunteers. This helps both morale and effectiveness.
You don't need to go overboard though. The amount of detail required depends on the criticality of the tasks. Those building space shuttles need pretty detailed documentation, those describing how to run a market stall need less! A good rule of thumb is that most organisations should spend about 5% of time on documentation and filing.
Some possible documentation activities for a conservation group
- Long, medium and short term plans
- Procedure manuals and (more detailed) work instructions- these should be recorded in a central register, should be kept up to date and should give the reasons for procedures
- Monitoring and evaluation reports
- Training notes
- Signs on machines which explain tips for use
- Direction signs on walls
- Filing systems and box labelling - including lists of contents
- Detailed notes on organising repeat events - ideal timetable, budget, useful contacts, key tasks and volunteers
- Donor liaison notes - information about donor history, requests, preferences, contacts
- Committee role and responsibilities
- Committee policy decisions
- Fundraising contacts
- Consistent reporting (eg reporting by objectives)
- Job induction package
- Office procedures on prominent display in relevant areas
- Historical records for events and campaigns to compare effectiveness
- Consistent and comprehensive job descriptions
- Documented grievance procedures
- Documented recruitment and termination procedures
- Use of reporting to keep people focussed on their tasks
- Accurate donor and network records including requests and discussions with supporters
- Ideas newsletter between intra and interstate conservation groups
- A single and up-to-date client database
- Any other suggestions?
- Who should be responsible for DME?
All staff should take some responsibility for ensuring that their work is documented and evaluated but key responsibility lies with the coordinators/managers of projects. Effective documentation, monitoring and evaluation are essential parts of successful management.
Organisations with ten or more staff should consider appointing one or more skilled people with specific responsibility for coordinating DME in the organisation. While DME responsibilities will still lie with general staff such positions help to ensure consistency in data collection and documentation, provide the time and skills for more complex evaluation and give energy and support to a DME program. These positions usually prove to be very cost effective.
Conclusion
Documentation, monitoring and evaluation can be very powerful tools in helping to build more effective organisations. The goal is to encourage a spirit of critical self inquiry and a willingness to record and apply the lessons learnt while still maintaining spontaneity in the organisation.
To achieve this it is important to realise that DME is an important part of the broad skills and strategies needed for effective management - but that it must be complemented by other 'softer' skills such as inspiration, negotiation, respect, listening, compassion, courage and humour.
Useful reading
D, Fry & L.King 1991 A manual of standards for community health (AGPS, Canberra) - this is a very detailed set of standards developed for community health centres, provides an interesting documentation and monitoring model
P. Hawe, D. Degeling, J. Hall 1992 Evaluating Health Promotion (Maclennan & Petty, Sydney) - good overview of planning and evaluation with very useful chapters on survey method and focus groups
Office of Public Management, NSW 1992 Planning and monitoring your program
(NSW Government) - a good and easy to read summary of the outcomes hierarchy approach to planning, monitoring and evaluation.
VCOSS 1984 Do it yourself Social Research (VCOSS, Melbourne) - a useful guide for community groups
Yoland Wadsworth 1991 Everyday Evaluation on the Run (Action Research Issues Association, Melbourne) - provides an overview of the wide range of monitoring and evaluation methods, and provides a very good comparison of both objective based review and the open inquiry method
Gail Wilson 1989 Self Evaluation Kit (Victorian Association of Citizens Advice Bureaus, Melbourne)
Garth Luke
C/- Post Office The Channon, NSW 2480
Phone: 02 66886198
Email: garthboy@ozemail.com.au
Top
|
