Splash and Ripple: Using Outcomes to Design and Guide Community Work [Health Canada, 2002]

Using Outcomes to Design & Guide Community Work

• Introduction
  What is Outcome Measurement?
  How can Outcome Measurement help your organization?
• Outcome Measurement
  The Basics
• Splash & Ripple
  Another image of Outcome Measurement

The Essential Parts of an Outcome Measurement Framework

• Situation Assessment
  Defining priorities for your organization
• Inputs
  The resources you need to create the results you seek
• Activities
  What you do to create the change you seek
• Outputs
  Creating the potential for desired results
• Outcomes
  Achieving desirable changes for people, organizations and communities
• Impact
  Toward longer term change
• Enablers And Constraints
  What's helping and hindering your project
• Indicators
  How you know you're achieving the results you seek
• Outcome Measurement Framework
  Putting Outcome Measurement to work
• Getting Started
• Website Resources
  on Outcome Measurement
• Appendix

Introduction

What is Outcome Measurement?

It is an approach to planning and managing projects that encourages us to be clear both about what our projects are DOING and what they are changing. Outcome Measurement largely stems from a global change in the way governments work.

In Canada, and most industrialized countries, government agencies are re-organizing the delivery of social services and the non-governmental organizations that supply these services to demonstrate and measure their results. Now, more than ever before, these agencies - our funders - are under pressure to show value for the public funds they spend.

The trend toward greater accountability is just beginning and is expanding throughout and beyond the public service.

How Can Outcome Measurement Help Your Organization?

Outcome Measurement is not just a way to strengthen accountability for the use of resources. Used effectively, it can help organizations make wise planning and management decisions. It will help your organization:

• know what to expect from project Activities;
• identify who will benefit from the expected results;
• gather just the right information to know whether the project is achieving what you want;
• know how to improve project activities based on this information;
• know how to maximize positive influences (Enablers), and to avoid or overcome negative influences (Constraints);
• communicate plans and achievements more clearly to people and other organizations;
• gain from the knowledge, experience and ideas of the people involved;
• provide accurate and convincing information to support applications for funding.

A Word of Caution...

First, the language of Outcome Measurement can vary slightly from one funding body to another. Take heart in knowing that current Outcomes focused planning and management approaches are centered around a 'logic chain' that links resources (Inputs) and Activities to desired results: changes in people, families, organizations or communities. Outputs, Outcomes, and Impact are the results terms used in this handbook.

Secondly, funding bodies each have their own styles of implementing Outcome Measurement. Some funders are more direct about the Outcomes they seek and fund projects largely on their own terms. Others are more open to negotiation.

The best way to prepare for the introduction of Outcome Measurement is to have a solid command of the language and concepts and how to apply them inside your organization.

Outcome Measurement Framework

The Basics

This handbook is organized around the nine terms presented here. It takes you through the terminology, concepts and examples to help you apply Outcome Measurement inside your own organization.

Situation Assessment
Defining priorities for your organization.

Inputs
The resources you need to create the results you seek.

Activities
What you do to create the change you seek.

Outputs
Creating the potential for desired results.

Outcomes
Achieving desirable changes for people, organizations and communities.

Impact
Toward longer term change.

Enablers and Constraints
What's helping and hindering your project.

Indicators
How you know you're achieving the results you seek

Outcome Measurement Framework
Putting these Outcome Measurement terms to work.

Splash & Ripple

Another Image of Outcome Measurement

For every project idea there are a multitude of people with ideas of what should be done and what can be changed. The more that diverse points of view are included in the planning process, the richer the project will be. Unfortunately the language of Outcome Measurement is often a barrier to participation. This is where images, or metaphor, can help. Here is one image to help people understand and use Outcome Measurement.

The rock is like a material Input, the person holding the rock is like a human resource Input. The act of dropping the rock is like an Activity. When the rock reaches the water, it creates a SPLASH. These are your Outputs.

The ripples, spreading out from the splash are like your Outcomes, and then later your Impact. The edge of the pond represents the geographic and population boundaries of your project.

There are five guiding ideas inside this image:

Time
Splashes (outputs) become Ripples (outcomes) which move outward over time. The idea is that Outputs are immediate and flow directly from Activities. Outcomes take the lifetime of your project to show and are at least one step removed from Activities. Impacts take longer than the life of your project to show and are many stages removed from Activities.

Spread Effect or Reach
Splashes cover a smaller area than ripples do. This suggests that an Activity and its Output involve a relatively small number of people, but that just as a splash yields ever widening ripples, the benefits of the activity and its output spread beyond the initial group of participants to include other people. It is in the zone of the ripples that a project generates important social changes.

Control
As splashes become ripples, control diminishes. You have considerable control over Inputs, Activities and even Outputs up to when the splash occurs in the pond, but after that you have less and less control.

Context
The ripples (Outcomes) take their own course, affected by other disturbances in the pond. You can influence the ripples, at least those ripples that are closer to the original splash. To use this influence, you must know how to deal with the disturbances, i.e. Constraints and Enablers.

At the outer edges of the pond, where the ripples are wide and distant from the splash, your influence has diminished further. Here, at the level of Impact, all you can do is contribute to the big picture, long-range change you seek.

Learning and Improvement
Every splash and ripple sequence is an experiment. The more you know about the Splash and Ripple effect of your project, the better able you are to add to or alter your Inputs and Activities to yield better results. Using the image, you can drop the rocks in differently, drop bigger ones or drop more or fewer of them.

Situation Assessment

Defining Priorities for your Organization

Looking Outward

• What is happening in your environment?
• What issues/opportunities most need to be addressed?
• Who will benefit?
• What timeframe is needed to make a difference?
• Who are possible partners/ funders/clients?

Looking Inward

• What is our mission/mandate?
• What are our best skills/ resources?

Before charting Activities, Outputs, Outcomes and Impact statements for a project, it is important to do some preparatory thinking. This involves an "outward" look at the issues and opportunities you could address, and an "inward" look at your organization, its mission, skills and resources. Take some time to do this; it will speed up your planning process and sharpen your focus. In doing so, you should be clear about the following:

• The priority issues/opportunities that exist, and those that fall within your organization's mission/ mandate and capability, e.g. - you may see that your organization is well-positioned to address community concerns about "teenage smoking". The issue is "hot," you have the contacts, and it fits your mandate
• Spatial/geographic boundaries for the project (if any), such as a municipality, city neighbourhood or region, e.g. - in a project to address "teenage smoking", the boundaries might be confined to a particular school and its catchment area.
• Population boundaries (if any), such as an ethnospecific community or gender group like teenage girls, e.g. - in the same project, the boundaries might be a particular school population.
• Particular groups (individuals, other organizations/ institutions) that might have a role to play in the project's Outputs, Outcomes or Impact, e.g. -school students, community leaders, teachers, school trustees, or key social service organizations.
• The amount of time you may need to address your priority issues/opportunity, e.g. - six months to participate in the teenage smoking study, two months to engage the public in discussion of findings for a total of eight months.

Meet New Beginnings, our fictitious Refugee Support Project. As you read through the handbook, you'll find examples of how their work would be described using Outcome Measurement language.

Each component is put together at the end in an Outcome Measurement Framework. This is a word you'll see again - it just refers to the overall plan.

You'll find the complete New Beginnings Outcome Measurement Framework in the Appendix. Throughout this handbook, whenever you see this symbol, New Beginnings will be used to help illustrate a concept.

SITUATION ASSESSMENT
for New Beginnings Refugee Support Project

A political crisis in their homeland has forced many to seek refugee status. 20,000 citizens of this country have arrived in Canada over the past 6 months. 200 households are currently living in our region (pop. 500,000). Households are isolated and lack an orientation to services and supports. Language is a barrier for the majority. Community organizations are helping out where they can, but a more intensive and coordinated approach is needed to help newcomer households re-establish themselves.

As the principal settlement service provider in the region, New Beginnings is well positioned to respond. Our decision to put forward this project has been made on the basis of consultation with community groups and the informal leadership of the newcomer community.

Timeframe: 2001-2003

Inputs

The Resources You Need to Create the Results You Seek

Inputs include:

• Human Resources
  For example: staff, volunteers, consultant time, etc.
• Material Resources
  For example: supplies, room rental, equipment, etc.

This is where you create a list of what you need to carry out the project. Consider, for example:

Staffing - What skill sets? How many full- time equivalents (FTE's), how many volunteer hours?

Office or meeting space - Size? Location?

Equipment - Computers? Chairs? etc.

Materials - Stationery? Training text books? Bus tickets?

Budgeting - Having a good understanding of your INPUT requirements helps you to draft a budget. Remember that your Outputs, Outcomes and Impact are based on receiving a certain amount of money. If you receive less funding, you'll need to modify these results.

INPUTS
for New Beginnings Refugee Support Project

The New Beginnings team designing this settlement support project has not yet made a complete list of inputs, but it would include:

• staff FTE's (Full Time Equivalents);
• volunteer time;
• consultant time for staff training workshops;
• facility costs including rental and utilities;
• meeting costs such as coffee and flipcharts;
• associated office costs such as fax, phone and photocopier;
• equipment costs such as desks, chairs and VCR; and
• material costs such as paper and books.

Activities

What you do to create the change you seek

Activities are what you do with the Inputs that you have.

The challenge is to group your list of activities into between five and eight 'sets' or 'clusters'. If you have too many activities listed, it's very difficult to keep track of your progress.

Common headings include:

• promotion/public relations
• counselling
• group formation
• networking
• advocacy
• training

Under these headings, you can write short paragraphs describing what the project is DOING.

When the project team did their Situation Assessment, they met with community groups and the informal leadership of the newcomer community. Therefore, initial consultation does not appear in their current Activities.

Here are the activity sets New Beginnings came up with:

• Connect with Community Groups
• Assess Situation
• Reach Out to Newcomer Households
• One-on-One /Family Counselling & Referral
• Launch Peer Support Groups
• Networking and Advocacy

At the New Beginnings Outcome Measurement planning session...

Let's put those into related activity sets of ...

Outputs

Creating the potential for desired results

Usually each Output relates back to one Activity. With each Output you create potential for your Outcomes to occur.

Output Characteristics

Sphere of Influence - the creation of products, or immediate changes to those participating in the Activity.

Timeframe - usually coincide with the completion of the Activity.

Degree of Control - a lot.

• Outputs are the most immediate results of your project.
• Each Output relates directly to one of your Activities, so you should have as many Outputs as Activities.
• Just as splashes yield ripples, Outputs create the conditions for project Outcomes. Outputs describe the potential for change created through your activities.

Sphere of Influence

Sometimes the Output is simply a product - such as a clearly argued policy submission, or an eye- catching poster. Sometimes the Output relates more to the behaviour of people or groups at the center of the activity. Here are two examples:

1. Your catering collective staff, equipped with an attractive poster and menu, visit local businesses to promote your new lunch hour service - this is just one of your activities. The Output is, "business contacts agree to pass your information along to their staff". Getting the word out about your new services is an important condition for a successful lunch-time business (Outcome). Of course, there are other conditions too, but they relate to other activities.
2. Your coalition for equal access to public services holds monthly meetings for member individuals and organizations. This is just one of your coalition activities and its Output is, "member representatives perceive monthly meetings as a good use of time". Meetings must be a good use of time if the coalition is to gain political influence and high quality, coordinated programming (outcomes). As above, other conditions are needed for outcomes, but they relate to other activities.

Outputs usually occur at the completion of Activities. You have a lot of control at this point. For example: An abusive spouse takes a course in anger management, and has learned to walk away in an escalating situation and how to identify those situations. The participant acquired this knowledge in the workshops you offered. This is the Output. Whether the individual can apply this knowledge when s/he returns home is less certain, but very important to the success of the project. This is the Outcome.

Outputs for New Beginnings Refugee Support Project

The project team has developed the Outputs for each of the activity sets. Notice how they're worded in the active (present) tense.

They show potential being created which will be unleashed in the Outcomes.

Activity: Connect with Community Groups

Output: There is growing mutual trust, an agreement to collaborate and a common sense of purpose between "New Beginnings", other participating organizations, and leaders of the newcomer community.

Activity: Assess Situation

Output: A baseline understanding of the demographics, education, skills and language profiles of the newcomer community exists; there is an agreement on the most pressing issues and priorities for action.

Activity: Reach Out to Newcomer Households

Output: Each household receives a flyer, the Outreach initiative is discussed at community meetings, and posters are displayed at key locations where family members might visit, e.g. selected doctors offices, religious organizations or grocery stores.

Activity: One-on-One/Family Counselling and Referral

Output: Newcomers leave with their most immediate questions answered, with a referral (if required), and a clear understanding of what steps they can take to address the issues discussed in the session, e.g. enroll in an ESL program, go to a doctor, meet with child's school teacher.

Activity: Launch Peer Support Groups

Output: Newcomer participants share life experiences with each other, learn new skills and take on daily tasks with more confidence.

Activity: Networking and Advocacy

Output: Relevant, well argued messages are delivered through submissions to task forces or funders, regional media, one-on-one meetings or direct correspondence with decision-makers.

Outcomes

Achieving desirable changes to people, organizations and communities

Outcomes are more than one step removed from Activities, are fewer in number (two to four) and flow from a combination of several immediate Outputs.

Outcome Characteristics

Sphere of Influence - Expanding into wider settings.

Timeframe - Should be observable at or shortly after the end of the project.

Degree of Control - Much less than over Outputs, but still have direct influence.

• Each flows naturally from several of the immediate Outputs - there will be fewer Outcomes than Outputs.
• Outputs point to the key changes that directly relate to your project.
• Each change represents the unleashing of the potential created by your Activities and their Outputs - the "ripple effect."

Outcomes relate to the individuals who directly participate in activities, but like ripples in a pond, they go beyond those individuals to include, for example, families or organizations surrounding those individuals.

In counselling projects, Outcomes may relate to what individual clients actually do with the advice or referral they receive (the activity and its output, or Splash!). They may also relate to desired changes in relations between the client and the family or colleagues. And, if you add up these individual outcomes, you can demonstrate the coverage your project has achieved

within a particular community or target group. This is the case in our New Beginnings example; we know roughly how many refugee households the Project must support.

In coalition-type projects, Outcomes may relate to the way in which individual member organizations work more effectively; how they work with their peers - more collaboratively; or how they influence decision-makers - more forcefully.

In projects designed to build stronger, more equitable organizations, Outcomes may relate to: what individual project participants do differently in their workplace with the knowledge, tools, or strategies gained in training activities; or expected changes in those who surround those participants -their peers and managers. You might be looking for changes in policies or in service delivery, or else for something less concrete but equally important like a more inclusive and welcoming work environment.

In projects designed to build stronger communities you might be looking for Outcomes like: more people actively involved, better connections to social services and supports, or innovative group initiatives, e.g. small businesses, co-operative day care or lending schemes, which solve problems and spin off additional ideas and adaptations.

These changes often take more time to see, but should be achievable within the life of your project.

You don't have full control over Outcomes because they are at least one step removed from your Activities, and with this kind of distance, outside influences can easily get in the way. Nevertheless, it is very important to manage your project to achieve these Outcomes because these represent the concrete changes you are trying to bring about through your work.

Outcomes
for New Beginnings Refugee Support Project

Here are three Outcomes the New Beginnings team have identified. You can see that they don't exactly match the number of Outputs, in the way that Outputs match up with Activities. Instead, there are three Outcomes flowing from six Outputs.

Two things to notice

• The potential created through activities is unfolding into desirable and achievable changes.
• Like ripples, the benefits generated by the activities and other splashes are spreading outward to involve others.

Sample Outcomes

1. Newcomer men and women enroll in locally available English language training and job development programs; obtain medical, dental and legal assistance as required; and utilize other community services and supports.
2. Newcomer women and men establish new, trusting, mutually supportive relationships, both within the same newcomer community and beyond; they establish their own community organizations.
3. Targeted local institutions identify and make changes that make it easier for newcomers to access services and supports.

Impact

Toward longer term change

This describes your vision of a preferred future and underlines why the project is important.

Your project alone cannot make this change, but it should contribute to bringing it about.

Impact Characteristics

Sphere of Influence: expanding into yet wider settings.

Timeframe: beyond the lifetime of your project.

Degree of Control: very little, only indirect influence.

Describe what this community will look like after this project.

• This is the goal or vision that your project is aiming for; it describes the bigger picture, longer term changes that your organization is working toward.
• It answers the question "Why is this work important?"
• Aim for one statement that describes this preferred future.

The Impact is like the outer ripples of a splash. Like a distant ripple, an Impact takes time to come about and tends to affect a wider audience.

Ripples at this stage are usually not powerful enough to be dominant, but they do have some influence on the pattern of the water. This gets across the idea that projects, on their own, cannot bring about impacts, they can only contribute to them.

Impact for New Beginnings Refugee Support Project

Here's the Impact statement they've come up with:

There are two defining words in this statement that provide direction:

"self-reliant" suggests that the newcomers are applying their skills and knowledge so they can take care of themselves;

"contributors" signifies that the newcomers are active in the community, school, etc.

Perspective

The view of Outcome Measurement from the Government of Canada

We hope by now you are getting excited about the benefits of using Outcome Measurement Frameworks in your organization to shift the focus from managing for short-term results (outputs) to managing for long-term results (outcomes). Building a framework together is also a useful way to open a dialogue about what changes you'd like to see, and what will be needed to get there with project implementers.

Realistically, however, most community initiatives need some outside funding to help them achieve their outcomes. Having a command of Outcome Measurement language, and a solid framework in hand, is a good foundation on which to build your funding support. Many funders have websites to help you understand "their language". For information how the Canadian government views outcome measurement click on the Treasury Board of Canada Secretariat Web Site and go to "Results for Canadians" This is the web site of the Treasury Board Secretariat, the government agency responsible for helping all federal agencies adopt a results focus. Here, you will see just how central an Outcomes orientation is to the Government's vision for the public service. Close attention to outcomes is central to maintaining: 1) a focus on citizens; 2) a clear set of public service values including self-reliance, compassion and respect for democracy; and 3) responsible spending.

There are three particularly important messages in the Results for Canadians document for non-profits:

1. The Government's commitment to results spans both the public service and those who partner with the Government and use public funds, including non-profit organizations.
2. Implementation of outcomes-based management must be done in a flexible manner; "one-size-fits-all management as prescribed by a central body does not work". "Knowledge partnerships" between public institutions and their partners in non-profit and private sector organizations are critical to achieving effective, citizen focused public sector management, "No longer do public institutions have a monopoly on relevant knowledge."
3. A challenge for the future is to foster a performance culture throughout the system that accepts "learning by doing" and "continuous improvement" as a way of working both within and outside government.

Enablers & Constraints

What's helping and hindering your project?

These forces can be found affecting every part of your project - (Inputs/ Activities/ Outputs/ Outcomes/ Impact)

They can be found at every scale (family, community, national, international) and within your organization as well as outside.

They can be generated by humans, or by forces of nature e.g. the weather.

The more you know about these influences, the better prepared you can be to manage for them by doing Activities differently.

You may use the positive Enablers to increase the effect of the changes you bring about. For example:

Your Mayor is a "champion" for more low cost housing; invite her to chair a task force on homelessness and moderate a panel at a property development forum.

You may be able to revise your plans to minimize the effect of the negative Constraints. For example: Change workshop dates to avoid religious holidays.

List the Enablers and Constraints, then decide if you can modify your plans to maximize the effect of the Enablers and minimize the effect of the Constraints. You may find that some of them are too big for you to influence. If this is so, make them conditions that need to be in place for your project to be successful. Call these assumptions.

Assumptions for New Beginnings Refugee Support Project

• Federally and Provincially funded ESL and Job Development programs will continue. Risk: Low
• The regional labour market continues to support skill sets offered by the members of the newcomer community. Risk: Medium

Assumptions and Risks These are the necessary conditions for success.

Assumptions refer to influences that are on such a large scale that you have little or no control over them.

They are the conditions that need to be in place for your project to have the intended results. For example:

The city zoning by-law review will not change the status of your agreement to build new housing units for your cooperative housing project.

The local labour market will continue to need new workers like the ones enrolled in your trades training project (TV repair workers, sheet metal workers).

The risk can be stated as high, medium, or low that these conditions will not be in place. Add a short explanation of the risk.

Assessing risk involves bringing together your best collective understanding of the project and its context. Occasionally with large, complex projects, funding bodies may require a more formal risk analysis.

Indicators

How you know you're achieving the results you seek

These point to the information that you need to gather to know if the project is making a difference.

They should be identified for: Outputs, Outcomes and Impact.

Each must be calibrated to measure a specific Output, Outcome, or Impact.

Information should be a mix of qualitative and quantitative and can be gathered in a variety of ways.

Each should provide information that helps the project improve.

Each must be cost-effective to use.

It's like an Airplane Cockpit
Imagine the dials and displays inside the cockpit of an airplane. These provide important information to the pilots about the performance status of the plane - its position in relation to its destination, windspeed, altitude, fuel level and much more. Without these indicators, the pilots have very little to guide them on their journey. The pilots don't use all the dials at once, and sometimes it takes a combination of dials and displays to give them all the information they need at a specific point in time. For pilots and project managers alike, Indicators are important for navigation.

There are a variety of data collection methods (see following page for a further explanation of methods).

Establish a Baseline
Often called pre and post testing (the pre- test establishes your baseline)

You need to have information before you begin so that you have something to compare your results to. This is how you'll demonstrate there's been a change.

Quantitative Indicators
Express indicators as a ratio, a percentage, a comparison, or a number. For example:

The ratio of the total population of newcomer men and women who enroll in ESL and Job Development.

Qualitative Indicators
Express indicators as a change or a comparison between two states or situations. For example:

Changes in perceived levels of self- confidence among newcomers in the sewing group.

Gathering data does not have to be a separate activity that everyone dreads. Work it into the Activities you're already doing.

Target Indicators
A target indicator (or success indicator) includes a level that you are aiming to achieve. It's difficult to set attainable targets unless you've been gathering data for years and are repeating activities in a familiar setting. For example:

75% of counselling clients will not return for more counselling for the same problem within 6 months of last session.

Sampling
When it is difficult to conduct a census of an entire population, a researcher will select a portion of that population, a sample, which is thought to be representative of that population. Sampling is a common method for collecting quantitative and qualitative data. It does not require as much staff or volunteer time for collection, but can still provide reliable information.

Under ideal conditions, a sample should be the exact representation (in smaller) of the population that you wish to probe. The bigger your sample, the more reliable your information. For example:

Three people's responses may give you a skewed impression of how 100 people feel, whereas responses from 25 of those people will yield a more accurate picture.

It can be used for most methods of data collection:

Instead of asking every client, a receptionist asks every 100th client questions to determine if the counselling/referral sessions were effective.

Many different kinds of sampling techniques have been developed and it is important to find the method that is most appropriate for your organization. For all you ever wanted to know about sampling techniques and tools, visit Lancaster University's Statistics Glossary.

Methods of Data Collection

You can combine methods and be creative. Collect data that will have meaning for your organization, and don't forget that you'll need to tabulate your results.

Interviews

• Informal conversations, e.g. At the drop-in centre, in the community garden while people are working.
• Guided interviews, where your questions are fixed (this helps you collect comparable information, especially important if you have different people conducting interviews), e.g. What are your plans for the future? Do you plan to take any kind of job training?
• Closed, fixed choice questions, where you offer biased statements, and the respondent indicates the level to which they agree or disagree with the statement, e.g. My ethnicity affects how I'm served by local merchants (Strongly Agree, Agree, Disagree, Strongly Disagree).
• Standardized open-ended questions, e.g. Please add any personal comments you'd like to make on the situation of refugees in your community.

Observation
No direct questions, you're just collecting data or stories e.g. checklists or frequency counts done by fieldworker or caseworker, oral histories or anecdotal information.

Documents/Records
Gather the information you need from existing documents such as participant diaries, organizational logs, test scores, literature searches, minutes from meetings, statistics from government departments and reports.

Focus Groups
A sample or portion of participants are brought together for a discussion. You are not building a consensus, you are simply trying to understand the range and depth of opinion on a few choice questions. Make sure you have someone to record what people say.

Surveys
Either written or done over the telephone, surveys are good if you need to ask a larger number of participants for their feedback. Response rate is usually not as high as with face- to-face methods.

Indicators

How you know you're achieving the results you seek

INDICATORS
for New Beginnings Refugee Support Project

The Indicator Test
You could use a very large number of indicators to gather information - enough to require hiring more staff just to collect and summarize data. It is better not to go overboard - choose the indicators that provide the most meaningful information. Indicators should be cost-effective, accurate, and useful to your organization.

Generate a list of candidate indicators for your output, outcome or impact statement(s). Use this checklist to help you decide which candidates work best.

For each indicator statement, ask...

(Number each statement you wish to test, then for each question post the number at the appropriate place on the line.)

1. Accuracy - Does it measure the result? YES NO
2. Is it cost-effective to collect the information? YES NO
3. Can information be gathered without invading privacy? YES NO
4. Does the information help project managers understand how the project is affecting men and women, and/or specific subgroups of people differently? YES NO
5. Does it give useful information with which to make management decisions? YES NO
6. Will the information communicate well to stakeholders (including funding bodies)? YES NO

They've developed quite a few indicators, both qualitative and quantitative, to measure Outputs, Outcomes, and Impact. There are too many to list here, so go to the Appendix to see them.

They still have to choose which ones are best for their organization... for that, they'll need to use "The Indicator Test".

As you test your candidate indicators with these questions you may realize that you can:

1. Improve your indicator - make it clearer to understand, or more specific to the output, outcome or impact you want to measure;
2. Improve your output, outcome or impact statement.

Feel free to do either. Even then, you may not have the best possible indicator. Look for improvements over time as you put them into practice.

Outcome Measurement Framework

Putting Outcome Measurement to work

We have introduced the first eight terms of Outcome Measurement. These help to identify actions and desired results, and to decide how to measure progress toward those results. There is one more term. You now have all you need to complete an Outcome Measurement Framework. This is the tool you use to summarize the design of your project. For a model, see the completed Framework for New Beginnings in the Appendix.

What can you do with this framework?

Communicate Intent
Use your Framework to plan or clarify the project internally, as well as explain your project idea to potential sponsors. A framework can provide a powerful summary of:

• What you intend to DO;
• What you intend to CHANGE;
• WHY the project is important;
• Your resource requirements.

Monitor and Evaluate
Once the project is underway, use your Outcome Measurement Framework to MONITOR your progress. Monitoring, or collecting and tabulating data, is a task that you build into your regular routines. This provides information for continuous improvements or the fuel for modifying future work and establishes a historical marker from which the organization can measure whether things have improved or not. Your Indicators, in particular, help you focus on the information you need.

For each indicator you must determine:

• Where you will get the information;
• What method you will use;
• Who will gather it;
• When will it be gathered.

Sometimes you have Indicators that do not fit well into regular project monitoring; they require periodic and intensive assessment. This is an EVALUATION; it can be done internally, externally or jointly. Evaluations look more comprehensively at the project -examining the day to day monitoring information, changes in the project's context, and progress toward the longer term results.

Modify
Information generated through monitoring and evaluation provides insight at many levels within a project.

• Information may be very important to those directly involved in Activities such as community members or clients. Involving participant or beneficiary groups can enrich their understanding and create additional momentum toward the desired change.
• There is information that can best be used by those carrying out the project - managers, field staff, volunteers. Your examination of the information may lead to modifications in the design of your Activities, changes in the allocation of Inputs, or new strategies to address Constraints and Enablers. Or, you may want to revise part of your Outcome Measurement Framework based on your new understanding of the project.
• At a more general level, there is information about the state of the project overall. This is particularly useful to funding organizations or private donors. Using the Activity, Output, Outcome, and Impact statements from your performance framework, you can write reports that show progress against what was agreed upon and expected.

Getting Started

How to get started on an Outcome Measurement Framework

Outcome Measurement is a Cyclical Process
It's hard to construct an Outcome Measurement Framework in one sitting - it usually happens over several sessions.

Make the most of the different learning styles in your group - there are always some people who like to think "long range" and "big picture" first (Impact), and then work backward to identify the more concrete Activities and Outputs. There will also be people who like to start concretely with Activities, and then work toward the big picture. Accommodate both learning styles by moving back and forth along the "logic chain".

1. Brainstorm
Form a group. Brainstorm the things your organization does and why you're doing them. Consider the questions set out under Situation Assessment on Page 4. Write down all your ideas.

2. Categorize
Try putting the results of your brainstorming exercises into the categories you've become familiar with by reading this handbook. Some groups like to start with Activities because you're most familiar with talking about what you do. Do not dwell here. Get some rough categories down and move on. You'll want to nail down your Impact as soon as possible, so that as an organization, you know where you're heading.

3. Refine
Refine the information in each category, group activities into sets, check that your Outputs flow into Outcomes and Impact, and that all the statements accurately reflect what you're doing and planning to change. List all your Enablers and Constraints.

4. Develop Indicators
You probably already have some indicators. Add more. Brainstorm different ways of collecting the information. Then run all the indicators through the checklist on page 17 and choose the best ones for your organization, or this particular project.

Theory and Practice are Always Different
There are just too many variables for your organization to control them all. Even when you think you have a sound Framework (smooth, logical flow linking Inputs to Impact, accompanying indicators and well anticipated Constraints and Enablers), projects rarely, if ever, unfold as expected. That's okay, Frameworks should be living documents. They should guide your work, but be adaptable as your experience builds and the context continues to change. The degree to which you can alter your framework during a project is a matter between you and your funding organization.

Resources

Website Resources on Outcome Management

Government of Canada

• Treasury Board Secretariat - "Results for Canadians"
• Office of the Auditor General

Outcomes Focus in Canada's Voluntary Sector

• 1999 Panel on Accountability and Governance in the Voluntary Sector, Ed Broadbent (chair)

Outcomes Focused Evaluation

• United Cerebral Palsy, Greater Utica (NY) Area
• University of Ottawa, School of Medicine

Selected Foundations Interested in Outcome Measurement

• Benton Foundation
• Canadian Centre for Philanthropy
• Harvard Business School
• The Centre for Accountability and Performance
• United Way of America
• United Way of Canada
• W.K. Kellogg Foundation

General Planning and Management for Organizations

• Free Management Library

Appendix

Fictitious Settlement Service Outcome Measurement Framework

Outcome Measurement Framework

Organization: 'New Beginnings'
Project Name: Refugee Support Project
Time Frame: 2001 - 2003

Situation (e.g. problem; need; opportunity; mandate; goal)

A political crisis in their homeland has forced many to seek refugee status. Twenty thousand citizens of this country have arrived in Canada over the past six months. Two hundred households are currently living in our region (pop. 500,000). Households are isolated and lack an orientation to services and supports. Language is a barrier for the majority. Community organizations are helping out where they can, but a more intensive and coordinated approach is needed to help newcomer households re-establish themselves. As the principal settlement service provider in the region, New Beginnings is well positioned to respond. Our decision to put forward this project has been made on the basis of consultation with community groups and the informal leadership of the newcomer community.

*or long term outcomes

Appendix

Working with Indicators

Potential Indicators

Organization: 'New Beginnings'
Project Name: Refugee Support Project
Time Frame: 2001 - 2003