Friday, March 16 , 2018, 11:50 pm | Overcast 51º


Claudia Armann, Fran Forman and Sara Miller McCune: Measuring Impact and Effectiveness

[Note: Fifth in a series of a community discussion from the 2013 Partnership for Excellence Conference. Click here for a related article from Tina Frontado, Belen Vargas and Emilie Neuman. Click here for a related article from Jon Clark, Erik Talkin and Kim Davis. Click here for a related article from Dean Zatkowsky, Jessica Tade, Sigrid Wright and Allison Bailey. Click here for a related article from Dave Clark, Colette Hadley and Ernesto Paredes. Click here to learn more about this community conversation at the Nonprofit Resource Network.]

Why is it acceptable for a private company to spend years raising and spending money, often operating at a loss, for a long-term future goal (profitability/market share) while the nonprofit sector must deliver its services or products immediately?

How do we define impact in our work? What values does the nonprofit sector reward around effectiveness in service and organizational management? How can we move toward evaluating impact beyond numbers served (quantitative measures)?

What demonstrated models exist for evaluating short and long-term impacts? Where is there common ground or common metrics for funders and nonprofits? What are the opportunities and challenges of considering multisector (business, government, nonprofit) roles in critical local social issues?

In this column, guest authors Fran Forman, executive director of the Community Action Commission of Santa Barbara County, and McCune Foundation executive director Claudia Armann and chairwoman Sara Miller McCune weigh in with a variety of viewpoints on the issue.

                                                                          •        •

Fran Forman on Measuring Impact and Effectiveness

As a nonprofit leader of an organization that serves thousands of people, one of my major concerns is the impact of our work. I am avidly interested in whether the quality of life of our clients is improved — for that is the raison d'être of the Community Action Commission of Santa Barbara County. We recognize that each and every one of the people on whose behalf we organize programs is vulnerable to the vagaries of our economic system.

Fran Forman
Fran Forman

Our clients are poor in a society that assumes all Americans have access to social and economic mobility. Whether we are offering them Head Start, a Healthy Lunch Program for Seniors, youth programming or energy assistance and education, our assumption is that the program will help them right now. We also assume that either the tangible support or personal experience that we are creating will assist them in increasing their confidence so they may access a success trajectory. We believe we are equipping our clients with the ability to successfully navigate an American system that (presumably) rewards educational achievement, hard work, frugality and discipline.

How do we measure the impact of work? How does one measure the impact of an excellent Head Start experience for the 4-year-old son of two agriculture workers when the measurable impact of that may not be obvious for another 12 years? How does one measure the impact of a month of delicious, nutritious meals for a frail, diabetic older man who most likely will not have the wherewithal to prepare those meals for himself? How do we measure the impact on the life of a young person and his or her family who will be assisted to and be the first one in their family to attend college? How do we measure the positive impact in the life of a family whose gas bill was paid right before the service was to be shut off? How do we measure the impact that having a healthy birth after an unplanned pregnancy has on the life of that teen and her newborn?

The impact of work that is done in what is called “the private sector” is easily measured in terms of profit margins and an increase in wealth. However, our work is much less easily measured because it may take years to assess our impact. This fact does not perturb me. I have personally experienced the impact of a decent public school system, community programming and an active, vibrant civic life in my own life. For this reason I have great confidence in the work that I and my wonderful staff of 400 people do each and every day.

Impact in the “private sector” is often assessed by whether there is an increase in wealth or profits for a group of investors or owners. The human service sector, a term I much prefer to the misnomer “nonprofit sector,” exists to meet people’s most basic needs — for nutritional food, for education, for the sense of companionship and community that is critical to societal well-being. These goals cannot be measured by metrics like profitability.

And yet we must measure the impact of our work in order to accumulate the resources so that the work can go on. We need both governmental and “private” support. We need data that will convince well-intentioned citizens to serve on our boards of directors, to help us raise money, to open up avenues of advancement in the workplaces and schools of our community.

I favor collecting data directly from the people on whose behalf we work. I like well-designed client satisfaction surveys for every program in which we are involved. Program participants know best what our income is. What they say can be supplemented by data about economic and social mobility in order to arrive at a comprehensive assessment of the value of our work.

We must resist mightily a simplistic, short-term approach to the assessment of our work.

— Fran Forman is executive director of the Community Action Commission of Santa Barbara County.

                                                                          •        •

Sarah Miller McCune and Claudia Armann on Investing and Evaluating Change that Matters

Real impact and systemic change requires a long-term investment. Twelve months — the timeframe of the typical grant — is woefully insufficient. And often, the community transformation that funders truly crave is not easily measurable; the evidence of change is not always calculated in numbers nor does it fit neatly in columns.

Sara Miller McCune
Sara Miller McCune

And yet, evaluation matters. The definitive textbook on program evaluation in the social sciences, Evaluation: A Systemic Approach — now in its seventh edition — states that “the allocation of scarce resources makes it more essential than ever to evaluate the effectiveness of social interventions.” The authors also contend that the most fundamental reason for evaluation is to inform action — by nonprofits in adapting, expanding or abandoning a program strategy or by funders in choosing to invest or discontinue funding.

Sometimes funders lack patience and expect profound change much sooner than it can be reasonably delivered. According to the book previously referenced, “It can easily take a year or more for a new program to establish facilities, acquire and train staff, make contact with the target population, and develop its services to the desired level. During this period, it may not be realistic to expect much impact on the social conditions that the program is intended to affect.”

In developing outcomes for evaluation, a frank conversation between grantees and funders is necessary so that realistic expectations are established. How honest the conversation is depends on the level of trust in the grantee-funder relationship. Creating a range of outcomes — from initial outcomes, through intermediate outcomes and agreement on long-term outcomes can provide a more flexible framework.

A longer timeframe for evaluation is only one consideration. USC professor Manuel Pastor argues for “developing metrics to capture both quantity and quality, both numbers and nuance, both transactions and transformation.” His solution? A new framework for measuring both transactions and transformations in social movements that mixes “quantitative metrics with the art of story-telling to create epiphanies not just for community leaders, but for foundation grantees.” (Pastor will introduce his model at the next Partnership for Excellence conference on April 17). Making space for nonprofits to share the meaningful indicators that point to the value of their work can be enlightening.

                                                                          •        •

More than a decade ago, local foundations made grants to Future Leaders of America (FLA) to support leadership development and college readiness for Latino teenagers. After a year’s work, the organization could report how many youth had attended a week-long leadership camp and the percentage who reported a change in attitude about their academic plans. These are no doubt tangible and measurable results, but not nearly as meaningful as the stories McCune Foundation representatives hear at our annual site visits with FLA.

Claudia Armann
Claudia Armann

The mother of one participant shared that the program had transformed her timid son into a confident young man now serving an internship for a congresswoman. Another youth told us that the camp he attended a few years prior inspired him to stay in high school; he previously had considered dropping out to support his family after his father was deported. The young man then proudly showed us his college acceptance letter.

Three or four years after the camp, data on the number of youth who graduate high school and enroll in college is available for funders. But fast forward a decade or more and powerful stories of community transformation begin to emerge — graduates of the FLA program are now community leaders — on city councils and school boards, they are school counselors inspiring the next generation of Latino youth, and they are impactful staff members at many key regional organizations throughout the Central Coast.

Two FLA alumni in particular are community organizers in Santa Maria and Ventura and can be credited for mobilizing residents to secure better bus services, a new park in a low-income neighborhood, and increased voter turnout for a statewide tax increase to improve funding for their schools.

At the McCune Foundation, we fund community organizing — mobilizing residents to create systemic change. We not only ask grantees to report the number of participants, but we also inquire about what roles these individuals played in advocacy campaigns, how much training they received, and what responsibilities they were expected to assume after their training. More important, we meet with the emerging community leaders, so we can hear from the youth, the farmworkers, and the bus riders themselves.

Not only do we want to know what progress was made toward policy goals in the grant proposal, but we also want to understand what strategies and partnerships were most effective. At the end of a grant period, a group may have a significant policy win — a new living-wage law or full-time interpreters at local clinics. Most likely, an organization may be able to document less-dramatic indicators of the change to come: 300 Latino parents attending a community’s first bilingual school board candidates forum — an indication of the growing engagement and power of Latino parents.

To demonstrate for the McCune Foundation board the annual return on their investments, staff gleans data from grant reports to document progress in Policy/Neighborhood Wins (the satisfying and tangible outcomes we seek), but also in three areas that show progress toward outcomes and growing community engagement: Raising Awareness, Leadership Development and Grassroots Mobilization.

Depending on their respective mission, each funder can devise the indicators that point toward the transformations they seek. The challenge for funders may come in the need for a deeper level of engagement with grantees, beyond the traditional transactional approach. Even more difficult may be a willingness to invest in promising organizations over an extended period of time, and to also fund the necessary technical assistance and training to grow organizational capacity.

The challenge to nonprofits is to take evaluation seriously. The trend is toward evidence-based giving in philanthropy, so the question is not “Should we evaluate?” but “What’s the best framework for measuring our impact?”

— Sara Miller McCune is chairwoman of the McCune Foundation and Claudia Armann is its executive director.

Related Stories in This Series

» Tina Frontado, Belen Vargas and Emilie Neuman: Nonprofit Overhead and Infrastructure

» Jon Clark, Erik Talkin and Kim Davis: Taking Risks in Pursuit of New Ideas

» Dean Zatkowsky, Jessica Tade, Sigrid Wright and Allison Bailey: Investing in Advertising, Marketing

» Dave Clark, Colette Hadley and Ernesto Paredes: Nonprofit Compensation

  • Ask
  • Vote
  • Investigate
  • Answer

Noozhawk Asks: What’s Your Question?

Welcome to Noozhawk Asks, a new feature in which you ask the questions, you help decide what Noozhawk investigates, and you work with us to find the answers.

Here’s how it works: You share your questions with us in the nearby box. In some cases, we may work with you to find the answers. In others, we may ask you to vote on your top choices to help us narrow the scope. And we’ll be regularly asking you for your feedback on a specific issue or topic.

We also expect to work together with the reader who asked the winning questions to find the answer together. Noozhawk’s objective is to come at questions from a place of curiosity and openness, and we believe a transparent collaboration is the key to achieve it.

The results of our investigation will be published here in this Noozhawk Asks section. Once or twice a month, we plan to do a review of what was asked and answered.

Thanks for asking!

Click here to get started >

Support Noozhawk Today

You are an important ally in our mission to deliver clear, objective, high-quality professional news reporting for Santa Barbara, Goleta and the rest of Santa Barbara County. Join the Hawks Club today to help keep Noozhawk soaring.

We offer four membership levels: $5 a month, $10 a month, $25 a month or $1 a week. Payments can be made through PayPal below, or click here for information on recurring credit-card payments.

Thank you for your vital support.

Reader Comments

Noozhawk is no longer accepting reader comments on our articles. Click here for the announcement. Readers are instead invited to submit letters to the editor by emailing them to [email protected]. Please provide your full name and community, as well as contact information for verification purposes only.

Daily Noozhawk

Subscribe to Noozhawk's A.M. Report, our free e-Bulletin sent out every day at 4:15 a.m. with Noozhawk's top stories, hand-picked by the editors.

Sign Up Now >