Archive for the ‘Evaluation’ Category

The costs of evaluation

April 30, 2008

A grantor-grantee consortium known as ProjectStreamline.org has just issued a very smart report called “Drowning in Paperwork, Distracted from Purpose.” As its title suggests, the report offers a skeptical take on the current craze for evaluation while pointing out the day-to-day costs of seeking grant support. Those costs are the reason the Nonprofiteer urges her clients to spend their fundraising time soliciting individual gifts instead of writing grants. But the report also makes clear that grantseeking could be a much lower-cost activity without depriving grantmakers of information they require to judge success or its likelihood.

For one thing, foundations could adopt a uniform grant application. It’s not clear why (other than ego) previous efforts to standardize application forms have ended up producing nothing but additional variation. Surely there’s someone in the grantmaking world with enough clout to push through this simple change, which makes up in utility what it lacks in sex appeal.

The report’s most provocative observation is that grantors’ evaluation demands of grantees are the philanthropic equivalent of outsourcing–that is, securing services from underpaid and unrepresented workers. Not only are grantors better positioned than grantees to assess outcomes by the simple fact of their seeing more of them, they’re uniquely positioned–being that it’s their money–to determine what constitutes success. By all means bring grantees to the table in deciding whether “success” is, e.g., having victims of domestic violence leave their homes or remain in their homes, and by all means require grantees to document in what proportions their clients are leaving or remaining. But any more detailed analysis of consequences is properly the work of people who have resources with which to assess them rather than those who–up to their ass in alligators–are trying to remember that the objective is to drain the swamp.

Saddest but truest in the report? The observation that charities get tempted to do things irrelevant to their mission if there’s a big grant available for doing so. The good news is that charities can cure this condition all by themselves, regardless of what the grantors do. Stop letting the fundraising tail wag the operating dog. Do what you do, and find people who want to support that.

Which leads us back to where we started: in the not-very-long run, individual giving is the better deal.

There’s nonsense, and then there’s damned nonsense

March 24, 2008

Here’s a truly thoughtless piece from David Brooks (not in itself an especially newsworthy phenomenon) about the "new" trend in nonprofits: social entrepreneurship!  Note that the only consideration he gives to the role of the commonwealth in determining the common good is to shudder at the prospect:

There’s obviously a
danger in getting government involved with these entrepreneurs.
Government agencies are natural interferers, averse to remorseless
competition and quick policy shifts.

The danger is in getting government involved with providing for the welfare of its citizens?  And the solution is having private individuals determine what public-welfare programs work best and should be supported?  That must be because privatizing government functions has worked so well in Iraq, and at the State Department (where contract employees are reading your passport files), and in the public schools that have been captured by for-profit companies.

Their problem now is scalability. How do the social entrepreneurs
replicate successful programs so that they can be big enough to make a
national difference?

America Forward, a consortium of these
entrepreneurs, wants government to do domestic policy in a new way. It
wants Washington to expand national service (to produce more social
entrepreneurs) and to create a network of semipublic social investment
funds….to invest in
community-run programs that produce proven results. The government
would not operate these social welfare programs, but it would, in
essence, create a network of semipublic Gates Foundations that would
pick winners based on stiff competition.

If the problem is replicating successful programs so they can be big enough to make a national difference we could use, uh, what’s that called?  Public funding?  But that would mean all these wealthy people would have to pay taxes.  And worse, that would mean all these brilliant people–and they must be brilliant, right?  Otherwise they wouldn’t be wealthy, because the market allocates rewards perfectly and life is completely fair–would have to consult something other than their own attitudes, prejudices and needs before deciding what’s best for the rest of us.

 

Division of labor, multiplication of results

February 6, 2008

Flying, as ever, in the face of conventional cut-your-expenses wisdom, the Nonprofiteer wonders why all nonprofits don’t adopt the bifurcated leadership model common in the arts: an Artistic Director to lead program, a Managing Director to handle resource acquisition and allocation.   

Wouldn’t social service agencies operate better with someone at the helm whose expertise was effective service to clients and someone at the rudder whose expertise was squeezing every dime til it shrieked?  These are not identical skills–they’re not even complementary–and for charities to insist on combining them into a unitary Executive Director means one part of what they need done will almost inevitably be done badly.

And if donors are serious about wanting to see rigorous metrics of charities’ effectiveness, they’ll recognize that it takes two: one leader to innovate, experiment and rethink client services and another to measure, evaluate and assess the results. 

H/T The Smithsonian

January 30, 2008

Responding to a report anatomizing revenue-generation schemes and their discontents at the Smithsonian, the Institution’s acting Secretary offered this morsel for thought to everyone who imagines that all our sector needs is a good stiff dose of for-profit goals and systems:

“We clearly need to restructure the Smithsonian Business Ventures,” Mr.
Samper said Monday in a phone interview. “We want to make sure that
decisions are not only driven by profit but also by the mission.”

Reporting back: Counting results, and which results count

January 29, 2008

A letter accompanying the annual report from the US Fund for UNICEF manages in the space of three paragraphs to get its appeal both absolutely right and completely wrong. 

Right: "For the first time in history, the death rate for children under 5 from preventable causes has dropped below 10 million a year to 9.7 million."  ["Here's what we do, and we're winning!"]

Wrong: "The first year of our campaign was incredibly successful, netting more than $3 million towards our goal of $6.5 million by 2010."  ["Here's how we pay for it, and we're not even halfway there yet."]

The error?  Putting the money in paragraph 1 and the children in paragraph 2.

Being halfway through a 4-year campaign in a single year is great; but that’s not the kind of progress donors care about.  Success in raising money is just a means to the end of saving children’s lives.  So saving children’s lives (probably complete with boldface) is what belongs in the lead.

Why isn’t it?  For two reasons, probably neither of them conscious:

1) UNICEF saves lives; the US Fund for UNICEF raises money.  If you work for the fundraising group, your goal is precisely to get from $3 million to $6.5 million, and that’s what you think about day-in day-out under enormous pressure with scarce resources, and when you look back over the year you have reason to congratulate yourself on how you’re managing it.  It’s hard to remember (probably even hard to acknowledge) that for the most part your audience doesn’t care about what you do–it cares about what other people do with the money you’ve worked so brutally hard to extract.  Still, any good fundraiser–and these people are obviously top-notch–bears in mind that the reader’s priorities trump the writer’s every time.  (To which end, all fundraising writers should have a sign over their desks: "So what?")

2) [The Nonprofiteer speculates] Recent attention to the issue of evaluation/assessment/measurement among nonprofits has created a demand for information that will always outstrip the supply.  That is, we’ve succeeded in producing among donors the constant nagging feeling that if they really knew what was going on, they’d feel disappointed/ripped off/ill-served/justified in not giving.  (See Charity Governance’s clear-eyed piece about the Red Cross for an account of the consequences of that success.)  So the hardest of hard numbers–300,000 saved children’s lives–raises more questions than it answers.  Is 300K out of 10M a good "save" percentage?  Can you maintain that momentum, or were you just picking low-hanging fruit?  And what was your cost per life?  Can you reduce that? 

In Geneva and New York and among scholars of international poverty reduction, these are serious questions whose serious answers are being sought; among fundraisers in Chicago and Kansas City and Denver, they’re just a cloud of black flies between you and your prospective donors.  Whereas reports of dollars appear complete unto themselves (inevitable consequence of a capitalist upbringing).  If you say, "We have more than $3 million," no one will ask, "Why aren’t you using the yuan?" 

Naturally you’d rather put the solid, the unquestionable, the unchallengeable, in Paragraph 1.  But take it from the Nonprofiteer: unless you use that lead paragraph to brag about saving 300,000 children’s lives, there won’t be any donors to question and challenge you.

In other words: Damn the detailed metrics.  Full speed ahead!

Foundation Friday: They’re Kidding, Right?

January 18, 2008

GiveWell has disciplined the second of its two co-founders for misrepresenting himself on the Internet, but the New York Times reports that the Hewlett Foundation remains open to the possibility of making a grant to the charity evaluation group.

Mr. Karnofsky said in mid-December that the Hewlett Foundation had
expressed interest in putting money into the organization, and Eric
Brown, a spokesman for Hewlett, said Monday that it continued to be
interested.

“We’re going to wait and see,” Mr. Brown said, “but we think the concept has a lot of value.”

"The concept."  Would that be the concept that it might be relevant to funding decisions if a group devoted to judging charities based on transparency and management excellence proves itself deceptive and ill-managed?  Or the concept that someone might be concerned about entrusting charitable reputations to an organization whose self-promotion included anonymous attacks on the reputations of its charitable competitors?

 

Who’s offsetting whom?

January 11, 2008

A reminder that the nonprofit world is not, in fact, a world unto itself: the FTC is looking into assertions about carbon offsets advertised by manufacturers and sold by for-profits and and nonprofits alike

This is a reason for skeptics like the Nonprofiteer to be grateful for the notion that for-profit companies can be significant sources of social good: as long as there’s profit (and assuming we elect a Democratic President before this gang can finish repealing the New Deal), there will be federal regulation.  This may not assure that much social good gets done, but it will minimize the amount of out-and-out lying and stealing that takes place.

Of metrics and mission, and a modest proposal

January 7, 2008

The cries for evaluation and measurement in the nonprofit sector are, if anything, more deafening now that the much-ballyhooed Givewell.net evaluation site/system has embarrassed itself with deceptive marketing practices.  (Brief summary here; if you want more, there are discussions on GiftHub, on the blog of GiveWell Board member Lucy Bernholz, on PhilanTopic, and of course at MetaFilter itself.)  Apparently transparency is as transparency does, or not.
 

Here’s the Nonprofiteer’s idea about measurement: that every nonprofit agency spend an hour writing a mission statement (stop groaning) in the following form:

    We do [activity] so that [result will occur].

One of these for Starbuck’s might read:

    We make coffee, so that communities have a place in which to gather.

Someone deciding whether or not to invest in Starbuck’s would then have two items of information which could be evaluated independently.  They might think, "What the world needs is a good $2 cup of coffee.  It won’t lead to community, but who gives a damn?" and so they’d invest.  Or they might think, "Coffee generates communities, and what the world needs is a good $2 community" and so they’d invest.  Or they might think, "Coffee could generate communities, and if it doesn’t at least it won’t damage communities," and so they’d invest.  Or they might think, "The most important thing in the world is community, and nothing destroys community like an infestation of corporate capitalism," and so they’d take their money elsewhere.

In the nonprofit world, you’ll get to do the same thing.  "The YMCA provides athletic opportunities for neighborhood residents, so they’ll be safe from violence."  Donors might think, "Athletic opportunities really are a terrific way to take kids off the
street, where they might perpetrate or be victims of violence," and
invest because they believe in process and result alike.  Or they might think, "Well, athletic opportunities will reduce obesity, and I’m in favor of that, so I’ll support them even though I doubt they reduce violence in any way"–that is, invest because they believe process will produce a different, but also good, result.  Or, "Athletics don’t seem to have much to do with reducing violence, but it couldn’t hurt and the goal is worthy enough to gamble that the people who think it works know something I don’t"–indifference about process coupled with enthusiasm for result and faith in actors=investment.  Or donors might think, "Athletics just teach kids socially-approved means of violence, but the YMCA building is certainly a safe house in a tough neighborhood" (doubt about process, support of result=investment).  Or "Athletics teaches kids to value the wrong things, and the time they spend in the Y would be better spent in a local library," and direct your money there (rejection of process, support of result=investment elsewhere).

Say the words "mission statement" to your staff and there will be instant weeping and wailing and gnashing of teeth, presumably because creating one is generally as difficult and unrewarding as having root canal.  So DON’T say those words or (god forbid) "visioning;" just get your staff and Board together and for one hour–60 minutes!  No more!  If you can do it in 15, so much the better!–compel the group to fill in the blanks:

    We do X [activity] so that Y [result] will occur.

    We conduct educational campaigns so the public will demand legislation outlawing smoking.

    We purchase and serve meals to poor people so they won’t be hungry enough to steal from their neighbors, that is, us.

    We commission and perform new plays so we can keep alive a sense that it’s important to experience things in other people’s company and not just alone at our computers.

    We research cures for cancer so people will stop dying of it.

Note: "so that" NOT "because".  If you use the word "because" you’ll end up with statements like, "We produce plays BECAUSE drama is important," that is, ideas at a level of generality too great to be useful.  "Because" means "why?" which is one of those huge, cloud-shape questions.  You want "So that,"  which means "to create X result".  Let your donors decide why that result matters, or doesn’t.

Why bother?  Because (the Nonprofiteer fearlessly predicts) people will give you money more often.  EITHER they’ll think your activity is valuable in and of itself (and will pay for it); OR they’ll agree with your reasoning that your activity will lead to your outcome and they like both (and will pay for it); OR they’ll be so gung-ho about your outcome that they’ll be willing to gamble that your activity will actually produce it.

Someone–Woody Allen?–said "Reality is a collective hunch."  Your goal is to make your hunch the reality–and all our talk about "evaluation" and "assessment" and "effectiveness" and "efficiency" boils down to making a reasonable case that your hunch–about activity, outcome, or both–is a good one.

And that’s something you can do without spending a lot of time answering questionnaires from researchers who imagine they’ve figured out a single answer to the question, "What should all charities do, and how?" 

The best answer to that question should be provided by every charity to every prospective donor.  It should be the headline on your Website:

    "We do BLANK so that BLANK will occur.  Won’t you help us make that happen?"

Every charity able to fill in those two blanks will be able to fill in its coffers, too.

Dear Nonprofiteer, I’m making a list and checking it twice

December 6, 2007

Dear Nonprofiteer,

I stumbled across you while looking for some
experts in the field of non-profit organizations. As someone with experience in
this arena, I would greatly appreciate your feedback regarding a site that I
recently developed, www.Vote4Cause.org . The concept is simple. Visitors vote
for their favorite non-profit organizations. These votes generate ad revenue.
100% of this ad revenue is then distributed to the top vote receiving
organizations.

I have currently populated the system with several
non-profits that I am familiar with. However, I am happy to add any
organization that can confirm its non-profit status. It could be an interesting
marketing tool for smaller non-profits to get their message out. Additionally,
even if the cause that you are trying to support doesn’t receive the
purse at the end of the voting session, you will know that the money that you
generated will ultimately be going to a good cause.

I recently graduated from Cal and I am in
the process of applying to medical school. I created this site in my spare time
because I felt like this model would benefit a lot of great organizations that
exist today. At some point when the site has grown, I would like to cover the
costs associated with hosting this site, but for now I am happy giving all of
the revenue away as a donation.

Sincerely, Secret Santa

Dear Santa,

I have the same problem with your list of endorsed or approved charities as with the one the Better Business Bureau published in the November 12 New York Times, namely, no one really knows what goes on inside charities until it’s too late.

Want proof?  Here’s the story of the latest upheaval at the Red Cross–which appears large as life and twice as natural (and predictably enough) in the BBB’s Times ad.  And (ever the curmudgeon) the Nonprofiteer scanned the BBB list looking for the Red Cross and shook her head sagely when she found it–and that was 36 hours before the news of its most recent scandal.  Is she psychic?  No, just cynical.

At the same time–and in apparent contradiction–some people know more about what goes on in charities than others.  So asking random visitors to your site to direct your (or your advertisers’) charity with their votes based on prejudice, ignorance, hearsay ("UNICEF hates Jews!") is a lot like asking some stranger to take money out of your pocket to make paper airplanes.  Ditto for permitting random nonprofits to solicit through your site based on nothing more than proof of their tax status.  The IRS has awarded 501c3s to lots of agencies that do nothing, or do more harm than good, or do less good for their clients than their employees–or, as one of my colleagues got in lots of hot water for saying, "Some nonprofits just suck!"

So is the Nonprofiteer saying "Don’t bother"?  "Don’t give money away" or, conversely, "It doesn’t matter where you give your money?"  Not at all.  It’s just that any list of endorsed charities needs to come accompanied with a statement about the basis for the endorsement, with answers to two pairs of basic questions:

  • What do they do?  Are you sure?  (Does the Humane Society provide services to animals?  Does the Cancer Society provide treatment?  The fact that the answer to these questions is "no" doesn’t demonstrate that the agencies don’t deserve support–advocacy and research are essential, too; but people are entitled to know what they’re paying for.)  If the Red Cross spends–COMPLETELY MADE-UP NUMBERS; JUST FOR ILLUSTRATIVE PURPOSES–50% of its money on first-aid training and 50% of its money on disaster relief, people who only want to supply blankets should know that before they write the check.  And if the Red Cross spends 10% of its money–LIKEWISE COMPLETELY MADE-UP NUMBER–on public relations to manage its scandals, maybe what it does with the other 90% of the treasury makes that worthwhile; but again, people are entitled to know.
  • Is that worth doing?  Why do you think so?  (Sample response to this question: "Providing bednets prevents malaria").  You don’t have to argue that it’s the single best way to prevent malaria, or the only way to prevent malaria, or that malaria is the very worst possible thing that could happen to a person in the developing world.  You simply need to draw a straight line between what’s being done and an indisputable benefit, such as preserving health.   

I urge you to take the time to answer these two pairs of questions for each of the charities on your site.  That way you’ll be educating people who will someday reach not just for their mouses (mice?) but for their wallets when the subject of charity is mentioned.  An additional benefit: you’ll be really clear about why you donate to the charities you do, which is another way of being really clear about the good you want to accomplish in the world.

(The Nonprofiteer’s own list?:

The United Nations Population Fund.  The Fund provides reproductive health services (including contraception and surgery for obstetric fistula) to women in the developing world.  Women shouldn’t have to sacrifice their lives or their health to have babies.

The Chicago Abortion Fund.  The Fund subsidizes abortions for poor women.  Women shouldn’t have to sacrifice their lives or their health to have babies.

Various theater and dance companies in Chicago.  Beauty makes life worthwhile.)

Bad data, or none?

November 21, 2007

At a phone-in press conference last Thursday, Clara Miller of the Nonprofit Finance Fund and Robert Otterhoff of Guidestar cautioned donors against using a charity’s Form 990 to determine whether it’s worthy of support.  By way of commenting on proposed revisions to the 990, Miller suggested that the IRS and everyone else:

  • Eliminate the emphasis on overhead expenses: focus on a charity’s outcomes, not its inputs.  Because administrative and fundraising expenses are assigned differently by different charities (not to say completely randomly), they’re neither revealing nor a good basis for comparison.
  • Eliminate the requirement to report salaries about $100,000, which suggests that paying a charity executive that much is per-se profligate
  • Acknowledge that larger organizations tend to have lower administrative costs, but that growing organizations should have high expenses because they’€™re investing.

Miller urged donors to stop treating the 990 as gospel, noting that it "penalizes nonprofits for having reserves, paying people adequately, and investing in systems."  And Ottenhoff, whose agency posts 990s online, nonetheless suggested that donors begin their investigations not with the 990 but with an  agency’s mission and programs.

No dispute here that the 990s offer a distorted and distorting picture of nonprofit operations.  But nature abhors a vacuum, and donors are a force of nature.  They’re not
going to stop using the "wrong" information until we’re willing to give
them a replacement.  And neither Miller nor Ottenhoff had one to suggest. 

It won’t do just to ridicule donors’ "obsession" with overhead
expenses, and to compare it unfavorably to the forms of evaluation used
in the for-profit world: "A businessperson would ask, ‘What do you
care what the overhead is if we’re delivering to our customers?’"  But donors mostly have no way of knowing whether, or what, nonprofits are
delivering to their customers.  We ask about cost because if the whole thing is a waste, we want it at least not to be much of one. 

How should donors secure information about outcomes to replace the 990 data about inputs?  Miller was vague: it would be useful to have data on "what did organizations actually do, capacity metrics or production metrics"–but those don’t belong in the 990.

Where can donors could go for information about a charity’s effectiveness?  Ottenhoff mentioned Guidestar, of course, and the Better Business Bureau, but ended up recommending a charity’s own Website–not because of what’s likely to be there, but because of what ought to be there:

An organization should be able to say how many people have we served, what was the benefit we provided–did they learn to read, did they get fed?–let’s quantify the benefits we’ve provided to the world.  Organizations should be able to connect the dots between mission and money with good numbers.  Let’s look at the bigger picture of what they’re trying to accomplish.  Let’s say they have a $1 million budget and employ 12 people, what is the output of that investment and what are you as a donor enabling them to do?

That’s the question, all right.  But don’t tell people to ask it if you can’t help them answer it.

——————–

The Nonprofiteer will take the long weekend to celebrate Thanksgiving.  Publication resumes Monday, November 26.  Happy holidays!


Follow

Get every new post delivered to your Inbox.

Join 41 other followers