Philanthropy and the Challenge of Quantifying Success

If you looked at how many people check books out of libraries these days, you would see failure. Circulation, an obvious measure of success for an institution established to lend books to people, is down. But if you only looked at that figure, you’d miss the fascinating transformation public libraries have undergone in recent years. They’ve taken advantage of grants to become makerspaces, classrooms, research labs for kids, and trusted public spaces in every way possible. Much of the successful funding encouraged creative librarians to experiment and scale when successful, iterating and sharing their learnings with others. If we had focused our funding to increase just the number of books people were borrowing, we would have missed the opportunity to fund and witness these positive changes.

I serve on the boards of the MacArthur Foundation and the Knight Foundation, which have made grants that helped transform our libraries. I’ve also worked over the years with dozens of philanthropists and investors—those who put money into ventures that promise environmental and public health benefits in addition to financial returns. All of us have struggled to measure the effectiveness of grants and investments that seek to benefit the community, the environment, and so forth. My own research interest in the practice of change has converged with the research of those who are trying to quantify this change, and so recently, my colleague Louis Kang and I have begun to analyse the ways in which people are currently measuring impact and perhaps find methods to better measure the impact of these investments.

As we see in the library example, simple metrics often aren’t enough when it comes to quantifying success. They typically are easier to measure, and they’re not unimportant. When it comes to health, for example, iron levels might be important, but anemia isn’t the only metric we care about. Being healthy is about being nourished and thus resilient so that when something does happen, we recover quickly.

Iron levels may be a proxy for this, but they aren’t the proxy. Being happy is even more complicated; it involves health but also more abstract things such as feelings of purpose, belonging to a community, security, and many other things. Similarly, while I believe rigor and best practices are important and support the innovation and thinking going into these metrics when it comes to all types of philanthropy, I think we risk oversimplifying problems and thus having the false sense of clarity that quantitative metrics tend to create.

One of the reasons philanthropists sometimes fail to measure what really matters is that the global political economy primarily seeks what is efficient and scalable. Unfortunately, efficiency and scalability are not the same as a healthy system. In fact, many things that grow quickly and without constraints are far from healthy—consider cancer. Because of our belief in markets, we tend to accept that an economy has to be growing for society to be healthy—but this notion is misguided, particularly when it comes to things we consider social goods. If we examine a complex system like the environment, for instance, we can see that healthy rainforests don’t grow in overall size but rather are extremely resilient, always changing and adapting.

There is more to assessing a complex system than looking at its growth, efficiency, and the handful of other qualities that can be quantified and thus measured.

As biologists know, healthy ecosystems are robust and resilient. They can tolerate reductions in certain species populations … until they can’t. Scholars in ecology and biology have tried to model the robustness and resilience of systems in an effort to understand how to build and maintain such systems. Scientists have tried to apply these models to non-biological systems like the internet and ask questions, such as “How many and which nodes can you remove from the internet before it stops functioning?” These models are different from the mathematics economists use. Instead of relying on aggregate numbers and formulae, they use network models of nodes and links to ponder dynamics among connections in the system, rather than stocks and flows of economies.

Maybe there is something to learn from biologists and ecologists—the people who study the complex and messy real world of nature—when philanthropists are thinking about how to save the planet. We know from ecology and biology, for instance, that monocultures and simple approaches tend to be weak and fragile. The strongest systems are highly diverse and iterate quickly. When the immune system goes to war against a pathogen, the body engages in an arms race of mutations, deploying a diversity of approaches and constant iteration, communication, and coordination. Scientists also are learning that the microbiome, brain, and immune system are more integrated and complex than we ever imagined; they actually understand and tackle the more complex diseases currently beyond our scientific abilities. This research is pushing biology and computational models to a whole new and exciting level.

Many diseases, just like all of the systems that philanthropy tries to address, are complex networks of connected problems that go beyond any one specific pathway or molecule. Obesity is often described as simply a matter of managing one’s calories and consequently cast as a lack of willpower on the part of an overweight individual. But it is probably more accurately understood in the context of a global food system that is incentivized by financial markets to produce low cost, high-calorie, unhealthy, and addictive foods. Calorie counting as the primary way to lose weight has been a rule of thumb, but we are learning that healthy fats are fine while sugar calories cause insulin resistance, which often leads to diabetes and obesity. So solving the obesity problem is going to require much more than increasing or reducing any one single thing like calories. It’s our food system that is unhealthy, and one result is overweight individuals.

In such a complex world, what are we to do? We need respect for plurality and heterogeneity. It’s not that we shouldn’t measure things, but rather that we should measure different things, have different approaches and iterate and adapt. This is how nature builds resilient networks and systems. Because we as a society have an obsession with scale and other common measures of success, researchers and do-gooders have a natural tendency to want to use simple measures (as described in our blog post) and other “gold standards” to gauge the impact of the money spent and effort expended. I would urge us to instead support greater experimentation, smaller projects, more coordination and better communication. We should surely measure indicators of negative effects—blood tests to measure what may be going wrong (or right) with our bodies are very useful for instance.

We also need to consider that every change usually has multiple effects, some positive and others negative. We must constantly look for additional side effects and dynamically adapt whatever we do. Sticking with our obesity example, there is evidence that high fat, low sugar diets, generally known as ketogenic diets, are great for losing weight and preventing diabetes; the improvement can be assessed by measuring one’s blood glucose levels. However, recent studies show that this diet might contribute to thyroid problems and if we adhere to one, we must monitor thyroid function and occasionally take breaks from it.

Coming up with hypotheses about causal relationships, testing them and connecting them to larger complex models of how we think the world works is an important step. In addition, asking whether we are asking the right questions and solving the right problems, rather than prematurely focusing on solutions, is key. Jed Emerson, who pioneered early attempts to monetize the economic value of social impact, makes the same point in his recent book The Purpose of Capital.

For the last 1,300 years, the Ise Shrine in Japan has been ritually rebuilt by craftspeople every 20 years. The lumber mostly comes from the shrine’s forest managed in 200 year time scales as part of a national afforestation plan dating back centuries. The number of people working at Ise Shrine isn’t growing, the shrine isn’t trying to expand its business, and its workers are happy and healthy—the shrine is flourishing. Their primary concern is the resilience of the forest, rivers, and natural environment around the shrine. How would we measure their success and what can we learn from their flourishing as we try to manage our society and our planet?

It is heartening to see impact investors developing evidence-based methods to tackle the complex and critical challenges that face us. It’s also heartening that capital markets and investors are supportive of investing, and in some cases even accepting reduced returns, in an effort to help tackle our big, complex challenges. We must, however, make changes in the way we fund potential solutions so that it supports a diversity of disciplines and approaches. That, in turn will require new methods of measurement and perhaps we can take advantage of some very old ones, such as the data from Shinto priests who have been measuring ice on a lake for 700 years. We must use these all of this to approach our complex challenges taking into account local cultures, unforeseen side effects and resist oversimplification. If we don’t, we risk wasting these funds or, even worse, amplifying existing problems and creating new ones.