Complexity in the Non-Profit Sector: Evaluation

Enormous amounts are invested by foundations each year into the non-profit sector. The traditional model of evaluation of foundation-funded programs is a very linear one:

inputs=>outputs=>outcomes=>impact

 

Human systems are inherently complex and require a different approach –a complex adaptive systems (CAS) approach. Foundation-funded interventions and programs do not take place in a vacuum but in a highly interconnected human ecosystem where one action can have multiple effects, can be amplified or be dampened, and can cascade into large events for the better or for worse. With a CAS approach, evaluation could become more than just an ineffective, backward-looking/post-mortem tool, it could also serve as a pro-active, forward-looking method for the assessment of future investment opportunities to maximize impact: what interventions can we design and/or fund that will have the most impact in a cost-effective manner? If success is appropriately defined and the parameters of success are better understood, more effort can be directed toward factors that improve the odds of success. The application ofcomplexity science to the issue of evaluation in the non-profit sector has the potential to be a game changer.

A simple example, to provide the beginning of a framework, is the Earned Income Tax Credit (EITC) program, the adoption of which has been supported by multiple foundations, such as the Annie E. Casey Foundation. In principle, the success of the program can be assessed by looking at the numbers of people enrolled each year and the amount of money they represent. Following this metric, the diagram below from theTax Policy Center, a joint-venture between the Urban Institute and the Brookings Institution, suggests that the program has been incredibly successful.

Photo Courtesy of Tax Policy Center

 

Other metrics have been used to assess the program’s reach (a 2006 report by EITC expert Steve Holt provides a great overview), but they all tend to be focused on numbers of people participating (including number of people in different socio-economic-ethnic categories) and amounts of money received.

It all sounds very one-dimensional. Does this money really go back into the community, and if so, how? Do the children of EITC families benefit? Does EITC really improve asset building over time? What happens to all the families that enroll only once, never to return? The simplistic diagram below illustrates what a systemic view of the EITC ecosystem may look like, with positive and negative synergies among the many different ecosystem components and stakeholders. That is the whole point: the EITC can have as many, if not more, unintended than intended consequences due to all these constituent units interacting. It raises many more questions than it answers and I view it as a starting point for thinking about evaluating a program like EITC by taking the entire ecosystem in which it operates into account. If the ultimate objective of the EITC is poverty alleviation, or, even more narrowly, asset development, we have to take many, many other things into account. The aggregate result is emergent.

(click for larger view)

 

A more recent, and equally insightful, report by Steve Holt suggests that the success of EITC goes beyond the simple quantitative metrics. But the evidence is more anecdotal than systematic. Even some of the quantitative metrics point to a more holistic view, for example, “having a tax refund directly deposited to a financial institution account”, which may be the beginning of asset development and savings. But data needs to be collected over time across multiple touchpoints to see if that is really the beginning of asset development or just an illusion. A CAS model will help determine what data needs to be collected to understand how the various factors contribute to the ultimate objectives, not just proxies that may or may not, in the end, reflect those objectives. Lastly, synergies among various components of the system have been discovered over time mostly by accident, and probably at a great cost. A systemic model can help uncover synergies in silico and dramatically accelerate the costly and lengthy trial-and-error process. Identifying leverage points means we can also determine where to invest, or how to design a more effective ecosystem.

Want to read more about agent-based modeling? 

Download our free e-paper!

 

index.php

Icosystem Corporation • 222 Third Street, Suite 0142, Cambridge, MA 02142 • Voice: (617) 520 1000 • Fax: (617) 492 1505

Terms of UsePrivacy Policy