���˶���

LSC grantees in more than 10 states use similar outcomes measure systems developed by Interest on Lawyers’ Trust Accounts (IOLTA) programs, bar foundations  or similar state funders (or “IOLTA" funders).

These IOLTA funders require grantees to collect and report similar types of data, which include outcomes and outputs data as well other quantitative, qualitative, and narrative data. These funders use these data as part of their grantmaking and grant oversight processes.

Grantees have participated in the design and implementation of these data collection and reporting systems to ensure that they were useful for grantees and not unduly burdensome.

���˶��� of Virginia (LSCV) implemented an outcomes reporting system in 1998. All of its grantees—including six LSC grantees—now use this system. See the sections below to learn about how the LSCV data reporting system and how one of these grantees, Blue Ridge Legal Services (BRLS), uses the system.

• Why LSCV and BRLS Use Outcomes Measures
• How LSCV and Virginia’s Legal Services Programs Developed the LSCV Outcomes Data System
• Major Outcomes Categories and Metrics
• Components of the Outcomes Reporting System
• Examples of the Ways BRLS and LSCV Use Outcomes Measures
• Lessons Learned ���˶��� Using Outcomes Data In Virginia

Why LSCV and BRLS Use Outcomes Measures

LSCV engaged BRLS and its other grantees in the development and implementation of its outcome measures system to ensure that the data collected was useful for both LSCV and the grantees.

Outcomes data are collected only for extended service cases (LSC CSR case closing codes F-L), not for limited services cases (LSC CSR case closing codes A and B).

Although LSCV does not make funding decisions based on outcomes data, those data have enabled LSCV to enhance its grantmaking decisions and grant oversight functions, better ensure grant funds are used effectively and efficiently, and successfully demonstrate the value and benefits of state funding of legal services.

For example, identifying and documenting the results of grantees’ work improves:

• Grantees’ ability to formulate more concrete and measurable objectives in their funding proposals
• The capacity of LSCV and grantees to assess programs’ success in accomplishing these objectives
• Assessments of the alignment of grantees’ advocacy work and their objectives and priorities
• The identification of areas for program improvement

LSCV also uses outcomes data to improve its ability to show the impact legal services programs have in the community and to demonstrate the value and cost-effectiveness of legal services programs’ work to the public and the Virginia General Assembly. This is essential since the General Assembly funds legal services filing fees and general appropriations. LSCV used outcomes data in its publications, including the "" and its recent report, "."

BRLS has used the LSCV outcomes data system since it was first implemented in 1998. It has found that its outcome metrics are increasingly valuable for both external and internal purposes, because outcomes data enable the program to better “tell the story” of the benefits of its work much more powerfully than outputs data alone.

BRLS reports that outcome measures help improve its client and program services by:

• Helping it better align its advocacy work with its strategic goals and priorities
• Generating reports that show the accomplishments of entire program, each office, and each case handler
• Assess the results of advocacy strategies and identify approaches to improve the benefits the program provides clients and the client community
• Informing the board’s decision-making by profiling the focus and accomplishments of the program’s work

More detailed information about the range of reports with outcomes data that LSCV and BRLS produce and assessments of the value of these data are in "Components of the outcomes reporting system" and "Specific examples of the ways BRLS and LSCV use outcomes measures."

It should be emphasized that outcomes data alone do not meet LSCV and BRLS' data needs. Both rely on the combination of a wide range of measures and datasets. Both also use data such as CSR data and numbers of persons data, Census data and data from other federal and state agencies regarding the client’s population characteristics (e.g., geographic location, economic and demographic data, education and employment status)and financial and staffing data. Equally important is BRLS’ on-going engagement with client and community groups, surveys and interviews with staff and community members, and GPS or similar data to align services with the needs of the client community.

How LSCV and Virginia's Legal Aid Programs Developed the Outcomes Measures

LSCV partnered with its grantees to develop the LSCV outcomes system. LSCV initiated what ultimately became a year-long process in 1997. Throughout this process, LSCV and its grantees worked with the staff of , which facilitated this process and played the lead role in designing a system that LSCV and its grantees agreed would be most effective. The Resource had previously worked with New York’s Interest on Lawyers’ Accounts (IOLA) program to develop an outcomes system for its grantees. At the same time that it was working with LSCV, it was working with the  to develop a similar outcomes system in that state. The Resource subsequently worked with other states to develop similar systems.

The development process began with a meeting that included the LSCV executive director, the executive directors of Virginia’s legal aid programs, and the director of The Resource. At the meeting, the group reviewed the outcome categories used in the New York IOLA system and discussed the benefits and shortcomings of using a similar system.  Over the next several months, Virginia's program directors and LSCV staff developed suggestions for ways the New York IOLA system could be modified to best meet the needs of LSCV and the Virginia programs. LSCV then worked with The Resource staff to develop a system that incorporated these elements.

During this period, management of the individual programs engaged their staff in a process to identify how the system would be most useful and efficient.

In addition to developing the substantive content of the system (see "Major outcomes categories and metrics" and "Components of the outcomes reporting system") the partners also had to identify the case management system (CMS) or other data systems that would be most effective in capturing the data. ���˶��� two years after the implementation of the outcomes system, LSCV purchased the same CMS for all of its grantees. The CMS had all of the capacities needed for the effective and efficient collection, analysis, and reporting of the outcomes system.

Major Outcomes Categories and Metrics Used by Virginia Legal Service Programs

The LSCV outcomes data system used by Virginia’s legal aid programs has four major outcomes data categories. Outcomes data are collected only for extended service cases (LSC CSR case closing codes F-L), not for limited services cases (LSC CSR case closing codes A and B).

Outcomes Category 1: Major benefits from direct legal representation of individuals

The system collects outcomes for cases closed in 12 areas that correspond to the LSC CSR legal problem categories (e.g., Consumer/Finance, Education, Employment, Family, Housing). The number of possible outcomes in these categories varies.

For example, the Consumer/Finance category has 20 different possible outcomes. Two examples of the outcomes in this category are: “Obtained federal bankruptcy protection,” and “Stopped or reduced debt collection activity.” The family category has 32 possible categories. Outcomes in this category include: “Obtained or maintained custody of children,” “Obtained protection from domestic violence,” and “Obtained, preserved, or increased child support.” Information is also provided for the “Number of persons directly affected” for each of these outcomes. See in each of the twelve categories.

Outcomes Category 2: Direct dollar benefits for clients

These are for “affirmative dollar awards” to clients, not cost savings from judgments of payments avoided, which are reported separately (see next bullet). Affirmative benefits are reported for the amounts of “lump sum awards/settlements” and “monthly benefits.” Specific financial benefits categories include, but are not limited to, “Social Security, SSI,” “Child Support,” and “Affirmative consumer judgments.” In addition, the “Other Benefits” category can include “Insurance settlements” and “Housing allowances.” See .

Outcomes Category 3: Dollar savings for clients

These outcomes are for “amount of dollar savings achieved for clients through judgments or payments avoided,” such as “Defensive Consumer Law Matters” (e.g., bankruptcy, garnishment). See .

Outcomes Category 4: Benefits from direct legal representation of groups

Only one outcome is captured in this area: “Obtained incorporation/tax exempt status.” All other data collected in this area are for outputs (e.g., “Obtained assistance with regulatory/licensing issues,” “Obtained assistance with other structural or governance issues"). See .

Blue Ridge Legal Services uses the LSCV outcomes data categories and supplements them with the results of its ongoing surveys of clients served by the program. It uses three separate surveys:

• One survey 
• Another
• And a

There are English and Spanish language versions of all of these surveys. The surveys are emailed to clients who provide an email address; others receive the survey via U.S. mail (with a BRLS-addressed, postage page envelope). Each survey is coded with the case number to match the client’s responses with other case data.

Components of Virginia Legal Services Programs' Outcomes Reporting System

The case management system that LSCV purchased for its grantees provides three specific key functionalities. The CMS:

• Enables case handlers to input required outcomes data about the case at the time of closing
• Generates basic reports for the grantees’ use
• Allows the grantee to seamlessly report the information to LSCV

At case closing, BRLS advocates complete . Once advocates enter the outcomes and other data into the CMS, the program can seamlessly transfer the necessary data into an Excel file to submit outcomes data (and other data) data to LSCV as well as to “slice and dice” the data in a variety of ways to produce reports for its own purposes.

Specific Examples of the Ways BRLS and LSCV Use Outcome Measures

BRLS' use of outcomes data

BRLS uses outcomes data—along with outputs and other data—to produce reports to assess and improve various aspects of its operations. Examples of these reports are profiled below.

• To assess and evaluate its client services, BRLS generates reports that show:
  • Program-wide client services data for all substantive law areas.
  • Client services data .
  • Client services data .
  • Outcomes based on clients’ demographic characteristics.
  • “Success rates” data: Whether the goals the clients and advocates identified at the time of opening the case were achieved.
• The outcomes data are key components of the quarterly reports to the BRLS board of directors about the program’s activities and accomplishments.

• Reports are also generated . When BRLS first initiated the use of outcomes data, some staff were concerned that the data would be used inappropriately for performance evaluations. However, staff now embrace the use of these data. Having information that more fully shows the results of their work has increased their morale and has also led them to take on cases that provide the greatest benefits to clients. They are not “cherry picking” cases that can yield high “success rates.” Instead, they are addressing more complicated issues and increasing the number of extended service cases that have greater impact on the lives of their clients.

• BRLS considers outcomes data critical to its fundraising success in its largely rural service area. There are nine relatively small United Ways in BRLS’ service area and a limited number of small private foundations. These funders want to assess the impact their limited funding has on the community, not just outputs such as cases closed. Outcomes data more fully demonstrate to funders how supporting legal services is a cost-effective investment. The program estimates that the use of outcomes data has significantly increased the funds it receives from the United Ways. For example, in 2013, a United Way allocations panel awarded BRLS twice the amount it had requested because it was impressed by the outcomes BRLS was able to document. For some funding requests, BRLS will go beyond the requirements of the application and include  of their work.

• BRLS has also used outcomes data to help inform difficult decisions to adjust its allocation of resources to ensure the program’s advocacy provides the greatest benefits to its client community. For example, the analysis of case outcomes and information from other sources indicated that certain types of family law cases do not provide the same level of client benefits as cases in other substantive law areas. This led the program to reduce its volume of family law cases while increasing its volume of housing and consumer cases.

LSCV's use of outcomes data

LSCV does not make funding decisions based on outcomes data, but it does use these data along with a range of other data in desk reviews and associated grant oversight activities. LSCV staff reports that outcomes data improve the ability of grantees and LSCV to assess a programs’ success in accomplishing grant goals and to identify strategies to better achieve these goals.

The outcomes data LSCV collects from grantees are integral to LSCV’s and provide the foundation for a comprehensive analysis of the .

LSCV cannot quantify how important its use of outcomes data has been to its success in sustaining and increasing these appropriations. It believes that these data  contributes greatly to LSCV’s efforts to make the “business case” for legal services—and the data may have increased bipartisan support for legal services. Even though it cannot affirm that outcomes have been instrumental in sustaining support in the General Assembly, LSCV  has concluded that it would be remiss if it did not marshal as much data as possible to demonstrate the need for and value of civil legal services for low-income people.

Lessons Learned ���˶��� Using Outcomes Data In Virginia

• You get what you measure. The use of outcomes and other performance measures can have the unintended consequence of driving program services rather than providing a means of evaluating and improving program operations. BRLS management recognized that the measures the program used could significantly shape staffers views of what is and is not important, and influence the type of cases they took and the strategies they pursued. BRLS' executive director John Whitfield has .  

• Outcome measures are necessary, but not the only tools needed to assess effectiveness. Both BRLS and LSCV stress that programs need a wide range of data, not just outcomes data, to assess and enhance program operations and improve client services. Both use outputs data such as CSR data and numbers of persons data; Census data and data from other federal and state agencies regarding the client’s population characteristics (e.g., geographic location, economic and demographic data, education and employment status); and financial and staffing data. BRLS also considers essential the ongoing engagement with client and community groups, surveys and interviews with staff and community members, and GPS or similar data to align services with the needs of the client community need.

• The outcomes data gap for limited services is substantial. BRLS is striving for better outcomes data for cases involving only advice and brief counsel. Conducting the follow-up necessary to obtain sufficient data for these cases could require more time and resources than those that were needed to provide the services in the first place. The program has sought to overcome this limitation by revising its client satisfaction surveys for advice and counsel clients to include:
  1. Whether the advice provided answered their questions
  2. Whether they understood the advice provided them
  3. Whether it was helpful in resolving their legal problem

The staff maintains these data in the CMS along with the other outcomes data, so that it can generate customized reports for a particular funding source. (See Major outcomes categories and metrics and Components of the Outcomes Reporting System.)

• Ongoing evaluation and improvement of the system is essential. Since the system’s establishment, LSCV, BRLS, and Virginia's other legal services programs have engaged in a continuous assessment of the outcomes system to identify ways it is and is not working. Based on its own needs and feedback from grantees, LSCV has “tweaked” its system nearly every year.  And BRLS has modified it reports and generated new reports to fit its needs.

• Many important aspects of the program’s work cannot be quantified. As with other legal services programs, LSCV and BRLS recognize that critical components of a program’s work cannot be meaningfully quantified. Limitations include: the quality of an advocate’s casework; the impact of the program’s work on a client’s sense of well-being (e.g., from avoiding homelessness, escaping from an abusive partner) or empowerment; and the impact of the program’s work with community organizations.

• Every measure is imperfect. LSCV and BRLS recognize that the outcome measures they use reflect assumptions and practices that can limit their validity and value. Questions they ask when assessing what measures to use include:
  1. Are we measuring the right things?
  2. Are the definitions of success appropriate?
  3. Are the measures based on the appropriate indicators?
  4. Are the factors used to calculate the financial benefits valid and appropriate?
  5. Are the advocates accurately reporting the information case outcomes?

As a quality assurance mechanism, when supervising attorneys review the case file at case closing they ensure that the case results information in the case closing memo is consistent with the case outcomes data entered into the CMS.

LSCV and BRLS address these concerns through their ongoing evaluation of the system and their recognition and transparency about methodologies used to calculate outcomes and the limitations of these meth