Using Lagged Outcomes to Evaluate Bias in Value-Added Models / Raj Chetty, John N. Friedman, Jonah Rockoff.

By:

Chetty, Raj

Contributor(s):

Material type: Text

TextSeries: Working Paper Series (National Bureau of Economic Research) ; no. w21961.Publication details: Cambridge, Mass. National Bureau of Economic Research 2016.Description: 1 online resource: illustrations (black and white)Subject(s):

Online resources:

Available additional physical forms:

Hardcopy version available to institutional subscribers

Abstract: Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today's inputs cannot influence yesterday's outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.

Tags from this library: No tags from this library for this title. Log in to add tags.

Average rating: 0.0 (0 votes)

Holdings
Item type	Home library	Collection	Call number	Status	Date due	Barcode	Item holds
Working Paper	Biblioteca Digital	Colección NBER	nber w21961 (Browse shelf(Opens below))	Not For Loan

Total holds: 0

Collection: Colección NBER Close shelf browser (Hides shelf browser)

Previous	No cover image available	No cover image available	No cover image available	No cover image available	No cover image available	No cover image available	No cover image available	Next
Previous	nber w21959 The Bright Side of Patents /	nber w2196 Privatization, Information and Incentives /	nber w21960 A Jury of Her Peers: The Impact of the First Female Jurors on Criminal Convictions /	nber w21961 Using Lagged Outcomes to Evaluate Bias in Value-Added Models /	nber w21962 Partners in Crime: Schools, Neighborhoods and the Formation of Criminal Networks /	nber w21963 The Limits of Meritocracy: Screening Bureaucrats Under Imperfect Verifiability /	nber w21964 Patents and Innovation in Economic History /	Next

February 2016.

Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today's inputs cannot influence yesterday's outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.