+ - 0:00:00
Notes for current slide
Notes for next slide

An Open Policy Analysis for Deworming Interventions

BITSS Annual Meeting

Fernando Hoces de la Guardia, BITSS

07 January 2021 | slides

1 / 14

Credits



Core BITSS Team
Aleks Bogdanoski
Fernando Hoces de la Guardia
Katie Hoeberling
Teng Aleksandra Ma
Edward Miguel

Undergraduate RAs
Sophia Bai
Orianna Jia



Keanu Lim
Charles Love
Emma Ng
Ryan Scholes
Jiayu Shi

Evidence Action
Grace Hollister
Grace Morgan

BITSS is part of the Center
for Effective Global Action (CEGA).

1 / 14

Table of Contents


  1. Introduction

  2. Demo

  3. Lessons

1 / 14

Table of Contents


  1. Introduction

  2. Demo

  3. Lessons

1 / 14

Motivation: Rise of Alternative Facts









Senator discussing facts on unemployment insurance

Interviewer: “We I have looked at what economist are saying, and [...] there is no measurable evidence that people are staying at home because of [$600 unemp. insurance]”

Congressman: “[scoffs] I don’t know which economist you are talking about, but ...”

2 / 14

Possible Mechanism: Opaqueness of Policy Analysis

  • Incredible Certitudes (Manski, 2011)

  • Report wars (Wesselink et al, 2013)

  • Low overall credibility of PA → credibility based on reputation ("serious") → assignment of reputation varies across political positions

3 / 14

Possible Mechanism: Opaqueness of Policy Analysis

  • Incredible Certitudes (Manski, 2011)

  • Report wars (Wesselink et al, 2013)

  • Low overall credibility of PA → credibility based on reputation ("serious") → assignment of reputation varies across political positions

  • Parallels to "Reproducibility Crisis" and Open Science reponse (Hoces de la Guardia, Grant, Miguel 2020). Propose:

    • Core principles for Open Policy Analysis (OPA) in outputs, analysis and materials (reproducibility)
    • Agenda to implement and document how OPA can be applied into several policy issues

  • Growing adoption of open science methods into policy analysis

3 / 14

Deworming Interventions

  • Parasitic worm infections are endemic in many countries, disproportionately affecting the poor
  • They interfere with regular bodily processes by decreasing nutrient uptake and can thus lead to serious consequences on human health, education outcomes, and long-term economic well being
  • Mass deworming interventions, at school level, have been propose as a cost-effective approach to tackle this problem
4 / 14

Deworming Interventions

  • Parasitic worm infections are endemic in many countries, disproportionately affecting the poor
  • They interfere with regular bodily processes by decreasing nutrient uptake and can thus lead to serious consequences on human health, education outcomes, and long-term economic well being
  • Mass deworming interventions, at school level, have been propose as a cost-effective approach to tackle this problem

Different settings for deworming

  • Context of original study (Kenya, 1998-99) had very high prevalence rates of worm infections
  • Implementation costs where very low ($0.42 per round of treatment)
  • Length of treatment was relatively short (2.4 years)
  • Current deworming settings have lower prevalnce rates, varying implementation costs and length of treatments
4 / 14

Deworming Interventions

  • Parasitic worm infections are endemic in many countries, disproportionately affecting the poor
  • They interfere with regular bodily processes by decreasing nutrient uptake and can thus lead to serious consequences on human health, education outcomes, and long-term economic well being
  • Mass deworming interventions, at school level, have been propose as a cost-effective approach to tackle this problem

Different settings for deworming

  • Context of original study (Kenya, 1998-99) had very high prevalence rates of worm infections
  • Implementation costs where very low ($0.42 per round of treatment)
  • Length of treatment was relatively short (2.4 years)
  • Current deworming settings have lower prevalnce rates, varying implementation costs and length of treatments

Strong debate around initial results

  • Ozier (2020) summarizes differences between original findings (Miguel and Kremer 2004) and a re-analysis (Aiken et al, 2015). Emphazises the role of communication of results in a reanalysis.
  • This type of debate (result/re-analysis) can be seen in several other topics. For example: minimum wage, inmigration, taxation.
  • OPA need not guarante agreement on key research finding, but should help avoid multiple policy reports
4 / 14

A Framework for Open Policy Analysis

Hoces de la Guardia,
Grant, Miguel (2020)












Contributions of OPA to deworming:

  1. Selected one policy estimate among several alternatives and establish a clear link between it and underlying assumptions
  2. Added documentation to increase reproducibility
  3. Created a public repository with all materials for one-click reproducibility
5 / 14

Table of Contents


  1. Introduction

  2. Demo

  3. Lessons

5 / 14

write notes here

Open Output

Demo


Main features

  • One clear output
    previously agreed in
    consultation with
    policy partner
  • Two additional tabs to modify assumptions
    (key assumptions and all assumptions)
  • Each source is classified into research,
    data, or guesswork
  • High level equations added to illustrate location of components
  • Added feature to modify standard deviations
  • Track values of each component
6 / 14

Open Analysis

Demo


Main features

  • Complete narrative description of the methodology
  • Translation of each narrative step into an equation
  • Implementation of each equation into code
  • Combine all of the above into using a dynamic document (RMarkdown)
  • Presentation of narrative, equations, and code in layered fashion to avoid overwhelming the reader Icon figure
7 / 14

Open Materials

Demo


Main features

  • One-click reproducible
    documentation and app
  • Extensive readme files
  • Clear folder structure
  • Version controlled
  • Open data
  • Acknowledgment to all contributors
8 / 14

Table of Contents


  1. Introduction

  2. Demo

  3. Lessons

8 / 14

write notes here

What happens if one principle fails: Opaque Output

  • Policy estimates of approach 2
    of deworming (Baird et al, 2016)
  • Assume Open Analysis and
    Open Materials
  • Good for researchers
  • Unclear for policy analysts
  • Allows policy makers to
    cherry pick results
  • Does not disclose uncertainty
  • Unclear separation of roles
    between policy analyst and
    policy maker (Truman's request
    for a "one-handed" economist)
9 / 14

Opaque Analysis/Materials


  • Assume open output and open materials
  • In this scenario we can have two policy analyses: one from advocates and one from opponents
  • Each analysis can claim to be open source. Code is available, reproducible, and each as an interactive app.
  • But each analysis can bury in their code analytical choices that move the final policy estimate in their favor
  • Same though exercise can be done with opaque materials
10 / 14

Lessons for future OPA projects: Timeline

  1. Reviewed existing documentation
  2. Review code/spreadsheets and write down missing documentation.
  3. Add equations to reflect any additional clarifying steps.
  4. Translate analysis to code scripts (ideally open source software)
  5. Check for computational reproducibility and consult with original analysts for discrepancies/further questions
  6. Discuss with policy partner on target policy estimate
  7. Draft doodle of app
  8. Incorporate any potential new analysis (repeat steps 1 - 3)
  9. Present draft of OPA (DD and doodle of app) to policy partner
  10. Incorporate comments into documentation of the OPA
  11. Build app based on analysis of dynamic document
  12. Incorporate any additional features requested by policy partner
  13. Verify that all three components produce the same output
  14. Publish v1.0
11 / 14

Useful practices developed during the project


Documentation of analysis and app are connected

  • Each analytic step is declared in the documentation and wrapped in a function
  • All this steps are used to produce the results in the documentation
  • The steps are also stored in a file that is later sourced in the app.
  • If something changes in the report, it will update the app.

Style guide

  • Group objects according to use: only within the documentation, for simulations, and for the app
  • Use diagram trees to depict nested relationships
12 / 14

Lessons for future OPA projects: Costs

  • Costs (approx bandwidth over a year at full time):

    • Principal Investigator 30-50%
    • Research assistant/programmer 100-150%
    • Program Manager 20%-30%
    • Original researcher: 1-5%

  • Not all policy analysis justify this level of effort

  • Characteristics that might justify an OPA:

    • Topics with strong disagreement on the facts among analysts
    • Recurrent reports (eg. ex-ante economic analysis from development banks/agencies)
    • Topics that have large expected welfare effects (eg. tax reform, social cost of carbon)

  • With each new OPA project, templates will emerge and costs will likely fall

13 / 14

Additional Benefits of OPA

Easy to update and reuse

After deworming OPA is released, anybody can modify and improve into a newer version

Clearer connection of how evidence from research is used in policy analysis

Researchers can see clearly where their estimates are being used in a policy analysis. For example, the OPA can be used to justify power calculations of potential new studies.

Connection with forecasting

When there is little information for a parameter used in an OPA, a forecasting exercise can be carried out to elicit expert knowledge (DellaVigna, Pope, Vivalt 2019).

14 / 14

Credits



Core BITSS Team
Aleks Bogdanoski
Fernando Hoces de la Guardia
Katie Hoeberling
Teng Aleksandra Ma
Edward Miguel

Undergraduate RAs
Sophia Bai
Orianna Jia



Keanu Lim
Charles Love
Emma Ng
Ryan Scholes
Jiayu Shi

Evidence Action
Grace Hollister
Grace Morgan

BITSS is part of the Center
for Effective Global Action (CEGA).

1 / 14
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow