Open Policy Analysis

class: center, middle, inverse, title-slide

# Open Policy Analysis
## Development Impact West Africa
### Fernando Hoces, Berkeley Initiative for Transparency in the Social Sciences CEGA
### 13 July 2020 | <a href="https://bitss-opa.github.io/DIWA2020_slides/#1">slides</a>

---

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
 height: 100%;
 background-color: blue;
}
*/
</style>

</style>

# Today's Presentation

### Introductions

### A Framework for Open Policy Analysis

### Case Studies on OPA

### Discussion
---
class: inverse, center, middle
name: introductions

# Introductions

---
background-image: url("Images/BITSSlogo.png"), url(Images/cega.png)
background-size: contain, 200px
background-position: 50% 100% , 0% 100% 
name: about-bitss
# About Us
  
  
### [BITSS](https://bitss.org)

.pull-left[
The Berkeley Initiative for Transparency in the Social Sciences works to improve the credibility of science by 
advancing transparency, reproducibility, rigor, and ethics in research.

### OPA

Aleks Bogdanoski

Fernando Hoces  
 
 Katie Hoeberling

]
.pull-right[ .right[
We are part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)). 
 
 Edward Miguel

Emma Ng

]]
 
---
# About Me

.font140[
- Long time aspiration to contribute to bringing more empirical rigor to policy discussions
 - Admire academic pursuit of rigor and global scope of the endeavor
 - Not so interested in novelty, struggle with academic contempt towards policy analysis
 - Admire relevancy of policy analysis, but struggle with lack of rigor
- Work as a policy economist for 4 years in Chile (Poverty, Education)
- Migrated to the US for family reasons 8 years ago
]

---
# About You
 
.font140[
- In one word: which area do you work on? 
 - Example: health, poverty, labor, energy, etc.
- What policy issue has occupied most of your time lately? 
 - Example: identifying vulnerable populations, providing essential goods or services, building infraestructure, etc.
- What type of analysis did you used, or would have liked to use, to inform the previous policy issue?
 - Example: geografical estimates of poverty, accurate costs of provision of goods and services, context relevant cost-benefits analysis. 
]
---
class: inverse, center, middle
name: framework

# A Framework for Open Policy Analysis

.font170[Based on [Hoces, Grant and Miguel 2020](https://osf.io/preprints/metaarxiv/jnyqh/)]
---
name: motivation

# .font80[Policy Analysis And The Evidence-Based Policy Movement]
.font130[
Evidence-Based movement is growing.  
- The golden age of evidence-based policy (Haskins 2017).  
- Credible causal evidence (Angrist & Pischke, 2010).  
- Transparency and reproducibility of research (Miguel et al. 2014).  
- Commission on Evidence-Based Policymaking (CEBP, 2017)  
]
--

.font130[
Policy Analysis is a fundamental link. 
- As many definitions as textbooks (Dunn, 2015; Weimer & Vining, 2017; Williams, 1971)  
- Common denominator: client-oriented empirical analysis meant to inform a specific policy debate  
- Aspires at scientific rigor. (Wildavsky 1979),  
]
---
background-image: url(Images/policy_agencies.png)
background-size: cover
# Examples of Policy Analysts

---
background-image: url(Images/figure_1_1.png)
background-size: contain
# One Link From Evidence to Policy

---
background-image: url(Images/figure_1_1_black.png)
background-size: contain 
# One Link From Evidence to Policy

---

# Reproducibility Crisis In Empirical Research

.font140[
 
- Large magnitude of publication bias (Franco et al 2014). 
]
--
.font140[
- Evidence of extensive p-hacking across social science disciplines (Gerber et al 2008, Brodeur et al 2016). 
]
--
.font140[
- Replication rates are low (Collaboration et al, 2015 , Camerer et al, 2016, 2018). 
]
--
.font140[
- Computational reproducibility is also low (Stodden et al 2016, Chang and Li 2015, Gertler et al 2018). 
]
---
# The Open Science Movement

.font140[
 
- Definition of principles of Open Science/Research Transparency (Miguel et al 2014)
]
--
.font140[
- Development of guidelines to operationalize principles of Open Science (Nosek et al 2015)
]
--
.font140[
- Journals and funders: Journals (Science + 5k other journals), Registries (AEA), Funders (NIH, NSF and multiple donors)
]
---

# Credibility Crisis Of Policy Analysis
.font140[
 
- Incredible Certitudes (Manski, 2013) 
]
--
.font140[
- Report wars (Wesselink et al, 2013)
]
--
.font140[
- Alternative facts (“The Death of Expertise” Nichols, 2017; “The Death of Truth”, Kakutani 2018; “Post-Truth”, McIntyre 2018)
]
---
# How This Affects The Evidence Based Policy Link?

---
background-image: url(Images/figure_1_1.png)
background-size: contain
count:false

# How This Affects The Evidence Based Policy Link?

---
background-image: url(Images/fig2.png)
background-size: contain 
count:false
# How This Affects The Evidence Based Policy Link?

---

# Relevance

.font140[
 
Main consequences of policy analysis that lacks openness:

- 1 - Cherry picking evidence.
]
--
.font140[
- 2 - Challenging to automate and improve systematically recurring reports.
]
--
.font140[
- 3 - Difficulty understanding how research informs policy analysis.
]
---

# Cherry Picking Evidence
 
.font150[
 <blockquote>
 “When I was director of the CBO, I was very frustrated when we would write a policy report [saying] a certain policy would have these two advantages and these two disadvantages, and the advocates would quote only the part about the advantages, and the opponents would quote only the part about the disadvantages. That encourages the view that there are simple answers. There aren’t generally simple answers. There are trade-offs.”
 
.right[ <cite> Douglas Elmendorf (Director of CBO, 2009-2015)
Harvard Magazine, [2016](https://harvardmagazine.com/2016/09/a-moral-conscience-for-economics))</cite> ]
 </blockquote>
]
---
count: false
# Cherry Picking Evidence
 
.font150[
 <blockquote>
 “When I was director of the CBO, I was very frustrated when we would write a policy report [saying] a certain policy would have these two advantages and these two disadvantages, and the advocates would quote only the part about the advantages, and the opponents would quote only the part about the disadvantages. That encourages the view that there are simple answers. There aren’t generally simple answers. There are trade-offs.”
 
.right[ <cite> Douglas Elmendorf (Director of CBO, 2009-2015)
Harvard Magazine, [2016](https://harvardmagazine.com/2016/09/a-moral-conscience-for-economics))</cite> ]
 </blockquote>
]

---

# Open Science

.font180[

| 	| Empirical Research 	| Policy Analysis 	|
|:---------:	|:-------------------------------------------------------:	|:----------------------------------:	|
| **Problems** 	| Reproducibility Crisis 	| Credibility Crisis 	|
| **Solutions** 	| Open Science Principles, Guidelines, Applications 	| ...	|
]
---

# Open Policy Analysis

.font180[

---

# Principles for Open Policy Analysis
 
## 1 - Open Output

## 2 - Open Analysis

## 3 - Open Materials

---
background-image: url(Images/main_pe.png), url(Images/open_output1.svg)
background-size: 500px, 300px
background-position: 80% 50%, 0% 40%
count:true
# Open Output: One Single Output
---
background-image: url(Images/output-input.gif), url(Images/open_output2.svg)
background-size: 700px, 300px
background-position: 80% 50%, 0% 40%
count:true
# Open Output: Clear Input-Output Link 
---
background-image: url(Images/open_analysis.gif), url(Images/open_analysis.svg)
background-size: 900px, 300px
background-position: 100% 50%, 0% 40%
count:true
# Open Analysis

---
background-image: url(Images/open_materials.gif), url(Images/open_materials.svg)
background-size: 900px, 300px
background-position: 100% 50%, 0% 40%
count:true
# Open Materials 
---
background-image: url(Images/opa_framework.svg)
background-size: contain
count:true
# A Framework for Open Policy Analysis

---
class: inverse, center, middle
name: case_studies

# Case Studies on Open Policy Analysis

---

# \#1 Minimum Wage (US)

.font140[
- 2017
- Proof of concept
- Stackeholders involved: none
- Result: [Dynamic Document](https://rpubs.com/fhoces/dd_cbo_mw), and [Sensitivity Analysis](https://www.rand.org/pubs/rgs_dissertations/RGSD394.html)
- Impact: maybe influenced new policies at mayor policy analysis organization (CBO) ]

---

# \#2 Wealth Tax (US)

.font140[
- 2019
- [First full OPA](https://www.bitss.org/opa/projects/progressive-wealth-tax/)
- Stackeholders involved: original policy analysts
- Impact: praised in social media
]
---

# \#3 Deworming Interventions (Kenya+)

.font140[
- 2020 (August - September)
- Stackeholders involved: original policy analysts and some key policy makers
- Expected impact: will directly support policy decisions in several countries
]
---
# Our Plan for OPA
 
.font140[
- Develop framework to support OPA
- Support transition/adoption of OPA, and develop **case studies**
- Train students and analysts 
- Build a comunity of practice
]

---
background-image: url(Images/aspiration.png)
background-size: contain
count:true
# An Aspiration

---
class: inverse, center, middle
name: framework

# Discussion

---

# Open Discussion

.font140[
- Who are the key providers of policy analysis in your country?

- Is there an example of a previous analysis that you think could have benefited from the OPA framework?

- Can you think of a topic in the near future (up to 2 years) where a policy analysis will be generated and might benefit from the OPA framework?

Feel free to paste references in the chat, or send them to me by email: fhoces@berkeley.edu

]