Written by: M. A. Saeed, PhD and Paul Eder, PhD
The government is modernizing in the way it thinks about and uses data. This modernization has been in part spurred by the signing of The Foundations for Evidence-Based Policy Making Act (“Evidence Act”) in January 2019. When President Biden took the office, his administration fully supported the Evidence Act, which stresses advancing data and evidence-building functions in the Federal Government. Evidence is broadly defined and includes foundational fact-finding, performance measurement, policy analysis, and program evaluation.
The Evidence Act requires Federal agencies to institute Evaluation Officer (EO) and Chief Data Officer positions, form a multi-year learning agenda and an Annual Evaluation Plan, and complete a capacity assessment. Think of learning agendas as “evidence-building plans” with their first iteration due from agencies to Congress in early 2022.
On June 30, 2021, the Office of Management and Budget (OMB) published a new memo M-21-27 to provide guidance on Evidence Act. The memo reaffirms and expands on previous OMB guidance by asking agencies to design and coordinate evidence-building, data management and data access functions strategically to ensure integrated and direct connection with the need for evidence. OMB emphasizes that efforts to implement the Evidence Act should not be treated as a compliance exercise, and that agencies should develop the Learning Agenda, Annual Evaluation Plan, and Capacity Assessment for Statistics, Evaluation, Research and Analysis to align with overall organizational missions and strategies.
According to the memo, the strategy of evidence building is comprised of Learning Agendas and Annual Evaluation Plans. In building Learning Agenda, the agency will:
- List questions to develop evidence to support policy making;
- Outline the data that needs to be collected, used or acquired to facilitate the use of evidence in policy making;
- Detail all the methods and analytical approaches that will be used to develop evidence to support policy making.
Once Learning Agendas and Annual Evaluation Plan are developed agencies should use them to execute the identified evidence-building activities. An Evidence building culture will be established through multi-year approaches to evidence generation. Similarly, the Annual Evaluation Plan should be executed and lead to progress that builds upon the prior year’s evidence.
To address their unique needs, agencies have the flexibility to design their learning agendas in any format. However, we recommend agencies adopt five principles from the emerging field of data science to support their agenda:
- Knowledge management: Each agency has a huge amount of structured and unstructured data which can serve as evidence, but the data are scattered across different branches and offices. Establishing a well-documented data center or data lake (centralized cloud repository) will help stakeholders to access the evidence easily and use it for evaluation purposes.
- Identifying knowledge gaps: Although data may be able to be identified, it may be difficult to integrate in a coherent way. Accordingly, agency experts must ask key questions to identify linking fields between datasets. In addition, agencies must determine which data sets are readily usable, which need to be transformed, and which must be defined and collected for the first time. The benefits of any new data collection must be weighed against any projected new costs incurred.
- Hypothesis-driven data collection: As government agencies start stitching their existing evidence together through proper linkages, the gaps of their knowledge will be revealed. Agencies should initiate pilot action plans that generate evidence based on actionable hypotheses and research questions. A hypothesis-driven action plan will produce a coherent landscape of knowledge that will lead to evidence-based decisions.
- Data-driven hypothesis formulation: In additional to formal hypotheses, agencies may want to investigate what insights exist in the data itself. While not as “scientific,” when data are combined in new ways, expert analysts can unlock insights that had not been previously investigated. Accordingly, data should be structured in a way that allows for data sets to be integrated and investigated through common linking fields. Until actionable meaningful insights have been mined, time and cost-intensive efforts may be viewed as useless. But once insights are achieved, there can be an exponential return-on-investment.
- Technical transformation: In the modern age, every government agency should aspire to run with state-of-the-art technology capabilities. Agencies should modernize the technology functions which will help them to achieve evidence-driven policy making. In addition, they should address the knowledge gaps among their staff to ensure they have the capability to gain insights from advanced and emerging tools.
At a high level, these five principles can build the foundation for an effective learning agenda. The American people have an increasing desire to see data drive government effectiveness. Accordingly, Congress and the executive branch have shifted to an orientation of making decisions based on actionable evidence. Agencies must adapt by using emerging scientific methods and evidence-building to drive policies to unlock the promise of the government’s data assets.
