Strengthening Transparency in Regulatory Science

R. Samuel Herron1*, Jonathan Klonowski2*, Cassandra Rios2* 1University of Pittsburgh School of Medicine, Department of Pathology, Pittsburgh, PA 2University of Pittsburgh School of Medicine, Department of Developmental Biology, Pittsburgh, PA *All authors contributed equally http://doi.org/10.38126/JSPG170113 Corresponding author: jok119@pitt.edu


I. Background
The scientific community is experiencing a global increase in overall publication output alongside biased data caused by selective interpretations or incorrect analysis (White et. al. 2017;Horgan 2015). Furthermore, there is concern over the open availability of data and a growing body of irreproducible studies threatening public confidence in science (NAS 2018). In direct response to the challenges posed by this replication crisis, the Office of Management and Budget, Executive Office of the President (OMB) released the 2004 Final Information Quality Bulletin for Peer Review detailing the need for stringent peer review processes under the Federal Government (Bolten 2004). This bulletin is cited as the basis for the 2018 rule, proposed by the Environmental Protection Agency (EPA) under Administrator Scott Pruitt, named "Strengthening Transparency in Regulatory Science" (EPA-STRS, 83 FR 18768). EPA claimed that the authority for the rule change is granted under 5 USC §301, which states "an Executive department … may prescribe regulations for the government of his department".
In April 2018, EPA-STRS was submitted to the Federal Register and solicited public comment as mandated by the Administrative Procedures Act (5 USC § 551). In the initial proposal of the rule, EPA sought to resolve the scientific transparency issue by requiring all raw data be available, without exclusion, as a prerequisite for use in the regulation. Further plans were made for these data to be publicly available on an online database. Additionally, EPA noted their intention to revalidate dose-dependent based models of scientific information to corroborate regulatory actions. In response, EPA received over 590,000 comments, many of which criticized the lack of clarity throughout the document, as well as directives contradicting current data protection laws (Reilly 2019;Brugger 2020).
In March 2020, the current EPA Administrator Andrew Wheeler submitted a Supplemental Notice of Proposed Rulemaking (SNPRM) for EPA-STRS, addressing a number of complaints regarding the poorly defined scope, as well as expanding the science covered by the rule to include all data and models used for the creation of regulation(s)(85 FR 15396 § 30.5). The comment period deadline was extended from April 2020 to May 2020 due to the Covid-19 pandemic. Comments pertaining to EPA's authority for this rule -whether to singularly call 5 USC §301 as the authority or in conjunction with additional environmental statutory provisions -were of particular interest to the agency. The SNPRM received 397,000 comments, adding to the nearly 590,000 comments received on the initial 2018 proposal. Once again, an overwhelming proportion of the comments expressed opposition to the rule.
Currently, EPA-STRS stands in the proposal stage from which a finalized rule could be implemented, scrapped, or another SNPRM posted. If the rule is implemented, it is expected to receive legal challenge under the pretext that: (1) all unique comments have not been addressed, (2) the proposal still lacks a viable rationale, (3) the alternative weighting system is arbitrary and unnecessary, (4) the changes do not fall under the authority provided by 5 USC §301, and (5) the rule did not address a need (Departmental Regulations 1958;Bloomer 2020;Bergeson and Campbell, PC 2020).

II. Potential pathways for EPA-STRS
i. Accept the proposal as is Proponents of EPA-STRS note that EPA is within its rights to execute housekeeping authority. This proposed rule addresses the "replication crisis" faced by the scientific community as independent validation "will lead to better outcomes, and strengthen public confidence in the health and environmental protections underpinning EPA's regulatory actions" (85 FR 15396). Considering the breadth of data and models used in the development of significant EPA regulations, it is crucial to independently validate key findings and ensure the quality of published information before use. Furthermore, EPA's proposed steps toward transparency are an agenda shared with experts who state that methods used to obtain and analyze data, as well as availability of the data used, are critical factors to reproducibility in science (Miłkowski, Hensel, and Hohol 2018).
After the initial comment period, critics opposed to EPA-STRS expressed concerns regarding the infringement of data protected by the Freedom of Information Act Exemption 4 (FOIA, 5 USC § 552), designed to protect privileged information, and Title II of the Health Insurance Portability & Accountability Act of 1996 (HIPPA 69 FR 78719), that outlines a person's right to confidentiality. However, EPA addressed these matters by proposing a tiered access system, thereby protecting personal data and other sensitive information. Such a system has already been implemented at the Research Data Center (RDC), thus serving as a possible scaffold for EPA's venture into storage of publicly available data (NIH 2018).
ii. Reject the proposal As presently proposed, the decision-making surrounding implementation of peer-review and potential exceptions to EPA-STRS rests solely in the hands of the Director of EPA (Allison and Fineberg 2020). Critics have noted that this structure not only precludes checks and balances but also that the directorship is a politically appointed position which does not require the appointee to have any scientific background (Oreskes 2019;Mcintyre 2020). As such, this allows for EPA to circumvent scientific assessment regarding data usage due to this lack of structure and selective implementation of data standards and expertise. Furthermore, as currently defined this system leaves the fate of many viable studies, which no longer have accessible data sets due to their age being ambiguous (Allison and Fineberg 2020).
While ostensibly designed to address issues stemming from biases in scientific work, the proposal is structured only in regard to computational reproducibility, and fails to recognize or address other aspects of scientific work that contribute to biases, such as issues stemming from the study design, assumptions taken, or the merit of the conclusions drawn from the work (Allison and Fineberg 2020;Ioannidis 2018). All scientific studies suffer from limitations inherent in their design and scope (Lilford et al. 2001). An appropriate EPA-STRS proposal would address multiple aspects of scientific bias, something the currently proposed EPA-STRS fails to do.
In addition to only addressing a limited scope of scientific bias, the current structure of EPA-STRS presents a potential new source of bias and politicization that undercuts the stated goals of the proposal. Commenters expressed concern with the meaning and methods of independent verification of previous work. Furthermore, EPA-STRS does not offer a publicly transparent methodology for independent verification to avoid bias (Allison and Fineberg 2020). While EPA has taken steps to clarify the meaning of the terminology "independent validation," the proposal does not provide criteria for selection or vetting the entities responsible for independent validation (85 FR 15396).
Likewise, the concept that EPA-STRS increases oversight by enhancing the peer review process is an erroneous claim, as EPA-STRS does not substantively add to preexisting peer review criteria. EPA-STRS articulates peer review guidelines consistent with the 2004 "Final Information Quality Bulletin for Peer Review" (Bolten 2004). The proposal does not set forth any specifics for enhancing this peer review process -leaving broad discretion to the agency with regards to the extent of peer review needed -as well as the identity of the reviewer (85 FR 15396). Such guidelines fail to adequately address potential misuses of the peer review process that could arise from insufficient peer review validation (as defined by consensus in each scientific field) (Nature Neuroscience 1999). Nor does it address concerns over potential for abuse of the peer review process arising from insufficient oversight during reviewer selection. Therefore, EPA-STRS fails to properly implement a regimented peer review pipeline fortified against manipulation or misuse. These issues are unlikely to be resolved as EPA does not have the expertise to unilaterally solve these issues of peer-review.
iii. Alter the proposal The original EPA-STRS rule proposal (83 FR 18768) stunned people of the scientific community and beyond (see submitted comments to EPA-STRS; Allison and Fineberg 2020;Brugger 2020;Friedman 2019;Reilly 2020). A synthesis of the feedback suggests that, at best, the proposal was ill-conceived, and, at worst, it was a blatant attack on the scientific underpinning of the agency's environmental protections. Scientists are well aware of reproducibility and accessibility issues within the scientific profession, as well as the potential impacts of these issues on society. In response, the community has been adamantly working to overcome such challenges (Baker 2015;Berry 2016;Nature 2016;Nature 2014). Despite this, it is reasonable to implement additional governmental safeguards to both ensure accuracy and maintain public trust when developing regulations.
While the March 18, 2020 supplemental revision (85 FR 15396) marked improvements over the original, if implemented as is, the rule would only exacerbate issues it is claiming to tackle. Currently, EPA-STRS suffers from two fatal flaws, both of which can be fixed with diligent revisions. First (1), as written the rule can be exploited to undermine regulations by arbitrarily restricting scientific studies such as landmark historical studies. In the 2020 supplemental changes, EPA attempted to address criticism (1) by revising § 30.9 to include an opportunity for the exception of data and models released before the effective date of the current rule (85 FR 15396). Despite this amendment, the language states that exceptions are only granted on a case by case basis by the Administrator, leaving all decisions to a politically appointed official. Changing this process to allow for appropriate scientific experts to make the assessment can easily solve this issue.
Second (2), the vagueness surrounding the terms "independent peer review" and "independent validation," leaves a lack of clarity that can be exploited. Once again, EPA began to address the procedural and syntactical vagueness of the initial proposal by replacing the terms "replicate" and "reproducible" with "reanalyze" (Section 1C), as defined by the National Academy of Science Workshop Report (2016). Moreover, definitions for "independent peer review" and "independent validation" were adopted from the "Final Information Quality Bulletin for Peer Review" document released by OMB (Bolten 2004). These updates contributed clarity to the conceptual process by which EPA intends to evaluate studies. However, these improvements only began to address the issue, as there remains no description for how the review process would be executed. Further detailing the process will be an important step in placating critics.

III. Recommendation: Alter the proposal
We recommend that the proposal be further developed for reasons outlined in the section "Alter the Proposal." By addressing issues raised above and further developing the rule, it can truly tackle the issues it claims to be addressing.
First, EPA should take steps to build EPA-STRS in a way that removes science-dependent decisionmaking powers from political appointees, instead bestowing that power onto scientific experts. As recorded by a poll by the Pew Research Center, the general public trusts scientists at a higher rate than any other professional sector (see Figure 1, Funk et al. 2019). Therefore, the appointment of scientific experts to address scientific issues within the EPA would be a logical action, one that would bring the additional byproduct of building sustainable public trust with the rule, the regulatory process, and EPA as a whole. Specifically, we suggest that the rule could promote open data while addressing historical changes in best practices by granting an automatic exemption to all EPA funded data and models released FOUR (4) years prior to the date of final rule implementation -for research cited in current regulation -and FIVE (5) years prior to the conception of each new regulation. This is based on EPA's own guidelines for retention of data (5 years), minus one year for current rules to allow for a transition period for recent rules/studies. Further, we suggest that these exemptions could be revoked, on a case by case basis, but only as deemed by a panel of appropriate scientific experts. This panel would be tasked with the assessment of "historical" data and/or models that may come into question, thus leaving the decision-making power to qualified experts. Finally, to stimulate public trust, any revocation of an exception, and the accompanying reasoning, should be made public to promote an open and transparent regulatory process. Another pertinent step is clarifying the problematic vagueness in the rule proposal. Namely, EPA must detail a mechanism by which "independent peerreview" would be conducted. We suggest that EPA partner with the National Institutes of Health (NIH) for this purpose. The NIH has a wealth of experience and expertise in the peer-review processes and could support developing both the conceptual underpinnings for enhanced peer-review processes and the procedure itself. Even though EPA has narrowed the scope of their intended "peer-review" to "reanalysis" in the SNPRM for EPA-STRS, a more meaningful and holistic approach could be developed with the NIH. In addition, the NIH's knowledge should be leveraged to construct a non-overlapping, robust, transparent, and conflict of interest free peer-review process.
Our recommendations take a constructive approach toward EPA-STRS, recognizing the merit in its goal to both confirm the conclusions of scientific studies for