Journal of Science Policy & Governance
​Volume 22, Issue 01 | March 6, 2022

Technology Assessment: The Performance Gap of Policy Information Systems: A Knowledge Infrastructure Assessment Framework

Eduardo Carrillo (1), Catalina Frigerio (2), María Jesús Valenzuela (3), Alessia Aquaro (4), Jean-Christophe Mauduit (4), Ine Steenmans (4), María Paz Sandoval (5)​
  1. Asociación TEDIC, Asunción, Paraguay
  2. Corporación Wikimedia Chile, Santiago, Chile 
  3. Fundación Paz Ciudadana, Santiago, Chile
  4. University College London, Department of Science, Technology, Engineering and Public Policy, London, United Kingdom 
  5. ​University College London, Department of Computer Science, London, United Kingdom

Corresponding author: [email protected] ​
Keywords: knowledge information systems; knowledge infrastructure; next generation repositories; digital platform assessment; policy instrument ​
https://doi.org/10.38126/JSPG220105 ​

Executive Summary

Digital technologies’ development and their ubiquity has significantly changed the way information is collected and shared. These changes are also observed in the ways that knowledge and information used to design and implement Science, Technology, and Innovation (STI) policies are increasingly made accessible via digital platforms. Lacking, however, are evaluation frameworks to measure the performance and effectiveness of public information systems used for STI policy work. This limits what we know about what aspects work for whom, when, and why. As a response to this gap limiting our collective capacity for improving their utility, this paper presents an assessment framework tool for STI policy-focused digital knowledge platforms. Our proposition is informed by theoretical lessons from the areas of work on Knowledge Infrastructures (KIs) and Next Generation Repositories (NGRs), and practical experiences from policy professionals working in STI policy domains. The tool’s architecture is structured around three interdependent thematic pillars of performance in the production, aggregation, distribution, and maintenance of knowledge in digital information platforms: the communities pillar, the technical systems pillar, and the sustainability pillar. To test the design and utility of our proposed evaluation framework, we applied it on a digital platform of STI policy instruments maintained by the United Nations Educational, Scientific, and Cultural Organization (UNESCO), the Global Observatory for Science, Technology, and Innovation Policy (GO-SPIN) platform. We conclude with reflections on future areas for evaluation framework development.

-Read the full article through download.-

Background header image courtesy of Emeritus
Eduardo Carrillo. Earned a Bachelor’s in International Relations from the American University (Paraguay) and a Master’s in Public Administration with specialization in Digital Technology and Policy from University College London (UCL). He has over nine years of work experience in civil society and international organizations. As a researcher in science, technology, innovation, and digitization of public policies, he is interested in the economic and social development of Paraguay. He currently works at the digital rights organization TEDIC, based in Paraguay.

Catalina Frigerio. Earned a Bachelor’s degree in Legal and Social Sciences from Universidad de Chile, a European Master’s in Law and Economics from the Erasmus Mundus Programme, and a Master’s in Public Administration with a Specialization in Digital Technology and Policy from University College London (UCL). She is a lawyer specialized in technology and privacy and currently works as a Public Policy Manager for the Chilean chapter of the Wikimedia Foundation. Her research focuses on the intersection between technology, public policies, and regulation.

María Paz Sandoval. Currently a Cybersecurity and Policy PhD student at the Centre for Doctoral Training in Cybersecurity at University College London (UCL) and a Master’s in Public Administration student with a specialization in Digital Technologies and Policy at UCL. She earned a Bachelor’s in Socioeconomics, a Diploma in Public Policies, and has 14 years of experience in public, private, and civil society institutions. She also currently works as a Public Affairs, Digital Technologies, and Policies consultant.

María Jesús Valenzuela. Studied social anthropology at Universidad de Chile and earned a Master’s in Public Administration with a specialization in Development Policies, Technology, and Innovation from University College London. She specializes in issues regarding sustainable development goal 16 on peace and justice access with a focus on criminal justice, security, and human rights. Currently works as a researcher in the Justice, Reinsertion and Organised Crime Area at Fundación Paz Ciudadana.

Alessia Aquaro. Has a Bachelor’s degree in social and political science from the University of York and a Master’s in Public Administration with a specialization in digital technologies and public policy from the University College London (UCL). She works as a policy advisor for the UK Government in digital and telecoms policy.

Dr Jean-Christophe Mauduit. Having earned a B.Sc. (Physics), an M.Sc. and Ph.D. (Astronomy & Astrophysics) as well as an M.A. (International Relations), Dr. Mauduit is currently a Lecturer in Science Diplomacy at University College London in the Department of Science, Technology, Engineering, and Public Policy. 

Dr Ine Steenmans. Has a Master’s of Engineering in Civil and Environmental Engineering from Cambridge University, an MSc in International Planning from the Bartlett School at UCL, and an EPSRC-funded Engineering Doctorate (EngD) from the UCL Centre for Urban Sustainability and Resilience. Currently, she is an Associate Professor in Futures, Analysis, and Policy at the University College London in the Department of Science, Technology, Engineering, and Public Policy.

References

  1. ​Borgman, Christine L. 2012. “The Conundrum of Sharing Research Data.” Journal of the American Society for Information Science and Technology 63, no. 6: 1059-78. https://doi.org/10.1002/asi.22634. 
  2. Borgman, Christine L., Peter T. Darch, Irene V. Pasquetto, and Morgan F. Wofford. 2020. “Our Knowledge of Knowledge Infrastructures: Lessons Learned and Future Directions”, June. https://escholarship.org/uc/item/9rm6b7d4. 
  3. Borgman, Christine L., Paul N. Edwards, Steven J. Jackson, Melissa K. Chalmers, Geoffrey C. Bowker, David Ribes, Matt Burton, and Scout Calvert. 2013. “Knowledge Infrastructures: Intellectual Frameworks and Research Challenges.” (May). https://escholarship.org/uc/item/2mt6j2mh. 
  4. Borgman, Christine L., Andrea Scharnhorst, and Milena S. Golshan. 2018. “Digital Data Archives as Knowledge Infrastructures: Mediating Data Sharing and Reuse.” (February). https://doi.org/10.1002/asi.24172. 
  5. Cash, David W., William C. Clark, Frank Alcock, Nancy M. Dickson, Noelle Eckley, David H. Guston, Jill Jäger, and Ronald B. Mitchell. 2003. “Knowledge Systems for Sustainable Development”. Proceedings of the National Academy of Sciences 100, no. 14: 8086–91. https://doi.org/10.1073/pnas.1231332100. 
  6. Denzin, Norman K. 1978. The Research Act: A Theoretical Introduction to Sociological Methods. 2nd ed. New York: McGraw Hill. 
  7. Karasti, Helena, Florence Millerand, Christine M. Hine, and Geoffrey C. Bowker. 2016. “Knowledge Infrastructures: Part I”. Science & Technology Studies 29, no. 1: 2–12. https://doi.org/10.23987/sts.55406.
  8. Lin, Dawei, Jonathan Crabtree, Ingrid Dillo, Robert R. Downs, Rorie Edmunds, David Giaretta, Marisa De Giusti, et al. 2020. “The TRUST Principles for Digital Repositories”. Scientific Data 7, no. 1 (May): 144. https://doi.org/10.1038/s41597-020-0486-7. 
  9. Mulgan, Geoff. 2020. “A world dominated by platform companies lacks good platforms for the SDGs: so what can be done?” geof mulgan (blog), March 9, 2020. https://www.geoffmulgan.com/post/the-vitalmissing-support-that-s-needed-for-the-sdgs-an d-how-it-could-be-provided. 
  10. Pasquetto, Irene. 2018. “PhD Dissertation - From Open Data to Knowledge Production: Biomedical Data Sharing and Unpredictable Data Reuses”. https://escholarship.org/uc/item/1sx7v77r. 
  11. Paunović, Katarina. 2008. “Data, Information, Knowledge”. In Encyclopedia of Public Health, edited by Wilhelm Kirch, 203–7. Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-1-4020-5614-7_685. 
  12. Rodrigues, Eloy, Andrea Bollini, Alberto Cabezas, Donatella Castelli, Les Carr, Leslie Chan, Chuck Humphrey, et al. 2017. “Next Generation Repositories: Behaviours and Technical Recommendations of the COAR Next Generation Repositories Working Group”, (November). https://doi.org/10.5281/zenodo.1215014. 
  13. Rogers, Patricia. 2014. “Methodological Briefs- Theory of Change”. https://www.unicef-irc.org/publications/pdf/b rief_2_theoryofchange_eng.pdf. 
  14. Schatz, Bruce R. 1991. “Building an Electronic Community System”. Journal of Management Information Systems 8, no. 3: 87–107. https://doi.org/10.1080/07421222.1991.1151 7931. 
  15. Shearer, Kathleen, Eloy Rodrigues, Paul Walk, and Pandelis Perakakis. 2016. “Next Generation Repositories”. (October 25). https://doi.org/10.5281/zenodo.163264. 
  16. Silva, Renan Gonçalves Leonel da, Roger Chammas, and Hillegonda Maria Dutilh Novaes. 2021. “Rethinking Approaches of Science, Technology, and Innovation in Healthcare during the COVID-19 Pandemic: The Challenge of Translating Knowledge Infrastructures to Public Needs”. Health Research Policy and Systems 19, no. 1: 104. https://doi.org/10.1186/s12961-021-00760-8. 
  17. Star, Susan Leigh, and Karen Ruhleder. 1996. “Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces”. Information Systems Research 7, no. 1: 111–34. https://doi.org/10.1287/isre.7.1.111. 
  18. Stemler, Steve. 2001. “An Overview of Content Analysis”. Practical Assessment, Research, and Evaluation 7, no 1. https://doi.org/10.7275/z6fm-2e34. 
  19. Walk, Paul. 2018. “#COAR2017 What Does the Next Generation Repository Look Like? By Paul Walk.” Video, 18:03. From COAR 2017 Annual Meeting, posted by "COAR e.V.," September 23, 2018. https://www.youtube.com/watch?v=U9QsGyeJ mmo&t=243s. 
  20. Walsh, P. P., E. Murphy, and D. Horan. 2020. “The Role of Science, Technology and Innovation in the UN 2030 Agenda”. Technological Forecasting and Social Change 154 (May): 119957. https://doi.org/10.1016/j.techfore.2020.11995 7. 
  21. Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship”. Scientific Data 3, no. 1: 160018. https://doi.org/10.1038/sdata.2016.18.

DISCLAIMER: The findings and conclusions published herein are solely attributed to the author and not necessarily endorsed or adopted by the Journal of Science Policy and Governance. Articles are distributed in compliance with copyright and trademark agreements.

ISSN 2372-2193
© 2022 Journal of Science Policy & Governance, Inc. All rights reserved. The opinions, findings and conclusions from JSPG publications, additional article commentaries and related events do not necessarily reflect the views of the journal.