USGS has a long history of risk projects that engage the public to enhance science, a method often referred to as “citizen science.” There is a broad spectrum of public engagement and community-based research (e.g., citizen science, crowdsourcing, and prize competitions). “Open Innovation” (OI) has recently become an umbrella term to reference these participatory methods. Such methods are ways to meaningfully engage communities at risk and transform users of risk products into vital contributors. However, the lack of bureau-wide guidance, resources, and policy on OI has led to ad hoc efforts giving rise to concerns and misconceptions about data quality and validity that need to be addressed and dispelled. A comprehensive strategy is needed to provide practical and consistent guidance to direct USGS scientists, managers, and leadership on how to use and evaluate OI methods effectively while meeting USGS science needs. A USGS Open Innovation Strategy is currently being developed that will result in USGS-specific Guidance and Policy as well as a Toolkit and Catalog of USGS OI activities. Supplementary to this strategy, our proposed project will focus on surveying past and potential OI risk projects by engaging USGS representatives from various mission areas, regions, and science support offices. Ethnographic research methods and user-centered design techniques will be employed to co-design and co-produce an OI Playbook for Risk. The goal is to build institutional capacity for using OI methods through bureau-wide engagement that will ultimately increase awareness and effective use of OI methods for future USGS risk research and applications.
The project outcomes are: (1) a catalog of past and potential USGS open innovation projects related to risk; (2) three case studies of OI projects related to earthquakes, climate change, and floods that will also be summarized through graphical and video products; and (3) a USGS Open Innovation Playbook for Risk that packages together the catalog, case studies, and multimedia products, which will be targeted towards the Risk Community of Practice and inform future Risk RFPs. Content and key insights from the Playbook will directly inform and be integrated into the broader USGS guidance and policy on open innovation.
Role in Project
Sophia B Liu
USGS, NE Region, Science and Decisions Center, Innovation Specialist
USGS, Office of Enterprise Information, Physical Scientist, Emeritus
USGS, Water MA, Water Cycle Branch, Branch Chief
USGS, OSQI, Youth in Education in Science, AAAS Fellow
Co-PI, Advisor on EQ & Climate Change
USGS, Rocky Mt Region, National Earthquake Info Center, Seismologist
Advisor on Earthquake (EQ) Case Study
USGS, Land MA, Alaska Climate Adaptation Science Center, Hydrologist
Advisor on Climate Change Case Study
USGS, Water MA, Decision Support Branch, Social Scientist
Advisor on Climate Change Case Study
USGS, Midcontinent Region, Dakota Water Science Center, Statistician
Advisor on Flood Case Study
USGS, Water MA, Extreme Hydrologic Events Coordinator
Advisor on Flood Case Study
USGS has a long history of projects that engage the public to enhance risk research and applications using a method that is often referred to as “citizen science.” For example, the inclusion of public felt reports of earthquakes first appeared in the appendix of a USGS report (Dutton 1889), then through postcards starting in 1926, which eventually led to the development of the Did You Feel It? online questionnaire in 1999. There is a broad spectrum of public engagement and community-based research methods (e.g., citizen science, crowdsourcing, and prize competitions). “Open Innovation” (OI) has recently become an umbrella term to reference these participatory methods (GAO 2016, 2017). Such methods are ways to meaningfully engage communities at risk and transform users of risk products into vital contributors. Open innovation methods have become 21st century engagement techniques that can enable economies of scale and be a force multiplier for conducting smart science to tackle scientific challenges, enable meaningful and democratic engagement with users, and foster scientific literacy that can increase public understanding of risks (NASEM 2018; Nielson 2011; OSTP 2015, 2019; Robinson et al. 2018). As social media, mobile phones, and low-cost sensors have become more ubiquitous, there is growing interest within USGS to harness these emerging technologies and “leverage social media, citizen science, and crowdsourced observations” (p. 35) to develop potential projects like “Hazards in My Backyard” (p. 44) in response to current and potential disasters (Ludwig et al. 2018).
However, the ad hoc uses of OI methods at USGS and lack of bureau-wide guidance, resources, and policy on using OI methods has hindered effective use and methodological advancement of these techniques, and has given rise to concerns and misconceptions about data quality and validity that need to be addressed and dispelled (Crimmins et al. 2017, Herman-Mercer et al. 2018, Lukyanenko et al. 2019, Wiggins et al. 2011). A comprehensive strategy is needed to provide practical and consistent guidance to better direct USGS scientists, managers, and leadership on how to effectively use and evaluate OI methods while meeting USGS science needs. A USGS Open Innovation Strategy is currently being developed that will result in USGS-specific Guidance and Policy as well as a Toolkit and Catalog of USGS OI projects. Supplementary to this strategy, our proposed project will focus on surveying past and potential OI projects related to risk from geophysical, hydrological, and climatological events that impact society. The goal is to build institutional capacity for using OI methods by engaging USGS representatives from various mission areas, regions, and science support offices that will ultimately increase awareness and effective use of OI methods for future USGS risk research and applications. Our aim is to show how OI enhances hazard science by improving the acquisition of data, and the development of risk products and decision support tools for affected communities to better communicate and reduce risk more effectively. OI methods also incorporate social science methods that reimagine how we conduct surveys with the public and engage users by advancing partnerships with the public and other stakeholders through meaningful and effective engagement on how hazards affect their livelihoods.
The project objectives are to: (1) survey past and potential USGS open innovation projects related to risk; (2) conduct three case studies of OI projects related to earthquakes, climate change, and floods; and (3) produce a USGS Open Innovation Playbook for Risk targeted to the Risk Community of Practice using ethnographic methods and user-centered design techniques. In addition to traditional methods for conducting interviews and participant observations in-person, online platforms will also be used to efficiently engage more USGS participants in a cost effective way. Online interviews, forums, and focus groups will be conducted through the USGS Open Innovation Strategy Teams platform. Surveys will be conducted using Microsoft Forms, SurveyMonkey, and/or Mentimeter. An Open Innovation Risk Forum will be organized in conjunction with the June 2020 Risk Meeting to obtain input and feedback on the Playbook with graphical outputs through Graphic/Visual Facilitation. The goal of the survey is to identify a broad range of risk projects that span across the bureau, the variety of OI methods, and the diverse types of participants in these projects. Such participants will expand the typical users identified in Figure 3 of the Risk Plan (Ludwig et al. 2018) by including enthusiasts (e.g., naturalists, techies) as well as underrepresented and vulnerable communities (e.g., indigenous, special needs, and youths). The goal of the case studies is to provide in-depth examples through text, videos, and graphics to better explain the use and value of OI methods for risk projects. A crowdsourcing video production platform will be used to produce short videos for each case study in a more cost effective way with a short turnaround.
For the earthquake case study, USGS Did You Feel It? (DYFI) (Atkinson and Wald 2007, Wald and Dewey 2005, Wald et al. 1999, Wald et al. 2012), USGS Tweet Earthquake Dispatch (TED) (Earle et al. 2010, Earle et al. 2012), Quake-Catcher Network (Cochran et al. 2009a, Cochran et al. 2009b), NetQuakes, Earthquake Detective, and Quest for Quakes Challenge projects will be surveyed for how OI methods can help detect hazards, collect hazard impacts, deploy a network of low-cost sensors, and improve hazard models. This case study will inform the Powell Center Working Group for Earthquake Monitoring and potential risk projects like Watson’s risk project with Hawaiian Volcano Observatory (HVO) on Bridging Local Outreach and Seismic Signal Monitoring (BLOSSM), which proposes to deploy low-cost seismometers called Raspberry Shake (Anthony et al. 2018, Calais et al. 2009) in schools across Hawai‘i to engage youth in science and to increase public awareness of earthquake risk. In-person interviews and participant observations will be conducted at NEIC and HVO to understand their projects.
For the climate change case study, Indigenous Observation Network (ION) (Herman-Mercer et al. 2016, 2018, 2019; Toohey et al. 2016; Wilson et al. 2018), Local Environmental Observer Network (Brubaker et al. 2013, Mosites et al. 2018), ISeeChange (Drapkin et al. 2016, Drapkin 2018), IceWatch USA, EyeOnWater, Globe Observer Land Cover (Hayden et al. 2019, Janney 2019), and Climate Resilience Data Challenge projects will be surveyed to understand how community-based monitoring, engagement with indigenous communities, Traditional Ecological Knowledge, data collected online versus on-the-ground, and ground truthing remote sensing data can inform research on climate impacts. This case study will focus on the ION project by conducting online and in-person interviews at the Alaska Climate Adaptation Science Center (AKCASC) and Yukon River Inter-Tribal Watershed Council (YRITWC) in Anchorage, AK. Participant observations will also be conducted by observing and video recording how USGS scientists engage with indigenous communities during their fieldwork in the summer. This study will also inform Watson’s proposed OI risk projects related to climate change with the Pacific Islands Climate Adaptation Science Center (PICASC) and Northern Rocky Mountain Science Center (NOROCK).
For the flood case study, CrowdHydrology (Fienen and Lowry 2012, Lowry and Fienen 2013, Lowry et al. 2019), CrowdWater (Seibert et al. 2019, Strobl et al. 2019), What’s Your Water Level?, iFlood - Flood Reports, and Lowering the Cost of Continuous Streamflow Monitoring projects will be surveyed to understand how OI can be used to understand floods for hydrological risk reduction (Paul et al. 2018), measure water levels and high water marks, monitor hydrologic changes, detect flooding, improve flood and hydraulic models, as well as identify and deploy low-cost sensors. This case study will inform proposed OI risk projects like Ryberg’s “Historical Floods - Stakeholder Engagement and Data Acquisition” crowdsourcing project, which engages local communities to find and share old records with historic flood information to improve flood forecasting and increase public awareness of flood risk.
Each case study will look at 7 related OI projects that have already been identified and extract project information from existing inventories. The POCs for these projects will be contacted to answer a series of questions through a mix of structured and unstructured interviews (e.g. purpose of project, the problem OI helps solve, benefits to using OI, implementation of OI, tools that were used, potential partnerships, types of participants, outcomes, successes/failures/barriers) through an online survey and/or phone interview. If feasible, 1-2 participants from each project will be asked about their experience participating in the OI project. This will be done prior to the proposed travel and then a more in-depth interview and participant observations will be conducted during the proposed trips with other relevant stakeholders that were involved with these projects or have a potential OI risk project in mind. The Risk CoP will be engaged prior to and during the in-person meeting to identify past and potential OI risk projects, get feedback (like, hate, wish list) on existing catalogs and toolkits, and how they envision what the OI Playbook for Risk would look like and how they would interact with it. The data from the interviews and participant observations will be coded based on the following categories and subcategories as sections of the OI Playbook for Risk: Guidance (Terms/Definitions, Use Cases, Implementation Process, Evaluation Metrics), Policies (info related to PRA/ICR and other policies), Catalog (past/potential projects, info in existing inventories), Toolkit (tools, publications), and Case Study (project information, project data, tools, publications, photos, videos).
Sophia Liu is the Project Lead that will develop and manage the project and conduct the ethnographic research proposed. She is the USGS Crowdsourcing and Citizen Science (CCS) Coordinator, Co-Chair of the Federal Community for CCS, an interdisciplinary scientist that has used ethnographic and user-centered design methods for disaster projects, conducted OI risk projects like Tweet Earthquake Dispatch (TED) and iCoast - Did the Coast Change?, and helped FEMA initiate a Crowdsourcing Unit. David Govoni, Pierre Glynn, and Sheree Watson will be Co-PIs guiding the project and reviewing the Playbook that will inform the USGS Open Innovation Strategy. Govoni is an Emeritus and citizen scientist that previously lead the CDI Citizen Science Working Group and will inform how we might engage USGS Emeriti working on risk projects. Glynn led a review of the USA National Phenology Program (Glynn and Owen 2015) and has been an advocate for OI, including participatory modeling (Sterling et al. 2019; Voinov et al. 2018, 2016). Watson is currently developing various OI risk projects for the Youth and Education in Science program. David Wald and Sheree Watson will advise the earthquake case study based on their involvement with the DYFI and BLOSSM projects respectively. Ryan Toohey and Nicole Herman-Mercer will advise the climate change case study based on their involvement with the ION project and other community-based monitoring projects engaging indigenous communities. Sheree Watson will also advise the climate change case study based on her proposed risk projects with PICASC and NOROCK. Karen Ryberg and Robert Mason will advise the flood case study based on their Historical Floods risk project. Even if Watson’s and Ryberg’s risk proposals are not funded, these potential risk projects will still be examined and inform the development of the Open Innovation Playbook for Risk.
Participants in this project will be encouraged to attend and present at Risk monthly meetings on their OI risk projects. An Open Innovation Risk Forum will be organized in conjunction with the June 2020 Risk meeting to further engage the Risk COP. The OI Playbook for Risk is targeted for the Risk COP to inform future Risk RFPs and will be integrated into the broader USGS Open Innovation Strategy.
Anthony, R.E., Ringler, A.T., Wilson, D.C., and Wolin, E. (2018). Do Low‐Cost Seismographs Perform Well Enough for Your Network? An Overview of Laboratory Tests and Field Observations of the OSOP Raspberry Shake 4D. Seismological Research Letters, 90(1): 219–228. https://doi.org/10.1785/0220180251.
Atkinson, G.M., and Wald, D.J. (2007). “Did You Feel It?” Intensity Data: A Surprisingly Good Measure of Earthquake Ground Motion. Seismological Research Letters, 78: 362-368. https://doi.org/10.1785/gssrl.78.3.362.
Brubaker, M., Berner, J., and Tcheripanoff, M. (2013). LEO, the Local Environmental Observer Network: A Community-based System for Surveillance of Climate, Environment, and Health Events. International Journal on Circumpolar Health, 72: 22447. http://dx.doi.org/10.3402/ijch.v72i0.22447.
Calais, E., Boisson, D., Symithe, S., Momplaisir, R., Prépetit, C., Ulysse, S., Etienne, G.P., Courboulex, F., Deschamps, A., Monfret, T., Ampuero, J. Mercier de Lépinay, B., Clouard, V., Bossu, R., Fallou, L., and Bertrand, E. (2019). Monitoring Haiti’s Quakes with Raspberry Shake. Eos, 100. https://doi.org/10.1029/2019EO123345.
Cochran, E.S., Lawrence, J.F., Christensen, C., and Chung, A. (2009a). A Novel Strong-Motion Seismic Network for Community Participation in Earthquake Monitoring. IEEE Instrumentation & Measurement Magazine, 12(6): 8-15. https://doi.org/10.1109/MIM.2009.5338255.
Cochran, E.S., Lawrence, J.F., Christensen, C., and Jakka, R.S. (2009b). The Quake-Catcher Network: Citizen Science Expanding Seismic Horizons. Seismological Research Letters, 80 (1): 26-30. https://doi.org/10.1785/gssrl.80.1.26.
Crimmins, T.M., Crimmins, M.A., Gerst, K.L., Rosemartin, A.H., and Weltzin, J.F. (2017). USA National Phenology Network’s Volunteer-Contributed Observations Yield Predictive Models of Phenological Transitions. PLOS ONE, 12(8): e0182919. https://doi.org/10.1371/journal.pone.0182919.
Drapkin, J. K. (2018). Co-Producing Climate Science with Residents: ISeeChange Methods for Building Partnerships, Innovating, and Creating Decision-Relevant Information. AGU Fall Meeting Abstracts. https://ui.adsabs.harvard.edu/abs/2018AGUFMPA51A..01D/abstract.
Drapkin, J. K., Ramamurthy, P., Vant-Hull, B., Yuen, K., Glenn, A., Jusino, C., Corbin, C., Schuerman, M., Keefe, J., and Brooke, H. (2016). Tools and Techniques to Collaborate and Connect with At-Risk Climate Communities Using Sensors, Remote Sensing Data, and Media. AGU Fall Meeting Abstracts. https://ui.adsabs.harvard.edu/abs/2016AGUFMPA31E..02D/abstract.
Dutton, C.E. (1889). The Charleston Earthquake of August 31, 1886. U.S. Geological Survey, Ninth Annual Report 1887-88: 203-523.
Earle, P.S., Bowden, D.C., and Guy, M. (2012). Twitter Earthquake Detection: Earthquake Monitoring in a Social World. Annals of Geophysics, [S.l.], 54(6). http://dx.doi.org/10.4401/ag-5364.
Earle, P.S., Guy, M., Buckmaster, R., Ostrum, C., Horvath, S., and Vaughan, A. (2010). OMG Earthquake! Can Twitter Improve Earthquake Response? Seismological Research Letters, 81(2): 246-251. https://doi.org/10.1785/gssrl.81.2.246.
Fienen, M.N., and Lowry, C.S. (2012). Social.Water—A Crowdsourcing Tool for Environmental Data Acquisition. Computers and Geosciences, 49: 164-169. American Geophysical Union (AGU) https://doi.org/10.1016/j.cageo.2012.06.015.
Glynn, P.D. and Owen, T.W., Eds. (2015). Review of the USA National Phenology Network.
U.S. Geological Survey Circular 1411. http://dx.doi.org/10.3133/cir1411.
Government Accountability Office (GAO). (2016). Open Innovation: Practices to Engage Citizens and Effectively Implement Federal Initiatives. GAO-17-14.
Government Accountability Office (GAO). (2017). Open Innovation: Executive Branch Developed Resources to Support Implementation, but Guidance Could Better Reflect Leading Practices. GAO-17-507. https://www.gao.gov/products/GAO-17-507.
Hayden, L., Taylor, J., and Colon Robles, M. (2019). GLOBE: Connecting a Community of Observers Directly to NASA Satellites. IEEE Geoscience and Remote Sensing Magazine, 7(1): 98-99. http://doi.org/10.1109/MGRS.2019.2891930.
Herman-Mercer, N., Antweiler, R., Wilson, N., Mutter, E., Toohey, R., & Schuster, P. (2018). Data Quality from a Community-Based, Water-Quality Monitoring Project in the Yukon River Basin. Citizen Science: Theory and Practice, 3(2). http://doi.org/10.5334/cstp.123.
Herman-Mercer, N., Matking, E., Laituri, M., Toohey, R., Massey, M., Elder, K., Schuster, P., and Mutter, E. (2016). Changing Times, Changing Stories: Generational Differences in Climate Change Perspectives from Four Remote Indigenous Communities in Subarctic Alaska. Ecology and Society, 21(3): 28. http://dx.doi.org/10.5751/ES-08463-210328.
Herman-Mercer, N., Laituri, M., Toohey, R., Matkin, E., Massey, M., Schuster, P., and Elder, K. (2019). Vulnerability of Subsistence Systems Due to Social and Environmental Change: A Case Study in the Yukon-Kuskokwim Delta, Alaska. Arctic. https://doi.org/10.14430/arctic68867.
Janney, D. (2019). Engaging Underserved High School Students in Science Investigations using NASA's GLOBE and GLOBE Observer Resources. American Geophysical Union (AGU) Fall Meeting Abstracts. Earth and Space Science Open Archive (ESSOAr). https://doi.org/10.1002/essoar.10501246.1.
Lowry, C.S. and Fienen, M.N. (2013). CrowdHydrology: Crowdsourcing Hydrologic Data and Engaging Citizen Scientists. Groundwater, 51: 151-156. https://doi.org/10.1111/j.1745-6584.2012.00956.x.
Lowry, C.S., Fienen, M.N., Hall, D.M., and Stepenuck, K.F. (2019). Growing Pains of Crowdsourced Stream Stage Monitoring Using Mobile Phones: The Development of CrowdHydrology. Frontiers in Earth Science, 7(128). https://doi.org/10.3389/feart.2019.00128.
Ludwig, K.A., Ramsey, D.W., Wood, N.J., Pennaz, A.B., Godt, J.W., Plant, N.G., Luco, N., Koenig, T.A., Hudnut, K.W., Davis, D.K., and Bright, P.R. (2018). Science for a Risky World—A U.S. Geological Survey Plan for Risk Research and Applications. U.S. Geological Survey Circular 1444, 57 p., https://doi.org/10.3133/cir1444.
Lukyanenko, R., Wiggins, A., and Rosser, H.K. (2019). Citizen Science: An Information Quality Research Frontier. Information Systems Frontier, 1-23. https://doi.org/10.1007/s10796-019-09915-z.
Mosites, E., Lujan, E., Brook, M., Brubaker, M.A., Roehl, D., Tcheripanoff, M., and Hennessy, T.W. (2018). Environmental Observation, Social Media, and One Health Action: A Description of the Local Environmental Observer (LEO) Network. One Health, 6: 29-33. https://doi.org/10.1016/j.onehlt.2018.10.002.
National Academies of Sciences, Engineering, and Medicine (NASEM). (2018). Learning Through Citizen Science: Enhancing Opportunities by Design. Washington, DC: The National Academies Press. https://doi.org/10.17226/25183.
Nielsen, M. (2011). Reinventing Discovery: How Online Tools are Transforming Science. Princeton: Princeton University Press.
Office of Science & Technology Policy (OSTP). (2015). Memorandum: Addressing Societal and Scientific Challenges through Citizen Science and Crowdsourcing. Washington, DC: The U.S. Government. https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/holdren_citizen_science_memo_092915_0.pdf.
Office of Science & Technology Policy (OSTP). (2019). Report: Implementation of Federal Prize and Citizen Science Authority: Fiscal Years 2017-18. Washington, DC: The U.S. Government. https://www.whitehouse.gov/wp-content/uploads/2019/06/Federal-Prize-and-Citizen-Science-Implementation-FY17-18-Report-June-2019.pdf.
Robinson L., Cawthray, J., West, S., Bonn, A., and Ansine, J. (2018). Ten Principles of Citizen Science. In S. Hecker, M. Haklay, A. Bowser, Z. Makuch, J. Vogel, and A. Bonn. Citizen Science: Innovation in Open Science, Society and Policy. London, UCL Press. 271-4023. https://doi.org/10.14324/111.9781787352339.
Seibert, J., Strobl, B., Etter, S., Hummer, P., and van Meerveld, H.J. (2019). Virtual Staff Gauges for Crowd-Based Stream Level Observations. Frontiers in Earth Science – Hydrosphere. https://doi.org/10.3389/feart.2019.00070.
Sterling, E.J., Zellner, M., Jenni, K.E., Leong, K., Glynn, P.D., BenDor, T.K., Bommel, P., Hubacek, K., Jetter, A.J., Jordan, R., Schmitt Olabisi, L., Paolisso, M. and Gray, S. (2019). Try, Try Again: Lessons Learned from Success and Failure in Participatory Modeling. Elementa Science of the Anthropocene, 7(1): 9. http://doi.org/10.1525/elementa.347.
Strobl, B., Etter, S., van Meerveld, I., Seibert, J. (2019). The CrowdWater Game: A Playful Way to Improve the Accuracy of Crowdsourced Water Level Class Data. PLOS ONE, 14(9): e0222579.
Toohey, R., Mutter, E., Schuster, P., and N. Herman-Mercer. (2016). Multi-Decadal Increases in Yukon River Basin Chemical Fluxes as Indicators of Changing Flowpaths, Groundwater, & Permafrost. Geophysical Research Letters. https://doi.org/10.1002/2016GL070817.
Voinov, A.A., Jenni, K., Gray, S., Kolagani, N., Glynn, P.D., Bommel, P., Prell, S., Zellner, M., Paolisso, M., Jordan, R., Sterling, E., Schmitt-Olabisi, L., Giabbanelli, P.J., Sun, Z., Le Page, C., Elsawah, S., BenDor, T.K., Hubacek, K., Laursen, B.K., Jetter, A., Basco-Carrera, L., Singer, A., Young, L., Brunacini, J., and Smajgl, A., (2018). Tools and Methods in Participatory Modeling: Selecting the Right Tool for the Job. Environmental Modelling and Software, 109: 232-255. https://doi.org/10.1016/j.envsoft.2018.08.028.
Voinov, A., Kolgani, N., McCall, M.K., Glynn, P.D., Osterman, F., Palaniappan, R., Pierce, S., and Kragt, M.E. (2016). Modelling with Stakeholders – Next Generation. Environmental Modelling & Software, 77: 196-220. https://doi.org/10.1016/j.envsoft.2015.11.016.
Wald, D. J., and Dewey, J.W. (2005). Did You Feel It? Citizens Contribute to Earthquake Science. USGS Fact Sheet 2005-3016. https://doi.org/10.3133/fs20053016.
Wald, D.J., Quitoriano, V., Dengler, L.A., and Dewey J.W. (1999). Utilization of the Internet for Rapid Community Intensity Maps. Seismological Research Letters, 70(6): 680-697. https://doi.org/10.1785/gssrl.70.6.680.
Wald, D.J., Quitoriano, V., Worden, C.B., Hopper, M., and Dewey, J.W. (2012). USGS “Did You Feel It?” Internet-based Macroseismic Intensity Maps. Annals of Geophysics, 54(6). https://doi.org/10.4401/ag-5354.
Wiggins, A., Newman, G., Stevenson, R.D., and Crowston, K. (2011). “Mechanisms for Data Quality and Validation in Citizen Science.” In e-Science Workshops (eScienceW). IEEE Seventh International Conference, 14-19. http://ieeexplore.ieee.org/abstract/document/6130725/?reload=true.
Wilson, N.J., Mutter, E., Inkster, J., and Satterfield, T. (2018). Community-Based Monitoring as the Practice of Indigenous Governance: A Case Study of Indigenous-led Water Quality Monitoring in the Yukon River Basin. Journal of Environmental Management, 210: 290-298. https://doi.org/10.1016/j.jenvman.2018.01.020.