Assessment

Overview

Teaching: 10 min
Exercises: 50 min

Questions

• How can I assess the FAIRness of myself, an organisation, a service, a community… ?

• Which FAIR assessment tools exist to understand how FAIR you are?

Objectives

• Assess the current FAIRness level of myself, organisation, service, community…

• Know about available tools for assessing FAIRness.

• Understand the next steps you can take to being FAIRer.

Reasons for assessment

FAIR is a journey. Technology and the way people work is shifting often and what might be FAIR today might not be months, years from now. A FAIR assessment now is a snapshot in time. Nevertheless, individuals, organizations, disciplines, services, countries, and communities will look to how FAIR they are. The reasons are various, including gaining a better understanding, comparing with others, making improvements, and participating further in the scholarly ecosystem, to name a handful. Ultimately, an assessment can be a helpful guide on the path to becoming more FAIR.

Mirror, mirror on the wall, who is the FAIRest one of all?

Mirror, mirror on the wall, who is the FAIRest one of all? - In March 2020, Theuringen FDM-TAGE offered awards to the FAIRest datasets based on the FAIR principles. The FAIRest Dataset winners were announced in June 2020. What is FAIR about the winning datasets? Is there anything else that can be done to make them FAIRer?

FAIR is a vision, NOT a standard

The FAIR principles are a way of reaching for best data and software practices, coming to a convergence on what those are, and how to get there. They are NOT rules. They are NOT a standard. They are NOT a requirement. The principles were not meant to be prescriptive but instead offer a vision to optimise data/software sharing and reuse by humans and machines.

Inconsistent interpretations

The lack of information on how to implement the FAIR principles have led to inconsistent interpretations. Jacobsen, A., de Miranda Azevedo, R., Juty, N., Batista, D., Coles, S., Cornet, R., … & Goble, C. (2020). FAIR principles: interpretations and implementation considerations describes implementation considerations.

Types of assessment

Depending on your needs, whether you want to assess yourself, a service, your organization or community, or even your country or region, FAIR assessment or evaluation tools are available to help guide you in your path towards FAIR betterment. The following are some resources and exercises to help you get started.

Individual assessment

How FAIR are you? The FAIRsFAIR project has developed an assessment tool called FAIR-Aware that both helps you understand the principles and also how you can improve the FAIRness of your research. Before taking the assessment, have a target dataset/software in mind to prepare you for the questions which include questions about yourself and 10 questions about FAIR. Each question provides additional information and guidance and helps you assess your current FAIRness level along with potential actions to take. The assessment takes 10 to 30 minutes to complete depending on your familiarity with the subject and issues covered.

Challenge

Encourage your workshop participants to review the episodes in this FAIR lesson and take the FAIR-Aware assessment ahead of time. In person (or virtual), ask the participants to split up into groups and to highlight some of their key questions/findings from the FAIR-Aware assessment. Ask them to note their questions/findings/anything else in the session’s collaborative notes. After a duration, ask the groups to return to the main group and call on each group (leader) to summarise their discussion. Synthesise some of the key points and discuss next steps on how participants can address their FAIRness moving forward.

Alternatively, the Australian Research Data Commons (ARDC) FAIR data assessment tool and/or the How FAIR are your data? checklist by Jones and Grootveld are also available and can be substituted for the FAIR-Aware assessment tool.

Evaluate the FAIRness of digital resources

How FAIR is your service and the digital resources you share? How can your service enable greater machine discoverability and (re)use of its digital resoruces? Evaluation of your service’s FAIRness lies on a continuum based on the behaviors and norms of your community. Frameworks and tools to assess services are currently under development and what options are available should be paired with the evaulation of what makes sense to your community.

FAIR Evaluation Services

The FAIR Evaluation Service is available to assess the FAIRness of your digital resources. Developed by the Maturity Indicator Authoring Group, FAIR Maturity Indicators are available to test your service via a submission process. The rationale for the Service are explained in Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud. To get started, the Group can also be reached via their Service.

As a service provider, for example a data repository, you might want to assess the FAIRness of datasets in your systems. You can do this by using one of the resources at FAIRassist or you can start your assessment manually (as a group exercise). Some infrastructure providers have provided overviews of how their services enable FAIR.

• Zenodo offers an overview of how the service responds to the FAIR principles.
• Figshare also published a statement paper on how it supports the FAIR principles.

Challenge

Encourage your workshop participants to review the episodes in this FAIR lesson and then review the FAIR principles responses/statements from Zenodo and Figshare above before the workshop. Again, ahead of the workshop, ask the participants to develop similar responses/statements for a service at their organisation, in their community. An outline with brief bullet points is best. Pre-assign workshop participants to groups and ask them to share their responses/statements with each other. Then in person (or virtual), ask the participants to split up into their pre-assigned groups and to discuss each other’s responses/statements. Ask them to note their questions/findings/anything else in the session’s collaborative notes. After a duration, ask the groups to return to the main group and call on each group (leader) to summarise their discussion. Synthesise some of the key points and discuss next steps on how participants can address their FAIRness moving forward.

Quantifying FAIR

In a recent DataONE webinar titled Quatifying FAIR, Jones and Slaughter describe tests that have been conducted to assess the FAIRness of digital resources across their services. The MetaDIG tool is referenced, used to check the quality of metadata in these services. Based on this work, DataONE also lists a Make your data FAIR tool as coming soon

Community assessment

Communities can also assess how FAIR they are and develop goals and/or action plans for advancing FAIR. Communities can range from topical to regional and even organisational. A recently published report from the Directorate-General for Research and Innovation (European Commission) titled “Turning FAIR into reality” is an invaluable resource for creating action plans and turning FAIR into reality for communities. The report includes a survey and analysis of what is needed to implement FAIR, concrete recommendations and actions for stakeholders, and example case studies to learn from. Ultimately, the report serves as a useful framework for mapping a community’s next steps towards a FAIRer future.

Challenge

Encourage your workshop participants to first read:

Collins, Sandra, et al. “Turning FAIR into reality: Final report and action plan from the European Commission expert group on FAIR data.” (2018). https://op.europa.eu/en/publication-detail/-/publication/7769a148-f1f6-11e8-9982-01aa75ed71a1

Leverage the recommendations (pictured below) and organise group discussions on these themes. Ask the participants to brainstorm initial responses to these themes ahead of the workshop and collect their initial responses in a collaborative document, structured by the themes. During the workshop, ask the participants to join their groups based on the themes they initially responded to and discuss each other’s responses. Ask them to note their questions/findings/anything else in the session’s collaborative notes. After a duration, ask the groups to return to the main group and call on each group (leader) to summarise their discussion. Synthesise some of the key points and discuss next steps on how participants can address their FAIRness moving forward.

Challenge

Alternatively, start your community off on the path to FAIR by setting up an initial study group. The group’s goal, to scan, and possibly survey, work that has been done by community members (or like communities) on example/case studies, policies, recommendation, guidance, etc to collect resources to help inform future FAIR discussions/initiatives. Consider structuring your group work to produce guidance, e.g. Top 10 FAIR Data and Software Things:

Paula Andrea Martinez, Christopher Erdmann, Natasha Simons, Reid Otsuji, Stephanie Labou, Ryan Johnson, … Eliane Fankhauser. (2019, February). Top 10 FAIR Data & Software Things. Zenodo. http://doi.org/10.5281/zenodo.3409968

Other assessment tools

To see a list of additional resources for the assessment and/or evaluation of digital objects against the FAIR principles, see FAIRassist.

Planning

Data and software management plans are also a helpful tool for planning out how you/your group will manage data and software throughout your project but they also provide a mechanism for assessment at different checkpoints. Revisiting your plan at different checkpoints allows you to review how well you are doing, incorporate findings, and make improvements. This allows your plan to evolve, be more actionable, and less static.

Resources and examples include:

Data:

• Data Stewardship Wizard
• DMPonline
• DMPTool
• ICPSR example pans
• LIBER Research Data Management Plan (DMP) Catalogue

Software:

• ELIXIR Software Management Plan
• SSI Software Management Plans
• CLARIAH Guidelines for Software Quality
• EURISE Network Software Quality Checklist

Challenge

Use one of the resources above to draft a plan for your research project (as an individual and/or group). As an individual, ask a colleague to review your draft, provide feedback, and discuss. As a group, outline the plan questions in a collaborative document and work together to draft responses, then discuss together as a group. Consider publishing your plan to share with others.

Resources

This is a developing area, so if you have any resources that you would like to share, please add them to this lesson via a pull request or GitHub issue.

Key Points

• Assessments and plans are helpful tools for understanding next steps/action plans for becoming more FAIR.