This lesson is still being designed and assembled (Pre-Alpha version)

Assessment

Overview

Teaching: 10 min
Exercises: 50 min
Questions
  • How can I assess the FAIRness of myself, organisation, service, community… ?

  • Which FAIR assessment tools exist to understand how FAIR you are?

Objectives
  • Assess the current FAIRness level of myself, organisation, service, community…

  • Know about available tools for assessing FAIRness.

  • Understand the next steps you can take to being FAIRer.

Reasons for assessment

FAIR is a journey. Technology and the way people work is shifting often and what might be FAIR today might not be months, years from now. A FAIR assessment now is a snapshot in time. Nevertheless, individuals, organizations, disciplines, services, countries, and communities will look to how FAIR they are. The reasons are various, including gaining a better understanding, comparing with others, making improvements, and participating further in the scholarly ecosystem, to name a handful. Ultimately, an assessment can be a helpful guide on the path to becoming more FAIR.

Mirror, mirror on the wall, who is the FAIRest one of all?

Mirror, mirror on the wall, who is the FAIRest one of all? - In March 2020, Theuringen FDM-TAGE offered awards to the FAIRest datasets based on the FAIR principles. The FAIRest Dataset winners were announced in June 2020. What is FAIR about the winning datasets? Is there anything else that can be done to make them FAIRer?

FAIR is a vision, NOT a standard

The FAIR principles are a way of reaching for best data and software practices, coming to a convergence on what those are, and how to get there. They are NOT rules. They are NOT a standard. They are NOT a requirement. The principles were not meant to be prescriptive but instead offer a vision to optimise data/software sharing and reuse by humans and machines.

Inconsistent interpretations

The lack of information on how to implement the FAIR principles have led to inconsistent interpretations. Jacobsen, A., de Miranda Azevedo, R., Juty, N., Batista, D., Coles, S., Cornet, R., … & Goble, C. (2020). FAIR principles: interpretations and implementation considerations describes implementation considerations.

Types of assessment

Depending on your needs, whether you want to assess yourself, a service, your organization or community, or even your country or region, FAIR assessment or evaluation tools are available to help guide you in your path towards FAIR betterment. The following are some resources and exercises to help you get started.

Individual assessment

How FAIR are you? The FAIRsFAIR project has developed an assessment tool called FAIR-Aware that both helps you understand the principles and also how you can improve the FAIRness of your research. Before taking the assessment, have a target dataset/software in mind to prepare you for the questions which include questions about yourself and 10 questions about FAIR. Each question provides additional information and guidance and helps you assess your current FAIRness level along with potential actions to take. The assessment takes 10 to 30 minutes to complete depending on your familiarity with the subject and issues covered.

Challenge

Encourage your workshop participants to review the episodes in this FAIR lesson and take the FAIR-Aware assessment ahead of time. In person (or virtual), ask the participants to split up into groups and to highlight some of their key questions/findings from the FAIR-Aware assessment. Ask them to note their questions/findings/anything else in the session’s collaborative notes. After a duration, ask the groups to return to the main group and call on each group (leader) to summarise their discussion. Synthesise some of the key points and discuss next steps on how participants can address their FAIRness moving forward.

Alternatively, the Australian Research Data Commons (ARDC) FAIR data assessment tool and/or the How FAIR are your data? checklist by Jones and Grootveld are also available and can be substituted for the FAIR-Aware assessment tool.

Evaluate the FAIRness of digital resources

How FAIR is your service and the digital resources you share? How can your service enable greater machine discoverability and (re)use of its digital resoruces? Evaluation of your service’s FAIRness lies on a continuum based on the behaviors and norms of your community. Frameworks and tools to assess services are currently under development and what options are available should be paired with the evaulation of what makes sense to your community.

FAIR Evaluation Services

The FAIR Evaluation Service is available to assess the FAIRness of your digital resources. Developed by the Maturity Indicator Authoring Group, FAIR Maturity Indicators are available to test your service via a submission process. The rationale for the Service are explained in Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud. To get started, the Group can also be reached via their Service.

As a service provider, for example a data repository, you might want to assess the FAIRness of datasets in your systems. You can do this by using one of the resources at FAIRassist or you can start your assessment manually (as a group exercise). Some infrastructure providers have provided overviews of how their services enable FAIR.

Challenge

Encourage your workshop participants to review the episodes in this FAIR lesson and then review the FAIR principles responses/statements from Zenodo and Figshare above before the workshop. Again, ahead of the workshop, ask the participants to develop similar responses/statements for a service at their organisation, in their community. An outline with brief bullet points is best. Pre-assign workshop participants to groups and ask them to share their responses/statements with each other. Then in person (or virtual), ask the participants to split up into their pre-assigned groups and to discuss each other’s responses/statements. Ask them to note their questions/findings/anything else in the session’s collaborative notes. After a duration, ask the groups to return to the main group and call on each group (leader) to summarise their discussion. Synthesise some of the key points and discuss next steps on how participants can address their FAIRness moving forward.

Quantifying FAIR

In a recent DataONE webinar titled Quatifying FAIR, Jones and Slaughter describe tests that have been conducted to assess the FAIRness of digital resources across their services. The MetaDIG tool is referenced, used to check the quality of metadata in these services. Based on this work, DataONE also lists a Make your data FAIR tool as coming soon

Community assessment

CESSDA/Turning FAIR into reality

Exercise - Try and find policies in your region, country, commonwealth that reference the FAIR principles and encourage their implementation

Wellcome guidance for organisations… https://wellcome.ac.uk/what-we-do/our-work/open-research

Other assessment tools

To see a list of additional resources for the assessment and/or evaluation of digital objects against the FAIR principles, see FAIRassist.

Planning

Data & Software Management Plans

Resources

This is a developing area, so if you have any resources that you would like to share, please add them to this lesson via a pull request or GitHub issue.

Also see https://doi.org/10.5281/zenodo.3678715

Additional tools…
Make Data Count https://makedatacount.org/
DANS https://www.surveymonkey.com/r/fairdat
ARDC https://ardc.edu.au/resources/working-with-data/fair-data/fair-self-assessment-tool/
DTL FAIRifier https://github.com/DTL-FAIRData/FAIRifier

Key Points

  • First key point.