Annotation protocols in Sens Public
This project seeks to put forward a fully open peer-review process for the journal Sens public.
In its current state, both reviewers and authors are aware of their peers’ identity, but the content of the evaluations conducted by reviewers is not made available for other reviewers.
We are therefore aiming to extend transparency to the entire review process.
Using the text editor Stylo and the annotation tool Hypothesis, Sens public wishes to enable all stakeholders in the review and assessment process to work together on a single document.
Concretely speaking, reviewers will be encouraged to make comments that will be visible not only to the author, but to the other reviewer(s) as well.
All participants can thus respond and react among themselves.
In this way, we hope to foster dialogue between researchers and make conversation a promising vehicle for improvement in the scientific writing process.
The goal is also to think about new protocols that would encourage scientific dialogue and to adjust the evaluation process along experimentations and reflections.
Issues
The double-blind peer-review process is not devoid of challenges, especially concerning the high acceptance rates of articles and the low number of substantial changes between the submitted article and the one that is eventually published.
Although we do not consider open review to be a panacea for the problems of double-blind peer-review—open-reviewed articles show an even lower rejection rate than double-blind articles—this change in the process helps destabilize the paradigm of knowledge validation in the scientific community.
Through this project, we aim to dynamize and humanize the review stage, so that it is less centered on forms, reports and decisions relayed by the editor and instead ressembles a discussion between colleagues around a topic of common interest.
Open annotations can also make the editing stage richer for the author by enabling reviewers to work collaboratively.
Authors will be able to bounce back on evaluation remarks, asking for clarification, making comments and justifying certain choices, all of which we believe will stimulate careful, targeted reworking of the articles.
Technical challenges
The current peer-review process is based on a general evaluation form, developed to suit Sens public’s needs, sent by email and to be filled directly in the email response, as well as the sharing of weblinks leading to Stylo previews of the article to be evaluated. Peer reviewers are thus free to leave comments on these preview pages of the article through Hypothesis open annotations, or to only fill in the general form. As targeted annotations are often more significant for authors, the aim is to encourage this form of reviewing by offering technical support, currently by way of video meetings on Jitsi, for the use of Hypothesis and Stylo. This technical chain will be modified as experimentations are conducted on the peer-review process.
Research activities
1. Evaluation form
- Assess the length and relevance of comments left in the form
- Test multiple evaluation instructions with varying levels of specificity
- Adjust form sections to encourage strong evaluations according to the article or the journal’s needs
- Take note of the flow between the form and Hypothesis annotations
2. Hypothesis annotations
- Provide peer reviewers with a single link towards the same webpage so that each can see comments left by the other
- Keep statistics on the number of annotations received, taking into account peer reviewer order (Peer Reviewer 1 and Peer Reviewer 2), and the number of interactions between peer reviewers
- Harvest soft data
- Questions recieved from peer reviewers
- Needs for technical assistance
- Responses to the review process
Deliverables
- Review protocol fitting of Sens Public's epistemological framework
- Exploitable documentation of the project with a view to possible similar experiments conducted by other journals
- Promotion of the theoretical and technical aspects of the experiment via a paper presented at an international conference in the field of digital humanities
