Video of the workshop:
https://api.nakala.fr/embed/10.34847/nkl.aeccww5g/b3b6a86689093332dab79f42a64d8024fb2c8b3c

Presentation of the project on AI-assisted literature review

The project’s Pink My Pad:
https://pinkmypad.net/libreon/0p58JaDESASJRGwIm3YytQ

**Feedback on AI-assisted literature review: project for the journal *Humanités Numériques***

Initially focused on bibliography because it seems like a simpler task due to its formal structure. However, the goal is to eventually extend this to other parts of academic articles.

The aim is to explore the use of tools that are not necessarily mainstream—and before that, to understand the stakes involved in literature review work.

Methodology:

• Position the journal in relation to perceived editorial uses of generative AI (GAI).

• Conduct fieldwork to map out the functions of literature review work and model this task.

• Test several scenarios and AI tools based on the identified issues.

Origin of the project:
The idea emerged during a National Training Action (ANF) organized by CNRS for scholarly journal editors. Participants were encouraged to launch AI-based projects, and it was noted that most projects proposed used ChatGPT.

Requirements of Humanités Numériques (this project offered the team a chance to reflect on AI through four angles):

• Editorial standards

• Working conditions

• Environmental concerns (e.g., running models locally)

• Practical concerns: is it even useful to use GAI when tools like Zotero already exist?

Three types of revisions done by Florence:

• Content revision (accuracy of references)

• Form revision (formatting, spelling)

• Concordance revision (ensure references aren’t redundant or irrelevant)

Interviews based on Florence Daniel’s work to identify usage scenarios:

Scenario 1:

• Author submits a .docx, with or without BibTeX.

• Bibliographic revision is done by the author.

Scenario 2:

• Editor receives a .docx, but without BibTeX.

• Bibliographic revision is done by the editor.

Experimentation paths:

• Transforming a poorly structured bibliography using ChatGPT directly.

• Transformation using specialized tools like AnyStyle, ReversedZotero.

• Retrieval-Augmented Generation (RAG) for concordance checks (not presented here).

Case study

Experiment on a 15-reference bibliography that does not follow Humanités Numériques style.
→ Manual work, classic method used to estimate time and needs and to produce a ground truth.

About 45 minutes of work:
15 minutes for formatting, and 30 minutes to check references.

First revision attempt using ChatGPT (o3-mini) and Duck.ia with the following prompt:

→ Time not necessarily reduced because:

• Instructions had to be split.

• Checking references was even more time-consuming.

• Limited reproducibility (more errors with each iteration).

Second strategy: attempt to automatically convert to BibTeX using ReversedZotero and AnyStyle.

• ReversedZotero: worked poorly, reversed names and surnames.

• AnyStyle: worked well.

→ But the results were reversed a few weeks ago. In all cases, corrections are necessary.

Comments:

• The core problem with bibliographies is authors’ lack of interest.

• This disinterest might favor ethically questionable tools, starting with ChatGPT (which many authors already use).

• Technical processes become invisible: formats as well as the labor of bibliographic research are hidden.

→ Raises deeper questions about the value and purpose of reference work: what role does it play in scientific discourse?

Discussion

Marcello Vitali-Rosati: quoting Matteo Treleani about the work of TV archivists (“subaltern” archivists at Mediaset): they fill in content by selecting images based on the day’s theme (hence repetitions).

The main condition for ChatGPT’s use is authors’ disinterest in bibliographic revision and the trivialization of this task.

If references are seen as mere filler, like TV images, then nobody cares. But citation styles used to indicate disciplinary belonging and had scientific weight. These practices raise questions about the epistemological/symbolic status of references. What is a bibliographic reference?

Servanne Monjour: Style still matters, especially in literature. The importance of the source is not just symbolic—it has practical value, which may be diminishing.

Marcello Vitali-Rosati: What is the purpose of a bibliographic reference? If it’s just about retrieving the source, isn’t the DOI enough? Is it aesthetic? Is it scientific?

Aurélien Berra: Beyond reproducibility, bibliographies help build reference systems, discourse authority, and situate texts within knowledge networks. Presentation format (margin, popup, endnote) affects understanding. Style matters.

Marcello Vitali-Rosati: Citations help contextualize a publication semantically. We must rethink the meaning of citing. Is bibliography the only or best way to do that?

Servanne Monjour: Beyond the act of citing, there’s the question of reading. A university professor once provocatively said that when approaching a thick book, one only needs to read the abstract and the bibliography. The symbolic weight of references (rigor and authority) can be faked by bloating the reference list. Limiting scientificity to references is reductive—but the art of it may be fading.

Margaux Jacques (in chat): I’ve definitely encountered this—certain authors who must be cited in a paper.

Tony Gheeraert: In the 80s–90s, building a bibliography was very difficult (sources, methods, imagination, etc.). Now it can be done in a few clicks → symbolic value has dropped. Two main functions of bibliography:

1. Verification

2. Offering readers ways to dig deeper

Michael Sinatra: Did you keep track of how ChatGPT and other tools evolved over time?

Servanne Monjour: We documented the experiments carefully: dates and tools must be specified. Energy usage indicators should also be included. Same for time.

Florence Daniel: Regarding time spent: 15min on formatting, 30min on verifying references one by one. The second part cannot be automated with ChatGPT. But for formatting references, it does save time. Checking links etc. is still needed.

Marcello Vitali-Rosati: The difference between expert systems and LLMs is that LLM results are plausible. Old systems produced totally incorrect results, making errors easier to spot. That correction work was undervalued. LLMs might invert the value system around this work.

Aurélien Berra: LLM tools (ChatGPT, etc.) become obsolete so quickly that even with documentation, we’ll soon be unable to replicate experiments. Epistemological question: what’s the point of documenting sources if they vanish in weeks?

Servanne Monjour: We’re already reworking texts and bibliographies partly generated by AI tools. Alternatives demand a certain level of digital literacy (ReversedZotero has a steep learning curve). We end up using heavy tools instead of readily available ones.

Tony Gheeraert (in chat): In my opinion, AI output should be verified by humans and/or automatically cross-checked (if we must use AI, though I believe improving regex might be better). The resistance to Zotero might be worth exploring. What if it’s actually valid?

Marcello Vitali-Rosati: Why make efforts to structure references when there’s no standard? Even harvesters (like Isidore) don’t use article metadata, they produce their own. Algorithms force us to rethink: why do we write bibliographies? What’s the value of this task?

Florence Daniel: We need to understand what we’re trying to structure. Even filling out Zotero requires an understanding of reference types. Knowing the difference between publication types is essential.

Servanne Monjour: In teaching, we don’t actually teach the meaning and function of bibliographic references. We should perhaps re-examine them as objects in their own right.

Call for papers for the symposium on constructing bibliographies, suggested by Servanne:
https://ebdf.hypotheses.org/2540

Tony Gheeraert (in chat): Can I talk about the joy of patiently and lovingly writing out footnotes by hand? 😉 Writing bibliographic notes is an intimate and personal engagement with the works. I don’t want my notes generated without full control over every character. Plus, I work with sources that don’t fit neatly into Zotero fields (though I still teach Zotero, just to reassure you).

Tony Gheeraert: Some authors are passionate about their bibliographies. It’s almost a philological devotion, which doesn’t preclude promoting Zotero use.

Aurélien Berra: We can’t simply fall back on the ethical imperative of “doing it by hand.” Science and editorial practices are largely moving toward automated bibliography structuring.

Servanne Monjour: There are also varied uses of Zotero: for example, adding personal notes beyond the database function. If the conclusion is that formatting revision is easier and more effective with AI...

Aurélien Berra: Are there GAI models specifically trained for bibliographic revision?