We have put together a list of 37 problems in research and publishing that are still open, each with the published evidence for it and a line on what “solved” would have to mean. Several of them sit close to metadata and the scholarly record:
Retracted papers keep being cited (problem 8)
Journals vanish from the web (9)
Links in reference lists die (10)
Preprints lose their journal version (11)
Hallucinated references entering the record (17)
Retraction status disagrees between databases (18). Solved means: for a random sample of 1,000 retracted items, PubMed, Crossref, Scopus, WoS and OpenAlex show the same status within 30 days of the notice in at least 98% of cases.
Stealth (unannounced) corrections to the Version of Record (19)
Research software rarely cited properly, and dataset citations not counted (26 and 27)
The full list, with its sources, is on the Open Problems page of the Smart Scholars website.
We would value this community’s view. Which problems are missing? Which “solved means” lines are wrong or unrealistic? The page takes suggestions.
There is also a prize: US$1,000 for the highest-scoring solution of round 1 (entries until 31 May 2027), and a year of our services for one journal for the highest-scoring entry that solves each problem. Entry is free.
On #18’s “solved means”: 98% agreement across five databases within 30 days may conflate latency with error. Crossref and OpenAlex largely reflect publisher deposits, PubMed has its own indexing rules, and Scopus/WoS curation lags; a database can be not yet updated without being wrong. A more testable target would compare each source against a curated ground-truth sample, report update-latency distributions separately, and distinguish missing from incorrect status. Otherwise the metric may penalise coverage differences rather than flag real failures. It would also help to state whether the sample covers retracted items only or includes expressions of concern.
Thank you, Matthew. That is a fair point, and it changes how #18 should be measured.
A database that has not yet caught up is a different failure from one that shows the wrong status, and our line counted both as disagreement. We plan to revise #18’s “solved means” before round 1 opens on 1 December, along these lines:
Ground truth first: a random sample of 1,000 retraction notices, each checked by hand against the publisher’s own notice (the Retraction Watch data that Crossref has made openly available is a natural place to start).
Correctness: once a database shows the item, its status matches the ground truth in at least 98% of cases. No retracted item is shown as current, and no item is wrongly shown as retracted.
Update time, reported separately: the days from the notice to each database’s update, as a distribution per source (median and 90th percentile), with 90% of items updated within 30 days.
Missing is not wrong: an item a database does not cover, or has not yet updated, counts as missing, not as incorrect.
On your question: the current line covers retractions only. Expressions of concern behave differently, because they can be lifted or turned into retractions, so we would report them as a separate sample rather than mix them in.