On the Abolition of the “ICSE Paper” and the Adoption of the “Registered [Proposal]” and the “[Results] Report” DOI: NA Conference: Submitted to Future of Software Engineering at ICSE 2026; January 2026; Rio, Brazil ISBN: NA TBC
Fabio Massacci, Winnie Bahati Mbaka · 2026
Read On the Abolition of the “ICSE Paper” and the Adoption of the “Registered [Proposal]” and the “[Results] Report” DOI: NA Conference: Submitted to Future of Software Engineering at ICSE 2026; January 2026; Rio, Brazil ISBN: NA TBC by Fabio Massacci, Winnie Bahati Mbaka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18566.txt.
Introduction
Fabio Massacci and Winnie Bahati Mbaka call for a deep change in how a top software meet shares new work. They blame the plain paper for two ills: a chase for new-ness at all costs, and a crisis in which finds cannot be run again to check them. Each ill feeds the next, they say. Fresh claims win space, while checks and re-runs get short shrift.
Their plan swaps the plain paper for a two-tier path. First comes a short plan, shared and judged on its worth as a plan. Then comes a report of what the work found, judged on how well it kept to the plan. The pair build on a form of report that some fields use now. A shift like this could reward care over flash in software work.
Catalog record
| CREATOR | Fabio Massacci, Winnie Bahati Mbaka |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18566
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Abolition of the “ICSE Paper” and the Adoption of the “Registered [Proposal]” and the “[Results] Report” DOI: NA Conference: Submitted to Future of Software Engineering at ICSE 2026; January 2026; Rio, Brazil ISBN: NA TBC Fabio Massacci, Winnie Bahati Mbaka Abstract . To address the ’novelty-vicious cycle’ and the ‘replicability crisis’ of the field (both discussed in the survey) we propose abolishing the “ICSE paper” as we know it and replacing it with a two-tier system that also evolves the existing notion of ‘Registered Report’. Authors proposing a new idea, experiment, or analysis would submit a “Registered [Proposal]” of their idea and the proposed experimental methodology to undergo peer review. The following year, anyone can submit (shorter) “[Results] Reports” on the realization of the empirical work based on the registered proposals of the previous ICSE (or FSE or ISSTA or ASE etc.). Both works should be first class citizens of the mainstream events. We argue that such a disruptive (heretical?) idea is supported and based on the responses of the community of the Future of Software Engineering pre-survey. 1. Introduction There is a broad agreement that there is a replicability crisis in software engineering ( 3 ; 1 ) . This is starkly visible in the field of Machine Learning for vulnerability detection, where each year new papers show that previous results are fundamentally wrong ( 2 ) . This feeling is also broadly reflected in the comments collected from the survey on the Future of Software Engineering ( 4 ) . We select two themes to highlight the two major problems of the field that we will address in the rest of the position paper. The first theme is indeed reproducibility: \circ We are not good at replication (despite better rigour in what we do), which is partly a publication thing (journals and conferences do not rate these). Too much effort goes into chasing ‘new’ rather than consolidating our knowledge. (192758693) We need, therefore, to transform our publication model in a way that still values conference submissions for novelty but makes it possible to validate the results by somebody other than the inventor . The second problem is the elitist nature of the research community. Although a derogatory interpretation might be that there is a mafia: \circ Research …favor established names and groups (192154506) \circ ICSE is very cliquey (192676762) We want to propose a more general interpretation and, in our opinion, closer to the truth interpretation: \circ there is still a strong fear of trying innovative ideas that might be perceived as challenging the work of well-established and renowned researchers in the field. (192295956) Combined with the replicability crisis this generate a negative feedback loop: all-important ideas are published by the all-important groups, nobody can essentially show that the original idea was wrong (as negative results are not appreciated) unless the challengers invent also something new, and the new idea cannot be replicated or negated as you again need to have something new…As pointed out also in the Discord discussion: \circ One notable “problematic area” is that we focus on "novelty" so much that we create many tool prototypes that are unusable in practice - ones thar require painful setup and work mainly on “toy examples”. (Discord) As a conclusion research results make very limited impact: \circ I think this makes SE research body of knowledge very shallow and with little relevancy to the industrial practice. (196067582) This is unsurprising: if researchers themselves cannot replicate new solutions, how should practitioners adopt them? To propose a solution to this impasse we need a bold move and our proposal is to abolish the traditional “ICSE paper” and replace it with two different type of papers (the “Registered Proposal”, which partly exists in the form of a “Registered Report”) and the “Results Report” which does not exist yet and we will introduce here. 2. [Taking stock] - Problems and Trade-Offs We review here some of the results of the survey on the Future of Software Engineering ( 4 ) . While an interesting theme is surely managing reviews and scaling down both numbers and stress 1 1 1 The word ’review*’ occurred 118 times just in the question about if you could change one thing what you could do with “write 1, review 3” proposals. we believe two interesting themes emerged; • Novelty vs Rigor • Replication vs Novelty In the first theme, a perceived trade-off between novelty and the rigor of empirical validation is implied. On the one hand, Software Engineering communities view novelty as either the use of new datasets, the development of a new tool, or a new technique. On the other hand, validation is evaluated after the fact and is conditioned on the strength of the published results. \circ the empirical evaluation seems to be more important than the approaches themselves. (196022336) \circ The excessive focus on rigor and big samples seems to have made it harder to connect with real practitioners (193757741) \circ Reviewers should look for new ideas in papers, now only for the most thorough evaluations. (195811076) \circ Be more open about new ideas and mitigate requirements on tooooo conventional and rigorous points (195844264) \circ reviewers …hold firmly to their own assumptions or preferred methodologies. This can lead to biased evaluations, where papers proposing new directions or challenging established norms face unnecessary resistance. (196077577) In a nutshell, we can summarize the problem as \circ The review and publication culture is becoming overly focused on incremental but sound engineering advances, at the expense of novel/disruptive ideas […] should be more open to contributions not yet fully validated in terms of external validity, but with great novelty potential (196196304) In the second theme, a perceived tension between the Software Engineering community’s commitment to rigor through replication and the structural process of peer reviews governing publications is implied. Overall, a much larger number of respondents of the survey ( 4 ) pointed out that replication studies remain systemically marginalized due to the focus being on novelty. \circ we are reluctant to accept papers that replicate previous study, or just provide evidence in the areas that were already studied (sometimes having single studies only) as we believe these are not new results. (196067582 - our emphasis) \circ A reviewing practice…accepting novelty also comes with replications. (196247974) \circ be more open for paper submissions that may be just about specific case studies, or have negative results, or just replicate previous studies under some different settings. (196290588) \circ too much focus on …moving to the next hot topic, instead of ensuring reproducibility (196144420) Of course there are increasing efforts towards replication and negative results (e.g. ICPC’26 session, the RENE session at some ICSEs, the MSR and ESEM registered reports) but they are not papers in the main track and sometimes not even in the proceedings . They are always children of a lesser God. \circ Not sure how they could be handled by conferences, but surely we can devise some way of giving researchers credit for such studies. (192758693) Since there are no established incentives to promote independent replication studies, there is a shift in how trust is established, via reputation, whereby papers from well-known authors and research