Abstract review process for FOSS4G 2016

The final selection of the presentations for the FOSS4G used to be a controversy. Reviewing so many abstracts takes a lot of effort, so it's not possible to tell the rejected presenters in detail why their abstract wasn't accepted. Though it's possible to be transparent about the process that was applied. That's what this blog post is about. Let's start with some numbers. There were 281 abstracts submitted, 181 were accepted. There was a community review and a separate review by the program committee. Both reviews were blind, without knowing the author's name. Next the program committee met several times using a Google Docs spreadsheet which had the community rank, reviewer's rank and the submitted details in it. This time it was including the author's name.

The selection process

Step 1: Get rid of the lowest ranked ones

We were picking a number to eliminate the abstracts that had *both* a low rank (less thank 180) by the community and by the reviewers. This way we got the numbers down by 47 to 233.

Step 2: Go through highest ranked presentations and limit those by author and company

Only the abstracts where both the community as well as the program committee were ranking them above 180 were taken into account, that were 142 talks. We went trough them to limit the number of presentations to a maximum of two per person (with exceptions: some people are so active in the FOSS4G community, that they represent so many different projects, so that it make sense to have them give more than two talks. We also limited the number of talks by company. This wasn't a hard-number, but depending on how many different topics were covered. This reduced the total talks by 11 to around 222.

Step 3: Go through the "undecided" ones

Those presentations where the community and the program committee didn't agree on got special care. Those were about 90 talks. We sorted them by the community voting. Then everyone from the committee was able to argue for talks being in our out. This also included talks being excluded because of the limitations outlined in step 2. This was the most time consuming step which lead to a lot of good discussions. In the end it brought the total count down by 41 to the 181 we accepted.

Step 4: Group talks by topic

As we were meeting already twice, we decided to take the grouping offline. All members were working on the spreadsheet of the final selection and adding tags of what the talks are about. The program committee chair then took the time to do the final grouping for the website.

Impact of the community vote

If you compare the top 181 talks from the community voting with the final program the overlap is 77% (when taking the talks removed in step 2 into account it's even 80%). When the committee made decisions that were contrary to the community voting, it had numerous reasons. For example all OSGeo project should get a stage to present, whether they were voted highly by the community or not. Other considerations were to include non-developer centric talks like case studies, about business models or talks matching one of our four key topics: remote sensing for earth observation, disaster management, open data and land information.