Notebook

Attention Report¶

31st July 2014 Neil D. Lawrence¶

Updated for new cmtutils May 2021¶

This notebook contains scripts for emailing area chairs notifying them of potential problems with the papers allocated to them. The script goes through the calibrated reviews (as generated by [this notebook](./Reviewer Calibration.ipynb)) and seeks papers which seem problematic in terms of either the span of the review scores, the confidence of the reviewers, the length of the reviews or those that are in the 'grey area' for publication. For each paper it creates a set of comments, which is then emailed out to area chairs.

In [ ]:

import cmtutils.cmtutils as cu
import pandas as pd
import os

Firstly we read in the processed reviews. The processing includes the calibrated review scores and the probability of accept.

In [ ]:

revs = pd.io.parsers.read_csv(os.path.join(cu.cmt_data_directory, 
                                           '2014-08-02_processed_reviews.csv'), 
                              dtype={'PaperID':object})
revs.set_index(keys='PaperID', inplace=True)
revs.fillna('', inplace=True)

We also load in the area chair assignments.

In [ ]:

report = cu.review_report(calibrated_reviews=revs)
report.generate_html_comments()
report.generate_comments()

Now that the reports have been generated, load the area chair assignments so we know who to notify about problems.

In [ ]:

a = cu.assignment()
a.load_assignment(filename='2014-07-14_area_chair_assignments.txt', reviewer_type='metareviewer')

Now we can email out the report. We use software written for using gmail for this purpose. The gmail user name that is used is set inside .ods_user.cfg in the users home directory or in defaults.cfg in the directory where pods is installed.

In [ ]:

from pods import email
mailer = email.gmail()
# need to check the console here for your password input!

You'll need to check the console where you launched the notebook from to input your gmail password. Now we can construct the reports for the individual area chairs from the overall attention report. Since for NIPS this command sends 92 emails to the area chairs, it is currently 'switched off' in this notebook with an if False: statement. Simply set False to True to send mails.

In [ ]:

if False:
    for reviewer in a.assignment_reviewer['metareviewer']:
        recipient=reviewer
        reply_to = cmtutils.program_chair_email
        body = ''
        subject = "NIPS 2014: Paper Attention Report"
        papers = a.assignment_reviewer['metareviewer'][reviewer]
        comments = report.attention_report.loc[papers].sort(columns='attention_score', ascending=False).comments
        for comment in comments:
            body+=comment
        if len(body)>0:
            mailer.send(recipient=recipient,reply_to=reply_to,body=body,subject=subject)

This report was used for NIPS before author feedback to ensure that the Area Chairs had a good overview of where their papers were, both in terms of their own reviews, and relative to other papers in the system. It is not always easy as an area chair to know where the decision boundary is going to fall. Calibrated review scores help in assessing this. It could have also been used more in the reviewer discussion stage, but we decided that it was more convenient to share information via google spreadsheets. That also allowed Area Chairs to store notes and feedback information to us. As well as organising and highlighting their papers in the manner which they thought most appropriate. Those sheets were created by [this notebook](./Spreadsheet Reports.ipynb) and then updated every day as the reviews evolved by [this notebook](./Update Spreadsheets.ipynb).