Example Workflow
Guide based on ICLR venue
Last updated
Was this helpful?
Guide based on ICLR venue
Last updated
Was this helpful?
How to use this document: This document lists the major steps of running a conference venue. Each step links to relevant documentation that explains it in depth.
Fill out the venue request form and choose settings for your venue.
OpenReview will review the request, ask for any necessary clarification, then you will receive an email notifying you that your venue has been deployed.
Also see:
Ensure all PCs have OpenReview accounts associated with the email listed in the venue request form in order to access venue pages.
You may edit many settings for your venue through the 'Revision' button of the request form.
Recruitment can happen at any point in the workflow. All participants (Area Chairs, Reviewers, etc.) must have an OpenReview account linked to the email used in the recruitment message.
5. (optional)
You may choose to add a task for program committee members to remind them to complete their registration.
Submissions automatically open on the date/time listed in the venue request form. If no date is given, submissions open as soon as the venue is deployed.
Note: All submitting authors must have an active OpenReview Profile.
Use this if you want to have an abstract deadline ahead of the main submission deadline.
Set up matching should be done in the direction Senior Area Chairs-> Area Chairs-> Submissions, then Reviewers should be assigned to submissions. The basic stages of this workflow is to:
Each of these steps needs to be completed for each group being matched (e.g. Area Chairs - Submissions , Reviewers - Submissions)- Described further in steps 11-16 below.
To assign SACs to ACs, choose 'Senior Area Chairs' as the matching group in paper matching setup. Then in the Paper Matching Stage, SACs will be the matching group, and ACs will be in the Submissions field.
Note: Matching between SACs and ACs will not calculate conflicts. Instead, the conflicts to the SACs will be transferred to the ACs and calculated at the AC matching stage.
It is very important to deploy Senior Area Chair assignments before assigning submissions to Area Chairs to allow conflicts to be transferred successfully.
If you decide to directly assign Senior Area Chairs to submissions, skip to step 12.
The next step is to assign the program committee to each submission. If your conference is smaller than 2000 submissions, this can be run by you directly, otherwise please contact OpenReview.
It is very important that all Program Committee members (Senior Area Chairs, Area Chairs, and Reviewers) have a complete OpenReview profile (an active profile with at least one active institution and one publication). We recommend removing from the committee groups all the profiles that are not complete before running the matching system.
In order for conflicts to be taken into account in paper matching, matching set up (Step 14) must be run before the bid stage.
The bidding console shows the submissions sorted by affinity scores of the logged user and filtering out the ones that are in conflict with the user.
Note: when the conference is large, only sparse scores are uploaded to the system, this means we only keep the first N (400) scores for each user and submission. Users will see at least 400 submissions sorted by their affinity scores. If they want to see other submissions that are not among the top 400 they should use the “search” functionality.
After computing affinity scores and optionally enabling the bidding process, Program Chairs can run the paper matching system to assign Senior Area Chairs/Area Chairs/Reviewers to Submissions. Program Chairs can assign Senior Area Chairs/Area Chairs/Reviewers at different stages because they are independent processes.
Some venues decide to share the Reviewer-Submission assignments with the Area Chairs to review before releasing them to the Reviewers.
After the proposed assignments were reviewed by the Program Chairs and/or the Senior Area Chairs/Area Chairs, they can be deployed and be visible to the Reviewers. Deploying assignments doesn’t automatically send emails to the Reviewers- it is recommended that you as PCs notify them.
16. (optional) Modify assignments
The deadline field of the Review Stage form will be the one shown to reviewers as the advertised deadline. The expiration date in the form (not seen by reviewers) will be the time after which no more reviews can be submitted. After this point, reviewers will need to contact the PCs for any late reviews.
If there are still pending reviews, setting Release Reviews to Authors to "Yes, reviews should be revealed when they are posted to the paper's authors", will release all posted reviews, and later will release pending reviews after they are posted.
You may optionally allow authors to revise their submissions, including limiting which fields can be edited. This stage can be enabled any time after the submission deadline has passed.
Use this if you have Senior Area Chairs and want them to review, confirm, or revise the meta review posted by the Area Chair.
For large venues (>2000 submissions) we offer a bulk upload process where the PCs can get the meta reviews values, edit them to meet the acceptance rates and then upload them to the system. The PC console will show the decision stats and decision notes can be edited after they are uploaded.
PCs can also (optionally) decide to release the submissions to the public (all the submissions or accepted only) and deanonymize the author names.
Authors will no longer be able to edit submissions after the Camera Ready period deadline set in the Submission Revision Stage. After this point, they will need to contact the PCs to allow any revisions after the deadline.
Submissions will automatically close on the date specified in the venue request form. To change the submission deadline, see .
(optional)
Look at, and potentially proposed assignments
assignments
that assigned program committee are readers of their assigned submissions
Most venues assign SACs to submissions by first assigning them to Area Chairs. Here you may decide whether to have SACs automatically assigned to ACs or based on affinity scores by following steps 1-5 above.
Program Chairs can make reassignments after the proposed assignments are deployed or assignments.
Program Chairs can optionally ask the Senior Area Chairs, Area Chairs and/or Reviewers to .
PCs must make all the existing submissions visible to all the members of the Senior Area Chairs/Area Chair/Reviewers group hiding the PDF, supplementary material and any other fields they don’t want Senior Area Chairs/Area Chair/Reviewers to see by using the stage.
In order to do this, Program Chairs need to the Area Chair-Submission assignments so that Area Chairs can see their own assigned submissions and choose a matching configuration to share the proposed assignments. Then you can .
. This must be requested to the support team.
Deployed assignments can be by the Program Chairs and Senior Area Chairs. Program Chairs can also decide if they want to . When a reassignment is done, an email notification is sent to the new Reviewer.
You can start the review period through the . You may also at this point the.
Note: There are two fields with default names “rating” and “confidence” that are used to compute stats in the different consoles. You may modify these fields but make sure to specify the names in the field of the Revision Stage.
It is also important to choose what groups a review should be visible to. A common configuration is that reviews are only visible to the assigned Senior Area Chair, Area Chair and Reviewers. You may change the readers of reviews at any time using the
Usually venues have a rebuttal period where Authors can reply to the Reviewers. In OpenReview, the rebuttal period can start at any time using the . They can choose between settings to allow a free number of rebuttal comments or require authors to have one rebuttal per Submission/Review.
PCs may also use the so that reviewers can optionally reply to the authors and keep threaded discussions.
When the review period has concluded, the PCs can start the where the ACs make their recommendations.
You may also optionally allow Area Chairs to submit ratings for their reviews (found in ).
This is the last step before releasing the decision to the authors. PCs need to submit the final based on the AC meta reviews and confirmations. Decisions are visible to the PCs only.
Once all the decisions are made and uploaded to the system, you may them to the authors and send email notifications using the . OpenReview offers a where the PCs can define the email template for each decision.
The camera ready period starts after the authors are notified about the submission decisions. To begin, open the . Make sure to enable the setting 'Enable revision for accepted submissions only'.