Example Workflow
Guide based on ICLR venue
How to use this document: This document lists the major steps of running a conference venue. Each step links to relevant documentation that explains it in depth.
Setting up a venue
Fill out the venue request form and choose settings for your venue.
OpenReview will review the request, ask for any necessary clarification, then you will receive an email notifying you that your venue has been deployed.
Also see:
Ensure all PCs have OpenReview accounts associated with the email listed in the venue request form in order to access venue pages.
You may edit many settings for your venue through the 'Revision' button of the request form.
Recruitment can happen at any point in the workflow. All participants (Area Chairs, Reviewers, etc.) must have an OpenReview account linked to the email used in the recruitment message.
5. (optional) Create a registration task for Senior Area Chairs, Area Chairs and Reviewers
You may choose to add a task for program committee members to remind them to complete their registration.
Submission Phase
6. Submissions Open
Submissions automatically open on the date/time listed in the venue request form. If no date is given, submissions open as soon as the venue is deployed.
Note: All submitting authors must have an active OpenReview Profile.
7. (Optional) Abstract Registration Stage
Use this if you want to have an abstract deadline ahead of the main submission deadline.
8. Submissions close
Submissions will automatically close on the date specified in the venue request form. To change the submission deadline, see here.
9. Update readers/ hide fields from authors using Post Submission Stage
Set up matches between submissions and program committee
Set up matching should be done in the direction Senior Area Chairs-> Area Chairs-> Submissions, then Reviewers should be assigned to submissions. The basic stages of this workflow is to:
(optional) Bidding
Look at, and potentially modify proposed assignments
Deploy assignments
Confirm that assigned program committee are readers of their assigned submissions
Each of these steps needs to be completed for each group being matched (e.g. Area Chairs - Submissions , Reviewers - Submissions)- Described further in steps 11-16 below.
10. Assign Senior Area Chairs to Area Chairs
Most venues assign SACs to submissions by first assigning them to Area Chairs. Here you may decide whether to have SACs automatically assigned to ACs or allow them to bid on ACs based on affinity scores by following steps 1-5 above.
To assign SACs to ACs, choose 'Senior Area Chairs' as the matching group in paper matching setup. Then in the Paper Matching Stage, SACs will be the matching group, and ACs will be in the Submissions field.
Note: Matching between SACs and ACs will not calculate conflicts. Instead, the conflicts to the SACs will be transferred to the ACs and calculated at the AC matching stage.
Program Chairs can make reassignments after the proposed assignments are deployed or undeploy assignments.
It is very important to deploy Senior Area Chair assignments before assigning submissions to Area Chairs to allow conflicts to be transferred successfully.
If you decide to directly assign Senior Area Chairs to submissions, skip to step 12.
The next step is to assign the program committee to each submission. If your conference is smaller than 2000 submissions, this can be run by you directly, otherwise please contact OpenReview.
It is very important that all Program Committee members (Senior Area Chairs, Area Chairs, and Reviewers) have a complete OpenReview profile (an active profile with at least one active institution and one publication). We recommend removing from the committee groups all the profiles that are not complete before running the matching system.
To make sure that the program committee can access their assignments, ensure that they are listed as readers in the Submission Readers field of the venue request form, othewise update this value in the Post Submission stage.
12. (Optional) Bidding period for Senior Area Chairs, Area Chairs and Reviewers
Program Chairs can optionally ask the Senior Area Chairs, Area Chairs and/or Reviewers to bid on papers.
PCs must make all the existing submissions visible to all the members of the Senior Area Chairs/Area Chair/Reviewers group hiding the PDF, supplementary material and any other fields they don’t want Senior Area Chairs/Area Chair/Reviewers to see by using the Post Submission stage.
In order for conflicts to be taken into account in paper matching, matching set up (Step 14) must be run before the bid stage.
The bidding console shows the submissions sorted by affinity scores of the logged user and filtering out the ones that are in conflict with the user.
Note: when the conference is large, only sparse scores are uploaded to the system, this means we only keep the first N (400) scores for each user and submission. Users will see at least 400 submissions sorted by their affinity scores. If they want to see other submissions that are not among the top 400 they should use the “search” functionality.
After computing affinity scores and optionally enabling the bidding process, Program Chairs can run the paper matching system to assign Senior Area Chairs/Area Chairs/Reviewers to Submissions. Program Chairs can assign Senior Area Chairs/Area Chairs/Reviewers at different stages because they are independent processes.
14. (optional) Share Reviewer-Submission proposed assignments with Area Chairs to review and edit.
Some venues decide to share the Reviewer-Submission assignments with the Area Chairs to review before releasing them to the Reviewers.
In order to do this, Program Chairs need to deploy the Area Chair-Submission assignments so that Area Chairs can see their own assigned submissions and choose a matching configuration to share the proposed assignments. Then you can allow ACs to do reassignment.
Area Chairs can make modifications to these assignments and they can optionally invite external reviewers. This must be requested to the support team.
After the proposed assignments were reviewed by the Program Chairs and/or the Senior Area Chairs/Area Chairs, they can be deployed and be visible to the Reviewers. Deploying assignments doesn’t automatically send emails to the Reviewers- it is recommended that you as PCs notify them.
16. (optional) Modify assignments
Deployed assignments can be edited by the Program Chairs and Senior Area Chairs. Program Chairs can also decide if they want to allow Area Chairs to make reassignments. When a reassignment is done, an email notification is sent to the new Reviewer.
Review Stage
17. Start review period
You can start the review period through the Review Stage. You may also at this point customize the default review form.
Note: There are two fields with default names “rating” and “confidence” that are used to compute stats in the different consoles. You may modify these fields but make sure to specify the names in the Rating/Confidence Field Name field of the Revision Stage.
It is also important to choose what groups a review should be visible to. A common configuration is that reviews are only visible to the assigned Senior Area Chair, Area Chair and Reviewers. You may change the readers of reviews at any time using the Review Stage.
18. Review Stage ends
The deadline field of the Review Stage form will be the one shown to reviewers as the advertised deadline. The expiration date in the form (not seen by reviewers) will be the time after which no more reviews can be submitted. After this point, reviewers will need to contact the PCs for any late reviews.
19. Release reviews to the authors and other reviewers
If there are still pending reviews, setting Release Reviews to Authors to "Yes, reviews should be revealed when they are posted to the paper's authors", will release all posted reviews, and later will release pending reviews after they are posted.
20. (optional) Rebuttal stage
Usually venues have a rebuttal period where Authors can reply to the Reviewers. In OpenReview, the rebuttal period can start at any time using the rebuttal stage. They can choose between settings to allow a free number of rebuttal comments or require authors to have one rebuttal per Submission/Review. PCs may also use the comment stage so that reviewers can optionally reply to the authors and keep threaded discussions.
21. (optional) Submission Revision Stage
You may optionally allow authors to revise their submissions, including limiting which fields can be edited. This stage can be enabled any time after the submission deadline has passed.
Meta-review Stage
When the review period has concluded, the PCs can start the meta review period where the ACs make their recommendations.
23. (optional) Start meta review confirmation period
Use this if you have Senior Area Chairs and want them to review, confirm, or revise the meta review posted by the Area Chair.
24. (optionaI) ACs rate reviews
You may also optionally allow Area Chairs to submit ratings for their reviews (found in venue request form).
Decision Stage and Camera Ready Revisions
This is the last step before releasing the decision to the authors. PCs need to submit the final decisions based on the AC meta reviews and confirmations. Decisions are visible to the PCs only.
For large venues (>2000 submissions) we offer a bulk upload process where the PCs can get the meta reviews values, edit them to meet the acceptance rates and then upload them to the system. The PC console will show the decision stats and decision notes can be edited after they are uploaded.
Once all the decisions are made and uploaded to the system, you may release them to the authors and send email notifications using the Post Decision stage. OpenReview offers a form where the PCs can define the email template for each decision.
PCs can also (optionally) decide to release the submissions to the public (all the submissions or accepted only) and deanonymize the author names.
27. Start camera ready period
The camera ready period starts after the authors are notified about the submission decisions. To begin, open the Submission Revision Stage. Make sure to enable the setting 'Enable revision for accepted submissions only'.
28. End camera ready period
Authors will no longer be able to edit submissions after the Camera Ready period deadline set in the Submission Revision Stage. After this point, they will need to contact the PCs to allow any revisions after the deadline.
Last updated