OpenReview
  • Overview
    • OpenReview Documentation
  • Reports
    • Conferences
      • OpenReview NeurIPS 2021 Summary Report
      • OpenReview ECCV 2020 Summary Report
  • Getting Started
    • Frequently Asked Questions
      • I accidentally withdrew a submission, what do I do?
      • How do I add a Program Chair to my venue?
      • When will I be able to withdraw my submission?
      • I want to delete my withdrawn or desk-rejected paper, what do I do?
      • An author of a submission cannot access their own paper, what is the problem?
      • What should I do if I find a vulnerability in OpenReview?
      • How can I report a bug or request a feature?
      • What is the difference between due date (duedate) and expiration date (expdate)?
      • Will Reviewers be notified of their Assignments?
      • What is the max file size for uploads?
      • Why are the "rating" and "confidence" fields in my PC Console wrong?
      • What should I do if my question is not answered here?
      • My Profile is "Limited". What does that mean?
      • What field types are supported in the forms?
      • How do I recruit reviewers?
      • How do I obtain a letter of proof for my services as a reviewer?
      • How do I complete my tasks?
      • Can I automatically transfer my Expertise Selection to another venue?
      • Why does it take two weeks to moderate my profile?
      • What do the different 'status' values mean in the message logs?
      • I am an Independent Researcher, how do I sign up?
      • How do I locate the date a submission is made public?
      • I am a reviewer but I can't access my assigned submissions, what do I do?
      • Reviewers for my venue cannot see their assigned submissions, what should I do?
      • I am a reviewer and I don't have papers for Expertise Selection, what do I do?
      • How do I upload a publication with a license that is not listed?
      • I didn't receive a password reset email, what do I do?
      • How do I add/change an author of my submission after the deadline?
      • How do I find a venue id?
      • Why can't I update my DBLP link?
    • Using the API
      • Installing and Instantiating the Python client
      • Groups
    • Hosting a venue on OpenReview
      • Creating your Venue Instance
      • Navigating your Venue Pages
      • Customizing your submission form
      • Enabling Supplementary Material Upload
      • Changing your submission deadline
      • Enabling an Abstract Registration Deadline
    • Creating an OpenReview Profile
      • Signing up for OpenReview
      • Resending an activation link
      • Expediting Profile Activation
      • Add or remove a name from your profile
      • Add or remove an email address from your profile
      • Finding your profile ID
      • Entering Institutional Data
      • Importing papers from DBLP
      • Manually adding a publication to your profile
      • Finding and adding a Semantic Scholar URL to your profile
      • Finding and adding your ACL Anthology URL to your profile
      • Merging Profiles
    • Customizing Forms
    • Using the New Forum Page
    • Live Chat on the Forum Page
  • Workflows
    • Example Workflow
    • ARR Commitment Venues
    • Exercises for workflow chairs
      • Prerequisites
      • Exercise: Posting LLM generated reviews
  • How-To Guides
    • Modifying Venue Homepages
      • How to customize your venue homepage
      • How to modify the homepage layout to show decision tabs
    • Managing Groups
      • How to Recruit and Remind Recruited Reviewers
      • How to have multiple Reviewer or Area Chair groups
      • How to Add and Remove Members from a Group
      • Publication Chairs
      • How to Copy Members from One Group to Another
    • Workflow
      • How to Programmatically Post Support Request Form
      • How to test your venue workflow
      • How to Post a Test Submission
      • How to support different tracks for a venue
      • How to Make Submissions Available Before the Submission Deadline
      • How to Change the Expiration Date of the Submission Invitation
      • Desk Reject Submissions that are Missing PDFs
      • How to begin the Review Stage while Submissions are Open
      • How to Change Who can Access Submissions After the Deadline
      • How to Enable Commenting on Submissions
      • How to Set a Custom Deadline for Withdrawals
      • How to Enable an Ethics Review Stage
      • How to Hide Submission Fields from Reviewers
      • How to modify the Review, Meta Review, and Decision Forms
      • How to release reviews
      • How to Enable the Rebuttal Period
      • How to Undo a Paper Withdrawal
      • How to enable Camera Ready Revision Upload for accepted papers
      • How to make papers public after decisions are made
      • How to enable bidding for Senior Area Chair Assignment
      • How to release the identities of authors of accepted papers only
      • How to enable the Review Revision Stage
    • Paper Matching and Assignment
      • How to Compute Conflicts Between Users
      • How to Post a Custom Conflict
      • How to create your own Conflict Policy
      • How to Bid on Submissions
      • How to add/remove bids programmatically
      • How to do manual assignments
      • How to do automatic assignments
        • How to setup paper matching by calculating affinity scores and conflicts
        • How to run a paper matching
        • How to modify the proposed assignments
        • How to deploy the proposed assignments
        • How to modify assignments after deployment
      • How to enable Reviewer Reassignment for Area Chairs
      • How to Sync Manual and Automatic Assignments
      • How to Compute Affinity Scores
      • How to Undo Deployed Assignments
      • How to Modify Reviewer Assignments as an Area Chair
      • How to Get all Assignments for a User
      • How to Update Custom Max Papers for Reviewers or ACs
      • How to Make Assignments using Subject Areas
    • Communication
      • How to send messages through the UI
      • How to customize emails sent through OpenReview
      • How to send messages with the python client
      • How to Send Decision Notifications Using the UI
      • How to view messages sent through OpenReview
      • How to email the authors of accepted submissions
      • How to get email adresses
    • Submissions, comments, reviews, and decisions
      • How to add formatting to reviews or comments
      • How to submit a Review Revision
      • How to add formulas or use mathematical notation
      • How to edit a submission after the deadline - Authors
      • How to upload paper decisions in bulk
      • How to hide/reveal fields
      • Update camera-ready PDFs after the deadline expires
    • Data Retrieval and Modification
      • How to check the API version of a venue
      • How to view Camera-Ready Revisions
      • How to Export all Submission Attachments
      • How to loop through Accepted Papers and print the Authors and their Affiliations
      • How to add/remove fields from a submission
      • How to manually change the readers of a note
      • How to post/delete an Official Review using Python
      • How to Get Profiles and Their Relations
      • How to Get All the Reviews that I have written and their Corresponding Submissions
      • How to Get All Registration Notes
      • How to Get All Submissions
      • How to Get All Reviews
      • How to Export All Reviews into a CSV
      • How to get all Rebuttals
      • How to Get All Official Comments
      • How to Get All MetaReviews
      • How to Get All Decisions
      • How to Get All Venues
      • How to Retrieve Data for ACM Proceedings
      • How to Get Reviewer Ratings
  • Reference
    • API V1
      • OpenAPI definition
      • Entities
        • Edge
          • Fields
        • Note
          • Fields
        • Invitation
    • API V2
      • OpenAPI definition
      • Entities
        • Edge
          • Fields
        • Group
          • Fields
        • Note
          • Fields
        • Invitation
          • Types and Structure
          • Fields
          • Specifiers
          • Dollar Sign Notation
        • Edit
          • Fields
          • Inference
    • Stages
      • Revision
      • Registration Stage
      • Bid Stage
      • Review Stage
      • Rebuttal Stage
      • Meta Review Stage
      • Decision Stage
      • Comment Stage
      • Submission Revision Stage
      • Post Submission Stage
      • Post Decision Stage
      • Ethics Review Stage
    • Default Forms
      • Default Submission Form
      • Default Registration Form
      • Default Comment Form
      • Default Review Form
      • Default Rebuttal Form
      • Default Meta Review Form
      • Default Decision Form
      • Default Decision Notification
      • Default Ethics Review Form
    • OpenReview TeX
      • Common Issues with LaTeX Code Display
      • OpenReview TeX support
    • Mental Model on Blind Submissions and Revisions
Powered by GitBook
On this page
  • Setting up a venue
  • Submission Phase
  • Set up matches between submissions and program committee
  • Review Stage
  • (Optional) Rebuttal Stage
  • (Optional) Submission revision stage
  • (Optional) Meta-review Stage
  • Decision Stage and Camera Ready Revisions

Was this helpful?

Export as PDF
  1. Workflows

Example Workflow

Guide based on ICLR venue

PreviousLive Chat on the Forum PageNextARR Commitment Venues

Last updated 3 months ago

Was this helpful?

How to use this document: This document lists the major steps of running a conference venue. Each step links to relevant documentation that explains it in depth.

Setting up a venue

1.

Fill out the venue request form and choose settings for your venue.

OpenReview will review the request, ask for any necessary clarification, then you will receive an email notifying you that your venue has been deployed.

Also see:

2.

Ensure all PCs have OpenReview accounts associated with the email listed in the venue request form in order to access venue pages.

3.

You may edit many settings for your venue through the 'Revision' button of the request form.

4.

Recruitment can happen at any point in the workflow. All participants (Area Chairs, Reviewers, etc.) must have an OpenReview account linked to the email used in the recruitment message.

5. (optional)

You may choose to add a task for program committee members to remind them to complete their registration.

Submission Phase

6. Submissions Open

Submissions automatically open on the date/time listed in the venue request form. If no date is given, submissions open as soon as the venue is deployed.

Note: All submitting authors must have an active OpenReview Profile.

Use this if you want to have an abstract deadline ahead of the main submission deadline.

8. Submissions close

Set up matches between submissions and program committee

Set up matching should be done in the direction Senior Area Chairs-> Area Chairs-> Submissions, then Reviewers should be assigned to submissions. The basic stages of this workflow is to:

Each of these steps needs to be completed for each group being matched (e.g. Area Chairs - Submissions , Reviewers - Submissions)- Described further in steps 11-16 below.

10. Assign Senior Area Chairs to Area Chairs

To assign SACs to ACs, choose 'Senior Area Chairs' as the matching group in paper matching setup. Then in the Paper Matching Stage, SACs will be the matching group, and ACs will be in the Submissions field.

Note: Matching between SACs and ACs will not calculate conflicts. Instead, the conflicts to the SACs will be transferred to the ACs and calculated at the AC matching stage.

It is very important to deploy Senior Area Chair assignments before assigning submissions to Area Chairs to allow conflicts to be transferred successfully.

If you decide to directly assign Senior Area Chairs to submissions, skip to step 12.

The next step is to assign the program committee to each submission. If your conference is smaller than 2000 submissions, this can be run by you directly, otherwise please contact OpenReview.

It is very important that all Program Committee members (Senior Area Chairs, Area Chairs, and Reviewers) have a complete OpenReview profile (an active profile with at least one active institution and one publication). We recommend removing from the committee groups all the profiles that are not complete before running the matching system.

To make sure that the program committee can access their assignments, ensure that they are listed as readers in the Submission Readers field of the venue request form, othewise update this value in the Post Submission stage.

12. (Optional) Bidding period for Senior Area Chairs, Area Chairs and Reviewers

In order for conflicts to be taken into account in paper matching, matching set up (Step 14) must be run before the bid stage.

The bidding console shows the submissions sorted by affinity scores of the logged user and filtering out the ones that are in conflict with the user.

Note: when the conference is large, only sparse scores are uploaded to the system, this means we only keep the first N (400) scores for each user and submission. Users will see at least 400 submissions sorted by their affinity scores. If they want to see other submissions that are not among the top 400 they should use the “search” functionality.

After computing affinity scores and optionally enabling the bidding process, Program Chairs can run the paper matching system to assign Senior Area Chairs/Area Chairs/Reviewers to Submissions. Program Chairs can assign Senior Area Chairs/Area Chairs/Reviewers at different stages because they are independent processes.

14. (optional) Share Reviewer-Submission proposed assignments with Area Chairs to review and edit.

Some venues decide to share the Reviewer-Submission assignments with the Area Chairs to review before releasing them to the Reviewers.

After the proposed assignments were reviewed by the Program Chairs and/or the Senior Area Chairs/Area Chairs, they can be deployed and be visible to the Reviewers. Deploying assignments doesn’t automatically send emails to the Reviewers- it is recommended that you as PCs notify them.

16. (optional) Modify assignments

Review Stage

17. Start review period

The deadline field of the Review Stage form will be the one shown to reviewers as the advertised deadline. The expiration date in the form (not seen by reviewers) will be the time after which no more reviews can be submitted. After this point, reviewers will need to contact the PCs for any late reviews.

If there are still pending reviews, setting Release Reviews to Authors to "Yes, reviews should be revealed when they are posted to the paper's authors", will release all posted reviews, and later will release pending reviews after they are posted.

(Optional) Rebuttal Stage

20. Start Rebuttal stage

Readers of the rebuttal must match the review readers. PCs can check the review readers selected in the Review Stage, in particular: Release Reviews To Authors and Release Reviews To Authors, and match the rebuttal readers to these options

(Optional) Submission revision stage

You may optionally allow authors to revise their submissions, including limiting which fields can be edited. This stage can be enabled any time after the submission deadline has passed.

(Optional) Meta-review Stage

23. (optional) Start meta review confirmation period

Use this if you have Senior Area Chairs and want them to review, confirm, or revise the meta review posted by the Area Chair.

24. (optionaI) ACs rate reviews

Decision Stage and Camera Ready Revisions

For large venues (>2000 submissions) we offer a bulk upload process where the PCs can get the meta reviews values, edit them to meet the acceptance rates and then upload them to the system. The PC console will show the decision stats and decision notes can be edited after they are uploaded.

PCs can also (optionally) decide to release the submissions to the public (all the submissions or accepted only) and deanonymize the author names.

27. Start camera ready period

28. End camera ready period

Authors will no longer be able to edit submissions after the Camera Ready period deadline set in the Submission Revision Stage. After this point, they will need to contact the PCs to allow any revisions after the deadline.

7. (Optional)

Submissions will automatically close on the date specified in the venue request form. To change the submission deadline, see .

9. Update readers/ hide fields from authors using

(optional)

Look at, and potentially proposed assignments

assignments

that assigned program committee are readers of their assigned submissions

Most venues assign SACs to submissions by first assigning them to Area Chairs. Here you may decide whether to have SACs automatically assigned to ACs or based on affinity scores by following steps 1-5 above.

Program Chairs can make reassignments after the proposed assignments are deployed or assignments.

11.

Program Chairs can optionally ask the Senior Area Chairs, Area Chairs and/or Reviewers to .

PCs must make all the existing submissions visible to all the members of the Senior Area Chairs/Area Chair/Reviewers group hiding the PDF, supplementary material and any other fields they don’t want Senior Area Chairs/Area Chair/Reviewers to see by using the stage.

13.

In order to do this, Program Chairs need to the Area Chair-Submission assignments so that Area Chairs can see their own assigned submissions and choose a matching configuration to share the proposed assignments. Then you can .

. This must be requested to the support team.

15.

Deployed assignments can be by the Program Chairs and Senior Area Chairs. Program Chairs can also decide if they want to . When a reassignment is done, an email notification is sent to the new Reviewer.

You can start the review period through the . You may also at this point the.

Note: There are two fields with default names “rating” and “confidence” that are used to compute stats in the different consoles. You may modify these fields but make sure to specify the names in the field of the Revision Stage.

It is also important to choose what groups a review should be visible to. A common configuration is that reviews are only visible to the assigned Senior Area Chair, Area Chair and Reviewers. You may change the readers of reviews at any time using the

18. ends

19. to the authors and other reviewers

Usually venues have a rebuttal period where Authors can reply to the Reviewers. In OpenReview, the rebuttal period can start at any time using the . They can choose between settings to allow a free number of rebuttal comments or require authors to have one rebuttal per Submission/Review.

PCs may also use the so that reviewers can optionally reply to the authors and keep threaded discussions.

21. Start

22.

When the review period has concluded, the PCs can start the where the ACs make their recommendations.

You may also optionally allow Area Chairs to submit ratings for their reviews (found in ).

25.

This is the last step before releasing the decision to the authors. PCs need to submit the final based on the AC meta reviews and confirmations. Decisions are visible to the PCs only.

26.

Once all the decisions are made and uploaded to the system, you may them to the authors and send email notifications using the . OpenReview offers a where the PCs can define the email template for each decision.

The camera ready period starts after the authors are notified about the submission decisions. To begin, open the . Make sure to enable the setting 'Enable revision for accepted submissions only'.

Abstract Registration Stage
here
Post Submission Stage
Set up matching
Bidding
Run matching
modify
Deploy
Confirm
allow them to bid on ACs
undeploy
Setup Paper Matching between Senior Area Chairs/Area Chairs/Reviewers and submissions
bid on papers
Post Submission
Run matching between Submissions and Senior Area Chairs/Area Chairs/Reviewers
deploy
allow ACs to do reassignment
Area Chairs can make modifications to these assignments and they can optionally invite external reviewers
Deploy proposed assignments for Senior Area Chairs, Area Chairs and Reviewers
edited
allow Area Chairs to make reassignments
Review Stage
customize
default review form
Release reviews
rebuttal stage
comment stage
Submission Revision Stage
meta review period
venue request form
decisions
Release decisions to the authors and notify authors
release
Post Decision stage
form
Submission Revision Stage
Rating/Confidence Field Name
Review Stage.
Review Stage
Start meta review period
Submit decisions by Program Chairs
Customizing venue homepage
Customizing submission form
Review venue pages
Change settings of your venue as necessary from the venue request form
Recruit Senior Area Chairs, Area Chairs, Reviewers
Create a registration task for Senior Area Chairs, Area Chairs and Reviewers
Submit venue request through the OpenReview Site