For OpenReview Profiles

Signing up for OpenReview

• I am an Independent Researcher, how do I sign up?

• I didn't receive a password reset email, what do I do?

Modifying an existing profile

• How can I add or remove a name from my profile?

• How can I add or remove an email address from my profile?

• How can I import papers from DBLP?

• How do I complete my tasks?

• How can I add a Semantic Scholar URL to my profile?

• My profile is "limited". What does that mean?

• How do I upload a publication with a license that is not listed?

• Why can't I update my DBLP link?

For Authors/Reviewers

• I am a reviewer but I can't access my assigned submissions, what do I do?

• How do I add/change an author of my submission after the deadline?

• How do I locate the date a submission is made public?

• How can I add formatting to my reviews/comments?

• Why is my LaTeX code not displaying properly?

• How do I obtain a letter of proof for my services as a reviewer?

• When will I be able to withdraw my submission?

• I accidentally withdrew a submission, what do I do?

• I am a reviewer and I don't have papers for Expertise Selection, what do I do?

• I want to delete my withdrawn or desk-rejected paper, what do I do?

For Program Chairs

Setting up a venue

• How can I access the request form for my venue?

• How do I find a venue id?

• How do I add a Program Chair to my venue?

• How can I override the information on my venue homepage?

• How do I recruit reviewers?

• How can I edit the Submission Form?

• How can I test my venue workflow?

• What customizations are available for emails sent through OpenReview?

• Reviewers for my venue cannot see their assigned submissions, what should I do?

• What field types are supported in the forms?

• What do the default submission, review, metareview, and decision forms look like?

Managing active venues

• How can I extend the Submission deadline?

• How can I release reviews?

• How can I enable Camera-Ready Revisions?

• How can I manually assign Reviewers/Area Chairs?

• How can I automatically assign Reviewers/Area Chairs?

• What do the different 'status' values mean in the message logs?

• How can I contact Reviewers, Area Chairs, and/or Authors?

• Where can I view the Camera-Ready Versions?

• How can I export all supplementary material/PDF/zip files?

• An author of a submission cannot access their own paper, what is the problem?

• What is the max file size for uploads?

• What is the difference between due date (duedate) and expiration date (expdate)?

• Will Reviewers be Notified of their Assignments?

Other

• What should I do if my question is not answered here?

• What should I do if I find a vulnerability in OpenReview?

• How can I report a bug or request a feature?

For Authors

The timeline for submission withdrawals is determined by the Program Chairs of each venue. Therefore, there is not a single overarching OpenReview withdrawal policy. If you have questions about when in a venue workflow you will be able to withdraw your submission, you should reach out to the Program Chairs of that venue directly.

For Program Chairs

Authors are able to withdraw their paper at any time after the submission deadline or after Post Submission Stage has been run. You can optionally restrict the withdrawal window from your venue request form. You can also use the venue request form to configure the visibility of withdrawn and desk-rejected papers, as well as the identities of their authors.

Individual conferences determine the policy on the visibility of withdrawn and desk-rejected submissions. Please refer to the conference website for further information on the policy, and contact the venue organizers with any questions or concerns.

The author must make sure that the email address associated with the submission is added to their profile and confirmed.

To revert a withdrawal, please contact the Program Chairs of your conference using the contact email on the venue homepage.

The OpenReview Documentation is divided into 3 main sections:

OpenReview ECCV 2020 Summary Report

(The Computer Vision Foundation, CVF, has the long-term goal of unifying the CVPR conference workflow tools under one integrated infrastructure. Seeing the success of OpenReview for ICLR over the past seven years, CVF has been providing the OpenReview Foundation with a multi-year financial gift towards this new software development. CVPR is hoping to move to OpenReview in the future.)

ECCV 2020 workflow was not fundamentally different from its previous years: double blind, closed reviewing, with area chairs, closed reviewer discussion, author reponses, and meta-reviews.

Workflow details and timing were planned extensively with shared Google Docs, and three video conference meetings with the OpenReview team. Through the submission and reviewing process OpenReview technical staff provided 24/7 support to the ECCV program chairs, including rapid responses and custom work over weekends and evenings.

• Reviewer recruiting. Based on a list provided by ECCV PCs, OpenReview invited over 5k reviewers, and automatically gathered their responses. We also coordinated with ICLR to invite additional reviewers from ICLR’s 2020 reviewer pool.

• Reviewer & author registration. OpenReview already had profiles for approximately 100k researchers. For ECCV we added an additional ~3k reviewer profiles, and incorporated their papers from DBLP, running our own version of author coreference, augmented by verification performed by OpenReview staff. ECCV required all authors to register with OpenReview (mostly for the purposes of conflict-of-interest resolution, and gathering multiple email addresses per person); this resulted in ~12k additional profiles being created.

• Conflicts-of-interest gathering. Author and reviewer profiles include not only current institution domain names, but DBLP url, Google Scholar url, past advisors, and other non-institutional conflicts. OpenReview could in the future also create conflicts based on paper co-authorship within the last N years; in future ECCV may use this feature also.

• Reviewer expertise modeling. Expertise models were built for all reviewers, using modern deep learning embedding methods run on titles and abstracts of reviewers’ papers. In future, OpenReview expertise modeling will also use paper full-text and citation graphs. ECCV 2020 decided to use a combination of both OpenReview reviewer-paper affinities as well as those from TPMS.

• Paper submissions. As requested by ECCV 2020 PCs, draft paper titles and abstracts were submitted one week before the full-paper deadline. OpenReview received 7646 paper submissions. In the 24 hours before the final deadline, OpenReview received over 24k submission updates, and had over 19k active users (over 3.7k active simultaneous users during the last hour of submissions). The OpenReview multi-server system never surpassed 50% CPU usage, and maintained smooth operation with rapid system response throughout. (In contrast, the ECCV static web server simply providing submission deadline information became unresponsive.) In addition, during the submission period over 55k email messages were sent to authors (sent to each author for each update).

• Paper double-submission check. ECCV used the OpenReview service that checks for double submissions against ICML 2020 and NeurIPS 2020.

• Bidding. Both ACs and reviewers bid on papers, assigned as a “task” that was not complete until a given number of bids had been entered. During reviewer bidding, ACs and reviewers were able to search submissions by keyword.

• Specialized consoles: OpenReview provided specialized consoles for reviewers, area chairs, and program chairs, including functionality such as task lists, reviewing status, search, reviewer re-assignment, aggregate statistics, status of bids for each revidewer, status of review completion, sending email to remind reviewers, the ability to dump data as downloadable CSV files.

• Review rating. ECCV PCs requested that area chairs be able to rate the quality of each review on the scale -1, 0, 1, 2. From this reviewers were assigned an aggregate rating, what also included information about their tardiness. These aggregated reviewer ratings are stored (privately) in OpenReview, so that they will be easily (and programmatically) available to future ECCV program chairs. (We are also hoping to encourage private sharing of these ratings across conferences.)

• Paper ranking. ECCV program chairs requested that ACs be able to enter a ranking of their assigned papers.

• Decisions. PCs downloaded various CSV files into Google Sheets, including AC decisions. Some decisions were modified by the PCs. Then OpenReview emailed and posted the decision based on this Google Sheet. (In future, OpenReview may provide browsing, sorting, and editing directly through its UI; avoiding the need for Google Sheets. Alternatively, we may more closely embrace Google Sheets––leveraging its features––with live bi-directional data updates between OpenReview and the Google Sheet.)

• Camera-ready revisions. OpenReview created additional upload invitations and tasks for accepted paper authors, including copyright form, supplementary materials (including videos), camera-ready LaTeX zip file.

• Conference track formation. OpenReview also provided affinity scores between accepted papers, as input to paper clustering, for conference track assignments.

Feedback from Thomas Brox, ECCV 2020 Program Co-chair: Very happy with how OpenReview worked, and would recommend it to future program chairs. Particularly liked: (a) very stable and reliable system, (b) great response time and availability of the team, (c) excellent custom service (even implementing custom features we needed), (d) expressive conflict management (this was a primary impetus for moving to OpenReview, (e) reviewer assignment tools. Improvements that would be helpful for next year: feature allowing program chairs to impersonate another user (as CMT allows); additional reviewer-assignment constraints limiting the number of papers from the same institution on one paper, and multiple of the new features listed below.

OpenReview team’s plans for improvement, including

new system features:

• Allow program chairs to impersonate another user (as CMT allows), for purposes of understanding reviewer and area chair questions.

• Additional reviewer-assignment constraints limiting the number of papers from the same institution on one paper.

new UI features:

• Reviewer re-assignment directly from the convenient OpenReview “Edge Browser” interface (without the need to visit the PC console).

• Faster load times of the PC console when there are >5k submitted papers.

• Improved UI and organization of the “forum” page containing per-paper reviews and discussion: Easier way to read one-to-one discussion and distinguish between different types of replies: reviews, comments, rebuttals. More self-documenting “idiot-proof” UI widget for discussion participants to select the readers of the comments they enter.

• Allow ACs to download all their assigned paper files in a zip file.

• Add the ability for ACs and reviewers to bid “in blocks,” for example, bidding (positively or negatively) on all submissions containing a keyword, or in an area.’

• Add additional UI options for filtering papers, area chairs, or reviewers by various criteria, and then taking actions (such as sending email) on those objects satisfying the criteria.

new data gathering features:

• Improved expertise data, by automatically gathering the most recent computer vision conference publications that are not yet in DBLP. Improved expertise model based on the full-text and citations of each reviewers’ papers.

• Provide summary statistics of the number of past computer vision publications authored by each reviewer.

simple, alternative configuration for the next ECCV (no new system features needed):

• Restrict the list of papers shown to reviewers during the bidding stage: only the top N relevant submissions to each reviewer (rather than allowing reviewers to see all submissions).

• Allow the reviewer to edit the review after the rebuttal stage without showing the change to the authors until final decisions are released.

OpenReview NeurIPS 2021 Summary Report

In 2021 the organizers of NeurIPS (one of the flagship conferences in machine learning) decided to move from CMT to OpenReview. This report provides a summary of the NeurIPS 2021 workflow, the OpenReview services provided, the system performance, and enhancements planned for the next NeurIPS.

The volume of NeurIPS paper submissions has been increasing dramatically over the past decade. NeurIPS also has a history of innovation in peer review.

NeurIPS 2021 workflow was very similar to its previous years: double blind, closed reviewing, with area chairs and senior area chairs, and meta-reviews, with the addition in 2021 of rolling reviewer discussion with authors (rather than a single author response).

Workflow details and timing were planned extensively with the OpenReview team, and coordinated through Google Docs, several video conference meetings, and conversations through a shared Slack channel. Throughout the submission and reviewing process OpenReview technical staff provided 24/7 support to the NeurIPS program chairs, including rapid responses and custom work.

Reviewer recruiting. NeurIPS PCs invited over 13k reviewers, 1k area chairs and 155 senior area chairs. With the permission of ICLR, OpenReview also shared with the PCs the list of accepted authors of the previous ICLR conference from 2016 until 2021.

• Reviewer & author registration. OpenReview already had profiles for approximately 228k researchers. During the reviewer recruiting and the paper submission 11k profiles were created, and incorporated their papers from DBLP, running our own version of author coreference, augmented by verification performed by OpenReview staff. NeurIPS required all authors (not just submitting authors) to register with OpenReview (mostly for the purposes of conflict-of-interest resolution, and gathering multiple email addresses per person). During the month of May 12,537 new user profiles were created, more than in any month of OpenReview’s history.

• Conflicts-of-interest gathering. Author and reviewer profiles include not only current institution domain names, but also a DBLP URL (from which authors imported all their publications), Google Scholar URL, and extensive conflict-of-interest information, including institutional history, advisors, other collaborators, and social connections, and other non-institutional conflicts. As requested by NeurIPS, we also added the ability to record private conflicts (not shown in the public web site). For NeurIPS review matching, OpenReview computed the conflicts based on institution history, all conflict relations listed above, and paper co-authorship within the last 3 years.

• Reviewer expertise modeling. Expertise models were built for all reviewers, using OpenReview’s own modern deep learning embedding methods run on titles and abstracts of reviewers’ papers. NeurIPS decided to use only our expertise model instead of TPMS or Semantic Scholar.

• Paper submissions. As requested by NeurIPS 2021 PCs, draft paper titles and abstracts were submitted one week before the full-paper deadline. OpenReview received 11,729 paper submissions. In the 24 hours before the final deadline, OpenReview received over 42k submission updates, and had over 28k active users (over 2.3k active simultaneous users during the last hour of submissions). The OpenReview multi-server system never surpassed 50% CPU usage, and maintained smooth operation with rapid system response throughout. In addition, during the submission period over 110k email messages were sent to authors (sent to each author for each update).

• Bidding. SACs bid on ACs and both ACs and reviewers bid on papers, assigned as a “task” that was not complete until a given number of bids had been entered. During reviewer bidding, SACs, ACs and reviewers were able to sort the ACs/papers by affinity scores or search by metadata.

• Specialized consoles: OpenReview provided specialized custom consoles for reviewers, area chairs, senior area chairs, ethic reviewers, ethic chairs, and program chairs––including functionality such as task lists, reviewing status, filtering entries with an filtering language (such as “papers with missing reviews”, or “papers where the average rating is higher than 3”), keyword search, reviewer re-assignment, aggregate statistics, status of bids for each revidewer, status of review completion, sending email to remind reviewers, the ability to dump data as downloadable CSV files.

• Reviewing and discussion. Reviews were entered directly into the OpenReview system, visible immediately to the ACs, then visible to authors and reviewers of the same paper after the reviewing deadline. An enhancement created specially at the request of NeurIPS, OpenReview implemented multiple tabs in the discussion forum of a paper (author discussion, committee discussion, all reviewing discussion, post-reviewing public discussion). OpenReview processed 37,284 reviews, 8,103 meta reviews and 452 ethics reviews. In addition, 101,112 confidential comments.

• Review rating. NeurIPS PCs requested that area chairs be able to rate the quality of each review. The PCs also allowed authors of the submissions to provide review feedback. The ratings and feedback were only visible to the Program Chairs.

• Ethics reviews. As requested by NeurIPS, for the first time OpenReview added configuration to handle ethics reviews. The Ethics Review Chairs assigned ethics reviewers to papers flagged with ethical issues. The OpenReview expertise matching system was used to suggest reviewers with the appropriate topical expertise.

• Decisions. OpenReview provides the ability to download various CSV files, which PCs downloaded into Google Sheets, including AC decisions. Some decisions were modified by the PCs. Then OpenReview emailed and posted the decision based directly on this Google Sheet. (In future, OpenReview may provide browsing, sorting, and editing directly through its UI; avoiding the need for Google Sheets. Alternatively, we may more closely embrace Google Sheets––leveraging its features––with live bi-directional data updates between OpenReview and the Google Sheet.)

• Camera-ready revisions. OpenReview created additional upload invitations and tasks for accepted paper authors, including copyright form, supplementary materials (including videos), camera-ready LaTeX zip file.

• Conference track formation. OpenReview also provided affinity scores between accepted papers, as input to paper clustering, for conference track assignments.

System Responsiveness

Throughout the submission period, the OpenReview system provided smooth service, with rapid response and smooth uptime.

Peer Review Experiments:

With the help and guidance of the team at OpenReview, NeurIPS 2021 ran the following experiments:

• To discourage resubmissions without substantial changes, authors were asked to declare if a previous version of their submission had been rejected from another peer-reviewed venue. Like the year before, authors of resubmissions were asked to describe the improvements made. This information was entered into OpenReview during the submission process. To evaluate resubmission bias, resubmission information was made visible to reviewers and area chairs only for a randomly chosen 50% of submissions. While the experiment allowed us to eliminate a significant bias, we can’t confidently ascertain there is none.

• Author perception experiment: OpenReview implemented a two-part author survey to help NeurIPS understand how well authors’ perception of their submissions agrees with reviewing outcomes. The results of this experiment are forthcoming.

Releasing the data to the public:

Submissions under review were visible only to assigned program committee members, and we did not solicit comments from the general public during the review process. After the notification deadline, accepted papers were made public and open for non-anonymous public commenting, along with their anonymous reviews, meta-reviews, and author responses.

By default, rejected submissions were not made public, but authors of rejected submissions were given 2 weeks to opt in to make their de-anonymized papers public and open for commenting in OpenReview. If they chose to do so, this also opened up the reviews, meta-reviews, and any discussion with the authors for these papers. This policy does give authors a mechanism to publicly flag and expose potential problems with the review process. In the end, only about 2% of rejected papers opted in.

Feedback from Alina Beygelzimer, NeurIPS 2021 Program Co-chair:

“As Program Chairs for NeurIPS 2021, we decided to shift the entire reviewing workflow to OpenReview. OpenReview is a flexible platform that allows heavy customization, and will be easy to adapt as the needs of the conference evolve. It brings a number of infrastructural improvements including persistent user profiles that can be self-managed, accountability in conflict-of-interest declarations, and improved modes of interaction during the discussion process. NeurIPS has a long history of experimentation with the goal of informing and improving the review process (e.g., the widely known “NeurIPS Consistency Experiment” of 2014). This year we took full advantage of the great flexibility of OpenReview’s workflow configuration to run several key experiments (including a version of the noise audit that hasn’t been done since 2014). We are grateful to the OpenReview team for supporting all requested experimentation.

Our experience with OpenReview has been a delight. Not only did the paper deadline proceed smoothly (with sub-second system response time throughout the arrival of thousands of submissions just before the submission deadline), but OpenReview gracefully handled more than 20K authors accessing the system roughly at the same time to read and respond to preliminary reviews, and enabled 10K reviewers and Area Chairs and 20K authors to engage in discussions in the weeks that followed. The feedback we received from our authors and program committee members has been overwhelmingly positive.

I hope that NeurIPS will continue to work with OpenReview for years to come. We are hugely grateful to the OpenReview team, for their unparalleled level of support to everyone involved in the review process. OpenReview has also supported the Data & Benchmarks track (new this year) as well as the Ethics Review process for both the main conference and the Data & Benchmarks track. It is also notable that over 20 of the NeurIPS workshops have chosen to use OpenReview for their reviewing workflow this year.”

OpenReview team’s plans for improvement. The OpenReview system is ready for re-use for future NeurIPS conference reviewing needs. The OpenReview team continues to make improvements and new features. Current work likely to be ready for NeurIPS 2022 includes

• We are currently designing a new version of the paper reviewing discussion forum, and would be eager for feedback and feature requests. NeurIPS concerns about “rolling discussions” could be addressed here.

• Further improvements to the reviewer-paper matching system.

• Deployment of a new API providing (1) additional flexibility for fine-grained per-field control of visibility, (2) ease of changing readership permission of content, (3) better storage and access of the history of changes to a paper, review, or comment, (4) creation of “CRON”-jobs for automated sending of reminders.

In future, we will also have support for synchronous chat-style communication among reviewers, area chairs, and program chairs––which we hope will encourage more interactive, open, scientifically-flexible communication during the reviewing period. We are also building support for live conferences, integrated into the OpenReview reviewing platform.

After your venue is deployed, add Program Chair(s) by going to your Venue Request form, then adding the emails of the additional program chair(s) to the Program Chairs Email field of the form.

Due date is the advertised deadline.

Expiration date is the hard deadline.

A vulnerability is a defect (bug) in the system that compromises the integrity of the application or its data. If you found one, please contact us at info@openreview.net as soon as possible so that it can be fixed. Please do NOT post about the vulnerability elsewhere.

If you removed the rating or confidence fields from your Official Review form, the PC console will show an average rating and/or confidence of 0 for each paper. If you replaced them with custom values, you can customize your PC console to show the average of those values instead.

1. From your venue request form, click Review Stage.

2. Select which field you want to be used in place of rating and/or confidence. It must be in the "Additional Review Form Options" field. The options reviewers can select for that field must follow the format "number: description", for example "1: Very Poor".

3. Enter the field name for the "Review Rating Field Name" (or "Review Confidence Field Name"). It must match the case as it is entered in Additional Review Form Options.

4. Click "Submit".

You can report a bug or request a feature by going to our issue tracker repository. Before you do so, please read the guidelines so that we can understand and address your bug report or request. Once you are familiar with the procedure, you can make use of one of these templates to help us better understand your issue or feature request.

The max file size for uploads is 100 MB.

Reviewers will see their assigned papers in their Reviewer console and in their Task list. Depending on what method you used to assign Reviewers to papers, they may have already been notified of their assignments:

Automatic Assignment Deployed through the Edge Browser

If you deployed an assignment configuration from the edge browser, Reviewers would not have received a notification of their assignments.

Manual Assignment through the Edge Browser

If you use the "Assign" button to add a Reviewer to a paper, the Reviewer will receive a notification of their new assignment.

Throughout the course of a conference/workshop, members of the committee groups are assigned tasks to complete, such as, Expertise Selection, Reviewing, and the Reviewer License Agreement. Users can find their tasks by clicking on the Tasks button (openreview.net/tasks) on the red nav bar at the top of the page. They can also find these tasks in their respective venue consoles.

They'll see a list of different venue IDs and the number of pending and completed tasks.

To view the tasks, click on the arrow next to the venue ID or "Show pending tasks and completed tasks". Clicking on a task link will redirect the user to another page to complete the task. When the task is completed it will be grayed out on the Tasks page. When the expiration date for that task expires the task will disappear from the Tasks page.

Note: Some tasks, such as Expertise Selection, will continue to show as pending until the task expires, at which time it will be removed from the queue.

The best way to get help with a specific issue is to contact the program chairs or organizers of the venue you are participating in. Contact info can usually be found on the venue's OpenReview page.

This is not something that is generated by OpenReview. Please contact the organizers of the venue where you served as a reviewer to see if they can assist you.

To see a history of all emails sent to you by the organizers, please login and go to openreview.net/messages. To see your recent activity, including any reviews you may have submitted, you can go to openreview.net/activity.

OpenReview’s goal is to provide scientific communication to the entire global community. Initially OpenReview was not available to users in Iran, Cuba, Syria, and several other countries because our cloud service provider blocked access as a simple approach to comply with United States economic sanctions and trade laws.

Recently the OpenReview staff has done extra work to make OpenReview available in all countries, including those above. In order to comply with relevant laws, OpenReview must guarantee that none of our registered users appear in the Office of Foreign Assets Control of the U.S. Department of the Treasury (OFAC) SDN list. We primarily accomplish this by looking for first-last-name matches in the list. When there is a name match, we ask for additional information that would disambiguate the OpenReview user from the person on the SDN list.

If you were notified that your profile has been limited, this means that the name on your profile matches that of a person on the SDN list. While in a limited state, you can log in to OpenReview, edit your profile, and view the same amount of data as before, but you will be unable to author any notes on the OpenReview system. This means you will be prevented from performing most common actions, including submitting to venues or posting reviews, meta-reviews, or comments. In order to reactivate your profile, you will need to do the following:

• Log in to OpenReview at https://openreview.net/login

• Navigate to your Profile at https://openreview.net/profile

• Click the Edit button.

• Find the field where you can enter your year of birth, and enter the information.

• Click Save at the bottom of the page.

OpenReview will routinely check for SDN matches, and if your birth year has been updated, your profile will return to active. Note that this change will not be immediate. If you believe you have been waiting longer than expected and your profile has not been reactivated, you can reach out to info@openreview.net to request that your profile be updated.

Forms are rendered in the UI using invitations. You can read more about invitations in the link below.

API V2 Invitations

To recruit reviewers, there is a feature located on the venue request form called "Recruitment", clicking this button will allow you to send emails to potential reviewers. To find more information, please see our documentation on Reviewer Recruitment and Reminders.

Participants signing up for an OpenReview profile will notice a warning if they sign up with a email domain that is public (ex, gmail.com, 163.com, qq.com, etc.) or if the domain is not included in our institution list. The warning states that signing up with a public domain may spend up to 2 weeks in moderation. This warning has understandably triggered a large number of emails sent to the support address requesting an early activation. We see these emails, we may not answer each depending on the load, but we review moderation daily.

The warning was added by the staff in response to an increase in the number of fake/spam profiles being submitted with public domains, delaying moderation.

We encourage users to signup with their institutional (industry/government/university/research org) email domains to bypass moderation. "Institutional" in this case is defined as an established organization or corporation. If your domain is not recognized, please contact us through the feedback form shared in the signup page warning to add your domain to our list.

Sent: The initial status value, meaning the message was sent to the email service.

Dropped: The email was not sent to the recipient for delivery.

Deferred: The email cannot immediately be delivered, but has not been completely rejected. The service will continue to try for 72 hours to deliver the deferred message. After 72 hours the defferal turns into a block.

Bounce: The server cannot or will not deliver the message. This is often caused by outdated or an invalid email address.

Delivered: The email has been accepted by the receiving server. This does not mean that the email made it to the recipient's inbox.

Blocked: The email was denied temporarily by the server. This is unrelated to the validity of the address.

Publication date(pdate) means when the venue marked the paper as accepted.

Modification date(mdate) means when the last time the paper was modified.

The odate means when the submission becomes public. To check the odate of a submission, see the note object https://api2.openreview.net/notes?id=

To find the id you can check the url of the submission, for example https://openreview.net/forum?id=KS8mIvetg2 . Use id=KS8mIvetg2 , the note object url will be https://api2.openreview.net/notes?id=KS8mIvetg2.

Ensure that you are logged in with an active OpenReview account that has the email the invitation was sent to added and confirmed to it. To make an OpenReview account, see Signing up for OpenReview. To add an email to your account, see Add or remove an email address from your profile.

If you are logged in and are still not able to see your assigned submissions (but can see assignments under your Tasks), then reviewers have likely not been given reader access for submissions by the organizers of the venue.

Please contact the venue organizers notifying them of the issue. The contact email can be found on the homepage of the venue.

The OpenReview system uses the reviewer's publications from Expertise Selection in order to match reviewers and papers via affinity scores. Reviewers who do not have existing publications should leave the Expertise Selection form blank, and the reviewer will not be assigned an affinity score for a paper.

Decisions on how to assign reviewers without affinity scores is at the discretion of the venue organizers. Please contact the venue organizers if you have further questions about how review assignment will work for a specific venue.

Note: There is no 'Submit' button for Expertise Selection, and the task will show in your pending queue (where you can make changes) until the task expires at which time the task will automatically leave the queue.

See also: How do I complete my tasks?

We've received many emails from concerned researchers who want to sign up quickly but aren't affiliated with an academic institution.

If you are not affiliated with an academic institution, but are employed at a company, you can use that domain and position information to sign up. You may need to request OpenReview add your company as an institutional domain.

If you do not have a company domain, you can use a personal email. All personal emails will be moderated - see Expediting Profile Activation for additional tips on how to make sure your profile is activated as quickly as possible.

When filling out the profile details for registration, there is a section that will ask for your Education and Career History, which will require a current position. For the role you can type “Independent Researcher” and set that as your current position and type in a domain. See the page Entering Institutional Data for an example of how to fill out institutional information as an independent researcher. Please include as much of your background as possible. If the "Education and Career History" section only lists "Independent Researcher" we will ask you to add another institution record.

Please do not use an old role or affiliation as your current position, as that may delay moderation.

There are a few different ways to find your venue id.

1. If you are a Program Chair, go to your venue request form and copy the information in the field Venue ID

2. Use the URL of the submission page. The venue id is the portion of the url after group?id:For example for ICLR 2024 the submission homepage is : https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-accept-oral, and the venue id is ICLR.cc/2024/Conference

In order for a reviewer to see their assigned submissions, they need to be logged into an OpenReview account that has read permissions for the submission.

If a specific reviewer cannot access their assigned submission, they may need to check that the email they received the invitation to is added and confirmed to their account- The email should be listed in their profile with (Confirmed) next to it. If this is not the case, they should follow the steps to add an email to their account.

If all reviewers cannot access the forum page of their assigned submission, check that they are readers of the submission they are assigned to. On the Venue Request Form, make sure that the Submission Readers field includes reviewers. If reviewers are not listed as a reader, update the Post Submission stage to add them as readers for their submissions.

There are a few common reasons why you might not have received the password reset email. You can try the following steps to debug the issue

1. Check that the email was not sent to a spam or junk folder.

2. Try sending the password reset email again and double check that there are no typos in the email.

3. Reset your password using a different email that is listed in your profile (such as personal email, previous institutional email).

If after these steps, you are still having trouble resetting your password, email info@openreview.net

Expertise Selection is a task for Program Committee members (such as reviewers) that gives them the opportunity to identify which previous publications they would like to be considered when being matched with submissions to review. If you want to use the same Expertise Selection configuration for multiple venues, it is possible to transfer the expertise from one venue to another using the OpenReview Python client using the following steps.

1. Download and install the Python Client

import openreview
from openreview.api import OpenReviewClient

live_client_v2 = OpenReviewClient(
    baseurl='https://api2.openreview.net',
    username=YOUR_USERNAME,
    password=YOUR_PASSWORD
)

2. Get Expertise Selection edges from the previous venue

OpenReview by default considers all publications as relevant expertise for paper matching and assignment. In Expertise Selection, edges are posted indicating the papers that should be excluded from paper matching. You can get the edges from a previous venue using the following method:

#set up venue and user information
profile_id = YOUR_TILDE_ID 
from_venue_id = FROM_VENUE_ID 
from_group_name = FROM_GROUP_NAME  #Typically 'Reviewers' or 'Area_Chairs'

#get edges for from_venue
expertise_edges = openreview_client.get_all_edges(invitation=f"{from_venue_id}/{from_group_name}/-/Expertise_Selection",
 tail=profile_id)

You will need the following information:

• profile_id: Your profile ID (such as ~First_Last1). Also see Finding your Profile ID

• from_venue_id: The id of the venue you want to transfer expertise from. Also see: How do I find a venue ID?

• group_name: The name of the group you are in on the Program Committee (typically this is Reviewers or Area_Chairs)

The number of edges should correspond with the number of papers that you are excluding from Expertise Selection.

3. Copy edges to new venue

Finally, post an edge to the new venue that you are doing Expertise Selection for:

to_venue_id = TO_VENUE_ID
to_group_name = TO_GROUP_NAME
invitation = f"{to_venue_id}/{to_group_name}/-/Expertise_Selection"

for edge in expertise_edges:
    openreview_client.post_edge(openreview.api.Edge(
        invitation = invitation,
        head = edge.head,
        tail = edge.tail,
        signatures = [profile_id],
        label = 'Exclude'
    ))

OpenReview requires every publication to have a license. If you have a paper that is under a license that is not listed, please email info@openreview.net.

Each venue decides its own policy for how to handle requests to change the authorship list, order, or author details after the deadline. This is sometimes listed on the venue website or FAQ. If you need to request a change to your submission after the submission deadline has passed, please contact the venue organizers.

You can try the following steps to debug issues related to adding your DBLP link to your profile:

3. Check for previous profiles. Sometimes users have previously created profiles with DBLP links. If you get an error stating another user is using your profile, check for previous accounts using past emails.

5. Contact support. If none of the above steps work, contact us at info@openreview.net. Include your profile id, the DBLP link, and the error message you receive when you attempt to update your profile.

1. You will need to install the openreview-py client.

pip install openreview-py

2. Create a client object with your OpenReview credentials. If you do not yet have an OpenReview profile, you will need to make one now.

import openreview

# API V2
client = openreview.api.OpenReviewClient(
    baseurl='https://api2.openreview.net',
    username=<your username>,
    password=<your password>
)

# API V1
client = openreview.Client(
    baseurl='https://api.openreview.net',
    username=<your username>,
    password=<your password>
)

There are currently two APIs supported. All new venue requests are set to API 2 by default.

While most operations will work on both APIs, pay careful attention when that is not the case, for example, the JSON format for each API is different.

When a user is created, it is automatically assigned to certain groups that give them different privileges. A username is also a group, therefore, groups can be members of other groups.

If you used an abstract registration deadline and want to change the submission deadline after the abstract deadline has passed, you will need to rerun the Post Submission stage after changing the Submission Deadline.

Submitting a Venue Request Form

This is where you will select many of the settings for your venue such as general information about the venue, the committee roles, submission visibility, deadlines, etc.

For example, some venues have an abstract and a full paper deadline while others just have a full paper deadline. This can be specified in the form. Venue organizers can also decide if they want to force all the authors of the submission to have a profile or if they can enter the name/email address. The default setting is that the submission force accepts both options. Usually venue organizers give the authors 24 hours after the deadline to sign up before the submission is desk rejected.

Deployment

After you submit the form, our team will review it, ask for necessary modifications and deploy it; making your venue live. The deployment process does the following:

• Creates a venueid of the form AAAI.org/2025/Conference.

• Creates a home page for the venue located at https://openreview.net/group?id=[venueid]. This page contains the general information about the venue and the submission button.

• Creates the submission invitation open to any logged in user.

• Creates the committee groups and author groups with their respective consoles. The default committee group id for each role is as follows: Program_Chairs, Senior_Area_Chairs, Area_Chairs, Reviewers and Authors. They can be accessed from the home page or following the url https://openreview.net/group?id=[committee_group_id]. The console pages show different information depending on the role: paper assignments, pending tasks, status of the reviews, etc.

You can then edit some of your selected settings from the venue request form. Note that if you do not enter a submission start date, submissions will open immediately upon venue deployment.

There are two ways you can set an abstract registration deadline:

1. When you submit your Support Request Form

2. After your venue request form has been submitted, add an Abstract Registration deadline using the 'Revision' button on your venue request form.

Until the Abstract Registration deadline, authors will have the option to modify their submissions using the ‘edit’ button in the top right corner of their submission forum page.

After the Abstract Registration deadline passes, you will need to run the 'Post Submission Stage' from your venue request form. This creates the 'Revision' button and paper groups, which allow authors to access and edit their submissions. It also creates blinded copies to anonymize submissions, if applicable.

If you change the Submission deadline after the Abstract Registration deadline, you will need to re-run the ‘Post Submission Stage’ in order to update the revision deadline. After the Submission deadline, authors will lose the ability to revise their submissions.

The resulting field in the submission form would look like this:

Common Customizations

Asking authors to agree to conference policies

Limit read-permissions for certain fields

The readers field can be used to list who will be allowed to read a specific field of the submission form. The example below limits the readers of the private comments field to just authors, Assigned Senior Area Chairs, and Program Chairs.

Note: Authors will not be able to read these fields if they are not in the readers list

Adding tracks to your venue

Users can keep multiple names in their profiles and select one as preferred, which will be used for author submission and identity display. Names can be replaced by new names in the profile and in some submissions as long as the organizers of the venue allow it.

Adding a name to your profile and mark it as preferred

1. Go to your profile page at https://openreview.net/profile

2. Click 'Edit Profile'.

3. Locate the 'Names' section and click the blue plus sign underneath your name.

4. Enter your name and then click ‘Save Profile Changes’ at the bottom of the page.

If you would like to make that name preferred, you can do so by clicking ‘Edit profile’ once more and selecting ‘Make Preferred’ next to your desired name. The option to make a name preferred will not appear until you have saved the new name to your profile.

Removing a name from your profile

1. Go to your profile page at https://openreview.net/profile

2. Click 'Edit Profile'.

3. Locate the 'Names' section and make sure you have a name marked as preferred that you want to keep.

4. Click on the button 'Request Deletion' next to the name you want to remove.

5. In the new window, explain why you want that name to be removed and click 'Submit'

The process for removing the name will take some time until has been reviewed and accepted. You will receive email updates regarding the status of the name removal. Once the process is complete the name will be removed and any publications with the previous name will be updated with the one you marked as preferred in your profile.

You can add supplementary material to the submission form by clicking on the 'Revision' button and adding the following JSON under Additional Submission Options:

{
    "supplementary_material": {
        "value": {
            "param": {
                "type": "file",
                "extensions": ["zip", "pdf"],
                "maxSize": 50,
                "optional": true,
                "deletable": true
            }
        },
        "description": "All supplementary material must be self-contained and zipped into a single file. Note that supplementary material will be visible to reviewers and the public throughout and after the review period, and ensure all material is anonymized. The maximum file size is 100MB.",
        "order": 7,
        "readers": [ 
            "ICML.cc/2023/Conference", 
            "ICML.cc/2023/Conference/Submission${4/number}/Senior_Area_Chairs", 
            "ICML.cc/2023/Conference/Submission${4/number}/Area_Chairs", 
            "ICML.cc/2023/Conference/Submission${4/number}/Authors"
        ]
    }
}

The field readers is optional and it can be used to restrict the readers of the field, if you don't specify the readers then all the readers of the submission will be able to see the supplementary material. Make sure you use the right group ids to specify the readers.

This will add a supplementary material field to upload zipped files of size up to 50 MB. You can also enable a Submission Revision Stage to allow a separate deadline for Supplementary Material.

If your venue is using the API 1 (api_version = "1") then you should use the following JSON example:

{
  "supplementary_material": {
    "description": "Supplementary material (e.g. code or video). All supplementary material must be self-contained and zipped into a single file.",
    "order": 10,
    "value-file": {
        "fileTypes": [
            "zip"
        ],
        "size": 50
    },
    "required": false
  }
}

Adding email addresses to your profile and mark them as preferred

1. Go to your profile page at https://openreview.net/profile

2. Click 'Edit Profile'.

3. Locate the 'Emails' section and click the blue plus sign underneath your name.

4. Enter your email and then click 'Confirm'. An email will be sent to your new address.

5. DO NOT close your current window/tab. The verification number sent to your inbox needs to be input in the same window/tab.

6. Confirm that email by pasting the verification code and clicking on Verify.

If you would like to make that email preferred, you can do so by clicking ‘Edit profile’ once more and selecting ‘Make Preferred’ next to your desired email. The option to make an email preferred will not appear until you have confirmed the new email to your profile.

Removing email addresses from your profile

OpenReview uses email addresses associated with current or former affiliations for profile deduplication, conflict detection, and paper coreference. For this reason, OpenReview prefers to keep all current and former institutional email addresses on each user's profile. OpenReview will only send messages to the address marked as “Preferred”. OpenReview only displays obfuscated emails (****@umass.edu) and never releases full email addresses to the public.

If there is an email address you want removed from your profile, please reach out to info@openreview.net. Please note that if you have changed institutions, we ask you to add and confirm your new institutional email before requesting that we remove an old or expired email.

1. Making or finding a profile

To create a profile, go to https://openreview.net/signup. Enter your full name as you would like it to appear in publications. A common format is: First Last

By checking the confirmation box, existing profiles with your name will be listed.

Then, take one of the following actions, depending on whether there is a pre-created account, you already have a partial or complete OpenReview Profile, or you are signing up for the first time.

Claim Profile

If a Claim Profile option appears, this means that there is a pre-created profile in OpenReview with your name. These are often associated with DBLP urls, and if you believe an account should belong to you, click the Claim Profile button. Enter your email, click claim, and enter a password, then proceed to the Profile Registration step.

Contact support if: you no longer have access to an email associated with the profile you need to claim, such as a past institution.

Send Activation Link

If there is a profile created with your name, but you didn't finish the activation process, you will see this option. Enter your email address and click ‘Send Activation Link’, you will receive an email with a link that will bring you to the OpenReview registration page (Step 2).

If you do not receive an activation email, refer to: Resending an activation link

Reset Password

Existing active accounts will show up with the 'Reset Password' option, and the email domains associated with the account will be displayed. If you believe one of these is your profile, click 'Reset Password', enter your email address and click 'Reset Password' again to send a password reset email.

If you do not receive a password reset email, refer to: I didn't receive a password reset email, what do I do?

Create a new account

If no existing profiles belong to you and you would like to create a new one, you can fill in your email address in the text field next to the ‘Sign Up’ option and click the 'Sign Up' button. We recommend that you sign up with your institutional (university, company, or organization) email.

If you sign up with a public domain or an email domain that is not included in our institution list you will see a warning:

If you are using an institutional email and get this warning, contact support using Feedback to ask that your domain be added to the list. Signing up with a public domain may take up to 2 weeks in moderation. See here for tips on expediting this process.

After typing your email you will then be prompted to enter a new password and send a confirmation email. Clicking the link in the confirmation email will bring you to the registration page.

2. Profile Registration

If you have claimed an inactive profile, resent an activation link, or created a new profile, you will receive an email with the subject 'OpenReview Signup Confirmation'. Follow the link in this email to complete your profile registration. Profile registration is a required step in order to have an active OpenReview Profile.

Providing as complete information as possible, especially an institutional email address and valid personal homepage, will help your account be activated quickly.

A valid homepage is a website that shows your name, affiliation, and email that you used to register for OpenReview. This could be a Google Scholar profile, personal webpage, GitHub, institutional profile page, or other professional homepage.

If you do not have an institutional affiliation, you may sign up as an Independent Researcher. See here for detailed instructions on what to include in your profile.

Common issues for Profile registration:

• Validation errors for career/education history or institutional data: See See Entering Institutional Data for a detailed description and examples.

• Relation not found in the dropdown menu: If none of the existing relations are appropriate, you may type in a custom relationship to the textbox.

• Homepage duplication: If you get an error that one of your links can't be added, it is either in the wrong format or already added to another account. Check that there are no exiting or claimable accounts with your email, and if you still don't recognize the account listed in the error, contact OpenReview Support (info@openreview.net).

3. Moderation

After clicking ‘Register for OpenReview’, your profile will either be activated immediately or sent to moderation. Moderation can take up to two weeks, but the support team reviews moderation regularly, and will follow up with additional actions necessary.

If you receive an email indicating that additional information is required to activate your OpenReview profile, return to the Profile Registration Step by resending the activation link, and complete the missing information. See Expediting Profile Activation for additional information.

To add institutional data to your OpenReview profile, go to your profile page at https://openreview.net/profile and click 'Edit profile'. You must enter at least one position under 'Education & Career History' for your profile to be saved.

Each position requires at least the following information: Position, Start date, Institution Info (domain, name, and country).

For the Position field, you can choose a position from the dropdown or, if none of the existing positions are a good fit, you may type in a custom role. Make sure to click in the custom role as it appear in the dropdown in order to select it for that field, otherwise no value will be entered.

Similarly for the institution name and domain, choose from the dropdown or type the name and domain yourself. If you are not sure what domain to type in for your institution, use the email domain used in institutional emails.

A start date is required for all affiliations. If your affiliation is current, leave the end date blank, otherwise include the end date for past affiliations

Below are some examples of complete Education and Career History sections:

Example - complete profile with institutional information

As an example of complete institutional information, this sample user has one past affiliation as a PhD student and one current affiliation. Note that the institution domain, name, and country are all required for the changes to be saved.

Example - complete profile with freeform data

This sample profile includes one past affiliation with a small company not listed in the drop-down menu, and is currently an independent researcher. Here the domain and position information are typed in.

The expiration date of an activation link is listed in the activation email. If you need to access the link again to complete your profile registration, confirm an email or update a pending profile, you can request another activation link by going to https://openreview.net/login entering your email and click the ‘Didn’t receive confirmation email?’ button.

If you received an email with the subject ‘OpenReview profile activation pending’, your profile is in moderation. The fastest way to activate an OpenReview Account is to ensure that you have an email associated with your school or company added and confirmed to your profile.

How to add an institutional email to a pending account

Please note that all three steps must be completed for the institutional email to be confirmed.

2. Add an email associated with your institution (university, lab, or company) to your profile. Make sure to press Confirm to save the email to your account.

1. DO NOT close your current window/tab. The verification number sent to your inbox needs to be input in the same window/tab.

2. Confirm that email by pasting the verification code and clicking on Verify.

Common questions about pending profiles

I want to publish / receive emails under a personal email, what do I do?

After creating an account, you can set your preferred email to your personal email account, and this will be the email used for submissions and notifications.

I have an institutional email in my profile, why is my profile still pending?

Please check that your email has (confirmed) next to it in your profile page. If your email is confirmed, it is likely that your institution has not yet been added to our list. Submit a request for us to add your institution to our list through our Feedback form.

I don’t have an institutional email, what else can I do to speed up the process?

If you don’t have any institutional emails, please ensure your other profile information is complete. Ensuring your homepage displays your email and career details, along with a fully completed profile, will facilitate the activation process.

Related pages:

If one or more publications are not present in your DBLP homepage, you can use our direct upload feature to manually upload your missing publications. Please go to the OpenReview.net Archive page and follow the instructions there.

Your OpenReview profile ID is a unique string made up of a tilde concatenated with your full name and a number, for example ∼First_Last1. If you go to your OpenReview profile, your ID will be at the end of the url (for example, https://openreview.net/profile?id=∼Your_Id1)

If you know you have multiple profiles (and access to them):

1. Log in using the email address associated with any one of your profiles. Preferably, the one you want to merge the other profiles into.

2. Add the email address that is already confirmed in your other profile to the profile you're currently logged into. (You can skip to step 3 if you have already added this email address.)

3. Click on "Confirm."

4. You will receive an email to confirm the profile merge. Click on this link - make sure you are logged into the account from which you requested to do the merge.

If you get an 'Merge Forbidden' message, try logging out, logging into the other OpenReview profile, then opening that link again.

If you got an error during profile registration:

Sample: [link] is already in use by [user]

This can happen because you have previously registered an account with this information with OpenReview, or because the publication information is associated with a pre-created account.

1. Finish profile registration without the link that caused the error

2. Email info@openreview.net with your profile ID and the profile ID that you wish to merge with.

Other cases

If you do not recognize a profile flagged as duplicate being yours, or do not have access to the email address of the other profile, please reach out to us at info@openreview.net.

To locate your Semantic Scholar URL, go to https://semanticscholar.org and search for the name you publish by.

If Semantic Scholar has your data, an author tile with your name will appear under the search bar. If your name is not immediately one of the top tiles, click the "Show All Authors" link to expand the tile section. Click on the author tile.

Once you have identified your author page with the associated papers, the URL in the browser address bar is the Semantic Scholar URL that you can use in OpenReview profile edit page.

If you would like to edit your Semantic Scholar author page or add additional metadata (e.g. affiliation data) you may use the "Claim Author Page" button located under your name at the top left of your Semantic Scholar author page.

After you have claimed your page and the claim has been approved, you will receive an email from Semantic Scholar with instructions to edit and update your author page. You will have the option to edit or add metadata, remove papers or add additional papers to your claimed Semantic Scholar author page (in case there are multiple author pages with your name).

To add existing DBLP publications to your OpenReview profile:

1. Login to OpenReview

You can click on the 'Login' button on the right of the navigation menu, or go to https://openreview.net/login

2. Navigate to the profile page

Click on your name on the right of the navigation menu and click 'Profile' from the dropdown options, or go to https://openreview.net/profile.

Click 'Edit Profile'.

3. Add a DBLP URL

Locate the 'DBLP URL' text field under the 'Personal Links' section.

You will need to get the 'Persistent DBLP URL' from your DBLP homepage. To do so, hover over the share icon to the right of your name in DBLP page heading and copy the persistent URL from the hover menu.

Paste this persistent url into the DBLP URL field.

4. Click the "Add DBLP Papers to Profile" button

If your persistent DBLP url was valid, the option to 'Add DBLP Papers to Profile' will appear. Click the button and your DBLP publications will appear in a modal window.

5. Select papers to upload

Use the checkbox in front of each paper to select those which you would like to import to your OpenReview profile.

6. Click the 'Add to your Profile' button

Click the 'Add to your Profile' button at the bottom of the modal window to import the selected papers.

If you get an error that says "please ensure the provided DBLP URL is yours", make sure that the name (or one of the names) in your OpenReview profile matches exactly with the name used in DBLP publications. If it does not, you can add a new name to your profile, click 'Save Profile Changes', and try again to import your papers.

To remove publications imported from DBLP

Go to your profile page and click 'Edit Profile'. Scroll to the bottom of the page and look for 'Publications' section. All publications associated with your profile will be listed here but those imported from DBLP will have a minus icon displayed after the title.

You can use this minus button to remove a DBLP publication from your profile. If you mistakenly remove a publication, you can click the icon again to reverse it.

When you are finished, click 'Save Profile Changes' in order to remove the selected papers from your profile.

An option is now availible to add your ACL Anthology URL to your OpenReview profile. This is an optional field, and if you don't have a personal ACL anthology page it can be ignored.

To locate your ACL Anthology URL, go to https://aclanthology.org, type your publishing name into the Search bar at the top of the page, and then click on the magnifying glass icon to initiate the search.

After the search results are returned, click on the Authors tab and select your name. The personal author pages are auto generated, so it's possible a page for you was not created. If you notice an error in the content of your author page, you can create an issue or submit a pull request https://github.com/acl-org/acl-anthology/.

The author page will list known published name variations, publication list, and links to coauthors as well as venues the author submitted papers to. Your ACL Anthology URL can be found in the address bar.

To add this link to your OpenReview profile, login and click on Edit Profile. Scroll down to the Personal Links Section and you'll see a textbox labeled "ACL Anthology URL". Copy the ACL Anthology URL and paste it in this textbox. Make sure to scroll all the way down the Edit page to submit your changes.

How to use this document: This document lists the major steps of running a conference venue. Each step links to relevant documentation that explains it in depth.

Setting up a venue

Fill out the venue request form and choose settings for your venue.

OpenReview will review the request, ask for any necessary clarification, then you will receive an email notifying you that your venue has been deployed.

Also see:

Ensure all PCs have OpenReview accounts associated with the email listed in the venue request form in order to access venue pages.

You may edit many settings for your venue through the 'Revision' button of the request form.

Recruitment can happen at any point in the workflow. All participants (Area Chairs, Reviewers, etc.) must have an OpenReview account linked to the email used in the recruitment message.

You may choose to add a task for program committee members to remind them to complete their registration.

Submission Phase

6. Submissions Open

Submissions automatically open on the date/time listed in the venue request form. If no date is given, submissions open as soon as the venue is deployed.

Note: All submitting authors must have an active OpenReview Profile.

Use this if you want to have an abstract deadline ahead of the main submission deadline.

8. Submissions close

Set up matches between submissions and program committee

Set up matching should be done in the direction Senior Area Chairs-> Area Chairs-> Submissions, then Reviewers should be assigned to submissions. The basic stages of this workflow is to:

Each of these steps needs to be completed for each group being matched (e.g. Area Chairs - Submissions , Reviewers - Submissions)- Described further in steps 11-16 below.

10. Assign Senior Area Chairs to Area Chairs

To assign SACs to ACs, choose 'Senior Area Chairs' as the matching group in paper matching setup. Then in the Paper Matching Stage, SACs will be the matching group, and ACs will be in the Submissions field.

Note: Matching between SACs and ACs will not calculate conflicts. Instead, the conflicts to the SACs will be transferred to the ACs and calculated at the AC matching stage.

It is very important to deploy Senior Area Chair assignments before assigning submissions to Area Chairs to allow conflicts to be transferred successfully.

If you decide to directly assign Senior Area Chairs to submissions, skip to step 12.

The next step is to assign the program committee to each submission. If your conference is smaller than 2000 submissions, this can be run by you directly, otherwise please contact OpenReview.

It is very important that all Program Committee members (Senior Area Chairs, Area Chairs, and Reviewers) have a complete OpenReview profile (an active profile with at least one active institution and one publication). We recommend removing from the committee groups all the profiles that are not complete before running the matching system.

12. (Optional) Bidding period for Senior Area Chairs, Area Chairs and Reviewers

In order for conflicts to be taken into account in paper matching, matching set up (Step 14) must be run before the bid stage.

The bidding console shows the submissions sorted by affinity scores of the logged user and filtering out the ones that are in conflict with the user.

Note: when the conference is large, only sparse scores are uploaded to the system, this means we only keep the first N (400) scores for each user and submission. Users will see at least 400 submissions sorted by their affinity scores. If they want to see other submissions that are not among the top 400 they should use the “search” functionality.

After computing affinity scores and optionally enabling the bidding process, Program Chairs can run the paper matching system to assign Senior Area Chairs/Area Chairs/Reviewers to Submissions. Program Chairs can assign Senior Area Chairs/Area Chairs/Reviewers at different stages because they are independent processes.

14. (optional) Share Reviewer-Submission proposed assignments with Area Chairs to review and edit.

Some venues decide to share the Reviewer-Submission assignments with the Area Chairs to review before releasing them to the Reviewers.

After the proposed assignments were reviewed by the Program Chairs and/or the Senior Area Chairs/Area Chairs, they can be deployed and be visible to the Reviewers. Deploying assignments doesn’t automatically send emails to the Reviewers- it is recommended that you as PCs notify them.

16. (optional) Modify assignments

Review Stage

17. Start review period

The deadline field of the Review Stage form will be the one shown to reviewers as the advertised deadline. The expiration date in the form (not seen by reviewers) will be the time after which no more reviews can be submitted. After this point, reviewers will need to contact the PCs for any late reviews.

If there are still pending reviews, setting Release Reviews to Authors to "Yes, reviews should be revealed when they are posted to the paper's authors", will release all posted reviews, and later will release pending reviews after they are posted.

(Optional) Rebuttal Stage

20. Start Rebuttal stage

(Optional) Submission revision stage

You may optionally allow authors to revise their submissions, including limiting which fields can be edited. This stage can be enabled any time after the submission deadline has passed.

(Optional) Meta-review Stage

23. (optional) Start meta review confirmation period

Use this if you have Senior Area Chairs and want them to review, confirm, or revise the meta review posted by the Area Chair.

24. (optionaI) ACs rate reviews

Decision Stage and Camera Ready Revisions

For large venues (>2000 submissions) we offer a bulk upload process where the PCs can get the meta reviews values, edit them to meet the acceptance rates and then upload them to the system. The PC console will show the decision stats and decision notes can be edited after they are uploaded.

PCs can also (optionally) decide to release the submissions to the public (all the submissions or accepted only) and deanonymize the author names.

27. Start camera ready period

28. End camera ready period

Authors will no longer be able to edit submissions after the Camera Ready period deadline set in the Submission Revision Stage. After this point, they will need to contact the PCs to allow any revisions after the deadline.

Live chat is a new OpenReview feature available to all venues to support synchronous dialog among reviewers and ACs. Similar to a Slack channel or chat room, messages will be visible to all participants and appear in real time as they are posted. In addition, there is an option to enable browser notifications so participants can be alerted when new messages are posted.

This feature was developed with encouragement and funding from the Alfred P. Sloan Foundation. For more information, see the Motivations section below.

Enabling Live Chat

Similarly, live chat can be disabled for all forum pages by clicking Comment Stage and selecting the disable chat option.

Once enabled, the live chat interface will be visible on all forum pages under the “Committee Members Chat” tab. Chat messages are hidden in the “Discussion” tab to prevent confusion and keep the discussions with paper authors and other users separate.

Chat Features

• Real-time: Type a simple message, and it appears immediately to other reviewers who have the same OpenReview forum page open in their browser, enabling interactive discussion.

• Private: By default, messages are visible all program chairs, area chairs, and reviewers of a paper. The messages will never be visible to the authors or the public. As an extra reminder, the chat participants are below the text input box. The ability to send private messages to a subset of the chat participants is planned for a future release.

• Rich text: Chat messages can use Markdown for formatting and LaTeX for math notation. When composing a message you can preview how it will look by clicking the Preview button. To copy the unformatted text of a posted chat message, hover over the message and click the View Raw button.

• Replies: Users can reply to specific message from earlier in the chat by hovering over the message and clicking the Reply button.

• Permalinks: Users can copy a URL pointing to a specific chat message by hovering over the message and clicking the Copy Link button.

• Notifications: Browser and email notifications are supported. User should enable browser notifications from their browser. Email notifications will be sent in bulk either every 5 messages or every 4 hours.

• Reactions: Users can add emoji "reactions" to chat messages, in the same way they can react to messages on apps like Slack and GitHub. To add a reaction, hover over the message and click the "Add Reaction" button, then select the emoji you want to use. All the reactions a message has gotten will be shown below the message content as separate buttons. Clicking one of these buttons will either add a reaction of that type, or remove your reaction if you have already added one.

Motivation

In the past, many peer-reviewed yearly conferences in computer science (such as NeurIPS and CVPR) held an in-person meeting among the 20-30 “area chairs” of the conference to discuss and debate each paper. These meetings enabled extensive synchronous communication and serendipitous discussion during the final decision-making phase of the peer review process.

Unfortunately, these practices of in-person area-chair meetings are increasingly rare due to growing conference size, travel expenses, and (more recently) risks from the pandemic. Given the current universal use of web tools for reviewing, one would hope that the scientific communities would take advantage of internet technology to greatly increase interactivity during the peer review process, but unfortunately, reviewing workflows remain largely unchanged.

One exception is the introduction of “rebuttal statements” (an opportunity for authors to write a response to first drafts of the reviews), and asynchronous messages among the reviewing team. However, due to the lack of synchronous communication, discussion of these rebuttals (and scientific debate among the decision-making team) is hampered – often lackluster or incomplete.

Synchronous communication creates beneficial social pressure for engagement and the opportunity for interactive clarification without the extensive “context switching” inherent in asynchronously juggling many tasks. One highly experienced computer science reviewer and OpenReview user told us, “The burden of context switching after 24 hours away from a conversation can sometimes be the bottleneck in writing a response; re-remembering the details of a paper often takes even more time than writing a reply to a co-reviewer. Synchronous communication could dramatically reduce the burden and increase quality of reviewing by removing this context-switching bottleneck.”

While reviewers may naturally find each other online at the same time, and benefit from the chat feature, scheduled chat discussion sessions may be even more helpful. Scheduling synchronous sessions will be one key aspect of recapturing the benefits of in-person synchronous discussion. The details and cultural considerations of this scheduling will be driven by the program chairs. The in-person meetings were a burden, but also provided a scientific/social occasion that was enjoyed and cherished; we anticipate that with some sensitive design, program chairs can provide a similar occasion. The burden can be reduced by culling from the process submissions that are clear rejections.

Why text messaging rather than audio/video? Although speaking is easy, typing has multiple advantages: typed messages can be more easily archived, read later, skimmed quickly, analyzed, and studied. Many people are used to interactive synchronous text chat tools, such as Slack, and find them very productive. Furthermore, text has high accessibility, and is recommended by the Web Content Accessibility Guidelines (WCAG 2.0). With text entry, users can naturally make some statements publicly and mark others as private; scientists are already used to this ability in typed chat systems such as Slack or Zoom. Typed chat messages support not only fully-synchronous interaction, but also interactive semi-synchronous interaction (e.g. with two minute interruption or delay), which fits modern work styles. If some users are slow typists, they can use voice recognition on their devices to type messages at speaking speed.

Submitting a Request

Venues using OpenReview that are "Full ARR" or "Hybrid" now follow the same workflow. Organizers should submit a venue request form and inform the OpenReview staff that they subscribe to ARR.

Full ARR venues: this commitment forum is the only venue to be created.

Hybrid venues: submit two venue request forms, one for non-ARR reviewing and another for commitment of ARR-reviews papers.

Venue Management

Once the commitment venue is deployed, there will be a field present in the submission form named "paper_link" for authors to commit their ARR processed papers. Here authors will simply provide the URL to the OpenReview forum of their ARR submission. This field is validated, so it will check whether the forum link exists in an ARR venue before allowing the author to submit.

Requesting ARR Reviews

Please contact OpenReview (info@openreview.net) when the submission deadline has passed and you're ready to begin the reviewing process. By default we provide readership permissions for the ARR submission forum, Official Reviews, and Meta Reviews. Please indicate whether you want us to include author responses to the official reviews.

OpenReview will provide read access to ARR forums of committed papers to the venue (assigned ACs and PCs) or import ARR reviews into the venue's OpenReview instance depending on the API used for the original ARR submission.

ARR API 1 cycles: After the ARR notes are imported, PCs and ACs will see a new field in the submission forum titled "Migrated Link, click this link to access the imported ARR notes.

ARR API 2 cycles: After read access has been granted, PCs and ACs should click on the "Paper Link" to access the previous ARR forum.

Although OpenReview can support multiple groups for each role, it cannot support distinct workflows, different deadlines, or different review forms for multiple groups within the same venue.

1. Contact OpenReview support at info@openreview.net to request the ability to support multiple groups for a particular role.

2. After we have customized your venue to support multiple groups, you can recruit members directly into each group with the 'Recruitment' button.

3. When it is time to assign the group members to submissions, you will need to run separate matchings for each group. When you deploy your assignments, the members of each group will be moved into the individual paper groups. For example, if a reviewer from Group A and Group B are each assigned to paper 1, they would both be moved into the Paper1/Reviewers group.

4. From this point forward the reviewers will be treated identically, regardless of their original group.

OpenReview is releasing a major update of the forum page for venues using the new API (v2). While the interface will look familiar, it should be faster, more flexible, and offer new ways to quickly find the content you are looking for.

Anatomy of a Forum Reply

Here is an example of a typical forum comment:

Every post on the new forum page contains the following information:

1. Title: title may be provided by the author of the post or it may be a generic title. Clicking the link icon next to the title copies a direct link to this post to your clipboard.

2. Reply Type: represents the type of forum reply, and comes from the name of the invitation that was used to create the post.

3. Signature: shows the identity of the user who posted the reply. If the reply comes from a group and you have permission to see the members of the group, their identities will be shown in parenthesis next to the signature.

4. Creation Date: shows when the reply was posted and when it was last modified

5. Readers: shows who this post is visible to

6. Revisions Link: see a list of all edits made (for more see "Editing Posted Content" below)

7. Edit & Delete: allows you to modify or hide the reply (for more see "Editing Posted Content" below)

8. Content: shows the complete content of the forum reply

9. Collapse Toggles: allows you to show more or less of the content of the note. The top button will collapse everything down to a single line displaying just basic information, the middle button will only show the first 5-10 lines of content, and the bottom button will display the entire contents of the note.

10. Reply Buttons: show all the available options for replying to this post. Clicking one of the buttons will open a form that allows you to submit your reply.

Sorting and Filtering Replies

The new forum page provides advanced controls for sorting and filtering replies such as comments, reviews, private responses, and PC decisions. The new controls look something like this:

1. Invitation Filter: show only replies of a certain type. Can select multiple invitations (types) to show replies matching any of those invitations.

2. Author Filter: show only replies signed by the selected user or group. Can select multiple authors to show all replies that include any one of the selections as a signature.

3. Keyword Filter: show all replies that match the phrase. Matches can come from the content of the reply, the title, or the invitation.

4. Sort Control: change the order of replies shown to either most recent first or oldest first

5. Layout Control: change the level of nesting of the replies. The three options are Linear, Threaded, and Nested

6. Collapse Control: change how much of all the replies is shown. By default the entire contents of the reply is visible, this corresponds to the three lines. Selecting the middle button (two lines) will only show the first 5-10 lines of content, and selecting the left button (one line) will condense the replies down to a single line displaying just basic information.

7. Link Button: copy a URL to your clipboard that includes all the currently selected filter and sort options. This is useful for sharing specific views of a forum page with other people or bookmarking for later.

8. Readers Filter: show all the replies that have the selected users or groups listed as a reader. Clicking a button twice will turn the button red – this means that only the replies that DO NOT include that group as a reader will be shown.

9. Preset Views: venues can configure sets of filters and layout options that are displayed as tabs above the filter form. Clicking on a given tab will switch the current filters over to those settings.

Forum Layout Options

As mentioned in the section above there are currently three layout modes available for forum pages. These are:

• Linear: all replies are shown at the same level of nesting (no indentation). This is useful to quickly see a chronological feed of all forum replies. If a given post is a reply to another reply (not a general reply to the submission note) the title of that reply will be shown in gray above the title. Clicking on this gray title will scroll the page to that reply.

• Threaded: one level of nesting is shown. This layout is useful to group conversations into overarching topics.

• Nested: two levels of nesting are shown. This is useful for breaking larger conversations down into sub threads.

Editing Posted Content

If you are logged into OpenReview and have permission to modify the content of a submission or a forum reply (aka a Note) you will see a dropdown button labeled Edit to the right of the title. Clicking this button will display a list of all the available ways to modify the note (aka edit Invitations). For a submission note this might include options to revise the submission or withdraw the submission, and for a reply it might include the option to edit the content of the reply.

You can see a list of all the edits of a given note by clicking on the Revisions link below the title.

Providing Feedback

We hope that you find the new functionality useful. If there is anything you would like to see changed or added please send an email to info@openreview.net with the subject "New Forum Page Feedback".

You can customize your venue homepage using Markdown. All Markdown should be validated to ensure that it will not break the layout of the page: https://stackedit.io/

Group Content

Go to your homepage https://openreview.net/group?id=Your/Venue/ID

1. Click on "Edit Group" button above the title of the venue.

2. Scroll down to "Group Content", click to open the edit box and find the field "instructions".

3. In the "value", you can include your instructions in markdown or html as a string.

Markdown example, to customize:

"# Important Dates\n
- Submission Deadline: Sept 11, 2021 11:59pm AOE\n
- Author Response Period: Oct 11, 2021 - Oct 18, 2021\n
- Decision Notification: Nov 1, 2021\n
- Camera Ready: Nov 15, 2021\n\n
# Submission Format\n
- Proceedings track: 8 pages. Findings (extended abstract) track: 4 pages. Excluding references and appendices.\n
- Submissions must be anonymous for the double-blind review process\n
- Latex template: [Proceedings](https://www.overleaf.com/latex/templates/proceedings-template/trtrvjxchjdw) or [Findings](https://www.overleaf.com/latex/templates/ml4h-2024-findings-template/vkvrxcgdkcmn)"

which will be displayed as:

Where to customize forms

Many OpenReview forms are customizable from the different buttons on the venue request form. You can find where to input your customizations by clicking on a button (for example, "Review Stage") and finding the large text box under "Additional _____ Options". Under "Revision", you'll find that this box modifies the submission form under the heading "Additional Submission Options". For the "Review Stage", the heading will be "Additional Review Form Options"

Essential structure of custom fields

These text boxes accept a valid JSON object with fields and values. The following is an example where the title field gets replaced with a radio button, like so:

{
    "title": {
        "order": 1,
        "description": "Title of the paper",
        "value": {
            "param": {
                "type": "string",
                "enum": ["Test Submission Title"],
                "input": "radio"
            }
        }
    }
}

Note that correct indentation levels and matched brackets are necessary for valid JSON.

The primary fields of the entry are:

title - This will be the name of the field in the form

order - Determines where in the form the field will appear

description - Will show up in the form as instructions or description

value - Will have subfields (under param) determining the format of the field and the options for responses

"value": {
    "param": {
        "type": "string",
        "enum": ["Test Submission Title"],
        "input": "radio"
    }
}

When making a field that is asking for user input, you will always see this pattern of "value": { "param": {...} }. Inside the param object are fields determining what the user sees in the input form along with what the user is allowed to submit: these are representation specifiers and validation specifiers.

Specifiers

This section will introduce common specifiers used in customizing forms. Further information about specifiers can be found here.

Representation Specifiers

Representation specifiers determine how the user will input their response into the field (for example a textbox or a checklist). These will be defined in the param object.

The input specifier determines the rendering on the form and can have the following values (see below for examples of how different input types render):

• text

• select

• checkbox

• textarea

• radio

default is the default value of the box that will appear when the widget is initialized

markdown is a boolean value (true/false) that enables markdown for this text field

scroll is a boolean that adds a scroll bar to a textarea input

Validation Specifiers

Validation specifiers are used by the back-end to ensure data submitted through the form conforms to certain requirements. In the example above, only a single string is allowed, and that string must be one of the values defined in the enum array. Specifically, a string that has the value "Test Submission Title"

optional is a boolean (true/false) value that indicates whether or not this field is required to be present when the form is submitted. By default all fields in the form are required, and you can add optional : true to indicate a required field.

type specifiers require the input to be of a specific type: options are string, string[] (string array) and file.

string fields can be further validated by using fields to describe the structure of a valid string input. Some of these field are:

• "maxLength": set the maximum number of characters of the input

• "minLength":set the minimum number of characters of the input

• "regex": use regular expressions to define acceptable string structures

• "enum": restrict the user to a predefined set of strings

• "items": an array of strings as indicated by its type (only used with type: string[] ) **

** All values in "items" will be considered required unless specified otherwise with "optional": true. Please see multiple choices for an example.

file fields are specifically validated with maxSize and extensions. maxSize is an integer that specifies the size of the largest file that can be uploaded on the form in megabytes. extensions is a list of strings that are extensions, for example "extensions": ["pdf", "zip"].

"extensions": ["zip", "tar.gz"]

Examples of Common Form Patterns

Text Input

{
  "title": {
    "order": 1,
    "description": "(Optional) Brief summary of your comment.",
    "value": {
      "param": {
        "type": "string",
        "maxLength": 500,
        "optional": true
      }
    }
  }
}

{
  "comment": {
    "order": 2,
    "description": "Your comment or reply (max 5000 characters). Add formatting using Markdown and formulas using LaTeX. For more information see https://openreview.net/faq",
    "value": {
      "param": {
        "type": "string",
        "maxLength": 5000,
        "markdown": true,
        "input": "textarea"
      }
    }
  }
}

Single Choice

{
  "recommendation": {
    "order": 10,
    "description": "Please provide your recommendation based on the manuscript, reviews, author responses and your discussion with the reviewers.",
    "value": {
      "param": {
        "type": "string",
        "enum": [
          "Reject",
          "Accept",
          "Nominate for best paper"
        ],
        "input": "radio"
      }
    }
  }
}

{
  "submission_track": {
    "order": 11,
    "description": "Please select a preferred track, which will be considered in the paper-reviewer assignment. Please note that your choice of a track will be taken into account, but the chairs may decide to move your paper to a different area when appropriate.",
    "value": {
      "param": {
        "type": "string",
        "enum": [
          "Commonsense Reasoning",
          "Computational Social Science and Cultural Analytics",
          "Dialogue and Interactive Systems",
          "Discourse and Pragmatics",
          "Efficient Methods for NLP",
          "Ethics in NLP",
          "Human-Centered NLP",
          "Information Extraction",
          "Information Retrieval and Text Mining",
          "Interpretability, Interactivity, and Analysis of Models for NLP",
          "Language Grounding to Vision, Robotics and Beyond",
          "Language Modeling and Analysis of Language Models",
          "Linguistic Theories, Cognitive Modeling, and Psycholinguistics",
          "Machine Learning for NLP",
          "Machine Translation",
          "Multilinguality and Linguistic Diversity",
          "Natural Language Generation",
          "NLP Applications",
          "Phonology, Morphology, and Word Segmentation",
          "Question Answering",
          "Resources and Evaluation",
          "Semantics: Lexical",
          "Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.",
          "Sentiment Analysis, Stylistic Analysis, and Argument Mining",
          "Speech and Multimodality",
          "Summarization",
          "Syntax, Parsing and their Applications",
          "Theme Track: Large Language Models and the Future of NLP"
        ],
        "input": "select"
      }
    }
  }
}

Multiple Choices

{
  "confirmation_of_submission_requirements": {
    "order": 17,
    "description": "Before you submit this paper, please make sure that the following requirements are met. If any of these requirements are not fulfilled, your submission will be rejected and will not be reviewed.",
    "value": {
      "param": {
        "type": "string[]",
        "optional": true,
        "items": [
          { "value": "Requirement 1", "description": "Requirement 1", "optional": true},
          { "value": "Requirement 2", "description": "Requirement 1", "optional": true}
        ],
        "input": "checkbox"
      }
    }
  }
}

{
  "submission_tracks": {
    "order": 11,
    "description": "Please select your preferred tracks, which will be considered in the paper-reviewer assignment. Please note that your choice of tracks will be taken into account, but the chairs may decide to move your paper to a different area when appropriate.",
    "value": {
      "param": {
        "type": "string[]",
        "items": [
            { "value": "Commonsense Reasoning", "description": "Commonsense Reasoning", "optional": true },
            { "value": "Computational Social Science and Cultural Analytics", "description": "Computational Social Science and Cultural Analytics", "optional": true },
            { "value": "Dialogue and Interactive Systems", "description": "Dialogue and Interactive Systems", "optional": true },
            { "value": "Discourse and Pragmatics", "description": "Discourse and Pragmatics", "optional": true },
            { "value": "Efficient Methods for NLP", "description": "Efficient Methods for NLP", "optional": true },
            { "value": "Ethics in NLP", "description": "Ethics in NLP", "optional": true },
            { "value": "Human-Centered NLP", "description": "Human-Centered NLP", "optional": true },
            { "value": "Information Extraction", "description": "Information Extraction", "optional": true },
            { "value": "Information Retrieval and Text Mining", "description": "Information Retrieval and Text Mining", "optional": true },
            { "value": "Interpretability, Interactivity, and Analysis of Models for NLP", "description": "Interpretability, Interactivity, and Analysis of Models for NLP", "optional": true },
            { "value": "Language Grounding to Vision, Robotics and Beyond", "description": "Language Grounding to Vision, Robotics and Beyond", "optional": true },
            { "value": "Language Modeling and Analysis of Language Models", "description": "Language Modeling and Analysis of Language Models", "optional": true },
            { "value": "Linguistic Theories, Cognitive Modeling, and Psycholinguistics", "description": "Linguistic Theories, Cognitive Modeling, and Psycholinguistics", "optional": true },
            { "value": "Machine Learning for NLP", "description": "Machine Learning for NLP", "optional": true },
            { "value": "Machine Translation", "description": "Machine Translation", "optional": true },
            { "value": "Multilinguality and Linguistic Diversity", "description": "Multilinguality and Linguistic Diversity", "optional": true },
            { "value": "Natural Language Generation", "description": "Natural Language Generation", "optional": true },
            { "value": "NLP Applications", "description": "NLP Applications", "optional": true },
            { "value": "Phonology, Morphology, and Word Segmentation", "description": "Phonology, Morphology, and Word Segmentation", "optional": true },
            { "value": "Question Answering", "description": "Question Answering", "optional": true },
            { "value": "Resources and Evaluation", "description": "Resources and Evaluation", "optional": true },
            { "value": "Semantics: Lexical", "description": "Semantics: Lexical", "optional": true },
            { "value": "Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.", "description": "Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.", "optional": true },
            { "value": "Sentiment Analysis, Stylistic Analysis, and Argument Mining", "description": "Sentiment Analysis, Stylistic Analysis, and Argument Mining", "optional": true },
            { "value": "Speech and Multimodality", "description": "Speech and Multimodality", "optional": true },
            { "value": "Summarization", "description": "Summarization", "optional": true },
            { "value": "Syntax, Parsing and their Applications", "description": "Syntax, Parsing and their Applications", "optional": true },
            { "value": "Theme Track: Large Language Models and the Future of NLP", "description": "Theme Track: Large Language Models and the Future of NLP", "optional": true }
        ],
        "input": "select"
      }
    }
  }
}

Miscellaneous

{
  "supplementary_material": {
    "order": 10,
    "description": "All supplementary material must be self-contained and zipped into a single file. Note that supplementary material will be visible to reviewers and the public throughout and after the review period, and ensure all material is anonymized. The maximum file size is 200MB.",
    "value": {
      "param": {
        "type": "file",
        "extensions": [
          "zip",
          "pdf",
          "tgz",
          "gz"
        ],
        "maxSize": 200,
        "optional": true
      }
    }
  }
}

Setting the Readers of a Field

If you want to limit who in the committee can see a particular field in a form, this is done by adding a readers field. Please follow this link for more detailed information on hiding or revealing fields. Below are two examples, one for the submission form and one for the meta review form. Notice the different use of dollar sign notation. The notation used for the meta review form will also work for other replies to the forum: reviews, comments, and decisions.

{
  "supplementary_material": {
    "order": 10,
    "description": "All supplementary material must be self-contained and zipped into a single file. Note that supplementary material will be visible to reviewers and the public throughout and after the review period, and ensure all material is anonymized. The maximum file size is 200MB.",
    "value": {
      "param": {
        "type": "file",
        "extensions": [
          "zip",
          "pdf",
          "tgz",
          "gz"
        ],
        "maxSize": 200,
        "optional": true
      }
    },
    "readers": [
    "Your/Venue/ID/Program_Chairs",
    "Your/Venue/ID/Submission${4/number}/Senior_Area_Chairs",
    "Your/Venue/ID/Submission${4/number}/Authors"
    ]
  }
}

{
  "recommendation": {
    "order": 10,
    "description": "Please provide your recommendation based on the manuscript, reviews, author responses and your discussion with the reviewers.",
    "value": {
      "param": {
        "type": "string",
        "enum": [
          "Accept (Oral)",
          "Accept (Poster)",
          "Reject"
        ],
        "input": "radio"
      }
    },
    "readers": [
    "Your/Venue/ID/Program_Chairs",
    "Your/Venue/ID/Submission${7/content/noteNumber/value}/Area_Chairs",
    "Your/Venue/ID/Submission${7/content/noteNumber/value}/Reviewers"
  ]
  }
}

After your venue is deployed, you will have access to multiple pages in OpenReview. These can be accessed through two methods:

1. Navigating to them from the Program Chair Console on the OpenReview website

2. Clicking the relevant link in the venue deployment confirmation email (subject: 'Your Venue is Deployed")

Venue Homepage

This is the public-facing page for your venue, and can be found under 'Active Venues' on the OpenReview homepage.

Program Chair Console

You can find a link to your PC console under ‘Active Consoles’ on the OpenReview homepage after you have logged into your account. Use this page to access your venue request form, committee consoles, and edge browser.

Venue Request Form

The venue request form contains the settings that were selected in your venue’s request form. You can get to the venue request form using the ‘Full venue configuration’ link at the bottom of your PC console. Almost all customizations, such as adding a Program Chair, review form modifications, deadline changes and workflow stages, should be made through the venue request form.

Edge Browser

The edge browser is the place where PCs can see reviewers, papers, and the links between them. This is where PCs will typically go to see and modify assignments for roles like reviewers and area chairs. To see the edge browser, you will first need to run the Paper Matching Setup workflow stage, then you will see the link to '[Role] Assignment' in your PC Console under the 'Timeline' section. Use that link to access the Edge Browser.

Committee Member Groups and Consoles

Each type of committee member (Reviewers, Area Chairs, and Senior Area Chairs) for your venue will have their own console. You can access these consoles through each of the links under ‘Venue Roles’ at the bottom of your PC console. This link will take you to the group edit page, then you can use the 'View Group' button to see the console.

Note that each key must be a valid decision option. For example, a venue with the decision options Accept (Oral), Accept (Spotlight), Accept (Poster) and Reject could choose to hide the rejected papers tab as follows:

Make sure to pay close attention to the Invitee Details on the Recruitment form. Invitees must be formatted in a specific way to send the messages otherwise the messages will fail to send and will return a status error.

Enter a list of invitees with one user per line. Either tilde IDs (∼Captain_America1), emails (captain_rogers@marvel.com), or email, name pairs (captain_rogers@marvel.com, Captain America) expected. If only an email address is provided for an invitee, the recruitment email is addressed to "Dear invitee". Do not use parentheses in your list of invitees.

• All invited reviewers will appear in the group venue_id/Reviewers/Invited

• Reviewers who accept the invitation will be added to the group venue_id/Reviewers

• Reviewers who decline the invitation will be added to the group venue_id/Reviewers_Declined.

You can find links to these groups on your PC console.

The program committee is represented as a group, like Reviewers, Area Chairs, Action Editors, etc. Each of these groups have a property 'members' that can contain a set of groups. Email addresses and profile ids are also represented as groups.

Adding Members to the Group

If you want to build you own group of reviewers, you can simply add them as members to the group.

Using the Group Editor

1. Access the group editor at the following URL, replacing group_id with the actual ID of your group: https://openreview.net/group/edit?id=group_id

2. Under the Group Members section select the text box area, add their profile ID or email, and click on Add to Group

Using the Python Library

To remove members using the python client, utilize the add_members_to_group function: client.add_members_to_group(group, ['~Jane_Doe1', '~Melisa_Bok1'])

Replace group with your target group's ID, and include the profile ids or emails of the members you wish to remove in the list.

Removing Members from the Group

Should you need to remove members from a group, you can also use the group editor or the python library.

Using the Group Editor

1. Access the group editor at the following URL, replacing group_id with the actual ID of your group: https://openreview.net/group/edit?id=group_id

2. In the Group Members section, locate the profiles/emails of the members you wish to remove.

3. Select the profile ID or email and click 'Remove' or 'Remove Selected' to delete them from the group.

Using the Python Library

To programmatically remove members using the python library, utilize the remove_members_from_group function:

Replace group with your target group's ID, and include the profile ids or emails of the members you wish to remove in the list.

Some venues with multiple deadlines a year may want to reuse the same reviewer and area chairs from cycle to cycle. In those cases, it is useful to use the python client to copy group members from one group to another rather than recruiting the same people each time. This can be done programmatically or from the UI.

Programmatically

2. Get the group you are taking members from. You can get an individual group by its id. If you are copying reviewers from one venue iteration to another, for example, do the following:

3. Each group has a field 'members' which is a list of profile IDs or emails belonging to the members of that group. Extract the members:

4. Finally, you can use add_members_to_group to add those members to your new Reviewers group.

From the UI

1. Navigate to https://openreview.net/group/edit?id=<first_venue_id>/Reviewers

2. Under Group Members section click on Select All

1. Then click on Copy Selected

1. Navigate to https://openreview.net/group/edit?id=<second_venue_id>/Reviewers

2. Under the Group Members section select the text box area and click on Add to Group

If your venue will be using publication chairs, there is an option for PCs to set up this group in the request form.

OpenReview does support a limited functionality for "tracks". If your workflow requires different deadlines and different review forms for each track, then please fill out a separate venue request for each track.

What we currently support:

• Create reviewer, area chair, and senior area chair group roles for each track. Please limit the group roles to a maximum of 10 tracks.

• Assign reviewers and ACs in these independent pools to the submissions with their track selected.

• ACs can modify reviewer assignments after the PCs have initiated the assignments.

• All tracks follow the same workflow, including deadlines and forms.

• Currently, PCs are responsible for recruiting to all track roles, as well as the paper matching setup to compute conflicts and affinity scores and create matching configuration.

If the needs for your venue are not addressed here regarding tracks, please email support at info@openreview.net. We can try to accommodate custom requests, however, we need to know at least one month before submissions open.

If you want to accept distinct types of submissions that will all follow the same workflow, you can add a ‘track’ field to your submission form. If you reach out to OpenReview support at info@openreview.net, we can then customize your PC console to allow you to filter and sort by track or add roles for reviewers and ACs if you need a group per track.

Test venues and profiles are not allowed on the live OpenReview site. You can test your venue workflow using the dev site: dev.openreview.net. In order to do so, create a profile and submit a venue request form, just as you would on the live site.

After submitting the request form, you must contact us at info@openreview.net so we can deploy the venue.

Note that sending emails through the dev site is not supported, except during profile signup so you no longer need to contact us for you activation link.

3. Choose your program chairs, and create a list of their email addresses.

4. Now it is time to choose the settings for your venue. These make up the 'content' field of your note. Go through the fields on the form in UI, and cross reference the JSON of the invitation to make sure each key matches that in the invitation exactly. For the Program Chairs field, you can enter your program_chair_emails list. For example:

6. Post your note.

If you are testing your venue on the dev site, you may want to generate some test submissions. This can be accomplished manually through the UI, but it may be faster to do it with python using the guide below.

Instantiate the python client using your dev profile credentials:

import openreview
client = openreview.api.Client(
    baseurl='https://devapi2.openreview.net',
    username=<your dev username>,
    password=<your dev password>
)

Familiarize yourself with the submission invitation. Every content field of your note will need to be in the format specified by the submission invitation, and all the fields that do not contain optional: true are mandatory.

Create your note. If you were submitting a note through the default submission form, it would look something like this:

note = openreview.Note(
    readers = [
        "Test/Venue",
        "~Test_Author1",
        "~Test_Author2"
    ],
    writers = [
        "Test/Venue",
        "~Test_Author1",
        "~Test_Author2"
    ],
    signatures = ["~PC_Id1"],
    content = {
        "title": "Test Submission 1",
            "~Test_Author1",
            "~Test_Author2"
            "~Test_Author1",
            "~Test_Author2"
            "machine learning", 
            "autonomous vehicles"
        "TL;DR": "A test submission",
    }
)
url = client.put_attachment('./paper.pdf', 'Test/Venue/-/Submission', 'pdf')
note.content['pdf'] = url

Invitation: The ID of your venue's submission invitation. It will take the form of your venue id + /-/Submission. Readers/Writers: A list of the venue ID + all author profile IDs. After the submission deadline, these values will be automatically updated to match the visibility settings selected in the venue request form. Signatures: Your profile ID. Content: The actual content of the submission, which must match the invitation. If the invitation calls for any file uploads, such as a pdf or zip file, you can build the url to the file using put_attachment and entering the path, the invitation ID, and the name of the field. You can then add that field to the note's content. The "authorids" field of the note's content should contain a list of the authors' profile IDs or emails, in the same order as the authors' names. If you registered dummy users, you can find their profile IDs by going to their profile page and copying the ID parameter of that page's url. For example, dev.openreview.net/profile?id=~Test_Author2 --> ~Test_Author2.

Post your note and output the resulting note's ID:

note = client.post_note_edit(
    invitation="Test/Venue/-/Submission",
    signatures="~Test_User1",
    readers=[
        "Test/Venue",
        "~Test_Author1",
        "~Test_Author2"
    ],
    writers = [
        "Test/Venue",
        "~Test_Author1",
        "~Test_Author2"
    ],
    note=note
)
print(note.id)

You can now view your note in the UI by going to dev.openreview.net/forum?id=<note_id>.

In order to begin reviewing while still accepting submissions, you will need to change the submission deadline to a past date, otherwise the Paper Matching Setup button won't be available. Run the Paper Matching Setup for reviewers and then you can access the matching from the PC console. Set your submission deadline back to its original time. You will need to run the Paper Matching Setup after the Submission Deadline expires to assign reviewers to new submissions and if you added new reviewers to the reviewer's group. You can use the Review Stage at anytime to set the configuration and the stage will run automatically when the Start Date passes.

On the request form for your venue, click on the ‘Post Submission’ button to make submissions available according to the settings selected in the fields ‘Author and Reviewer Anonymity’ and ‘Submission Readers’ of the venue request. This means that:

• You can choose which fields are kept hidden (author names are automatically hidden if the venue is double blind).

• If you select the option ‘Everyone (submissions are public)’ in ‘Submission Readers’, then all submissions will be public.

• If submissions should be private, then they can be released to the assigned program committee (only assigned reviewers, for example), to the entire program committee (all reviewers), or to PCs and authors only. However , assignments can't be made until after the submission deadline passes.

You can enable comments using the Comment Stage.

If you would like to restrict when authors can withdraw their submissions, you can do so from your venue request form. Go to your venue request form, click the "Revision" button, and enter a date and time in GMT for the field "Withdraw Submission Expiration".

The submission deadline set through the venue request form is actually only the advertised due date. The true deadline is the expiration date, which is set to 30 minutes after the submission deadline. If you would like to change the expiration date to allow the authors more or less wiggle room around the submission deadline, you can do so using the python client.

1. If you have not done so, you will need to install and instantiate the openreview-py client.

2. Choose your desired expiration date. You will need to convert it into epoch time in milliseconds using an epoch time converter. For example:

expdate = 1650639704000

Create an Invitation Edit

client.post_invitation_edit(
    invitations='<invitation-id>',  # This is the Invitation used to create this Invitation Edit
    readers=[venue_id],
    writers=[venue_id],
    signatures=[venue_id],
    invitation=openreview.api.Invitation(
        id="<Your/Venue/ID/-/Submission",
        expdate=expate
        signatures=[venue_id]
    )
)

PCs may want to programmatically desk-reject submissions that are missing PDF files once the submission deadline has passed.

Make sure to replace <venue_id> with the venue ID of your conference. For example, "ICLR.cc/2024/Conference".

venue_id = "<venue_id>"

submissions = client.get_all_notes(invitation=f'{venue_id}/-/Submission')
#gets the desk rejection name of the invitation
desk_rejection_name = client.get_group(venue_id).content['desk_rejection_name']['value']
    
# for each submission note that does not contain a pdf field, post a desk rejection note
for submission in submissions:
    if not submission.content.get('pdf', {}).get('value'):
        desk_reject_note = client.post_note_edit(
                    invitation=f'{venue_id}/Submission{submission.number}/-/{desk_rejection_name}',
                    signatures=[f'{venue_id}/Program_Chairs'],
                    note=openreview.api.Note(
                        content = {
                        'desk_reject_comments': { 'value': 'No PDF.' }
                        }
                    )
                )
        print(submission.number, "is desk rejected")

1. Select "Yes, our venue has Ethics Chairs and Reviewers" from the venue request form. This can be done when submitting a new support request form, or after your venue has been deployed by using the "Revision" button. This will generate an "Ethics Review Stage" button from the venue request form as well as other necessary customizations.

2. Recruit Ethics Chairs and Ethics Reviewers using the "Recruitment" button from the Venue Request form.

3. Once you have identified which papers need ethics review, create a comma-separated list of their paper numbers.

4. From the venue request form, run the "Ethics Review Stage". This will allow you to customize the Ethics Review form and pass in your list of papers that require review.

5. Now Ethics Chairs will be able to assign Ethics Reviewers to the flagged papers from their Ethics Chairs console. Ethics Reviewers will see the option to post Ethics Reviews to their assigned papers.

At any point during your venue's worflow, you can click on the ‘Post Submission’ button and use the field ‘Submission Readers’ to change the readers for all submissions:

• All program committee (all reviewers, all area chairs, all senior area chairs if applicable): all papers are private and only released to all reviewers, area chairs and senior area chairs (if your venue has them)

• Assigned program committee (assigned reviewers, assigned area chairs, assigned senior area chairs if applicable): all papers are private and only released to each paper's assigned reviewers, area chairs and senior area chairs (if your venue has them)

• Program chairs and paper authors only: papers are private and released only to program chairs and paper authors

• Everyone (submissions are public): papers are released to the public

You can use this field to change the submission readers as many times as needed.

The review, meta review, and decision forms should be modified only through the venue request form. Do not edit the review, meta review, or decision invitations directly from the invitation editor, or your changes will be overwritten.

Example: In order to edit the review form, run the Review Stage from the venue request form. Any additional fields can be added in valid JSON to the ‘Additional Review Form Options’ field, and any fields can be removed using the ‘Remove Review Form Options’ field. A similar process can be followed for meta-reviews and decisions using the Meta Review and Decision stages.

OpenReview supports enabling the Rebuttal Stage from the request form through the "Rebuttal Stage" button.

Before enabling the rebuttal stage, make sure authors have access to reviews.

From the venue request form, click the ‘Submission Revision Stage’ button to set up camera-ready revisions for papers. To view all camera-ready versions submitted to your venue, refer here.

When you are ready to release the reviews, run the Review Stage from the venue request form and update the visibility settings to determine who should be able to see them. This will change the readers of all existing reviews in bulk.

Please note that if you want to release the reviews to the public, you will first need to make all submissions public if they are not already. If you select to release the reviews to the public while trying to release them to authors, it will not work if the submissions are not already public. If your decision stage has passed, you can use the 'Post Decision Stage' to release submissions. If you need to make submissions public but have not yet posted the decisions, contact OpenReview support at info@openreview.net for assistance.

API 2 venues

If program organizers want to revert a submission withdrawal, they can do so by going to the forum of the withdrawn submission, locating the Withdrawal note posted on the forum page, and then clicking on the "Withdrawal Reversion" button located on the bottom right corner of the note.

Once decisions have been posted, you will see a ‘Post Decision Stage’ button on the request form for your venue. Click on this button to choose who should have access to submissions.

Once decisions have been posted, you will see a ‘Post Decision Stage’ button on the venue request form. Click on this button to choose between revealing identities of authors of all papers or only accepted papers to the public.

If you want to allow SACs to bid on ACs in assignment, you will need to run the bid stage from the python client. To do so you will need to:

1. Install and connect to the Python client, if you haven't already. Please use the API 1 client.

2. Find your request form id (in the id= portion of the URL of the venue request form) and add it to request_form_id

3. Find your venue ID (listed in the venue ID field of the venue request form) and add it to venue_id

4. Update the bid_start_date and bid_due_date

Then you can run the following block of code to start the SAC bidding stage

# Add the request form id as a string
request_form_id = ''

# Assign the venueID to a variable
venue_id = ''

# Change dates
bid_start_date = datetime.datetime(year=2024,month=9,day=30,hour=12,minute=0,second=0)
bid_due_date = datetime.datetime(year=2024,month=10,day=8,hour=12,minute=30,second=0)

venue = openreview.get_conference(client, request_form_id, setup=False)

sac_bid_stage = openreview.stages.BidStage(
    f'{venue_id}/Senior_Area_Chairs',
    start_date = bid_start_date,
    due_date = bid_due_date, 
    request_count = 25,
    score_ids=[f'{venue_id}/Senior_Area_Chairs/-/Affinity_Score']
)
venue.bid_stages = [sac_bid_stage]

venue.create_bid_stages()

The Review Revision Stage is run when PCs want reviewers to be able to modify a review by changing or adding fields.

PCs can include existing review fields in the revision fields to allow users to update those fields.

You could also add fields to a review for reviewers to fill out, such as after the rebuttal period. These new fields are often the final rating of the paper and the justification for changing the rating.

This stage cannot be run from the venue request page and must be requested.

To request the Review Revision Stage, please email info@openreview.net with the following information:

1. Start date, Due date, and Expiration date for the invitation, with the day, time, timezone for each.

2. Fields that reviewers should be able to change in the review, including existing and new fields