Evaluation campaigns, where researchers share important tasks, collaboratively develop test collections, and have discussion to advance technologies, are still important events to strategically address core challenges in information access research. The goal of this workshop is to discuss information access tasks that are worth addressing as a community, share new resources and evaluation methodologies, and encourage researchers to ultimately propose new evaluation campaigns in NTCIR, TREC, CLEF, FIRE, etc. The proposed workshop accepts four types of contributions, namely, emerging task, ongoing task, resource, and evaluation papers.
| 14:40 | - | 14:50 | Opening | Makoto P. Kato, Noriko Kando, Charles L. A. Clarke, and Yiqun Liu |
| 14:50 | - | 16:10 | Paper presentation (15min for each (+ 1min buffer time); 10~12min for presentation and 3~5min for Q&A) | Chair: Makoto P. Kato |
| Evaluating System Responses Based On Overconfidence and Underconfidence | *Tetsuya Sakai | |||
| Can We Use Large Language Models to Fill Relevance Judgment Holes? | *Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi and Mohammad Aliannejadi | |||
| Searching Unseen Sources for Historical Information: Evaluation Design for the NTCIR-18 SUSHI Pilot Task | *Douglas Oard, Tokinori Suzuki, Emi Ishita and Noriko Kando | |||
| Building Test Collections for Japanese Dense Information Retrieval Technologies and Beyond | Hideo Joho, Atsushi Keyaki, Yuuki Tachioka and *Shuhei Yamamoto | |||
| Understanding Tables in Financial Documents: Shared Tasks for Table Retrieval and Table QA on Japanese Annual Security Reports | *Yasutomo Kimura, Eisaku Sato, Kazuma Kadowaki and Hokuto Ototake | |||
| 16:10 | - | 16:30 | Coffee Break | |
| 16:30 | - | 17:20 | Group discussion - New evaluation campaigns - | |
| 17:20 | - | 17:50 | New task proposals | |
| 17:50 | - | 18:00 | Closing |
All the deadlines are the end of the day in Anywhere on Earth (AoE).
Each paper must contain two to six pages (including figures, tables, proofs, appendixes, acknowledgments, and any content except references) in length, with unlimited pages for references. The manuscripts must be written in English and in the PDF format. The paper format must follow the new ACM guidelines (e.g., using the ACM LaTeX template on Overleaf here) with the “sigconf” option. The PDF files must have all non-standard fonts embedded.
All papers will be peer-reviewed by the program committee. The review process is single-anonymized: authors should include author information (i.e., names and affiliations) in submitted papers and reviewers are anonymized for authors.
At least one author of each accepted paper must attend the workshop on-site and present their work. Papers should be electronically submitted by the deadline through EasyChair: https://easychair.org/my/conference?conf=emtcir2024.
A paper must be one of the following contribution types:
Papers in this category are expected to introduce new or emerging information access tasks. The authors are expected to describe how important the task is and identify technical challenges to be solved in the paper. Some approaches to the emerging task should be discussed, but are not necessarily fully matured. A primary experiment on this task would be helpful for further discussing potential challenges.
Evaluation criteria:
Ongoing task papers describe ongoing attempts that have already been accepted as an evaluation campaign or those that have recently been concluded. The authors are expected to describe the motivation of the ongoing task, highlight technical challenges, and explain the task design and evaluation methodology. We highly encourage task/track/lab organizers to take this opportunity for further discussing the task design and attracting more participants.
Evaluation criteria:
Resource papers are similar to those expected in SIGIR and CIKM: papers describing a new test collection. We especially welcome authors planning a new shared task based on the new test collection. Resource papers should include a motivation of a new test collection, its potential applications, the details of the dataset development, and test collection statistics. Some examples of applications (e.g., comparison of existing methods) are also expected.
Evaluation criteria:
A new task may require a new evaluation methodology. A new approach to replace traditional evaluation methods may also emerge due to new technologies. Existing evaluation methodology may require further discussion because of some technical problems. Evaluation papers are expected to include contributions regarding evaluation of information access technologies. Expected authors are researchers who are interested in EVIA (International Workshop on Evaluating Information Access), collocated with NTCIR.
Evaluation criteria:
The topics of this workshop include those of SIGIR-AP 2024 and those expected in IR evaluation campaigns such as NTCIR, TREC, CLEF, and FIRE.
The first half of the workshop mainly focus on the presentation of accepted contributions (except for ongoing task papers), while the latter half focuses on discussion for exploring new tasks. As successful examples, ongoing tasks are introduced by those who submit ongoing task papers or those invited by the organizers. We then have round-table discussion where new tasks are discussed. Authors of emerging task or resource papers are assigned to each table and serve as a leader of the table. The leaders may invite authors of the other contributions (e.g., evaluation or resource papers) to their table if they think evaluation methodology or resources presented in the workshop are useful to design a discussed task. After the round-table discussion, each table is expected to have a short presentation on a new task.