posted by user: lefterav || 1026 views || tracked by 2 users: [display]

WMT-Testsuites 2024 : 'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)

FacebookTwitterLinkedInGoogle

Link: https://www2.statmt.org/wmt24/testsuite-subtask.html
 
When Nov 12, 2024 - Nov 13, 2024
Where Miami, Florida
Submission Deadline Jun 12, 2024
Categories    machine translation   NLP
 

Call For Papers

The “test suites” sub-task will be included for the sixth time in the General MT Shared Task of the Conference on Machine Translation (WMT24).

*OVERVIEW*

Test suites are custom extensions to the test sets of the General MT Shared Task, constructed so that they can focus on concrete aspects of the MT output. They consist of a source-side test-set and a customized evaluation service. As opposed to the standard evaluation process which produces generic quality scores, test suites often produce separate fine-grained results for each phenomenon.

Since the usage of LLMs for translation is getting more popular, and we are expecting more LLMs participations in WMT this year, the theme of this year’s test suite sub-task is "Help us break LLMs", i.e. to reveal weaknesses and serious flaws of LLMs when translating, hidden within the overall high-quality generation.

*IMPORTANT DATES*

11th April: Test suite source texts may be submitted for a pre-run on SoTA MT systems
12th June: Test suite source texts must reach us
11th July: Translated test suites shipped back to test suites authors:
TBC - August: Test suite description and analysis paper
12th-13th November: Conference

Potential participants are kindly requested to fill in this form
https://forms.office.com/e/e4JuMTSWFF

Further information can be found in the dedicated page of the WMT website
http://www2.statmt.org/wmt24/testsuite-subtask.html

Related Resources

IEEE-Ei/Scopus-ITCC 2025   2025 5th International Conference on Information Technology and Cloud Computing (ITCC 2025)-EI Compendex
SDM 2024   SIAM International Conference on Data Mining
CCITT 2025   4th International Conference on Computing and Information Technology Trends
WMT-metrics 2024   WMT24 Metrics Task: Call for Participation
JEDT 2024   International Journal of Electronic Design and Test
LSIJ 2024   Life Sciences: an International Journal
MobiSys 2025   The 23rd ACM International Conference on Mobile Systems, Applications, and Services
IEEE ICCE 2025   43rd IEEE International Conference on Consumer Electronics
DEPLING 2023   International Conference on Dependency Linguistics