LR2Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems
LR2Bench is a novel benchmark designed to evaluate the Long-chain Reflective Reasoning capabilities of LLMs. LR2Bench comprises 850 samples across six Constraint Satisfaction Problems (CSPs) where reflective reasoning is crucial for deriving solutions that meet all given constraints. Each type of task focuses on distinct constraint patterns, such as knowledge-based, logical, and spatial constraints, providing a comprehensive evaluation of diverse problem-solving scenarios.
Note: We have released the LR2Bench dataset here. For evaluation, you can submit your model's answer here following the submission guidelines. The Leaderboard will automatically evaluate the performance with rule-based matching. If you have further questions, please feel free to contact us at chenjianghao2022@ia.ac.cn.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 96.3 | 58.7 | 23.6 | 61.7 | 11436 |
The Crossword task requires inferring correct words from given clues and filling them into a grid. A key challenge lies in satisfying the constraint of shared letter intersections between horizontal and vertical words. We collected 150 Crossword samples published in 2024 from Los Angeles Times and Vulture in three sizes: $5 imes5$, $10 imes10$, and $15 imes15$, with 50 ones for each size.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 99.3 | 77.7 | 24.7 | 89.3 | 10098 |
The Acrostic task involves word clues like Crossword, but its objective is to form a hidden quotation or sentence from the answers to the clues. This requires that the answer words not only satisfy the corresponding clues but also effectively integrate to construct the ultimate hidden message. We collected 50 easy and 50 hard Acrostic samples from Printable Puzzles with timestamps ranging from September 2024 to December 2024.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 100 | 62.2 | 0 | 83 | 10077 |
The Logic Puzzle task constitutes a problem that necessitates logical reasoning to deduce relationships between a set of entities based on the given constraints and clues. The objective is to systematically analyze the given information, employing techniques such as hypothesis formation, elimination, and deductive inference, to determine a unique solution that satisfies all given constraints. We collected 50 puzzles for each of the four sizes ($4 imes4$, $4 imes5$, $4 imes6$, and $4 imes7$) from Printable Puzzles, with timestamps ranging from September 2024 to December 2024.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 78.5 | 69.4 | 42.5 | 68.5 | 10242 |
The Cryptogram task involves the decryption of an encrypted quotation or sentence, where each letter of an original text is substituted with another, resulting in an apparently nonsense text. Decryption requires identifying patterns, common letter frequencies, and word structures to deduce the letter-to-letter correspondences, ultimately reconstructing the original content. We collected 50 easy and 50 hard samples from Printable Puzzles with timestamps ranging from September 2024 to December 2024.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 100 | 34.8 | 13 | 29 | 12567 |
The Sudoku task consists of filling a $n^2 imes n^2$ grid with digits from 1 to $n^2$, subject to the constraint that each row, column, and $n imes n$ subgrid contains all digits from 1 to $n^2$ without repetition. Success in Sudoku relies on logical deduction and careful consideration of the existing digits to determine valid placements for the remaining numbers. From 1sudoku, we collected 200 Sudoku samples in total: 50 easy and 50 hard samples for both $4 imes4$ and $9 imes9$ sizes.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 91.5 | 70.3 | 31.5 | 55.5 | 8277 |
The Drop Quote task comprises a grid of multiple rows and columns, with each column providing a set of candidate letters. The task requires determining the correct row for letters in each column, effectively "dropping" it into target place to reveal the hidden quotation. We created 50 easy samples by manually compiling common quotations, and collected 50 hard samples from Printable Puzzles, with timestamps ranging from September 2024 to December 2024.
Model | #Params (B) | Completion Rate | Subtask Accuracy | Exact Match | Partial Match (0.5) | Tokens |
|---|---|---|---|---|---|---|
671 | 100 | 38.8 | 13 | 38 | 13595 |
✉️✨ Submit your results here!
Submission Template
See submission_template.json for detail. The following is an example for the JSON structure.
{
"config": {
"model_name": "deepseek-ai/DeepSeek-R1", # your model name
"link": "https://huggingface.co/deepseek-ai/DeepSeek-R1", # your model link if available
"Params": 671, # number of parameters if available
"show_on_leaderboard": true, # whether to show your model on the leaderboard
},
"results": {
"crossword": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
],
"acrostic": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
],
"logic": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
],
"cryptogram": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
],
"sudoku": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
],
"drop": [
{"tag": "TAG", "level": "LEVEL", "answer": "ANSWER"},
]
}
}