Looking at the paper, the consistency function c(xi, yi, xj, yj) is frustratingly under-specified. The authors give only two concrete examples:
Math problems: Two solutions to the same problem can't both be "True" if they have different final answers
Comparisons: "A > B" and "B > A" can't both be "True" (asymmetry constraint)
The function returns 1 for inconsistencies, 0 for consistent pairs. But the paper doesn't explain how to detect "same math problem" or implement the asymmetry check in practice.
From the implementation hints, it seems they use simple pattern matching - probably checking if question text is identical for math problems, and detecting comparison pairs through linguistic patterns. The authors explicitly say they use "simple and general logical constraints" rather than fine-grained consistency checking.
This is one of those critical implementation details that makes reproduction difficult. You'd likely need to implement domain-specific heuristics: exact string matching for duplicate problems, regex patterns for comparisons, maybe basic semantic similarity for near-duplicates. The function seems designed to prevent obvious degenerate solutions (labeling everything the same) rather than enforce comprehensive logical consistency.
It's a significant gap in an otherwise interesting paper - the consistency term appears crucial for preventing the algorithm from gaming the mutual predictability objective, but they don't give enough detail to actually implement it reliably.
def generate_consistency_tests(examples, task_description):
prompt = f"""
Given this task: {task_description}
And these example pairs: {examples[:5]}
Write Python functions to detect when two examples should have:
1. The same label (duplicates, equivalent problems)
2. Different labels (contradictions, opposites)
3. Asymmetric constraints (A>B excludes B>A)
Return executable code for consistency_check(xi, yi, xj, yj).
"""
generated_code = llm.generate(prompt)
return compile_and_validate(generated_code)
Math problems: Two solutions to the same problem can't both be "True" if they have different final answers
Comparisons: "A > B" and "B > A" can't both be "True" (asymmetry constraint)
The function returns 1 for inconsistencies, 0 for consistent pairs. But the paper doesn't explain how to detect "same math problem" or implement the asymmetry check in practice. From the implementation hints, it seems they use simple pattern matching - probably checking if question text is identical for math problems, and detecting comparison pairs through linguistic patterns. The authors explicitly say they use "simple and general logical constraints" rather than fine-grained consistency checking.
This is one of those critical implementation details that makes reproduction difficult. You'd likely need to implement domain-specific heuristics: exact string matching for duplicate problems, regex patterns for comparisons, maybe basic semantic similarity for near-duplicates. The function seems designed to prevent obvious degenerate solutions (labeling everything the same) rather than enforce comprehensive logical consistency.
It's a significant gap in an otherwise interesting paper - the consistency term appears crucial for preventing the algorithm from gaming the mutual predictability objective, but they don't give enough detail to actually implement it reliably.