wnli

WNLI (Winograd NLI) is a conversion of the Winograd Schema Challenge into an NLI format, included in the GLUE benchmark — arguably the most problematic task in GLUE due to its small size and distribution issues.

Problem

• Structure: Sentence A: "The trophy didn't fit in the suitcase because it was too large." Sentence B: "The trophy was too large." (Entailment).
• Difficulty: The train/dev sets were disjoint in a way that made learning impossible for BERT.
• Hack: The standard baseline was simply to predict "Majority Class" (65%), which BERT failed to beat for years.
• Outcome: Generally ignored or treated as a "solved/broken" metric in GLUE analysis.

Why It Matters

• Cautionary Tale: Shows the difficulty of converting specific reasoning tasks (WSC) into generic formats (NLI).
• Winogrande: This failure led to the creation of Winogrande (larger, robust WSC).

WNLI is the broken benchmark — a well-intentioned attempt to include Winograd schemas in GLUE that ended up verifying very little.