nlvr (natural language for visual reasoning)

NLVR (Natural Language for Visual Reasoning) is a benchmark task requiring models to determine the truth of a statement based on a *set* of images — testing the ability to reason about properties, counts, and comparisons across multiple disjoint visual inputs.

What Is NLVR?

• Definition: Binary classification (True/False) of a sentence given a pair (or set) of images.
• Task: "The left image contains exactly two dogs and the right image contains none." -> True/False.
• NLVR2: The version using real web images (instead of synthetic ones) to test robustness.

Why NLVR Matters

• Set Reasoning: Unlike VQA (one image), NLVR requires holding information from Image A while analyzing Image B.
• Quantification: Heavily tests counting and numerical comparison ("more than", "at least").
• Robustness: Reduces the ability to cheat using language biases alone.

NLVR is a test of comparative visual cognition — validating that an AI can perform logical operations over multiple observations.