BENCHMARKS & METHODOLOGYTransparent evidence status

AI performance measured against enterprise complexity.

Business intelligence depends on more than generating syntactically valid SQL. A useful AI system must interpret the question, identify the right data, navigate complex relationships, and return results that stand up to verification.

Cayra-reported result

Spider 2.0-Lite execution accuracy

73.11%

Submitted to the official leaderboard; publication and verification pending

547

benchmark tasks

3

SQL environments

Execution-based evaluation

Query results compared with expected outcomes

Result reconciliation in progress

The reported 73.11% is being reconciled against the official 547-task denominator. Raw scorer output, task-level results, and the public leaderboard listing are not yet available for publication.

Benchmark performance does not represent an accuracy guarantee for individual customer databases, organizations, or use cases.

TWO LAYERS OF ACCURACY

Benchmark performance and customer-specific validation answer different questions.

A standardized benchmark measures performance on a fixed external task set. Customer-specific validation measures how a configured system performs after approved business rules, KPI definitions, and corrected query patterns are incorporated.

01Comparable external evaluation

Standardized benchmark accuracy

Measures execution performance under a fixed benchmark setup, before customer-specific business context is available.

02Scoped deployment validation

Customer-specific validated accuracy

Measures results against a defined customer validation suite after approved terminology, business rules, and corrections have been incorporated.

Use the right metric for each stage

01

First-pass execution accuracy

How often a new question produces the expected result on the first attempt.

02

Post-iteration validated outcome rate

How often the expected result is reached after allowed retries, corrections, and validation.

03

Approved recurring-query pass rate

Reliability when previously approved queries and definitions are run again.

04

Unseen customer-question holdout rate

Generalization performance on customer questions not used during configuration or correction.

SPIDER 2.0-LITE

Can AI turn business questions into the right SQL results?

Spider 2.0-Lite is an enterprise-focused benchmark comprising 547 text-to-SQL tasks across BigQuery, Snowflake, and SQLite. Its execution-based scoring evaluates generated query results against expected outcomes.

HOW WE APPROACH ACCURACY

From question to verifiable result

These are descriptions of Cayra's engineering approach. The benchmark score alone does not prove the contribution of each individual component.

01

Context-aware interpretation

Connect natural-language questions with relevant schema information, definitions, and approved business context.

02

Specialized reasoning

Use task-focused components to plan, generate, and refine analytical queries.

03

Validation

Check queries and outputs before presenting findings as business answers.

READING THE RESULT

What benchmark results do—and do not—mean

A benchmark summarizes performance on a defined test set under a specific evaluation setup. It is a standardized engineering signal, not a customer-specific service-level guarantee.

What it helps evaluate

  • Performance on common, documented tasks and scoring rules
  • Whether generated queries return expected benchmark outcomes
  • Progress between reproducible system configurations

What it does not guarantee

  • The same accuracy percentage on every organization's private data
  • Correctness when source data or business definitions are incomplete
  • Production reliability without permissions, governance, and human review

FREQUENTLY ASKED QUESTIONS

Understanding the benchmark

What is Spider 2.0-Lite?

Spider 2.0-Lite is an enterprise-focused text-to-SQL benchmark with 547 tasks spanning BigQuery, Snowflake, and SQLite. It tests whether generated SQL produces expected results.

What does execution accuracy mean?

Execution accuracy compares the output of a generated SQL query with the expected output under the benchmark's evaluation rules. It does not simply check whether the SQL runs without syntax errors.

Is Cayra officially ranked on the Spider leaderboard?

Cayra has submitted a self-reported result to the official leaderboard and is awaiting publication and verification. We do not present the result as an official rank or independently verified score.

Does this score guarantee the same accuracy on our database?

No. A benchmark is a standardized engineering evaluation, not a customer-specific accuracy guarantee. Results on your data should be validated against your organization's rules, permissions, and data quality.

How does Cayra account for company-specific definitions?

Cayra is designed to incorporate approved business definitions, schema information, and organizational context into its analysis workflow. Company-specific performance should also be assessed in a customer pilot.

How can customer-specific validation be higher than benchmark performance?

A benchmark uses a fixed external task set without a customer's approved definitions. Customer validation is performed after terminology, KPI logic, business rules, and corrected query patterns are incorporated. A defined validation suite can therefore reach a higher result, including 100%, without implying perfect accuracy on every unseen question.

EVALUATE ON YOUR DATA

A benchmark is a starting point. Your data is the real test.

Review Cayra's workflow with your own schemas, business definitions, and verification requirements.

Talk to SalesSee how Cayra works