AtScale, a leader in semantic layer technology, has introduced a public leaderboard for Text-to-SQL (T2SQL) solutions. This innovative platform sets a new standard for evaluating T2SQL solutions, enabling academia, vendors, and developers to measure performance using a replicable benchmark. With the rise of Generative AI, T2SQL solutions are transforming data accessibility, allowing non-technical users to query data with natural language. However, inconsistent evaluation standards have limited transparency—until now.
Key Highlights
1. Public Benchmarking Environment for T2SQL
- The leaderboard offers a GitHub repository with comprehensive guidelines for downloading the TPC-DS dataset, evaluation queries, KPI definitions, and scoring methods.
- This setup enables a consistent framework for T2SQL evaluations, creating an open resource for all participants.
2. Objective Complexity Metrics for Evaluation
- The leaderboard evaluates T2SQL solutions based on:
- Question Complexity: Ranges from basic selections to complex aggregations, accounting for the depth of query KPIs.
- Schema Complexity: Considers the number of tables used, with higher ratings for questions involving five or more tables.
3. Real-Time Leaderboard for Transparent Performance Tracking
- A first in the industry, AtScale’s real-time leaderboard publicly displays scores of participating T2SQL solutions.
- This live feature fosters transparency and healthy competition, encouraging continuous improvement.
4. Encouraging Community Collaboration
- As an open platform, AtScale invites the industry to provide feedback, submit solutions, and participate in refining the evaluation process.
- Community collaboration ensures ongoing improvements in T2SQL standards, making natural language data queries more accessible.
AtScale’s public T2SQL leaderboard addresses a critical need for standardization in evaluating Text-to-SQL capabilities. By providing a transparent, objective benchmark, AtScale is fostering industry collaboration and helping advance solutions that make natural language queries accessible for all.