An analysis of 828 database research papers published between 1999 and 2025 reveals a significant upward trend in the volume of benchmarking per paper, often at the expense of fundamental research. The author argues that this focus on quantity leads to pseudoscience, as many standard benchmarks fail to represent real-world traffic patterns or validate system performance effectively.
Key points
Modern database papers have increased their evaluation sections by over 40%, with a concurrent doubling of benchmarks per paper.
The reliance on closed-loop benchmarks fails to capture real-world system behavior under load, which requires open-loop testing models.
Standardized benchmarks like YCSB or TPC frequently rely on unrepresentative workloads that do not predict actual performance in production.
The current academic incentive structure forces researchers to prioritize benchmarking quantity over asymptotic performance improvements.