Back to Blog
    Performance
    GizmoData Team
    July 15, 2025
    5 min read

    GizmoSQL One-Trillion Row Challenge Results

    GizmoSQL completed the 1 trillion row challenge! Learn how we processed massive datasets using DuckDB and Apache Arrow Flight SQL.

    GizmoSQL
    DuckDB
    Big Data
    Performance
    GizmoSQL One Trillion Row Challenge

    GizmoSQL completed the 1 trillion row challenge! GizmoSQL is powered by DuckDB and Apache Arrow Flight SQL.

    Why We Took the Challenge

    We wanted to see what GizmoSQL was truly capable of. When we discovered the Coiled 1 Trillion Row Challenge, we knew it was the perfect benchmark to push GizmoSQL to its limits. Processing a trillion rows is no small feat—it tests everything from I/O throughput to query optimization to memory management. We were confident that the combination of DuckDB's columnar engine and Arrow Flight SQL's efficient data transfer would deliver impressive results.

    Infrastructure Setup

    We launched an AWS Graviton 4 (arm64) r8gd.metal-48xl EC/2 instance (costing $14.1082 on-demand, and $2.8216 spot) in region: us-east-1 using script: launch_aws_instance.sh in the attached zip file. We have an S3 end-point in the VPC to avoid egress costs.

    That script calls script: scripts/mount_nvme_aws.sh which creates a RAID 0 storage array from the local NVMe disks - creating a single volume that has: 11.4TB in storage.

    Docker Container Setup

    We launched the GizmoSQL Docker container using scripts/run_gizmosql_aws.sh - which includes the AWS S3 CLI utilities (so we can copy data, etc.).

    Data Preparation

    We then copied the S3 data from s3://coiled-datasets-rp/1trc/ to the local NVMe RAID 0 array volume - using attached script: scripts/copy_coiled_data_from_s3.sh - and it used: 2.3TB of the storage space. This copy step took: 11m23.702s (costing $2.78 on-demand, and $0.54 spot).

    Creating the Data View

    We then launched GizmoSQL via the steps after the docker stuff in: scripts/run_gizmosql_aws.sh - and connected remotely from our laptop via the GizmoSQL JDBC Driver - (see repo: https://github.com/gizmodata/gizmosql for details) - and ran this SQL to create a view on top of the parquet datasets:

    CREATE VIEW measurements_1trc
AS
SELECT *
  FROM read_parquet('data/coiled-datasets-rp/1trc/*.parquet');

    Row Count Query

    First, we ran a row count query to verify the data:

    Row count query result showing 1 trillion rows

    The Challenge Query

    We then ran the test query:

    SELECT station, min(measure), max(measure), avg(measure)
FROM measurements_1trc
GROUP BY station
ORDER BY station;
    Challenge query execution results

    Performance Results

    • First execution (cold-start): 0:02:22 (142s) - EC/2 on-demand cost: $0.56, spot cost: $0.11
    • Second execution (warm-start): 0:02:09 (129s) - EC/2 on-demand cost: $0.51, spot cost: $0.10

    Additional Results

    Here are the query results: 1trc_results.csv

    Scripts used: 1trc_gizmosql.zip

    Side note: Query SELECT COUNT(*) FROM measurements_1trc; takes: 21.8s

    What This Means

    These results demonstrate GizmoSQL's ability to handle massive datasets efficiently and cost-effectively. Processing a trillion rows in just over 2 minutes for under $0.60 showcases the power of combining DuckDB's columnar processing with Apache Arrow Flight SQL's efficient data transfer protocols.

    Try GizmoSQL Yourself

    Ready to experience lightning-fast data analytics? Try GizmoSQL or check out our GitHub repository to get started.

    Ready to Try GizmoSQL?

    Experience lightning-fast data analytics with our open-source SQL engine