Financial services firms are increasingly turning to Delta Parquet, an open-source, column-oriented file format designed to dramatically reduce the cost and time associated with processing large-scale datasets used in analytics, artificial intelligence (AI) and machine learning (ML) workflows.
Built on the widely adopted Parquet file format, Delta Parquet extends its columnar storage capabilities with transaction logs, schema enforcement and performance optimisations that bring database-like intelligence to the data layer. The result is a storage format that makes handling vast quantities of financial data significantly faster and more cost-effective.
LSEG Data & Analytics recently delved into the technology and how it is accelerating innovation.
The efficiency gains are striking, LSEG explained. A 1TB CSV file can be compressed down to roughly 130GB in Delta Parquet, an 87% reduction in size. Query run times for the same dataset shrink from 236 seconds to just 6.78 seconds, representing a 34x speed improvement. Perhaps most notably, compute costs fall from $5.75 to just $0.01 per query, a 99.7% reduction.
These gains stem from a combination of technical features built into the format. Columnar storage means data is held column-by-column rather than row-by-row, which is far more efficient for analytical queries that typically only need a fraction of available columns. Compression algorithms are then applied to this uniform data, further shrinking file sizes. The format also supports schema evolution, allowing users to add, remove or modify columns without rewriting entire datasets, a critical capability in fast-moving data environments.
Additional features such as row group statistics enable query engines to skip irrelevant data chunks entirely when applying filters, while the format’s design supports parallel processing across distributed frameworks like Apache Spark and Hadoop. Delta Parquet is also platform and language agnostic, meaning it integrates across a wide variety of systems and technologies without vendor lock-in.
London Stock Exchange Group (LSEG) has made several of its flagship data products available in Delta Parquet format via AWS, including Quantitative Analytics, Tick History, Tick History – PCAP, and Filings over S3 Direct. Firms accessing these datasets benefit from point-in-time and tick-level data that supports backtesting, research, advanced transaction cost analysis and regulatory compliance use cases, including the Fundamental Review of the Trading Book (FRTB).
For more insights, read the full story here.
Copyright © 2026 FinTech Global
