Processing Large Data Sets in Fine-Grained Parallel Streams with SQL

2 years ago thenewstack.io

Summary: This is a summary of an article originally published by The New Stack. Read the full original article here →

SQL is broadly used as a data access language, and Trino provides a powerful engine for SQL access to multiple data sources. However, as more and more real-time data becomes available, developers will need to process large data sets at an unlimited scale with predictable performance. The Aerospike Trino Connector leverages specific Aerospike mechanisms for accessing large data sets over parallel streams.

To process large data sets, a common scheme is to split the data into partitions and assign a worker task to process each partition.

Aerospike allows for data splits larger than 4,096 by allowing a partition to be divided into subpartitions efficiently.

DevOps Articles