Hero Image
Implementing Spark-Compatible json_tuple in Apache DataFusion

PR: apache/datafusion#20412 Background DataFusion-Comet accelerates Spark queries by offloading execution to Apache DataFusion. For this to work, DataFusion needs to support the Spark built-in functions that Comet encounters. json_tuple is one of them — it is commonly used in ETL pipelines to extract fields from JSON columns without defining a full schema. Comet had an open issue requesting this. Without native support, queries using json_tuple would fall back to Spark’s own execution path, defeating the purpose of using Comet.

Friday, February 20, 2026 | 2 minutes Read
Hero Image
Making gRPC Timeouts Configurable in Apache DataFusion-Ballista

PR: apache/datafusion-ballista#1337 Background Ballista is a distributed query engine built on DataFusion. It coordinates executors through a scheduler, with all inter-node communication going over gRPC. A previous PR (#115) had introduced gRPC timeout support, but all values were hard-coded. In production environments, different workloads require different timeout behavior — a long-running aggregation needs different settings than a quick metadata fetch. Without configuration options, the only recourse was to modify the source code directly.

Saturday, November 1, 2025 | 2 minutes Read