Logo Raymond's Portfolio
  • Home
  • About
  • Skills
  • Projects
  • Recent Posts
  • Posts
Logo Inverted Logo
  • Tags
  • Apache
  • Apache DataFusion
  • Apache DataFusion-Ballista
  • Apache DataFusion-Comet
  • Apache Iceberg
  • Code Quality
  • Contributions
  • Distributed Systems
  • ErrorProne
  • GRPC
  • Java
  • Open Source
  • Refactoring
  • Rust
  • Scala
  • Spark
Hero Image
Refactoring QueryPlanSerde in Apache DataFusion-Comet

PRs: #2028 and #2085 — part of tracking issue #2019 Background DataFusion-Comet translates Spark physical plans into DataFusion execution plans. The core of this translation lives in QueryPlanSerde, a Scala file responsible for serializing Spark expressions into Protocol Buffer messages. Over time, QueryPlanSerde had accumulated serialization logic for every expression type — comparisons, datetime operations, string functions, math — all in a single file. This made navigation difficult, PR reviews cumbersome, and adding new expressions error-prone.

  • Apache DataFusion-Comet
  • Scala
  • Refactoring
  • Open Source
Saturday, August 9, 2025 | 2 minutes Read
Navigation
  • About
  • Skills
  • Projects
  • Recent Posts
Contact me:
  • yuchuanhung.raymond@gmail.com
  • https://github.com/CuteChuanChuan
  • Yu-Chuan Hung

Toha Theme Logo Toha
© 2026 Raymond (Yu-Chuan Hung)
Powered by Hugo Logo