Process Optimisation / Big Data
ETL optimisation: reducing processing time by ~60%
I optimised PySpark and HBase ETL pipelines processing hundreds of GB in micro-batches. Total runtime fell by approximately 60%.
- My role
- Data Engineer · NTT DATA
- Reduction in ETL runtime
- ~60%
- Last updated
- 01Hundreds of GB
- 025–6 PySpark / HBase ETL pipelines
- 03~60% less processing time
The context
The project for Junta de Andalucía covered 5–6 large-scale ETL pipelines using PySpark and HBase. They processed hundreds of GB in micro-batches, making total processing time directly relevant to operations.
My contribution
I worked on implementing and optimising these pipelines at NTT DATA Europe & LATAM, with the aim of reducing end-to-end runtime.
The result
The result was an approximately 60% reduction in total ETL runtime. As a relative index, an initial duration of 100 becomes approximately 40; this comparison does not represent specific minutes or hours.
Measurement scope
The figure describes this project's result. Original run durations and infrastructure configuration are not published, so this is not a reproducible benchmark for other workloads.