Streaming large Excel reports with Bash

The limit was memory

I was working on a pipeline that generated CSV and Excel reports. The Excel libraries we used assembled files in memory. As worksheets grew, that approach became a constraint: generating a large report meant holding too much of it in memory at once.

The change that mattered was how the worksheet was written. I moved to streaming XML instead of assembling the complete worksheet in memory.

Inspect the format before changing the language

My earlier investigation started with different libraries and experiments in Python, Go and Rust. Looking inside the Excel file was the more useful step: it showed how worksheet data could be written separately from the rest of the workbook.

The original process sketch began with an .xlsx template containing placeholders. I opened the package as a ZIP archive to inspect its XML parts. Renaming the extension did not convert the data; it made the existing package easier to explore.

Wait for the data to be ready

The final pipeline used Bash, with Redis and queues for orchestration and Redshift unloads supplying the data. A completion file appeared after the other documents had been updated. Its arrival triggered report generation.

That separated the readiness signal from the report-generation step: the generator began when the upstream work had produced its completion file.

Write incrementally, then package

The pipeline generated CSV files and streamed the XML used by the Excel worksheets. It then packaged the Excel components into an archive. This avoided keeping an entire worksheet in memory while building it.

Streaming addressed the memory pressure that had limited larger worksheets, and report-generation performance improved. The final implementation used Bash; the earlier language and library experiments were part of the investigation.

The workbook-generation flow

The diagram below shows the workbook-writing stage. The completion-file trigger and Redis orchestration sit upstream.

Shared strings are an optional tradeoff

The original diagram also included an optional sharedStrings.xml step. Repeated text can be stored once and referenced by index from worksheet cells. Whether that reduces the final file size depends on the data.

Maintaining that index has its own bookkeeping cost. A streaming design needs to consider string handling alongside the worksheet writer, rather than assuming that every part of the generator has a constant memory footprint.

The engineering lesson

When a report grows beyond what an in-memory writer can comfortably handle, changing the data flow can matter more than changing the programming language. In this case, separating orchestration, data readiness and incremental output gave me a workable approach to larger reports.

A streaming writer still needs to produce a complete, valid workbook. The useful optimization is one that preserves the report people need while reducing the amount of data the process must hold at once.