All books

Professional

Apps About Coach Log in Start reading

Quantitative Finance · Glossary

What is Chunked processing, peak resident memory?

Also known as: chunked processing · peak resident memory

Definition 8.5 Research, Data and Risk Platforms · Chapter 8 — Python at Scale

Chunked processing runs a computation over a dataset one slice at a time, carrying between slices only the state the computation needs to continue exactly (a filter’s last value, a window’s tail), so that the memory it uses is set by the chunk size and not by the size of the data. The peak resident memory of a process is the largest amount of physical memory it occupied at any moment; it, not the average, decides whether a job fits a machine.

Peak memory of the streamed signal on a history of two million events against the chunk size, each point labelled with its time in seconds; the last point computes the whole array at once. Memory grows with the chunk; time is lowest for chunks of a hundred thousand events and within ten per cent of it from ten thousand to the whole array, once the per-chunk overhead is amortised. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, machine otherwise idle; memory counted by tracemalloc, the mapped input not included. Data: bench_pyscale.py.
Figure 8.3. Peak memory of the streamed signal on a history of two million events against the chunk size, each point labelled with its time in seconds; the last point computes the whole array at once. Memory grows with the chunk; time is lowest for chunks of a hundred thousand events and within ten per cent of it from ten thousand to the whole array, once the per-chunk overhead is amortised. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, machine otherwise idle; memory counted by tracemalloc, the mapped input not included. Data: bench_pyscale.py.
Read in context →