جميع الكتب

مهني

1 Markets I: The Ecosystem and Exchange-Traded Marketsالأسواق عبر الإنترنت 2 Markets II: Rates, FX and Creditالأسواق عبر الإنترنت 3 Markets III: Commodities, Energy and Cryptoالأسواق عبر الإنترنت 4 Quantitative Methodsالأساليب عبر الإنترنت 5 Derivatives and Volatilityالمشتقات عبر الإنترنت 6 Rates, Credit, XVA and Riskالفائدة والائتمان والمخاطر عبر الإنترنت 7 Research Craft: Predictors, Backtests, Measurement, Portfoliosالبحث عبر الإنترنت 8 Strategies I: Equities and Futuresالاستراتيجيات عبر الإنترنت 9 Strategies II: Volatility, Relative Value, Macro and the Bank Desksالاستراتيجيات عبر الإنترنت 10 Microstructure and Executionالتنفيذ عبر الإنترنت 11 Market Making and High-Frequency Tradingصناعة السوق عبر الإنترنت 12 Machine Learning for Marketsتعلم الآلة عبر الإنترنت 13 Low-Latency Softwareالتكنولوجيا عبر الإنترنت 14 Networks, Hardware and Trading Infrastructureالتكنولوجيا عبر الإنترنت 15 Research, Data and Risk Platformsالتكنولوجيا عبر الإنترنت 16 The Desk and the Firmالشركة عبر الإنترنت 17 The Industry: Firms, Roles and Careersالمسارات المهنية عبر الإنترنت 18 The Interview Bookالمسارات المهنية عبر الإنترنت
التطبيقات حول المدرب تسجيل الدخول ابدأ القراءة

Quantitative Finance · المسرد

ما معنى Chunked processing, peak resident memory؟

يُعرف أيضًا باسم: chunked processing · peak resident memory

Definition 8.5 Research, Data and Risk Platforms · الفصل 8 — Python at Scale

Chunked processing runs a computation over a dataset one slice at a time, carrying between slices only the state the computation needs to continue exactly (a filter’s last value, a window’s tail), so that the memory it uses is set by the chunk size and not by the size of the data. The peak resident memory of a process is the largest amount of physical memory it occupied at any moment; it, not the average, decides whether a job fits a machine.

Peak memory of the streamed signal on a history of two million events against the chunk size, each point labelled with its time in seconds; the last point computes the whole array at once. Memory grows with the chunk; time is lowest for chunks of a hundred thousand events and within ten per cent of it from ten thousand to the whole array, once the per-chunk overhead is amortised. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, machine otherwise idle; memory counted by tracemalloc, the mapped input not included. Data: bench_pyscale.py.
Figure 8.3. Peak memory of the streamed signal on a history of two million events against the chunk size, each point labelled with its time in seconds; the last point computes the whole array at once. Memory grows with the chunk; time is lowest for chunks of a hundred thousand events and within ten per cent of it from ten thousand to the whole array, once the per-chunk overhead is amortised. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, machine otherwise idle; memory counted by tracemalloc, the mapped input not included. Data: bench_pyscale.py.
اقرأ في الفصل →