सभी किताबें

पेशेवर

ऐप्स परिचय Coach लॉग इन पढ़ना शुरू करें

Quantitative Finance · शब्दावली

Data loader, memory-mapped dataset क्या है?

अन्य नाम: data loader · memory-mapped dataset

Definition 23.2 Machine Learning for Markets · अध्याय 23 — Training Infrastructure

A data loader is the code that turns stored data into training batches: it reads, decodes, samples, cuts and stacks examples, ideally while the device computes on the previous batch. A memory-mapped dataset stores the data in one binary file that the operating system maps into the process’s address space: nothing is read until a slice is touched, the operating system caches what is read, and several processes can share one copy.

Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
Figure 23.1. Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
अध्याय में पढ़ें →