Alle boeken

Professioneel

Apps Over Coach Inloggen Begin met lezen

Quantitative Finance · Begrippenlijst

Wat is Data loader, memory-mapped dataset?

Ook bekend als: data loader · memory-mapped dataset

Definition 23.2 Machine Learning for Markets · Hoofdstuk 23 — Training Infrastructure

A data loader is the code that turns stored data into training batches: it reads, decodes, samples, cuts and stacks examples, ideally while the device computes on the previous batch. A memory-mapped dataset stores the data in one binary file that the operating system maps into the process’s address space: nothing is read until a slice is touched, the operating system caches what is read, and several processes can share one copy.

Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
Figure 23.1. Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
Lees in het hoofdstuk →