Todos os livros

Profissional

Apps Sobre Coach Entrar Começar a ler

Quantitative Finance · Glossário

O que é Data loader, memory-mapped dataset?

Também chamado de: data loader · memory-mapped dataset

Definition 23.2 Machine Learning for Markets · Capítulo 23 — Training Infrastructure

A data loader is the code that turns stored data into training batches: it reads, decodes, samples, cuts and stacks examples, ideally while the device computes on the previous batch. A memory-mapped dataset stores the data in one binary file that the operating system maps into the process’s address space: nothing is read until a slice is touched, the operating system caches what is read, and several processes can share one copy.

Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
Figure 23.1. Where a training step’s time goes, by storage format and batch assembly: medians of five epochs of 100 steps of 256 windows. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, one thread, no isolated cores; other jobs may have run. Data: bench_train.py, measured_loader.csv.
Ler no capítulo →