All books

Professional

Apps About Coach Log in Start reading

Quantitative Finance · Glossary

What is Ingestion pipeline, schema validation?

Also known as: ingestion pipeline · schema validation

Definition 15.1 Machine Learning for Markets · Chapter 15 — Alternative-Data Pipelines

An ingestion pipeline is the code that receives a vendor’s deliveries, checks them, maps them to the firm’s identifiers and stores them with the time each value became known. Schema validation checks each record against a declared schema: every field present, of its type and in its range, and no record duplicated; records that fail are rejected with a reason, never silently repaired.

Read in context →