Data Engineering
109 flashcards by MentisLibrary
Building the pipelines analytics runs on: ingestion, batch against streaming, warehouses and lakes, modelling for analysis, idempotency and backfills, partitioning, data quality and what each choice costs.
- What does a data engineer build?
- Systems that move data from where it is produced to where it is analysed, reshape it into a model people can query, and keep it correct, timely and affordable while it runs unattended.
- Data pipeline
- A sequence of steps that extract data from a source, transform it, and load it somewhere it can be used, with defined triggers, dependencies and failure behaviour.
- Orchestrator versus scheduler
- A scheduler runs a job at a time. An orchestrator understands dependencies between jobs, so a task starts when its inputs are ready and a failure stops what depends on it.
- DAG
- A directed acyclic graph of tasks where edges are dependencies, which is the standard way an orchestrator describes a pipeline.
- Idempotent task
- A task that produces the same result whether it runs once or five times, which is what makes automatic retries and manual reruns safe.
- Why should a task never call now()?
- Because the result then depends on when the task happened to run, so a rerun of yesterday's job processes today's data and the output is not reproducible.
- Backfill
- Running a pipeline over a range of past periods, either to populate a new table or to correct a bug in historical output.
- Backfill concurrency
- Running many historical periods at once is faster but can overwhelm the source, the warehouse or a downstream table that expects ordered writes.
- Retry with exponential backoff
- Retrying a failed step after a delay that grows with each attempt, so transient failures recover without hammering a struggling dependency.
- Sensor and the polling trap
- A task that waits for an external condition, such as a file arriving, before downstream work starts.
- Task granularity
- How much work one task does. Smaller tasks give precise retries and better parallelism; larger tasks reduce scheduling overhead and coordination.
- Pipeline SLA
- A stated deadline for when a dataset must be ready, and an alert when it is not met, distinct from whether the job succeeded.
- Data lineage
- The record of which datasets and columns each dataset was derived from, and by which job.
- Extract, transform, load (ETL)
- Transform the data before loading it, so only the modelled result lands in the destination.
- Extract, load, transform (ELT)
- Load the raw data first and transform it inside the destination with SQL, keeping the untouched source as the starting point.
- Medallion layering
- A convention naming three layers: raw ingested data, cleaned and conformed data, and business-level aggregates.
- Change data capture (CDC)
- Capturing every insert, update and delete in a source database so a copy elsewhere can be kept current without repeatedly scanning the source.
- Log-based versus query-based CDC
- Log-based CDC reads the database transaction log and sees every change including deletes. Query-based CDC polls for rows whose timestamp changed.
- High-water mark
- The largest value of a monotonically increasing column seen in the last extraction, stored and used as the lower bound for the next one.
- Full refresh versus incremental load
- A full refresh reloads everything and is always consistent. An incremental load moves only new or changed rows and is far cheaper.
- Late-arriving data
- Records that reach the pipeline after the period they belong to has already been processed.
- Lookback window
- Reprocessing the last few periods on every run so that late-arriving data is picked up without a full backfill.
- Schema drift
- A source changing its schema over time: new fields, renamed fields, retyped fields, removed fields.
- Schema registry
- A service that stores the schema for each message type and enforces compatibility rules between versions before a producer can publish.
- Backward and forward compatibility
- Backward compatible means new consumers can read old data. Forward compatible means old consumers can read new data. Full compatibility means both.
- Dead letter queue
- A separate destination for messages that could not be processed, so one bad record does not stop the stream.
- Batch processing
- Processing a bounded set of records collected over a period, on a schedule.
- Stream processing
- Processing records continuously as they arrive, holding state across them rather than over a fixed file set.
- How do you choose batch or streaming?
- Ask what decision the data drives and how stale it may be. If nobody acts within an hour, batch is cheaper and simpler; if a system reacts in seconds, stream.
- Micro-batching
- Running a streaming job as a rapid sequence of small batches, which recovers batch semantics with much lower latency.