Apache Spark
111 flashcards by MentisLibrary
The Spark execution model, RDDs and DataFrames, transformations and actions, shuffles and partitioning, join strategies, caching, Spark SQL, Structured Streaming, and how to tune a slow job.
- What is Apache Spark?
- A distributed engine that runs a computation over a dataset split into partitions, keeping intermediate results in memory where it can and recomputing lost work from a recorded lineage of operations.
- Driver and executors
- The driver runs your program, builds the execution plan and schedules work. Executors are JVM processes on worker nodes that run tasks and hold cached data.
- Partition
- The unit of parallelism: a slice of the dataset that one task processes on one core.
- Task, stage, and job
- A task processes one partition. A stage is a set of tasks that need no shuffle between them. A job is everything triggered by a single action.
- Lazy evaluation
- Transformations only record what to do; nothing runs until an action asks for a result.
- Transformation versus action
- A transformation returns a new dataset and is lazy. An action returns a value to the driver or writes output, and it triggers execution.
- Narrow versus wide dependency
- A narrow dependency lets each output partition read from one input partition. A wide dependency needs data from many input partitions, so it forces a shuffle.
- Shuffle
- Redistributing data across partitions so that all records with the same key land together, by writing map-side output to disk and having reducers fetch it over the network.
- spark.sql.shuffle.partitions
- Sets how many partitions a shuffle produces for DataFrame and SQL operations.
- RDD
- The original Spark abstraction: an immutable, partitioned collection of objects with a recorded lineage of the operations that built it.
- Lineage
- The recorded chain of transformations that produced an RDD or DataFrame, used to recompute lost partitions after a failure instead of replicating data.
- When should you still use the RDD API?
- When the data is not tabular, when you need precise control over partitioning, or when you must work with an object model that has no schema.
- map versus flatMap
- map returns exactly one output element per input element; flatMap returns zero or more, and the results are flattened into one collection.
- mapPartitions
- Applies a function to a whole partition at once rather than to each element, so per-partition setup happens once.
- reduceByKey versus groupByKey
- reduceByKey combines values on the map side before shuffling, so only one partial result per key crosses the network. groupByKey shuffles every value.
- Partitioner
- The rule that maps a key to a partition number for a shuffle. Hash partitioning is the default; range partitioning keeps keys ordered.
- DataFrame
- A distributed table with a schema, described as a plan that Catalyst optimises before execution.
- Catalyst optimizer
- The query optimiser that turns a DataFrame or SQL query into a physical plan through analysis, logical optimisation, physical planning, and code generation.
- Tungsten
- The execution layer that stores rows in a compact binary format outside the JVM object heap and generates specialised bytecode for each query stage.
- Predicate pushdown
- Pushing a filter down into the data source so that rows are skipped before they are ever read into Spark.
- Column pruning
- Reading only the columns a query actually uses, which a columnar format such as Parquet or ORC supports natively.
- Explicit schema versus inferSchema
- Providing a StructType avoids the extra scan that schema inference needs and guarantees stable column types.
- Why is Parquet the default choice for Spark?
- It is columnar, compressed, splittable, and carries per-row-group statistics, so Spark can prune columns and skip blocks without reading them.
- collect()
- Brings every row of the result to the driver as a local list.
- Broadcast variable
- A read-only value shipped once to each executor and reused by every task there, instead of being serialised into every task closure.
- Accumulator
- A write-only counter that tasks add to and the driver reads, used for metrics and diagnostics.
- Closure serialization
- Everything a lambda references is serialised and shipped to executors, including the enclosing object if you touch one of its fields.
- repartition versus coalesce
- repartition(n) does a full shuffle and can raise or lower the partition count evenly. coalesce(n) only merges existing partitions with no shuffle and can only lower it.
- repartitionByRange
- Repartitions using sampled range boundaries so that ordered data lands in ordered partitions.
- Partitioned write (partitionBy)
- Writes output into a directory tree keyed by column values, so readers can skip whole directories.