Data Analysis Fundamentals
87 flashcards by MentisLibrary
The full workflow from question to conclusion: cleaning, joins, aggregation, metrics, experiments, and the pitfalls that quietly ruin analyses. Learn where real analyses go wrong and how to catch it.
- What are the stages of a typical data analysis workflow?
- Define the question, collect data, clean it, explore it, analyze or model, then communicate the result
- Why should an analysis start with a question, not a dataset?
- Because the question determines which data, which grain, and which comparison actually answer it
- What is tidy data?
- A table where each row is one observation, each column one variable, and each cell one value
- What is the difference between wide and long format?
- Wide spreads a variable's categories across columns, long stacks them into a category column and a value column
- What is the grain of a table?
- The level of detail one row represents
- What is a primary key?
- A column or set of columns uniquely identifying each row
- What is a foreign key?
- A column referencing the primary key of another table
- What does an inner join do?
- Keeps only rows with a match in both tables
- What does a left join do?
- Keeps every row from the left table, filling unmatched right columns with nulls
- What is a full outer join?
- Keeps all rows from both tables, filling in nulls where there is no match
- What is an anti join?
- Returns rows in one table with no match in the other
- What causes a join to produce more rows than either input?
- A many-to-many relationship, where a key repeats on both sides
- What is a cross join?
- Pairs every row of one table with every row of another
- What is data profiling?
- An initial survey of a dataset's structure, ranges, distributions, and missingness
- What is exploratory data analysis?
- Open-ended investigation of a dataset's structure using summaries and visuals before formal modeling
- What are the three types of missing data?
- Missing completely at random, missing at random given observed variables, and missing not at random
- What is listwise deletion and when is it risky?
- Dropping every row with any missing value. It is risky when missingness is related to the outcome
- What are common imputation strategies and their trade-offs?
- Mean or median imputation is simple but shrinks variance, model-based imputation preserves relationships but adds complexity
- Why is a missing value different from a zero?
- Missing means unknown, zero means measured and none
- How do nulls behave in aggregate functions?
- Most aggregates skip nulls, but counting rows and counting non-null values differ
- How do you find duplicate rows?
- Group by the intended key and look for groups with more than one row
- What is the difference between a full-row duplicate and a key duplicate?
- A full-row duplicate repeats every column, a key duplicate repeats the identifier with different values
- How should you handle an outlier?
- Investigate it first, then decide whether it is an error, a rare real event, or a sign the model is wrong
- What is winsorizing?
- Capping extreme values at a chosen percentile instead of removing them
- What is the difference between filtering and aggregating?
- Filtering selects which rows enter the calculation, aggregating collapses rows into a summary
- What does group by do?
- Splits rows into groups by one or more keys and computes an aggregate per group
- What is a window function?
- A calculation across a set of rows related to the current row, without collapsing them
- What is a pivot table?
- A cross tabulation summarizing a metric across two grouping dimensions
- Why is averaging averages usually wrong?
- Because each group average carries a different weight, so the unweighted mean misrepresents the whole
- What is the difference between a percentage and a percentage point?
- A percentage point is an absolute difference between two percentages, a percentage change is relative