📚 NumPy & Pandas – Advanced Study Notes (MIT‑Level)
Goal: Build a deep, rigorous understanding of NumPy and Pandas, from first principles to advanced usage.
Audience: Graduate students or senior undergraduates in Computer Science/Mathematics/Data Science.
Prerequisites:
- Basic Python (lists, dictionaries, functions).
- Elementary linear algebra (vectors, matrices).
- Familiarity with floating‑point arithmetic.
1. Introduction
| Library | Purpose | Core Data Structure | Typical Workflow |
|---|---|---|---|
| NumPy | Fast, memory‑efficient n‑dimensional array operations | np.ndarray |
1️⃣ Create array → 2️⃣ Vectorize → 3️⃣ Use ufuncs/BLAS |
| Pandas | Structured data manipulation (tabular, time‑series) | Series, DataFrame, MultiIndex |
1️⃣ Load → 2️⃣ Clean → 3️⃣ Transform/aggregate → 4️⃣ Export |
Why learn from first principles?
- Understand the why behind API choices.
- Avoid subtle bugs (e.g., broadcasting pitfalls).
- Optimize performance by exploiting memory layout.
2. NumPy Fundamentals
2.1 np.ndarray: The Core Object
A NumPy array is a contiguous block of memory that stores elements of a single dtype.
Formally:
A = {x ∈ ℝ^k | x is a vector of length k}
where k is the total number of elements.
- Shape:
tupledescribing dimensions, e.g.(3, 4). - Strides:
tupleof bytes to skip to move one step in each dimension. - Order:
'C'(row-major) or'F'(column-major).
Memory layout example:
A = np.arange(6).reshape(2,3)
# C-order
# 0 1 2
# 3 4 5
A.strides # (12, 4) → 12 bytes to move to next row, 4 bytes to next column
Key theorem (Linear Algebra):
For anynp.ndarray, there exists an isomorphism to a vector space over ℝ (or ℂ).
This permits us to apply linear‑algebraic operations elementwise.
2.2 Data Types (dtype)
- Numeric:
int64,float64,complex128, etc. - Object: arbitrary Python objects.
- Structured: composite types (
dtype=[('x', np.float64), ('y', np.int32)]).
Promotion rules (simplified):
| Left | Right | Result |
|---|---|---|
int64 |
float64 |
float64 |
float32 |
float64 |
float64 |
int32 |
int64 |
int64 |
| ... | ... | ... |
Why promotion matters:
It determines precision loss in mixed‑type operations.
2.3 Broadcasting – First Principles
Definition: Two arrays are broadcastable if for each dimension, the sizes are equal or one of them is 1.
Proof sketch (from numpy/doc/broadcasting.md):
Let A with shape s_A = (a1, a2, …, an) and B with s_B.
We right‑align shapes (pad with 1s on the left) and iterate dimensions:
for i in reversed(range(max(n_A, n_B))):
if s_A