Core Science – M.Tech.
Topic: Data Analysis
Data analysis is the systematic process of inspecting, cleaning, transforming, and modeling data to discover useful information, draw conclusions, and support decision‑making.
1. Simple Explanation
| Step | What Happens | Why It Matters |
|---|---|---|
| 1. Collect | Gather raw data from experiments, sensors, or databases. | Provides the material for analysis. |
| 2. Clean | Remove errors, handle missing values, and standardize formats. | Ensures reliability of results. |
| 3. Explore | Use descriptive statistics & visualisations to understand distributions, trends, and outliers. | Gives intuition and guides further steps. |
| 4. Model | Apply statistical or machine‑learning models (regression, clustering, classification). | Quantifies relationships or predicts outcomes. |
| 5. Validate | Test model performance using cross‑validation, error metrics, or hypothesis tests. | Confirms that the model generalises. |
| 6. Interpret | Translate numbers into scientific insight, write reports, and make recommendations. | Turns data into actionable knowledge. |
2. Key Points
| Concept | Definition | Example |
|---|---|---|
| Variables | Quantities measured or observed. | Temperature (continuous), Species (categorical). |
| Descriptive Stats | Mean, median, variance, standard deviation, percentiles. | Mean temperature of a dataset = 23.4 °C. |
| Correlation | Statistical association between two variables. | Pearson’s r = 0.78 indicates a strong positive relationship. |
| Regression | Predicts a dependent variable from one or more independent variables. | Linear regression: (y = \beta_0 + \beta_1x). |
| Hypothesis Testing | Determines if observed patterns are statistically significant. | t‑test for mean difference. |
| Confidence Interval | Range that likely contains the true parameter. | 95 % CI for mean = (21.8, 25.0). |
| Outlier | Observation far from others. | A temperature reading of 120 °C in a dataset of 20–30 °C. |
| Missing Data | Unrecorded values. | Use imputation or deletion. |
| Overfitting | Model captures noise instead of signal. | Occurs when too many predictors are used. |
| Cross‑Validation | Technique to assess model’s predictive power. | 5‑fold CV. |
3. Worked Example
Problem:
You have measured the chlorophyll content (µg g⁻¹) of 10 plant leaves and want to see if leaf area (cm²) predicts chlorophyll.
| Leaf | Area (cm²) | Chlorophyll (µg g⁻¹) |
|---|---|---|
| 1 | 12 | 45 |
| 2 | 15 | 52 |
| 3 | 9 | 38 |
| 4 | 20 | 64 |
| 5 | 14 | 50 |
| 6 | 18 | 60 |
| 7 | 11 | 42 |
| 8 | 17 | 57 |
| 9 | 13 | 48 |
| 10 | 16 | 55 |
Step 1: Exploratory Data Analysis
- Mean area: ( \bar{x} = 15.1) cm²
- Mean chlorophyll: ( \bar{y} = 52.4) µg g⁻¹
- Scatter plot shows a positive trend.
Step 2: Linear Regression
We fit ( y = \beta_0 + \beta_1 x ).
- Compute slope: [ \beta_1 = \frac{\sum (x_i - \bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{340}{84} \approx 4.05 ]
- Compute intercept: [ \beta_0 = \bar{y} - \beta_1\bar{x} = 52.4 - 4.05\times15.1 \approx 14.8 ]
Model:
( \hat{y} = 14.8 + 4.05,x )
Step 3: Model Assessment
-
R²: proportion of variance explained.
( R^2 = 0.92 ) → 92 % of chlorophyll variance explained by area. -
Residuals: small and normally distributed → model fits well.
-
Significance: p‑value for ( \beta_1 < 0.001 ) → strong evidence that area predicts chlorophyll.
Step 4: Interpretation
Leaf area explains most of the variation in chlorophyll content. For each additional cm² of leaf area, chlorophyll increases by ~4 µg g⁻¹.
4. Quick Revision Summary
| Topic | What to Remember |
|---|---|
| Data collection | Keep a clear protocol; document sources. |
| Cleaning | Check for duplicates, outliers, missing values. |
| Descriptive stats | Mean, SD, median, IQR. |
| Correlation | Pearson r, Spearman ρ, correlation matrix. |
| Regression | Equation, coefficients, R², assumptions. |
| Hypothesis testing | Null vs alternative, p‑value, significance level. |
| Validation | Train‑test split, k‑fold CV, overfitting warning. |
| Interpretation | Translate statistics to scientific conclusions. |
Tip for M.Tech. students:
Always start with a question. Your analysis should answer that question or refine it. Document every step, as reproducibility is key in high‑level research.