Data Sorting, Filtering, and Validation: Software Applications & Programming Notes for NCO (Cyber Olympiad)

Data Sorting, Filtering, and Validation

Program: NCO – Cyber Olympiad
Subject: Software Applications & Programming
Topic: Data Sorting, Filtering, and Validation

Goal: Understand how to organise data, pick the right subset, and ensure its correctness before processing or storing it.

1. Simple Explanation

Concept	What it Means	Why It Matters
Sorting	Arranging data in a specific order (ascending/descending).	Enables fast searching, easier reporting, and deterministic outputs.
Filtering	Selecting a subset of data that satisfies given criteria.	Reduces data volume, improves performance, and focuses on relevant information.
Validation	Checking that data meets rules or constraints before it is used or stored.	Prevents errors, protects data integrity, and guards against malicious input.

2. Key Points

2.1 Sorting

Algorithm	Time Complexity	Space	Typical Use‑Case
Bubble Sort	O(n²)	O(1)	Teaching; very small n
Insertion Sort	O(n²) avg, O(n) best	O(1)	Nearly‑sorted data
Selection Sort	O(n²)	O(1)	Simple implementation
Merge Sort	O(n log n)	O(n)	Stable, large data
Quick Sort	O(n log n) avg, O(n²) worst	O(log n)	In‑place, fast on average
Heap Sort	O(n log n)	O(1)	In‑place, stable? (no)

Use a stable sort when relative order of equal elements matters (e.g., sorting people by age but preserving original order for same age).

2.2 Filtering

Technique	When to Use	Typical Syntax (SQL/Python)
Selection	Choose rows based on a condition	`SELECT * FROM table WHERE age > 30;`
Projection	Choose columns	`SELECT name, salary FROM employees;`
Join	Combine related tables	`SELECT a.name, b.order_id FROM customers a JOIN orders b ON a.id = b.customer_id;`
Aggregation + Group By	Summarise data	`SELECT department, AVG(salary) FROM employees GROUP BY department;`
Set Operations	Combine results	`UNION`, `INTERSECT`, `EXCEPT`

Filters are evaluated before sorting in most query engines (SQL), but in programming you might filter first, then sort.

2.3 Validation

Validation Type	Purpose	Example
Syntax Validation	Check format (e.g., email, phone)	Regex: `^[\w.-]+@[\w.-]+\.[A-Za-z]{2,}$`
Domain Validation	Ensure value is within expected set	`if role not in ['admin', 'user', 'guest']: raise ValueError`
Range Validation	Numbers within bounds	`if age < 0 or age > 120: raise ValueError`
Cross‑Field Validation	Consistency between fields	`if start_date > end_date: raise ValueError`
Uniqueness Validation	No duplicates	`if id in seen_ids: raise ValueError`
Referential Integrity	Foreign key constraints	`if customer_id not in customers: raise ValueError`

Tip: Validate as early as possible (e.g., during input) and again before persisting.

2.4 Common Pitfalls

Forgetting to handle stable vs unstable sorts when order matters.
Applying filters after sorting may waste time on unnecessary comparisons.
Skipping null/empty checks during validation leads to runtime errors.
Relying on client‑side validation only; always enforce server‑side checks.

3. Worked Example (Python)

We’ll read a CSV file of employees, filter by department, sort by salary, and validate each row.

import csv
import re
from typing import List, Dict

# ---------- Validation helpers ----------
EMAIL_REGEX = re.compile(r'^[\w\.-]+@[\w\.-]+\.[A-Za-z]{2,}$')

def validate_row(row: Dict[str, str]) -> None:
    # 1. Required fields
    required = ['id', 'name', 'email', 'department', 'salary']
    for field in required:
        if not row.get(field):
            raise ValueError(f"Missing required field: {field}")

    # 2. Email syntax
    if not EMAIL_REGEX.match(row['email']):
        raise ValueError(f"Invalid email: {row['email']}")

    # 3. Salary numeric & positive
    try:
        salary = float(row['salary'])
        if salary <= 0:
            raise ValueError
    except ValueError:
        raise ValueError(f"Salary must be a positive number: {row['salary']}")

    # 4. Department must be known
    if row['department'] not in {'Engineering', 'HR', 'Sales', 'Marketing'}:
        raise ValueError(f"Unknown department: {row['department']}")

# ---------- Load, filter, sort ----------
def process_employees(csv_file: str, target_dept: str) -> List[Dict[str, str]]:
    employees: List[Dict[str, str]] = []

    with open(csv_file, newline='') as f:
        reader = csv.DictReader(f)
        for raw_row in reader:
            try:
                validate_row(raw_row)
            except ValueError as e:
                print(f"Row skipped: {e}")
                continue
            # Keep only target department
            if raw_row['department'] == target_dept:
                employees.append(raw_row)

    # Sort by salary descending (highest first)
    employees.sort(key=lambda r: float(r['salary']), reverse=True)
    return employees

# ---------- Example usage ----------
if __name__ == "__main