irahardianto / data-engineering
Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles.
0
SKILL.md2.7 KB
Data Engineering Principles
Guidelines for building reliable, scalable data pipelines and platforms.
When to Invoke
- Designing data pipelines (ETL/ELT)
- Evaluating batch vs stream processing
- Data quality and governance requirements
- Data warehouse/lake architecture decisions
Pipeline Architecture
Design Principles
- Idempotent pipelines — re-running produces same result. Use upserts, not inserts.
- Schema evolution — handle new fields without breaking consumers.
- Exactly-once processing — deduplication at ingestion, idempotency keys.
- Incremental processing — process only new/changed data, not full reloads.
Patterns
| Pattern | When to Use |
|---|---|
| Batch ETL | Scheduled, high volume, latency-tolerant |
| Streaming | Real-time, event-driven, low latency |
| Lambda | Both batch and stream (complexity trade-off) |
| Kappa | Stream-only, reprocessing via replay |
| Medallion | Bronze (raw) → Silver (cleaned) → Gold (curated) |
Data Quality
Checks (Non-Negotiable)
- Completeness — no unexpected nulls in required fields
- Uniqueness — no duplicate records on primary keys
- Referential integrity — foreign keys resolve
- Freshness — data arrives within SLA window
- Volume — row counts within expected range (±threshold)
Framework
Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)Orchestration
| Tool | Strength |
|---|---|
| Apache Airflow | Most mature, Python-native, DAG-based |
| Dagster | Type-safe, asset-oriented, modern |
| Prefect | Pythonic, flow-based, cloud-native |
Best Practices
- DAGs should be idempotent and retriable
- Separate orchestration from computation
- Use backfill capabilities for historical reprocessing
- Alert on SLA breaches, not just failures
Data Modeling
| Model | When |
|---|---|
| Star schema | Analytics, BI dashboards, simple queries |
| Data Vault | Enterprise, auditability, multiple sources |
| Dimensional | Aggregated reporting, OLAP |
Governance
- Data lineage tracked (source → transformation → destination)
- Access controls per dataset/table
- PII identified and masked/encrypted
- Retention policies documented and automated
Related
- Database Design Principles @.agents/rules/database-design-principles.md
- SQL Idioms @.agents/skills/sql-idioms/SKILL.md
- Logging Implementation @.agents/skills/logging-implementation/SKILL.md