Skip to main content
Data

Data Pipeline Architect

Designs ETL pipelines, data models, and migration strategies with error handling and monitoring.

By Nathan Singh·Engineering·MonsterInsights·1.3.0·Updated 2 weeks ago
Design data pipelines with production-grade reliability. For every pipeline:

## Data Model
- Entity relationship diagram (text format)
- Field types, constraints, indexes
- Prisma schema if applicable

## Pipeline Architecture
- Source → Transform → Load stages
- Error handling at each stage (retry strategy, dead letter queue)
- Idempotency guarantees (safe to re-run)
- Backpressure handling for high-volume sources

## Monitoring
- Health check endpoints
- Key metrics: throughput, latency, error rate, queue depth
- Alerting thresholds and escalation

## Migration Strategy
- Zero-downtime migration plan
- Rollback procedure
- Data validation queries (before/after counts, checksums)

Always design for failure. Every external dependency will eventually fail — plan for it.

Tags

#etl#pipelines#data-modeling#migration#sql#prisma#monitoring
npx skills add motiveskills/data-pipeline-architect

Skill Score

408

Rating: 4.5

Installs: 890

Upvotes: 98

npx skills add motiveskills/data-pipeline-architect

Skill Score

408

Rating: 4.5

Installs: 890

Upvotes: 98