QuantaFlow - Production Data Engineering & Medallion Lakehouse Platform

QuantaFlow is a production-grade incremental data engineering pipeline designed to ingest, validate, clean, enrich, aggregate, and monitor multi-source data with a Medallion Architecture (Bronze, Silver, Gold).

Key Architecture Highlights

• Contract-Enforced Ingestion: Strict YAML schemas, SHA-256 manifest tracking, and idempotent ingestion.
• Medallion Lakehouse: Raw Bronze partitioned storage, Silver layer with data cleaning and Dead Letter Queue (DLQ), Gold aggregated analytics.
• Write-Audit-Publish (WAP) Circuit Breakers: Automated 6-dimension data quality audits halting publish if score is under threshold.
• Column-Level OpenLineage: Full lineage tracking compliant with OpenLineage standards.
• Infrastructure as Code: Zero-hardcoded-secret Terraform configurations for AWS S3 lakehouse, RDS PostgreSQL 16, and ECS Fargate.

Built with modern Python (3.11+), DuckDB/Polars, Ruff, and full CI/CD test automation.