Senior Data Engineer - Python/SQL (Public Sector)
- Experiência
- 6+ anos
- Práticas
- CI/CD, MLOps
- Formação
- Ensino superior
Role Overview The Senior Data Engineer sits within the DSAD Platform and Architecture Team and is responsible for designing, building, productionising and operating trusted data pipelines and reusable data products on the SWDA data platform using AWS and Databricks.
will provide technical leadership across data ingestion, transformation, modelling, quality, lineage, orchestration, observability and performance, applying CRISPr principles of comprehensiveness, robustness and scalability so that data remains dependable for analytics, reporting and ML/AI workloads. Key
Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks. Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations. Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source.
Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity,