Senior Data Engineer (AWS & Databricks) | Contract

Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks. Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers. Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.

Implement data quality checks, schema validation, reconciliation, metadata management and data lineage. Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools. Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.

Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency. Implement data security, access controls, encryption and governance in line with agency requirements. Establish engineering standards, review code and architecture, and mentor junior engineers.

Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions. Maintain technical documentation, operating procedures and knowledge-transfer materials.

Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field. 6–10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms. Strong hands-on experience in Python,