Senior Data Engineer
- Experiência
- 5+ anos
- Práticas
- CI/CD
Senior Data Platform Engineer - Spark / Flink / Kafka
Design and deliver large-scale batch and streaming data pipelines. Build reliable processing and storage flows, improve data quality and recovery, and support the operation of distributed data workloads in Kubernetes environments.
Design and develop batch and streaming pipelines using Apache Spark, Apache Flink and Apache Kafka. Implement processing logic using Java or Scala and SQL. Deploy, scale and troubleshoot containerised data workloads on Kubernetes.
Build data-quality checks and manage schema, lineage, checkpointing and recovery. Integrate S3-compatible object storage and support pipeline delivery through GitLab and CI/CD practices.
5-8 years of data engineering experience, including delivery of large-scale data systems. Strong hands-on experience designing pipelines using Apache Spark, Apache Flink and Apache Kafka. Proficiency in Java or Scala for distributed processing, together with SQL.
Experience deploying and scaling data workloads on Kubernetes. Experience with S3-compatible storage, data-quality controls, schema and lineage management, checkpointing, recovery, GitLab and CI/CD. Comparable large-scale processing frameworks may be considered where transferable distributed-data-system capability is demonstrated.
Good to have SAFe and safety-critical or mission-critical data platforms. MinIO, layered bronze/silver/gold processing, metadata preservation and stora