← Todos los empleosDATA ENGINEER (Scala/Spark)
Detalles
- Prácticas
- Agile, CI/CD
Postular
Abre la empresa y el enlace de arriba: tu candidatura va directamente al sistema de selección de la empresa, sin intermediarios.
Descripción
Hybrid work mode in Madrid.
The main responsibilities are:
- Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service.
- Implement data transformation and quality to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations.
- Set up CI/CD pipelines to automate deployments, unit testing and development management.
- Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.
- Automate data pipelines and streamline data ingestion through the implementation of different orchestrators and scheduling processes (Airflow as a Service mainly).
- Writing technical documentation (specifications, operational documents) to ensure knowledge capitalization.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Foster a culture of continuous learning and improvement within the team.
Good knowledge of:
- Spark on Scala
- CI/CD tools (Gitlab, Jenkins…)
- HDFS and structured databases (SQL)
Apache Airflow
S3 storage/COS and parquet (and ORC) format
Strong knowledge of SQL and NoSQL databases
Oozie
- Shell script
- Software Development Lifecycle (SDLC) awareness
- Agile principles and ceremonies
- Design effective prompts to leverage Gen AI tools across IT domains (e.g., development, testing, data generation, document
Descripción completa, empresa y enlace para postular: después de iniciar sesión.
Iniciar sesiónLos contactos del empleador (correo/teléfono/telegram) están ocultos en la vista previa pública.