Sr Data Engineer - Platform Modernization & Migration

Market insights

Stack

Free. Sign in with email.

Apply

Upload your CV — we will connect you with the employer directly through our pool.

Description

Accountabilities - Data Pipeline Migration: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, following established migration blueprints and ensuring reliable, consistent data delivery. - Workflow Modernization: Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3, including adapting, testing, and debugging DAGs to maintain scheduling reliability and operational continuity. - Data Lake Modernization: Migrate legacy Hive-based data structures and datasets to Apache Iceberg, leveraging Snowpark Connect frameworks and connectors where applicable. - Streaming Infrastructure Migration: Reconfigure and migrate Confluent and Apache Kafka streaming connections across cloud regions, ensuring seamless connectivity and minimal disruption to data flows. - Pipeline Adaptation and Optimization: Adapt existing data pipelines for Snowflake environments, optimize SQL queries, and improve processing efficiency using advanced techniques such as common table expressions (CTEs), window functions, and complex joins. - Data Processing and Orchestration: Build, maintain, test, and troubleshoot data workflows using Python, Apache Spark, PySpark, Apache Airflow, and Apache Kafka. - Testing and Quality Assurance: Develop and execute testing strategies, perform hands-on debugging, and validate data accuracy, consistency, and integrity throughout the migration lifecycle. - Performance and Reliability: Identify bottlenecks and optimize distributed data pipelines,

Full description, company and the link to apply — after you sign in. Sign in

Employer contacts (email/phone/telegram) are hidden from the public preview — send your CV, and we will connect you directly.