Senior Data Engineer

Market insights

Stack

Apply

Upload your CV — we will connect you with the employer directly through our pool.

Description

Key Responsibilities Data Management: Extract, combine, transform, load, manipulate, and cleanse data from various sources to ensure high-quality datasets for analysis. This also includes data migration from legacy systems into next generation systems Collaboration: Work closely with data modelers and business analysts to understand the data model and source-to-staging data mapping. Data Structures: Create and integrate data structures suitable for analysis, including developing DDLs to create various MS Cloud SQL Server tables and views in Pond (a subset of the Data Lake) DB. Databricks Development: Design and develop Databricks notebooks for extracting data from various input files, joining related entities to create Shared File Formats (SFFs). Ingestion Framework: Configure the Ingestion Framework (IF) tables to utilize the existing framework for loading staging and Data Lake Core (DLC) tables. Algorithm Development: Develop and deploy algorithms in performance-efficient code across analytical and data platforms. Data Pipelines: Build large-scale batch and real-time data pipelines using data processing frameworks in cloud platforms or on-premises environments. Continuous Integration: Leverage best practices in continuous integration and delivery to enhance data workflows. Data Governance: Track data consumption patterns, collaborating with data governance and information stewards, and participate in vetting and promoting content for the curated data catalog. Automation: R

Employer contacts (email/phone/telegram) are hidden from the public preview — send your CV, and we will connect you directly.