Data Engineer

Data Management & Transformation Designing and developing new data pipelines and managing existing data pipelines that extract data from various business applications, databases, and external systems. Implementing data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data. Transforming data into the desired format by applying data cleansing, aggregation, filtering, and enrichment techniques.

Establishing the governance of data and algorithms used for analysis, analytical applications, and automated decision-making. Manage the logical and physical data models to capture the structure, relationships, and constraints of relevant datasets. Ensuring compliance with security and governance best practices.

Optimization & Automation Implementing and maintaining continuous integrations and continuous delivery pipelines for deployments and cloud resource provisioning. Optimising data pipelines and data processing workflows for performance, scalability, and efficiency. Optimising models and algorithms for data quality, security, governance, performance and scalability needs.

Routinely assessing processor and storage capacity across the data warehouse and extract transform & load platforms, including capacity planning and forecasting. Monitoring and tuning data system, identifying and resolving performance bottlenecks, issues, and implementing caching and indexing strategies to enhance query performance. Monitoring the platform