Senior Data Engineer

Markteinblicke

Stack

Kostenlos, nach der Anmeldung.

Bewerben

Laden Sie Ihren Lebenslauf hoch — Arbeitgeber und Recruiter aus verschiedenen Ländern, die Ihr Profil suchen, können Sie direkt kontaktieren.

Beschreibung

Accountabilities: - Design, develop, and optimize scalable ETL/ELT data pipelines for batch and real-time processing using Python, SQL, PySpark, Spark Structured Streaming, and Delta Live Tables. - Implement and maintain Medallion Architecture, organizing data into Bronze, Silver, and Gold layers to support data quality, performance, and analytical requirements. - Process terabyte-scale datasets using distributed computing techniques, ensuring pipeline reliability, scalability, and efficient resource utilization. - Manage Databricks environments, including workflows, clusters, Delta Lake, and related lakehouse components. - Implement Unity Catalog for metadata management, granular access controls, OAuth-based authentication, and enterprise-level data security and governance. - Develop and execute cloud migration strategies for legacy databases, including Oracle, Teradata, and SQL Server, to cloud-based data platforms and storage solutions such as Google Cloud Storage, Amazon S3, Azure Data Lake Storage, Snowflake, and Amazon Redshift. - Build and maintain CI/CD pipelines using Terraform and GitHub Actions to automate infrastructure provisioning, data pipeline deployments, testing, and monitoring. - Develop custom data processing scripts and data-driven APIs using Python and web frameworks such as Flask. - Use Docker to containerize data processing workloads and support portable, consistent deployment environments. - Evaluate emerging technologies and approaches for data-inten

Vollständige Beschreibung, Unternehmen und Bewerbungslink — nach der Anmeldung. Anmelden

Kontaktdaten des Arbeitgebers (E-Mail/Telefon/Telegram) sind in der öffentlichen Vorschau ausgeblendet. Senden Sie Ihren Lebenslauf — dann können Arbeitgeber Sie direkt kontaktieren.