Senior Data Engineer

Аналитика рынка

Стек

Откликнуться

Загрузите резюме — мы свяжем вас с работодателем напрямую через нашу базу.

Описание

Accountabilities - Own Data Platform Architecture: Design, build, and operate a production-grade data platform from initial architecture through deployment, optimization, and ongoing maintenance. - Build Multi-Source Ingestion Pipelines: Develop concurrent ingestion systems for provider APIs, webhooks, files, and web scraping, incorporating rate limiting, error handling, and reliable processing at scale. - Develop Batch and Event-Driven Workflows: Implement scheduled pipelines, historical backfills, and dependency management using orchestration tools such as Airflow, Dagster, or Prefect. Build event-driven processing with messaging and task queue services, including Pub/Sub or Cloud Tasks. - Ensure Reliable Failure Recovery: Implement retries, dead-letter queues, controlled replay mechanisms, and idempotent upserts to recover from failures without generating duplicate records or corrupting data. - Establish Raw-to-Curated Data Architecture: Preserve original source payloads, including provider JSON, in Google Cloud Storage (GCS), then transform and standardize data using dbt. Maintain clearly defined raw, standardized, and curated data layers. - Strengthen Data Quality and Governance: Define data contracts, automated validation rules, pipeline tests, retention policies, and monitoring systems to track data freshness, processing failures, and overall reliability. - Optimize Database Performance: Design and maintain PostgreSQL schemas, queries, and incremental loading strategie

Контакты работодателя (email/phone/telegram) скрыты из публичного превью — отправьте резюме, чтобы мы связали вас напрямую.