Data Engineer: market

Vacancy

an open job that cleared the quality gate and lists at least two technologies

Observed candidate

a candidate whose stack contains this technology; an aggregate — not a single record leaves the perimeter

Matchable candidate

not counted yet

Window

jobs open at the moment the snapshot was built

Junior openings are scarce — the market expects Python Middle + advanced SQL or a switch from Backend/Analyst. The Junior → Middle jump — Airflow + one DWH + Spark basics. Senior owns an entire data domain (e.g. event pipeline). Lead Data Engineer / Head of Data Platform — runs 3-8 engineers, owns architecture decisions + DWH model + data governance. Career flow: Backend Middle/Analyst Senior → Data Engineer Middle → Senior → either Analytics Engineering (dbt focus), Platform Engineering (infra focus), or Head of Data.

Top 5: python, cloud, sql, aws, azure. Python — must (pandas, PySpark, boto3, GCP SDK). SQL — advanced must (window functions, CTEs, query optimisation, EXPLAIN ANALYZE). Airflow — industry orchestration standard; Dagster/Prefect growing in new projects. Spark (PySpark/Scala) — big data batch + streaming. Kafka — mandatory for Middle+ in event-driven. Flink — premium skill for true streaming. dbt — transform layer standard in the modern data stack. DWH: Snowflake (cloud premium), BigQuery (GCP), ClickHouse (OLAP, dominates in CIS), Redshift (AWS legacy). Storage: S3/HDFS, Delta Lake/Apache Iceberg (lakehouse). Docker+Kubernetes. Terraform/Pulumi for IaC. Senior — observability (Datadog/Grafana + Great Expectations / Soda for data quality).

Spark — industry standard batch + micro-batch streaming. PySpark the largest job market, Spark Scala — premium niche. Used by 80%+ DWH teams. Downside: micro-batch latency (seconds-minutes), not true realtime. Flink — true streaming engine with millisecond latency, exactly-once semantics out of the box. Growing segment: FinTech/AdTech/IoT/real-time ML inference. Fewer jobs, but Senior Flink Engineer premium (+15-25% over Spark). Kafka Streams (Java/Scala-only) — lightweight streaming library, not a separate cluster. Good for microservice-level stream processing, poor for heavy joins/state. Strategy: Spark first (market size), then Flink for diversification and premium pay. Kafka Streams — a niche add-on for JVM backend engineers.

Apache Airflow — industry standard, dominates 80%+ of jobs. DAGs in Python, huge operator ecosystem. Downsides: schedule-based, not event-driven natively, poor dynamic DAG generation. Dagster — modern challenger (Elementl/Astronomer-like). Asset-centric, the best DX, native testing, data lineage out of the box. Growing segment: product startups, modern data stack. Senior Dagster — premium +10-15%. Prefect — between Airflow and Dagster in philosophy. Good for hybrid workflows (batch + streaming). Fewer jobs than Airflow, but growing. Strategy: Airflow first (market necessity), then Dagster as a modern alternative. Cron + Python scripts — outdated approach, don't learn as primary.

Yes, 28% of Data Engineer jobs are full-remote or hybrid. Work is fully cloud-based (AWS/GCP/Azure DWH stacks). Startups and international SaaS — full-remote. Russian product companies — hybrid (1-3 office days) or remote after probation. Sber — more often office due to compliance (data residency). Relocant hubs for Russian-speaking Data Engineers: Dubai, Cyprus, Lisbon, Berlin, Warsaw, Tbilisi. English opens Databricks/Snowflake/dbt partners with a premium +25-40% over salary.

Backend Engineer — REST/GraphQL API, business logic, transactional DBs (PostgreSQL/MySQL). Focus: serve product requests. OLTP systems. Data Engineer — ETL/ELT pipelines, DWH modelling, streaming, batch processing of terabytes. Focus: move and transform large volumes of data. OLAP systems. ML Engineer — deploy models to production (FastAPI + ONNX/TorchServe), feature store, MLOps, model monitoring. Focus: production-ready inference. Career switch: Backend Senior → Data Engineer Middle in 4-8 months (Airflow + dbt + one DWH). Data Engineer → ML Engineer in 6-12 months (PyTorch + MLOps stack). By pay: Data Engineer ≥ Backend Senior in Moscow; ML Engineer ≈ Data Engineer Senior on average.

Roadmap: 1) Python to Middle — pandas, numpy, typing, testing. 2) SQL to advanced — window functions, CTEs, subqueries, optimisation (EXPLAIN ANALYZE), indexes. PostgreSQL + one OLAP (ClickHouse recommended). 3) Airflow — official tutorial, write 3-5 DAGs. 4) One DWH: BigQuery (free tier) or Snowflake trial. Master modelling (star/snowflake schema, SCD). 5) dbt — analytics engineering layer. dbt tutorial + pet project with jaffle-shop dataset. 6) PySpark basics — DataFrame API + one pet project with 1-10 GB of data. 7) Kafka basics — producer/consumer + one streaming pet. 8) End-to-end pet project: pipeline from API/files through Airflow + dbt into DWH, dashboard on Metabase/Superset, deployed in the cloud. Courses: Karpov.Courses "Data Engineer", OTUS "Data Engineer", Yandex.Practicum "Data Engineer", DataCamp Data Engineer Track. Books: "The Data Warehouse Toolkit" Kimball, "Fundamentals of Data Engineering" Reis. Time to Middle (Junior rare) — 12-18 months full-time after Python Middle experience.

5551 active open Data Engineer positions in the Zorky CRM sample. Geography: Russia / Ukraine / Poland. The real market is broader than the sample thanks to internal-only positions at large banks and CIS product companies. Time to close a Senior Data Engineer role — 4-8 weeks.

A Senior Data Engineer owns the full data-pipeline cycle. SQL: mastery (window functions, recursive CTEs, query optimisation, partition pruning, materialized views). Python: pandas/PySpark mastery, memory profiling, async for I/O-heavy ingestion. Distributed systems: Spark execution model (shuffle, broadcast, partitioning), Kafka exactly-once, two-phase commit, idempotency. DWH modelling: Kimball (star schema, SCD), Data Vault, lakehouse architecture (Delta Lake/Iceberg). Orchestration: Airflow DAG design (idempotent tasks, backfills, sensors), dynamic DAG generation. Streaming: Kafka design (partitions, consumer groups, exactly-once), Spark Streaming or Flink. Cloud: one of AWS/GCP/Azure at the architect level (IAM + cost optimisation + networking + storage tiers). Data quality: Great Expectations / Soda, monitoring metrics + alerts. DevOps: Docker, Kubernetes operators (Spark/Kafka on k8s), Terraform/Pulumi for IaC. Soft: code review, mentoring Middle, working with Analyst/PM on requirements. English — must for Senior+ (the data stack is predominantly EN).