Senior Data Engineer/Data Engineer
- Erfahrung
- ab 10 Jahren
Job Description This role sits within NCS AI Central's (AIC) Forward Deployed Engineering (FDE) model — the combined capability that takes AI solutions from proof-of-concept through to hardened production systems. You will operate across both fast-moving FDE engagements (POC/POV, pilot deployments for strategic and lighthouse clients) and steady-state system development and maintenance work — bringing the same rigor and a reusable, asset-fed approach to both. What will you do: 1.
Data Pipeline Engineering & AI-Readiness • Design and build ingestion, cleaning, and transformation pipelines that turn messy, real-world client data into AI-ready datasets. • Build batch and streaming pipelines (Airflow/Prefect/Kafka) that keep data flowing reliably into AI systems without manual intervention. • Own data quality — deduplication, schema validation, completeness checks — upstream of any model or RAG pipeline.
• Proactively flag data gaps or quality issues that would degrade model/RAG performance downstream, before they surface as an AI Engineer's problem in testing. 2. RAG & Vector Store Architecture • Architect document/data ingestion and indexing pipelines for Retrieval-Augmented Generation (RAG) systems — chunking strategy, embeddings, hybrid/vector search.
• Design and operate vector database and search infrastructure (pgvector/Pinecone/OpenSearch) at production scale and query volume. 3. Data Governance & Compliance • Implement PII redaction, data residency, and access-control pa