Senior AI Data Engineer

Experteer Overview

You’ll shape data strategy within a high-impact R&D AI Innovation Program, ensuring reliability, governance, and cost-efficient cloud platforms. This role blends hands-on engineering with strategic data leadership to accelerate AI research and deployment.

• Design, develop, and maintain scalable data pipelines and ETL processes for AI R&D
• Build scalable data models (star schemas, feature-ready datasets, semantic layers) for analytics and ML training
• Collaborate with AI scientists to capture data requirements and ensure data quality
• Implement data governance and security measures to protect sensitive information
• Establish observability, lineage tracking, and monitoring to detect anomalies and freshness issues
• Apply data partitioning, indexing, and storage optimization for large AI datasets
• Monitor and troubleshoot data pipelines to ensure reliability
• Stay current with emerging data engineering and AI technologies
• Drive data platform reliability, scalability, and cost optimization across cloud infrastructure
• Design scalable architectures for AI agent training, fine-tuning, and inference workflows
• Develop streaming and event-driven pipelines for real-time agent telemetry and feedback
• Create high-performance pipelines using modern orchestration frameworks (e.g., Airflow)
• Store and retrieve vector embeddings, knowledge graphs, and symbolic reasoning data
• Automate data validation, schema testing, and quality checks for training datasets
• Ensure monitoring and governance for high-quality, privacy-compliant data

• 5+ years in data engineering with at least 2 years in ML/AI data infrastructure
• Advanced programming in Python and Scala; knowledge of Rust/Go/Java/Julia valued
• Expert SQL and NoSQL experience
• Hands-on with vector databases (Pinecone, Weaviate, Milvus)
• Proficiency with Airflow 2.x and cloud platforms (AWS/Azure/GCP)
• Experience with Docker and Kubernetes; Terraform for IaC
• Experience with distributed frameworks (Spark, Dask, Ray) and streaming (Kafka, Flink)
• Knowledge of data lakehouse architectures
• Experience mentoring and contributing to architectural decisions

•