Senior Software Engineer, Data & Orchestration

Overview

As Senior Software Engineer in Wayves Data and Orchestration team, you will help shape the data platform that powers autonomous driving at scale. You will evolve pipeline and orchestration infrastructure, including revamping Harness and building a new DAG-based workflow system for large vehicle fleets.

You will create tooling to make ingestion, processing, inference and evaluation across teams reliable and scalable. You will collaborate with partner teams to meet pipeline requirements while maintaining high reliability and cost efficiency.

This role offers a chance to impact foundational infrastructure used company-wide and support high-priority data runs and incident response.

Responsibilities

• Design, build and evolve scalable data pipeline and orchestration infrastructure for the next-generation data platform
• Contribute to revamping Harness and developing a new DAG-based workflow platform
• Develop tooling and libraries to simplify ingestion, processing, inference and evaluation workflows
• Ensure parallel pipeline execution with correct outputs, avoiding conflicts and prioritising resources
• Support batch inference and optimization workflows for cost-effective evaluations
• Improve platform reliability, quality and usability, including incident response aligned to SLOs
• Collaborate with partner teams to understand pipeline requirements and provide underlying infrastructure

Key requirements

• 7+ years of professional software engineering experience
• Strong software engineering fundamentals for production-grade systems
• Solid Python programming skills
• Experience with unit/integration testing, TDD and data-driven development
• Experience with distributed systems, large data volumes or high-throughput processing
• Familiarity with Apache Kafka
• Understanding of database principles and distributed data systems
• Experience with Postgres, ClickHouse or column-oriented databases
• Python
• Apache Kafka
• Postgres
• ClickHouse
• Ray
• Flyte

Report this job