Lead Data Engineer

Overview

In this Lead Data Engineer role, you will design, build, and operate a cloud-native data platform powering analytics, regulatory reporting, and data-driven applications. You’ll deliver reliable, scalable, observable, and secure data solutions while partnering with product, analytics, and engineering teams to translate business needs into robust designs. You’ll advance engineering excellence through mentoring, standards, and technical direction. This position offers opportunities to influence platform standards and grow technical and leadership impact.

Responsibilities

• Design scalable data processing and quality frameworks using Python, PySpark, and dbt
• Build and optimize batch and streaming pipelines for performance, fault tolerance, and observability
• Develop and operate workflow orchestration (Airflow) to schedule and monitor data movement
• Model and transform data with SQL and dbt to support BI and reporting
• Write production-grade Python/PySpark with testing and maintainable design
• Implement infrastructure-as-code (Terraform) to provision cloud components
• Containerize and deploy services with Docker and Kubernetes (and Helm)
• Collaborate with analysts, data scientists, and apps teams to turn requirements into designs
• Own critical data systems to improve reliability, scalability, security, and operations
• Mentor junior engineers and influence technical direction through reviews and knowledge sharing

Key requirements

• Degree in Computer Science or a STEM-related field (or equivalent)
• 8 years of hands-on data engineering, coding in production
• Strong software engineering fundamentals (system design, data structures, OOP, testing, end-to-end lifecycle)
• Strong Python skills with unit and integration testing
• Experience building and operating cloud-based data platforms (AWS, GCP, or Azure)
• Experience with large-scale distributed data processing and performance tuning
• Hands-on experience with modern data warehousing/lakehouse tech (Redshift, BigQuery, Snowflake; Spark, Flink, Trino; Iceberg, Hudi)
• Strong SQL and dbt for transformations
• Experience designing and operating Airflow-based pipelines
• Experience building streaming pipelines (Kafka, Pub/Sub)
• collaborative mindset
• mentoring and leadership
• strong communication
• Python
• PySpark
• dbt

Report this job