Lead Data Engineer - Python, Databricks

Overview

As a Lead Data Engineer, you will design, build, and maintain scalable data pipelines and architectures to support enterprise analytics. You will work with agile teams to deliver secure, reliable data solutions that align with business objectives and drive data-driven decisions. Your focus on data quality, resilience, and access will enable trusted insights across functions. This role offers the chance to shape data platforms and contribute to operational excellence within a global financial services firm.

Responsibilities

• Design and deliver scalable, secure, and reliable data pipelines using Python and Databricks
• Develop and maintain data models and architectures for high-quality analytics and reporting
• Drive root cause analysis and corrective action for data quality issues
• Implement and manage backup, recovery, and archiving strategies for data availability and resilience
• Evaluate and report on access control processes to assess data asset security
• Collaborate with cross-functional teams to define data requirements and deliver aligned solutions
• Identify opportunities to optimize data workflows and improve pipeline performance
• Contribute to a culture of diversity, opportunity, inclusion, and respect

Key requirements

• Formal training or certification on data engineering concepts and advanced applied experience
• Hands-on experience developing and maintaining data pipelines using Python
• Proficiency with Databricks for large-scale data processing and analytics
• Experience with relational and NoSQL databases
• Proficiency across the full data lifecycle (ingestion, transformation, storage, access)
• Experience implementing backup, recovery, and archiving strategies
• Strong data quality principles and experience driving root cause analysis for data issues
• team collaboration
• problem solving
• communication
• Python
• Databricks
• relational databases

Report this job