Lead Data Engineer

Our client is a public services provider. Lead Data Engineer Hybrid – Attached to our Solihull, London or Hook office Full Time, Permanent Band 3 / £75,000 - £85,000 (dependent on experience) Here at our client, we're looking for a Lead Data Engineer to act as the technical authority for data engineering, driving architecture, standards, and delivery across a modern multi-cloud ecosystem spanning Azure, AWS, and Databricks.

You'll spend around 80% of your time hands-on, designing and building complex, enterprise-scale data solutions. The remaining 20% is just as critical: leading people, influencing stakeholders, and shaping how data engineering operates across the organisation.

Databricks sits at the heart of our platform - this role requires someone who can deeply leverage the Databricks to enable scalable analytics, data science, and AI use cases across the business. As part of this you'll: Define and enforce data engineering standards, patterns, and best practices across the organisation.

Architect and deliver scalable, resilient data solutions across Azure, AWS, and Databricks, ensuring enterprise-grade performance and reliability. Lead complex engineering workstreams - unblocking teams, guiding technical decisions, and ensuring high-quality delivery outcomes.

Design and govern enterprise data models (conceptual, logical, and physical), including lakehouse zone strategies and schema evolution approaches. Champion Databricks as the core data platform, leveraging lakehouse architecture, Unity Catalog, Delta Live Tables, Structured Streaming, and MLflow.

Drive a consistent multi-cloud Databricks strategy, enabling scalable and interoperable data platforms across environments. Optimise performance across pipelines, compute, storage, and orchestration frameworks, embedding cost-efficiency and scalability.

Define and implement CI/CD for data platforms, supporting automated testing, secure deployments, and multi-environment promotion. Embed robust data governance, security, and lifecycle controls, ensuring compliance through classification, encryption, access management, and lineage frameworks (e.g., Unity Catalog, Purview, Lake Formation).

Act as a senior SME for troubleshooting complex issues, leading root-cause analysis and long-term remediation of data quality and platform challenges. Lead, mentor, and grow a community of Data Engineers, driving technical excellence through coaching, reviews, and knowledge sharing.

Operate effectively across a matrix organisation, influencing architecture, cyber, product, and senior stakeholders. Communicate complex technical concepts clearly, articulating trade-offs, risks, and strategic decisions to both technical and non-technical audiences. What you'll need...

Report this job