Databricks Lead Engineer - Hybrid - Contract
Anson Mccade · London, England ·
Job Description A hands-on Databricks Lead Engineer is required to lead the delivery of a modern enterprise data platform for a leading financial services organisation. The role will take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers.
Responsibilities will include implementing Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns while mentoring engineers and driving production-quality delivery.
Whats on offer Up to £575 per dayInside IR35Initial 6-month contract with potential extensionHybrid working with three days per week on-site in LondonASAP startTechnical leadership of a strategic Databricks implementationOpportunity to shape engineering standards, reusable patterns and delivery practicesWhat you need At least two years of recent, hands-on Databricks experienceEvidence of delivering Databricks solutions into production environmentsAn active Databricks Data Engineering, Machine Learning or Generative AI certificationStrong experience implementing Medallion Architecture across Bronze, Silver and Gold layersHands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharingExperience developing Declarative Pipelines with Expectations, including failure handling and observabilityKnowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patternsProduction-level PySpark and SQL development skillsStrong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUMExperience with Databricks Workflows, orchestration and CI/CDUnderstanding of monitoring, alerting, replay, backfill and operational support strategiesAbility to mentor engineers and act as the technical escalation pointExperience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficialExposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous