Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35

Staff Data Engineer | Contract | Up to £1,100/day | Inside IR35 Location: Hybrid – London (3 days per week on-site) Rate: Up to £1,100/day Contract: Initial 6 months IR35: Inside IR35 We're supporting a major UK financial services organisation through a significant programme of data engineering modernisation and transformation .

We're looking for a genuinely Staff-level Data Engineer who combines exceptional hands-on Python/PySpark engineering with strong architectural thinking and practical experience of AI-assisted software development. This is not a hands-off architecture role.

You need to be equally comfortable setting technical direction, challenging engineering approaches and independently writing, debugging and optimising production-quality code. The sweet spot is someone who can operate at a strategic level while remaining an outstanding hands-on engineer.

The roleYou'll play a key role in modernising a large-scale enterprise data estate, including the migration and transformation of thousands of data pipelines, development of new data products and adoption of modern, AI-assisted engineering practices.

Responsibilities will include: Providing Staff-level technical leadership across complex data engineering and modernisation initiatives. Remaining deeply hands-on with Python, PySpark and Apache Spark , independently building and proving engineering solutions.

Designing and optimising scalable data pipelines and processing frameworks across AWS . Tackling complex Spark performance challenges , including shuffles, joins, data skew, partitioning and memory optimisation. Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies.

Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality. Establishing reusable engineering patterns, frameworks and approaches to accelerate large-scale data migrations.

Applying AI-assisted development tools to improve engineering productivity, code quality and delivery speed. Challenging technical assumptions, influencing architectural decisions and raising engineering standards across multiple teams.

What we're looking for Core engineering capability Exceptional hands-on Python/PySpark and Apache Spark expertise, with the ability to independently write, debug and optimise complex production code. Deep understanding of Spark internals and performance optimisation , including execution plans, joins, shuffles, partitioning,...

Report this job