We’re partnering with a large, well-established enterprise organisation that is seeking a Lead Data Engineer to support a high-profile security insights platform. This role offers long-term stability, a strong team culture, and the opportunity to lead the development of secure, scalable data pipelines.

About the role

As a Lead Data Engineer, you will be responsible for designing, building, and maintaining robust data pipelines using PySpark. You'll be working within a collaborative engineering environment focused on data transformation, ingestion, and performance optimisation. You’ll also play a key role in governance and stakeholder engagement, acting as a technical leader within the team.

What you’ll be doing

• Designing and developing scalable data pipelines using PySpark and Python
• Ingesting data from various sources including REST APIs, SFTP, SQL, and flat files
• Working within a Linux environment using tools like Bash and managing network components such as proxies and firewalls
• Collaborating with teams to optimise CI/CD workflows and improve data platform operations
• Leading technical discussions and providing direction across key data initiatives
• Supporting internal stakeholders around data governance, access control, and quality management
• Tuning performance through techniques like partitioning, pagination, and broadcasting

What we’re looking for

• Proven experience with Apache Spark, ideally using PySpark
• Solid understanding of data ingestion, transformation, and distributed computing
• Experience working in Agile teams and familiarity with tools like Jira and ServiceNow
• Ability to manage and support data services within large, enterprise-scale environments
• Experience working as a senior or lead engineer, with strong stakeholder engagement skills
• Exposure to platforms like Databricks, Snowflake, or Cloudera is a bonus
• Familiarity with data governance, quality, and master data management is highly regarded