Lead Data Engineer

Sydney, Australia

$230,000 (including Super) + Bonus

About the Role:

Join a high-performing Data Team at a leading financial services institution to lead regulatory initiatives and drive innovation in data engineering, utilizing AI and automation to enhance platform efficiency.

Key Responsibilities:

• Lead the development and transformation of the regulatory reporting platform.
• Collaborate across business and technical teams to deliver complex data solutions.
• Implement and optimize data integration pipelines and data warehouse solutions.
• Oversee the design and architecture of large-scale data solutions using AWS services.
• Drive platform efficiency through automation and AI enhancements.
• Ensure data governance and compliance with internal and external regulations.
• Provide leadership and mentorship to data engineers, fostering a culture of innovation and excellence.

Required Skills and Experience:

• Proficient in Spark, Hive, Presto, and big data storage formats (Parquet, ORC, Avro).
• Experience with data pipeline orchestration tools such as Airflow.
• Hands-on experience with data integration tools (Talend, DataStage, or Informatica).
• Strong AWS experience, including EC2, S3, RDS, and EMR.
• Expertise in data modeling techniques and data architecture best practices.
• Knowledge of data lineage, technical metadata, data quality, and reconciliation processes.
• Advanced skills in Spark SQL and Python for data automation.
• Ability to tune and optimize data queries for performance.
• Strong communication and negotiation skills.
• Proven experience in leading cross-platform data solutions and complex projects.
• Ability to mentor and lead technical teams.

If you feel you possess the relevant skills to the above, please respond with most updated CV!