What You’ll Be Doing

Design, Build and maintain data pipelines for ingesting data from external APIs, using Azure Data Factory

Transform ingested raw JSONL data into structured formats using Python notebooks (pandas/dataframes) publishing to Databricks DeltaTables across a medallion architecture

Design and manage dimension tables (SCD2) and development of surrogate keys for linking common external datasets for pipeline redundancy

Ensure data quality through testing and validation of transformation logic.

Collaborate with data scientists and developers to support ML model development through engineering of features

Work within the Azure ecosystem, including, but not limited to, Data Factory, Databricks, ADLSv2, Azure Queues, Function Apps and CosmosDB

Use GitHub for version control and collaborate in an Agile environment using Jira.

What You Bring

You’re a problem-solver who enjoys working with data at scale, writing clean Python code, and building systems that are reliable and scalable.

Must-haves:

Strong hands-on Data Architecture and Design

Strong Python skills, especially for data transformation using pandas/dataframes.

Familiarity with the Databricks ecosystem, including Unity Catalog, Delta Tables, and Medallion architecture.

Experience with Azure Data Factory and cloud-based data ingestion.

Experience designing and maintaining dimension tables (SCD2).

Ability to develop and manage surrogate keys for data linking.

Experience testing and validating data pipelines to ensure accuracy and integrity.

Ability to conceptualize and design solutions to complex data transformation problems.

Nice-to-haves:

Interest in sports (NBA, AFL, NRL, NFL, soccer) and understanding of sports statistics.

Experience using GitHub for code management and collaboration.