Role Description
We are looking for an experienced Senior Data Engineer to join a growing government team and play a critical role in designing, building, and optimizing enterprise-scale data pipelines. This role is ideal for someone with deep expertise in the Microsoft Data Platform ecosystem, including SSIS, Synapse Pipelines, Spark, and Profisee for Master Data Management (MDM).

About You
The successful candidate will have a strong background in data ingestion, transformation, and processing, ensuring high performance, scalability, and data quality across complex data solutions. You will also contribute to Power BI report development within Microsoft Fabric to support data-driven decision-making.

About the Team
The Data Analytics and Insights team plays a vital role in enabling data-driven decision-making across the organization. Our team is highly collaborative, working with modern Microsoft technologies to deliver high-impact insights. We are currently transitioning to a Lakehouse architecture using Synapse Analytics, while continuing to enhance our existing SQL-based Data Warehouse.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines for ingesting data from various sources into SQL Dedicated Pools and Synapse Lakehouse.
• Optimize data transformation logic using T-SQL and Spark (Python) within Synapse Pipelines.
• Develop Spark notebooks for complex data transformations and processing in Synapse.
• Implement data quality checks and validation rules to ensure data accuracy and consistency.
• Build and deploy Synapse Pipelines for data ingestion, transformation, and loading within Azure Synapse Analytics/Microsoft Fabric.
• Collaborate with data architects to maintain data models and metadata management.
• Work closely with Power BI developers to deliver curated, performant datasets and develop reports within Microsoft Fabric.
• Implement Medallion Architecture principles (Bronze, Silver, Gold layers) within the data platform.
• Conduct code reviews and contribute to best practices.
• Monitor and troubleshoot pipeline performance and resolve data-related issues in non-production environments.

Additional Responsibilities

• Stay up to date with Microsoft data platform advancements, including Microsoft Fabric.
• Document data pipelines, data models, and other technical artifacts.
• Develop batch integration processes with Profisee for MDM, including defining matching and survivorship rules for golden record creation.
• Contribute to data governance policies and enforcement of data quality standards.

Required Experience & Skills

• 5+ years of experience in data engineering.
• Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT principles.
• Expertise in T-SQL and experience with SQL Server/Azure Synapse Dedicated SQL Pools.
• Hands-on experience with Azure Synapse Analytics, including Synapse Pipelines and Spark Pools.
• Experience using Spark (Python) for data transformation and processing.
• Familiarity with Microsoft Fabric components, including Data Factory, Synapse Data Engineering, and Power BI.
• Experience with version control (Git, Azure DevOps, or Integrated Visual Studio).
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.

Preferred Experience

• Experience with Master Data Management (MDM) tools, preferably Profisee.
• Knowledge of data governance and quality management, preferably in Microsoft Purview.
• Experience with Azure Data Lake Storage Gen2.
• Exposure to CI/CD pipelines for data engineering.
• Experience working in Agile environments.
• Proficiency in developing and deploying SSIS packages.

Certifications (Highly Regarded)

• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Fabric Data Engineer Associate
• Profisee Certifications

Consultant: Sam Baidya

keywords: Senior Data Engineer ,Azure Synapse Analytics ,ETL/ELT Pipelines ,Microsoft Fabric ,Spark (Python) & T-SQL ,Master Data Management (MDM) – Profisee ,Power BI & Data Governance