Data Engineer
Prefer a full account with a password? Sign Up to Apply
Job Description
Main objective of the job:
• Design, develop, and maintain secure and scalable data pipelines on Azure Databricks as an intermediate data engineer, heavily leveraging Python, PySpark, and AI-assisted validation tools to automate code quality checks, compliance, and routine engineering tasks
Main attributions and responsibilities:
• Design, develop, optimize, and maintain scalable data pipelines, primarily using Python and SQL on the Azure Databricks platform.
• Pioneer and refine the integration of AI agents to automate code reviews, verify work quality, and ensure strict adherence to internal development processes.
• Implement, test, and maintain robust data quality frameworks, validation tools, and pipeline monitoring systems.
• Collaborate closely with the Data Governance team to support the implementation, adoption, and scaling of data quality concepts.
• Support internal business stakeholders by debugging pipeline issues, optimizing query performance, and ensuring high data availability.
• Champion professional software engineering standards, clean code practices, and the continuous improvement of automation within the team.
Requirements
Function: Information Technology
Experience Level: Mid-Senior Level