Data Engineer (Azure Databricks)

Responsibilities

• Design and develop scalable data pipelines using Azure Databricks , Delta Lake, and PySpark.
• Build and optimize ETL / ELT workflows across Azure Data Lake Storage (ADLS) and other data sources.
• Implement Delta Lake for ACID transactions, versioning, and high-performance data processing.
• Integrate Databricks with Azure services such as Azure Data Factory, Azure Synapse, ADLS, Key Vault , and Event Hub.
• Develop and maintain PySpark notebooks , jobs, and workflows for batch and streaming data.
• Ensure data quality, reliability, and governance , including schema enforcement and validation.
• Monitor and optimize Databricks clusters for cost efficiency and performance .
• Implement CI / CD pipelines for Databricks workflows using Azure DevOps or GitHub Actions.
• Collaborate with data scientists, analysts, and business teams to deliver consumable datasets.
• Stay updated on Azure analytics ecosystem, Databricks features, and best practices .

Interested Candidates can share their updated resumes on sourabh.sood@carecone.com.au or can reach me on +61 251 103 879.

Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Information Technology

Industries

Information Services, Data Infrastructure and Analytics, and Technology, Information and Media

#J-18808-Ljbffr