Data Lead

Job Description

Status: Valid visa with full working rights

Key Responsibilities \n

\n
• Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
\n
• Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
\n
• Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
\n
• Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
\n
• CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
\n
• Governance & Security: Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.
\n

Required Qualifications \n

\n
• Experience: 8+ years in Data Engineering, with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
\n
• Core Databricks & Spark: Advanced proficiency in PySpark and Databricks SQL.
\n
• Ingestion Expertise: Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
\n
• Cloud Platforms: Strong experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
\n
• Engineering Standards: Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
\n
• Soft Skills: Excellent stakeholder management and the ability to translate complex technical architectures into business value.
\n

Preferred Qualifications \n

\n
• Experience with Kafka, dbt, or Delta Live Tables (DLT).
\n
• Databricks Certified Data Engineer Associate/Professional credentials
\n

\n

To read about our privacy policy privacy-policy

#J-18808-Ljbffr