Data Engineer

We are seeking a highly skilled Data Engineer to join our team and play a key role in building and maintaining scalable, high-performance data platforms.

This role will focus on designing and optimizing batch and real-time data pipelines, ensuring data quality, security, and compliance while supporting business and analytical needs in a fast-paced environment.

Key Responsibilities

Design, develop, and maintain batch and streaming pipelines using Apache Spark (Structured Streaming), Scala, and Python.

Ingest, transform, and model data from multiple sources into Cosmos DB, HBase, Azure Data Lake Storage (ADLS Gen2), and Synapse Analytics.

Build and optimize real-time streaming solutions leveraging Kafka and Azure Event Hub (AEH).

Implement and manage CI / CD pipelines with tools such as Jenkins, GitHub / Bitbucket, Maven, and Docker / Kubernetes.

Orchestrate and monitor data workflows using Control-M (good to have).

Ensure data quality, security, and governance, including secure credential management with Azure Key Vault.

Collaborate closely with data scientists, analysts, and platform engineering teams to deliver reliable and compliant data solutions.

Skills & Experience

Strong programming expertise in Scala and Python.

Proven hands-on experience with Apache Spark (batch and streaming).

Practical experience with Cosmos DB, HBase, Synapse, Snowflake, and ADLS Gen2.

Strong knowledge of event-driven architectures with Kafka and Azure Event Hub.

Familiarity with CI / CD pipelines and DevOps practices using Docker, Kubernetes, Terraform, Jenkins, and Git.

Experience with job orchestration tools such as Control-M or similar.

Strong problem-solving skills and ability to thrive in dynamic, fast-paced environments (preferably in financial services or similar regulated industries).

What We Offer

Opportunity to work on cutting-edge data engineering solutions at scale.

Collaborative culture with exposure to cloud-native platforms and advanced analytics.

Competitive compensation, growth opportunities, and a chance to shape the future of data-driven decision making.

#J-

• Ljbffr