Lead Data Engineer - Data Engineering 4C

Inviting applications for the role of Principal Consultant- Databricks Lead Developer!

In this role, the Databricks Developer is responsible for solving the real world cutting edge problem to meet both functional and non-functional requirements. Maintains close awareness of new and emerging technologies and their potential application for service offerings and products. • Work with architect and lead engineers for solutions to meet functional and non functional requirements. • Demonstrated knowledge of relevant industry trends and standards. • Demonstrate strong analytical and technical problem-solving skills. • Must have experience in Data Engineering domain .

• Bachelor's Degree or equivalency (CS, CE, CIS, IS, MIS, or engineering discipline) or equivalent work experience. • Overall 8+ years of experience in IT • Maintains close awareness of new and emerging technologies and their potential application for service offerings and products. • Work with architect and lead engineers for solutions to meet functional and non functional requirements. • Demonstrated knowledge of relevant industry trends and standards. • Demonstrate strong analytical and technical problem-solving skills. • Must have excellent coding skills either Python, Spark / Scala, preferably Python & Spark. • Must have experience in Data Engineering domain. • Must have implemented at least 2 project end-to-end in Databricks. • Must have at least experience on databricks which consists of various components as below  AutoLoader  Delta lake  Unity Catalog  Lakeflow Connect  db API 2.0  Databricks workflows orchestration • Must be well versed with the Databricks Lakehouse concept and its implementation in enterprise environments. • Must have good understanding to create complex data pipeline • Must have good knowledge of Data structure & algorithms. • Must be strong in SQL and sprak-sql. • Must have strong performance optimization skills to improve efficiency and reduce cost. • Must have worked on both Batch and streaming data pipeline. • Must have extensive knowledge of Spark and Hive data processing framework. • Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases. • Must be strong in writing unit test case and integration test • Must have strong communication skills and have worked on the team of size 15 plus • Must have great attitude towards learning new skills and upskilling the existing skills. Preferred Qualifications • Good to have Genie One, MosaicAI, DQX, and basic governance knowledge. • Good to have Databricks SQL Endpoint understanding. • Good To have CI/CD experience to build the pipeline for Databricks jobs. • Good to have if worked on migration project to build Unified data platform. • Good to have knowledge of DBT.

• Good to have knowledge of docker and Kubernetes.

Qualifications

Bachelors - Business Analytics, Bachelors - Computer Science, Bachelors - Statistics, Masters - Data Science