Senior Data Lead - Azure Data Factory (ADF), Databricks
HCLTech
Bangalore South, Karnataka, IndiaPosted 21 days agoDiscoveredMatch locked
Full-time
Job Summary
JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).
Key Responsibilities:
JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).
Key Responsibilities:
Data Pipelines: Build and optimize ETL/ELT pipelines using Azure Data Factory, Databricks (PySpark, SQL), and ingest data from diverse sources into ADLS and Snowflake.
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOp s
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOp
Key Responsibilities
- Lead and manage technical teams, providing guidance and support in the implementation of snowflake, azure data factory (adf), and data bricks projects.
- Design, develop, and maintain data pipelines using snowflake, azure data factory (adf), and data bricks to support business requirements.
- Work closely with stakeholders to gather requirements, identify opportunities for data analytics, and propose data driven solutions.
- Monitor and troubleshoot data pipelines, ensuring data quality, reliability, and performance.
- Stay updated with the latest trends and technologies in data management and analytics, incorporating best practices into projects.
- Collaborate with cross functional teams to integrate data solutions with existing systems and applications.
- Provide technical expertise and mentorship to team members, promoting a culture of learning and innovation.
Skill Requirements
JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).
Key Responsibilities:
Data Pipelines: Build and optimize ETL/ELT pipelines using Azure Data Factory, Databricks (PySpark, SQL), and ingest data from diverse sources into ADLS and Snowflake.
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOps
Other Requirements
1.Relevant certifications in Snowflake, Azure Data Factory (ADF), DataBricks are a plus.
Not included in the source posting: about the role, benefits.
Skills
azuresnowflakedatabricksdevopssqlci-cdpythonscalasparketl
Who can apply
The employer didn't state any visa, work authorization, citizenship or clearance requirements in this posting. Confirm with the employer before applying.
Read automatically from the employer's posting text. Always confirm with the employer — requirements can change after a job is published.