Senior Databricks Engineer – Lead

• Design, develop, and maintain scalable data pipelines using Azure Databricks
• Implement ETL/ELT workflows using PySpark, Spark SQL, and Python
• Optimize Spark jobs for performance, cost, and scalability
• Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
• Build and manage Delta Lake tables (ACID, time travel, schema evolution)
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)
• Develop complex queries and transformations using SQL
• Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases
• Ensure data quality, validation, and monitoring
• Follow best practices for security, access control, and governance in Azure

• 6+ years of experience in Data Engineering
• Strong hands-on experience with Azure Databricks
• Proficiency in Python for data processing
• Strong knowledge of SQL (joins, window functions, performance tuning)
• Hands-on experience with Apache Spark / PySpark
• Experience working with Delta Lake
• Knowledge of Azure Data Lake Storage (ADLS Gen2)
• Understanding of distributed computing concepts
• Experience with Git version control
• Experience with Azure Data Factory
• Exposure to CI/CD pipelines (Azure DevOps, GitHub Actions)
• Basic understanding of data modeling
• Familiarity with cloud security and RBAC in Azure
• Exposure to streaming data (Spark Structured Streaming, Event Hub, Kafka)

• Employees can work remotely