YOUR NEXT CHAPTER
Senior Data Engineer
About the role
• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake
• Design and implement Bronze, Silver, and Gold medallion architecture layers
• Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
• Develop reusable ingestion frameworks for onboarding new data sources
• Build batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution, and error recovery
• Configure and manage Databricks Workflows and implement pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
• Optimize Spark and Delta Lake workloads, query performance, partitioning, joins, data-skew handling, and compute costs
• Implement Unity Catalog governance, access controls, lineage, auditing, masking, and row-level security
• Maintain data workflows on Amazon Redshift and Amazon EC2
• Collaborate with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues
• Follow version control, code review, automated testing, CI/CD, and technical documentation practices
• Partner with Data Analytics and business stakeholders to deliver trusted, documented, analytics-ready datasets
• Provide technical guidance and improve automation, engineering standards, platform reliability, and resource utilization
• Report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders
• 5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines
• Strong hands-on expertise in Databricks, including developing, troubleshooting, and optimizing production workloads
• Strong proficiency in Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation
• Practical experience with Delta Lake, including table management, incremental processing, and performance optimization
• Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets
• Experience with Databricks Workflows or equivalent orchestration solutions
• Proven experience building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications
• Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills, and failure recovery
• Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms
• Experience with Git/GitHub, code reviews, automated testing, and CI/CD practices
• Strong problem-solving skills and the ability to independently design, build, deploy, and maintain production data solutions
• Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders
• Experience with Databricks Lakeflow Spark Declarative Pipelines, Auto Loader, and advanced incremental ingestion patterns is desirable
• Experience implementing Unity Catalog governance is desirable
• Experience with Databricks Genie, Genie Agents, and Genie Code is desirable
• Experience in AI/ML, Data Science, LLMs, contextualization, and Amazon Bedrock is desirable
• Remote work arrangement
• AI-powered Talent Matching tool used in recruitment
• Privacy and data protection compliance information provided