Senior Data Engineer

• Design, develop, and maintain scalable, efficient data pipelines
• Implement data ingestion, transformation, and integration processes using batch and streaming architectures
• Develop and optimize solutions using Cloudera, Databricks, and AWS services
• Create data processing and transformation routines using Python, SQL, and PySpark
• Develop and maintain ETL/ELT processes
• Work with Data Lake, Lakehouse, and distributed processing architectures
• Integrate diverse data sources while ensuring information quality, consistency, and reliability
• Monitor and optimize the performance of pipelines, jobs, and queries
• Apply best practices in data security, governance, cataloging, and lineage
• Contribute to the evolution of the data architecture and the continuous improvement of processes
• Collaborate with architecture, analytics, data science, and business teams

• Solid experience as a Senior Data Engineer
• Knowledge of Cloudera and Databricks, with experience processing and transforming large volumes of data
• Knowledge of AWS (Amazon Web Services) and its core services applied to data engineering
• Advanced SQL skills and experience with Python and PySpark
• Experience with Apache Spark and distributed processing
• Knowledge of ETL/ELT processes and tools
• Experience with Data Lake and Lakehouse architectures
• Knowledge of data modeling and integrating diverse information sources
• Experience with version control using Git
• Knowledge of data pipeline monitoring, troubleshooting, and optimization
• Experience with AWS data services such as Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda, and Amazon Athena
• Knowledge of data processing and orchestration services in cloud environments
• Experience with Delta Lake and Hadoop ecosystem technologies such as Hive, HDFS, and Impala
• Knowledge of Apache Kafka and real-time data processing
• Experience with Apache Airflow or similar orchestration tools
• Knowledge of CI/CD practices, infrastructure as code, and automation
• Experience with data governance, quality, security, and observability
• Experience working on data platform migration, modernization, and integration projects

• This position is also open to people with disabilities (PwD)