Senior Data Engineer – AWS, PySpark, Kafka

• Design and evolve event-driven architecture on AWS, using Kafka as the backbone between transactional systems and the analytics platform
• Develop Kafka producers and consumers with PySpark (Spark Structured Streaming), managing offsets, checkpointing, idempotency, late data, and delivery semantics
• Define and advocate for the design of topics, keys, partitioning, retention, compaction, and event contracts
• Govern schemas in Schema Registry, defining compatibility and evolution without breaking consumers
• Implement CDC for transactional databases via Kafka Connect or Debezium
• Integrate sources such as custodians, distributors, and market data providers’ APIs
• Build event persistence in the S3 Data Lake using Iceberg or Delta, integrating Glue, EMR, and Redshift
• Optimize performance and costs in Kafka and Spark, including lag, rebalancing, brokers, micro-batches, skew, and spill
• Instrument observability and automated quality controls for event flows
• Ensure lineage, traceability, access controls, encryption, sensitivity-based segregation, and compliance with Brazil’s LGPD and industry regulations
• Automate provisioning and deployment using infrastructure as code and CI/CD
• Document architectural decisions (ADRs), participate in code reviews and pair programming, and help define standards
• Collaborate with Wealth Management, Sales, Risk, Compliance, and Operations
• Serve as a technical reference for mid-level and junior engineers

• 6+ years of experience in data engineering, including at least 2 years with architectural responsibility for event-driven platforms
• Proven track record of launching and operating a Kafka streaming platform in production, including incident response and on-call support
• Deep knowledge of Apache Kafka: topics, partitions, keys, consumer groups, offsets, retention, compaction, rebalancing, producer and consumer configuration, and lag troubleshooting
• Hands-on experience with producers and consumers using PySpark (Spark Structured Streaming) integrated with Kafka, including checkpointing, triggers, watermarks, and state management
• Strong experience with the AWS ecosystem: Amazon MSK, S3, Glue, EMR, Lambda, IAM, and Redshift or equivalent technologies
• Python applied to data engineering, with modular, testable, and version-controlled code, type annotations, and automated tests
• Advanced SQL, including window functions, CTEs, and execution plan analysis
• Experience with Schema Registry and Avro, Protobuf, or JSON Schema formats, including schema compatibility and evolution
• Experience with Kafka Connect or Debezium for Change Data Capture (CDC)
• Infrastructure as code: Terraform, CloudFormation, or CDK
• Git, code review, and CI/CD applied to data engineering
• Experience with open table formats such as Iceberg or Delta and Lakehouse architecture on S3
• Strong focus on numerical correctness and delivery guarantees
• Clear written communication for documenting event architecture
• Ability to independently investigate issues end to end
• Ability to serve as a technical reference through code reviews, pair programming, and mentoring
• Ability to translate business needs into sustainable event designs
• Honesty when estimating scope, risk, and timelines
• Nice to have: financial markets experience; regulatory data involving CVM, BACEN, BSM, or ANBIMA; Confluent; Kafka Streams; ksqlDB; Apache Flink; Kinesis; Airflow; Step Functions; dbt; data quality, catalog, or observability tools; and AWS or Confluent certifications

• Ongoing professional development
• Dynamic and collaborative environment
• 100% remote work
• Opportunity to work on financial markets and investment projects
• Work with AWS, Kafka, PySpark, and data engineering technologies
• Inclusive company: applications from people with disabilities are welcome