Cloud Engineer – Azure Databricks

• Design, implement, and administer the Azure infrastructure foundation for an enterprise data and AI platform
• Author and maintain reusable Terraform modules provisioning Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML
• Own remote state, module versioning, drift detection, and infrastructure deployment pipelines
• Design private networking and security architecture, including private endpoints, hub-and-spoke topology, VNet injection, NSGs, firewall rules, managed identities, RBAC, and data exfiltration controls
• Own ADLS Gen2 storage architecture, ACLs, lifecycle and tiering policies, encryption, key management, and access patterns
• Operate Azure Data Factory integration runtimes, linked-service credentials, managed identities, environment promotion, deployment automation, and monitoring
• Provision and operate Azure ML workspaces, compute clusters, GPU capacity, MLflow model registry integration, model-serving endpoints, identity, and networking
• Build and maintain observability using Azure Monitor, Log Analytics, diagnostic settings, alerting, and operational runbooks
• Own FinOps visibility and optimization for DBU and storage spending, including tagging, chargeback/showback, budget alerts, reserved capacity, and capacity planning
• Own CI/CD pipelines and promotion across development, test, and production
• Establish Databricks account and workspace administration standards, topology, settings, role delegation, and multi-workspace strategy
• Design Unity Catalog metastores, permissions, storage credentials, external locations, lineage, audit, and Delta Sharing
• Manage Entra ID integration, SCIM, identity federation, groups, entitlements, service principals, and token policies
• Define compute governance through cluster policies, instance pools, node/runtime standards, autoscaling, autotermination, Photon/serverless evaluation, and SQL warehouse configuration
• Analyze system tables, usage attribution, and DBU forecasting to identify spend concentration and remediation paths
• Review platform configurations, guide architecture decisions, publish standards and self-service patterns, and support administration and data teams
• Diagnose job failures, cluster startup issues, permission errors, connectivity faults, and performance concerns across infrastructure, Databricks configuration, and workloads
• Production ETL/ELT development, Spark transformations, model development, dimensional modeling, dbt, and BI/semantic-layer work sit with other teams

• 6+ years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure
• Expert-level, hands-on Terraform experience, including production Terraform modules, remote state, and CI/CD infrastructure as code
• Demonstrable hands-on Databricks administration experience: account and workspace administration, Unity Catalog, cluster policies, identity federation, and cost governance
• Track record as a senior technical resource to adjacent teams, guiding stakeholders and building consensus
• Deep working knowledge of Azure networking and security: private endpoints, VNets, hub-and-spoke design, NSGs, Entra ID, managed identities, RBAC, and Key Vault
• Enterprise-scale ADLS Gen2 design and access control
• Azure Data Factory platform-side experience with integration runtimes, managed VNet, credential management, and deployment automation
• Strong Python, plus PowerShell and/or Bash, for platform tooling and automation
• Working knowledge of Spark and SQL for diagnosing infrastructure and configuration-level performance issues
• Databricks Asset Bundles, Terraform Databricks provider, and workspace-as-code patterns preferred
• Unity Catalog migration or multi-workspace consolidation experience preferred
• Azure Machine Learning, MLflow, or model-serving infrastructure experience preferred
• Kubernetes/AKS and containerized workloads preferred
• Policy as code, event-driven infrastructure, formal FinOps practice, and multi-region or multi-tenant Databricks experience preferred
• Certifications such as Databricks platform credentials, Azure certifications, or HashiCorp Terraform Associate preferred
• Advanced English

• Remote work
• Completamente remoto
• Advanced English practice in a remote client environment