Senior Data Engineer

About the role

AllCloud is seeking a Senior Data Engineer to design, develop, and maintain a modern data platform built on AWS and Databricks. You will work at the intersection of data engineering, database administration, and analytics engineering, enabling data‑driven decisions for global clients.

Key responsibilities

• Develop and maintain end‑to‑end ETL/ELT pipelines using Python and PySpark on Databricks.
• Implement Medallion architecture (Bronze, Silver, Gold) with Delta Lake and manage Snowflake data layers.
• Set up and optimise Databricks SQL Warehouses, perform query tuning, and create internal visualisations for rapid data exploration.
• Leverage AWS services (S3, IAM, EC2, etc.) for data storage, compute, and secure data movement.
• Act as custodian of the data platform, handling indexing, clustering, vacuuming, and performance tuning across Delta Lake and relational warehouses.
• Design star‑schema and snowflake models in the Gold layer to support BI tools such as Power BI and Tableau.
• Manage metadata, permissions, and lineage using Unity Catalog and enforce data governance.
• Integrate automated data‑quality tests into CI/CD pipelines.

Required profile

• 5+ years of experience as a Data Engineer.
• Strong hands‑on experience with AWS ecosystem (S3, IAM, EC2) and modern cloud data warehouses, especially Snowflake.
• At least 1 year of intensive experience with the Databricks platform, including notebooks and workflows.
• Proven ability to build scalable, cost‑efficient data pipelines and optimise performance.

Required skills

• Python
• PySpark
• Databricks
• Snowflake
• Delta Lake
• AWS (S3, IAM, EC2)
• SQL
• Unity Catalog
• Power BI
• Tableau
• CI/CD
• Data modeling (Star Schema, Snowflake)
• Data governance