Senior Data Engineer - Azure Databricks
- Hong Kong Island, Hong Kong
- Permanent
- Full-time
- Design, build, and optimize scalable end-to-end data pipelines for structured and unstructured data from diverse sources (APIs, databases, files).
- Develop and manage robust ETL/ELT workflows for both batch and real-time data processing, ensuring high data quality and reliability.
- Monitor and optimize pipeline performance using autoscaling, caching, and cost-effective strategies.
- Leverage Databricks and Apache Spark (PySpark, Spark SQL) for distributed data processing and transformation.
- Implement data storage solutions using Medallion Architecture and Delta Lake, ensuring schema evolution and ACID compliance.
- Collaborate with data scientists, analysts, engineers, and business stakeholders to align data solutions with business goals.
- Stay current with emerging technologies in data engineering, especially within the Azure and Databricks ecosystem.
- Bachelor’s degree in Computer Science, Information Systems, or a related field.
- 5–10 years of hands-on experience in data engineering, with a strong track record in building and optimizing data pipelines.
- Proven expertise in Databricks, Apache Spark (PySpark, Spark SQL), Python, and SQL.
- Experience integrating Databricks with Azure services (Data Lake Storage, Data Factory, SQL Database, Event Hubs).
- Familiarity with both batch and real-time data processing tools such as Azure Event Hubs and Kafka.
- Proficient in Git and Azure DevOps for CI/CD and source control, including Databricks Asset Bundles.
- Strong understanding of data quality, monitoring, and performance optimization.
- Excellent documentation, collaboration, and communication skills.
- Mature, independent, and customer-oriented with strong time management and problem-solving abilities.