| Description: |
Location: Toronto, ON
Work Arrangement: Onsite / 4 Days in Office
Salary: $150K–$180K
Experience: 12–18 Years
We are looking for a Principal Data Engineer – Databricks to lead enterprise-scale data modernization initiatives, with a strong focus on Cloudera-to-Databricks migration, Lakehouse architecture, finance/risk data, and large-scale data engineering.
Key Responsibilities & Requirements
- 12–18 years of overall data engineering experience.
- 8+ years of experience with enterprise Data Warehouse and Data Lake platforms.
- 5+ years of hands-on experience with Databricks and Spark at scale.
- Lead modernization of legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
- Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
- Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
- Design and govern enterprise Data Warehouse, Data Lake, and Lakehouse architectures.
- Implement Medallion Architecture (Bronze, Silver, Gold) and layered data platforms.
- Develop and optimize large-scale pipelines using PySpark, Spark SQL, Delta Lake, and Databricks SQL.
- Optimize workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
- Build semantic and consumption layers for BI, reporting, and analytics.
- Work with finance and risk data models, including General Ledger, Sub-ledger, financial hierarchies, and credit/liquidity/market risk exposure models.
- Support reporting capabilities including aggregation, drill-down, and drill-back.
- Implement data governance, data quality, reconciliation, lineage, metadata, security, and compliance controls.
- Work with AWS or Azure cloud platforms.
- Implement CI/CD using Git, Terraform, Jenkins, or Azure DevOps.
- Use orchestration tools such as Airflow or Databricks Workflows.
- Experience with dbt is an asset.
- Act as a technical authority, lead architecture decisions, mentor senior engineers, and establish engineering standards.
- Partner with finance, risk, analytics, and governance stakeholders to deliver business-ready data solutions.
Nice to Have
- Experience in BFSI, Capital Markets, or regulatory reporting.
- Exposure to SAP Finance, Oracle Financials, or S/4HANA.
- Experience supporting AI/ML workloads.
- Databricks or cloud certifications.
Impact
- Lead Cloudera-to-Databricks transformation initiatives.
- Shape enterprise finance and risk data platforms.
- Support regulatory, management, and analytical reporting systems.
|
