Senior Consultant (Databricks Engineer)

Job Description

·      The Data Engineer will be the backbone of our data-driven ecosystem, responsible for designing, developing, and maintaining scalable, reliable data pipelines on Databricks and leading cloud platforms. 

·      You will bridge the gap between raw data sources and actionable insights by integrating diverse data sets, ensuring pristine data quality, and powering analytics, reporting, and machine learning workloads. 

·      You will work at the intersection of Analytics, Product, and Infrastructure, collaborating with cross-functional teams to elevate our data platform while championing best practices for governance, monitoring, and system reliability. 

What You Will Do 

·      Pipeline Engineering & Development

·      Develop and maintain robust ETL/ELT pipelines for centralized storage solutions (e.g., Delta Lake)

·       Integrate data from a variety of sources: relational databases, REST APIs, log files, streaming platforms, and external vendors. 

·      Build sophisticated transformation routines to cleanse, normalize, aggregate, and enrich raw datasets. 

·      Apply advanced data processing techniques to handle complex, nested, or inconsistent data structures. 

·      Architecture & Governance Contribute to internal frameworks and best practices for code development, versioning, and deployment. 

·      Implement robust data governance policies (access control, lineage, retention) aligned with enterprise standards. 

·      Partner with infrastructure leaders to advance our cloud-native data platforms (Azure, AWS). 

·      Explore and pilot new tools and technologies leveraging Azure, Databricks, and related ecosystems. 

·      Analytics & Business Collaboration Partner with Analytics and Product leaders to translate business requirements into operationalized pipelines. 

·      Attend requirement grooming, refinement, and sprint planning sessions with end-users. 

·      Develop dashboards, reports, scorecards, and data visualizations to drive business intelligence. 

·      Perform rigorous SIT, data profiling, and data validation to confirm accuracy and integrity. 

·      Monitoring & Reliability Monitor production pipelines to detect, diagnose, and resolve issues promptly. 

·      Develop monitoring dashboards, alerting systems, and automated error-handling mechanisms. 

·      Optimize performance, batch scheduling, and resource utilization across the data stack.

·      Validate the completeness and consistency of ETL loads during UAT and production rollouts. 

Qualifications & Required skills 

·      3+ years of hands-on experience in data engineering, building large-scale, high-performance data pipelines.

·       Strong experience designing data solutions, including data modeling, normalization, and distributed computing architectures. 

·      Extensive hands-on coding with PySpark, Spark SQL, and Databricks Notebooks/Jobs. 

·      Proficiency in orchestrating pipelines using Azure Data Factory (ADF), Apache Airflow, or similar schedulers. 

·      Proven experience with both real-time (streaming) and batch processing paradigms. 

·      Solid experience building pipelines on Azure (with AWS knowledge being a significant plus). 

·      High-level proficiency in SQL, including window functions, CTEs, and performance tuning. 

·      Strong understanding of DevOps tools, Git workflows, and CI/CD pipelines. 

·      Familiarity with Scrum methodology and practical experience working within cross-functional Scrum teams. 

·      Excellent problem-solving skills and a collaborative mindset. 

·       Hands-on experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis. 

·      Proven ability to design and implement real-time data processing pipelines. 

·      Databricks Certified Data Engineer Associate (preferred). 

·      Databricks Certified Data Engineer Professional (highly preferred).