Role Overview
We are looking for a highly skilled Databricks Expert to design, build, and optimize scalable data pipelines and analytics solutions. The ideal candidate should have deep expertise in Databricks, Spark, and cloud data platforms, with strong experience in building production-grade data solutions.
Key Responsibilities
- Design and implement end-to-end data pipelines using Databricks
- Develop and optimize ETL/ELT workflows using PySpark / Spark SQL
- Work with Delta Lake for data reliability, versioning, and performance
- Build and maintain Medallion Architecture (Bronze, Silver, Gold layers)
- Optimize performance of large-scale data processing jobs
- Integrate data from multiple sources (APIs, databases, files, streaming)
- Implement data quality, governance, and monitoring frameworks
- Collaborate with business teams for data modeling and analytics use cases
- Ensure best practices in CI/CD, code versioning, and deployment
Required Skills & Experience
🔸 Core Skills:
- Strong experience with Databricks (6+ years preferred)
- Expertise in Apache Spark (PySpark / Scala)
- Hands-on experience with Delta Lake
- Strong SQL skills (advanced querying, optimization)
🔸 Cloud Platforms (any one required):
- AWS (S3, Glue, Redshift)
- Azure (ADLS, Synapse, ADF)
- GCP (BigQuery, Cloud Storage)
🔸 Data Engineering:
- Experience in building scalable ETL pipelines
- Knowledge of data modeling (Star/Snowflake schema)
- Experience with batch and streaming data processing
Good to Have
- Experience with Airflow / orchestration tools
- Knowledge of data governance & security
- Exposure to BI tools (Power BI / Tableau)
- Experience with real-time data pipelines (Kafka, Structured Streaming)
🔹 Soft Skills
- Strong problem-solving and analytical thinking
- Ability to work independently in a fast-paced environment
- Good communication skills for stakeholder interaction
🔹 Contract Details
- Duration: 6 months (possible extension)
- Engagement: Contract / Freelance
- Immediate or short notice candidates preferred
🔹 Bonus (For Expert-Level Candidates)
- Experience in performance tuning at scale (TB/PB data)
- Designing data lakehouse architecture
- Mentoring junior data engineers