About the role
Job Summary
Seeking an experienced Specialist with 5 to 7 years of expertise in Databricks on AWS to design optimize and support cloud data solutions
Job Description
Minimum 5 to 7 years of IT experience with significant handson expertise in Databricks on AWS Proficient in leveraging Databricks latest features such as Delta Lake and Unity Catalog for scalable data processing Strong understanding of AWS cloud services including S3 Lambda SNS Step Functions and Glue Skilled in job cost optimization techniques to minimize runtimes and improve performance Experience working with Python core libraries and PySpark for data engineering tasks Good analytical and logical skills in writing SQL queries stored procedures and creating martsviews in RDBMS Knowledge of ETL concepts and endtoend data pipeline architecture from source systems to data warehouse and reporting layers Familiarity with CICD pipelines for code deployment using tools like Terraform Experience with version control systems such as GitHub or equivalents Understanding of Agile methodology and software development lifecycle Excellent communication and presentation skills to collaborate effectively with stakeholders
Roles and Responsibilities
Collaborate with end users to understand challenges and provide effective Databricksbased solutions Research and evaluate new Databricks features conduct proof of concepts POCs and develop best practice documentation Provide instance support for Sparkrelated issues and participate in code reviews with recommendations Lead and mentor junior team members guiding them in development and troubleshooting activities Manage midsized teams and oversee work deliverables ensuring quality and timely completion Take ownership of assigned tasks and work in an agile environment promoting continuous improvement Participate actively in design and solution documentation based on requirements Drive job cost optimization and performance tuning to enhance system efficiency Support CICD processes and infrastructure automation to streamline deployments Engage in Agile ceremonies and contribute to improving team collaboration and productivity