About the role
Experience Required: 5+ Years
Location: Hyderabad, India
Job Type: Full-time / Permanent
Work Mode: [Onsite / Hybrid / Remote – specify as per organization policy]
Job Summary:We are looking for a skilled and experienced Data Engineer with a strong background in Hadoop ecosystem, Python programming, and PySpark. The candidate will be responsible for building and optimizing big data pipelines, transforming raw data into usable formats, and collaborating with data scientists, analysts, and engineers to deliver high-quality data solutions.
Key Responsibilities:- Design, develop, and maintain scalable data pipelines and ETL workflows using Hadoop, Python, and PySpark.
- Work with structured and unstructured data across distributed systems.
- Perform data cleansing, validation, and transformation.
- Optimize Spark jobs for performance and efficiency.
- Collaborate with stakeholders to understand business requirements and implement data engineering solutions accordingly.
- Manage large-scale data sets and ensure data quality, integrity, and availability.
- Document data flows, schemas, and pipeline logic.
- Troubleshoot issues in data pipelines and perform root cause analysis.
- Strong hands-on experience in Hadoop ecosystem (HDFS, Hive, HBase, Sqoop, etc.)
- Proficiency in Python for data processing and scripting
- Expertise in PySpark and working with Spark for large-scale data transformations
- Solid knowledge of SQL for data querying and validation
- Experience with data pipeline orchestration tools (e.g., Apache Airflow – preferred)
- Familiarity with Linux/Unix environment and shell scripting
- Understanding of data warehousing and big data architectures
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field
- 5+ years of experience in data engineering or big data development
- Strong analytical and problem-solving skills
- Excellent communication and team collaboration abilities
- Experience with cloud platforms (AWS, Azure, or GCP – especially with services like EMR, DataProc, S3, BigQuery, etc.)
- Knowledge of containerization tools like Docker and Kubernetes
- Exposure to Kafka or other messaging systems
- Familiarity with CI/CD pipelines and DevOps practices