About the role
Requirements:
- 4+ years of overall IT experience, including hands-on experience with big data technologies.
- Strong hands-on experience in Python and PySpark.
- Python should be used extensively for application development, ETL (Extract, Transform, Load) processes, and Data Lake curation.
- Experience in building PySpark applications using Spark DataFrames in Python with development tools such as Jupyter Notebook and PyCharm (IDE).
- Proven experience in optimizing Spark jobs that process large-scale datasets and high-volume workloads.
- Hands-on experience with version control systems, particularly Git.
- Experience working with AWS Analytics services, including Amazon EMR, Amazon Athena, and AWS Glue.
- Experience with AWS Compute and Storage services, including AWS Lambda, Amazon EC2 and Amazon S3
- Other AWS services such as Amazon SNS.
- Working knowledge of Bash/Shell scripting is an added advantage.
- Experience in designing and building ETL pipelines to ingest, copy, cleanse, and transform data across multiple formats, including CSV, TSV, XML, and JSON.
- Experience working with various file formats, including fixed-width files, delimited files, and multi-record file formats.
- Good understanding of data warehouse concepts, including facts and dimensions, star schema, and snowflake schema.
- Experience with columnar storage formats, such as Parquet, Avro, and ORC.
- Strong knowledge of data compression techniques, including Snappy and Gzip.
- Good to have experience with at least one AWS database service: Amazon Aurora, Amazon RDS, Amazon Redshift, Amazon ElastiCache, or Amazon DynamoDB.
- Good to have experience with AI/ML.