About the role
Key Responsibilities:
- Ab Initio Development & Optimization:
- Design, develop, test, and deploy high-performance, scalable ETL/ELT solutions using Ab Initio components (GDE, Co>Operating System, EME, Control Center).
- Translate complex business requirements and data transformation rules into efficient and maintainable Ab Initio graphs and plans.
- Optimize existing Ab Initio applications for improved performance, resource utilization, and reliability.
- Troubleshoot, debug, and resolve complex data quality and processing issues within Ab Initio graphs and systems.
- Data Modeling & Advanced SQL:
- Apply expertise in advanced SQL to write complex queries for data extraction, transformation, validation, and analysis across various relational databases (e.g., DB2, Oracle, SQL Server).
- Design and implement efficient relational data models (e.g., Star Schema, Snowflake Schema, 3NF) for data warehousing and analytics.
- Understand and apply big data modeling concepts (e.g., denormalization for performance, schema-on-read, partitioning strategies for distributed systems).
- Spark & Big Data Integration:
- Collaborate with data architects on data integration strategies in a hybrid environment, understanding how Ab Initio processes interact with or feed into big data platforms.
- Analyze and debug data flow issues that may span across traditional ETL and big data platforms (e.g., HDFS, Hive, Spark).
- Demonstrate strong foundational knowledge in Apache Spark, including understanding Spark SQL and DataFrame operations, to comprehend and potentially assist in debugging Spark-based pipelines.
- Collaboration & Documentation:
- Work effectively with business analysts, data architects, QA teams, and other developers to deliver high-quality data solutions.
- Create and maintain comprehensive technical documentation for Ab Initio graphs, data lineage, data models, and ETL processes.
- Participate in code reviews, design discussions, and contribute to best practices within the team.
Required Skills & Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.
- 5+ years of hands-on, in-depth development experience with Ab Initio GDE, Co>Operating System, and EME.
- Expert-level proficiency in SQL for complex data manipulation, analysis, and optimization across various relational databases.
- Solid understanding of relational data modeling concepts and experience designing logical and physical data models.
- Demonstrated proficiency or strong foundational knowledge in Apache Spark (Spark SQL, DataFrames) and familiarity with the broader Hadoop ecosystem (HDFS, Hive).
- Experience with Unix/Linux shell scripting.
- Strong understanding of ETL processes, data warehousing concepts, and data integration patterns.
- Excellent problem-solving, analytical, and troubleshooting skills.
Strong communication and collaboration skills, with the ability to work effectively in cross-functional teams.