About the role
Responsibilities:
- AI-Driven Automation: Design and implement AI systems that automate pipeline creation, data transformation logic, and workflow orchestration, pushing the boundaries of what's possible in data engineering.
- Pipeline Development: Build scalable, high-performance ETL pipelines that handle critical financial market data with reliability and efficiency.
- Event-Driven Architecture: Design event-driven ETL systems to support both real time and batch data processing needs Intelligent Infrastructure: Experiment with and deploy AI agents, code generation tools, and LLM-powered solutions to accelerate data platform development.
- Analytics and Insights: Run analytics on large datasets to uncover insights, validate models, and propose new platform features.
- Data Modelling: Architect data structures for both analytical (OLAP) and transactional systems using appropriate database technologies.
- Cross-Team Collaboration: Partner with engineers across Bangalore and New York to maintain data integrity, security, and performance standards.
Requirements:
- A bachelor's or master's degree in computer science or a comparable subject.
- 3+ years of software engineering experience with a focus on data engineering.
- Exceptional programming skills in Python and SQL.
- Expertise with data manipulation libraries (Pandas, Polars).
- Experience building production-grade ETL pipelines using orchestration tools (Airflow, Dagster, Prefect).
- Hands-on experience with DBT and DuckDB for data transformation and querying.
- Familiarity with event-based ETL architectures for data processing.
- Deep understanding of database technologies and when to use them; experience with Snowflake, MSSQL, Postgres, or similar.
- Passion for AI: Genuine curiosity about how AI can transform data engineering; experience using AI tools in development workflows (Claude, GitHub Copilot, or similar).
- Strong problem-solving abilities and comfort handling ambiguity.
- Excellent communication skills for collaborating across distributed, cross-functional teams.
Strongly Preferred:
- Experience building resilient data pipelines on Databricks using Delta Lake, Auto Loader, and workflow orchestration.
- AI Implementation Experience: Hands-on experience with AI agentic frameworks (Claude Code, LangGraph), LLM APIs, prompt engineering, or building AI-powered automation tools.
- Exposure to financial domain analytics or experience working with hedge funds/asset managers.
- Understanding of financial data models (positions, securities, market data, risk metrics).
- Experience with cloud platforms (Azure, AWS, GCP) and cloud-native data architectures.
- Familiarity with code generation, AI-assisted development, or building developer tools.
- Familiarity with data quality frameworks and monitoring systems.
- Strategic thinking with the ability to contribute to product roadmap discussions.
- Computer science, engineering, or related degree.