Role objective:
Support the Risk & Data team by providing guidance on enhancing filesystems and optimizing data pipelines. Assess the existing infrastructure and propose strategies for accommodating upcoming processes.
Task objectives:
- Evaluate the orchestration of data pipelines.
- Assess the data and risk infrastructure for potential enhancements.
- Examine Cloud architecture and propose industry best practices.
Task steps:
- Analyze data flow to suggest filesystem improvements and data pipeline optimization.
- Assess the infrastructure and recommend necessary steps to support upcoming processes.
Task deliverables & milestones:
- Comprehensive review of the Risk & Data infrastructure.
- Recommendations for enhancing data flow, filesystems, and data pipeline orchestration.
Desired experience:
The ideal candidate should possess the following qualifications:
- Proficiency in cloud infrastructure, particularly in GCP (Google Cloud Platform).
- Strong experience in data engineering.
- Sound knowledge of Python programming.
- Familiarity with Kubernetes.
