Our goal is building and managing multiple new high-quality data layers that contain correct, consistent high-quality data servicing business applications , based on unambiguous definitions, compliant with the GDPR legislation and that can be accessed at the right time in the right form and through the right channel. These data Layers are developed on a Big Data platform with their specific tools & technologies. The group has created a new big data platform in AWS. On top of the platform, we are building a platform that focuses on supporting data science and other data pipelines. As a capability department for this big data platform on AWS, we are responsible to provide high quality frameworks towards local entities of the group. Those local entities are using an instance of our big data platform on the cloud. The local entities are responsible for the data operations (run /maintenance) through their instance of the big data platform.
Assignment
Our local entity in Belgium has developed a capability for access layer generation to support a compliant consumption of their data. This was developed on their local on prem big data platform. The goal is to assess the current setup and architecture. Propose and implement a scenario to land this capability as a framework in our organization for our big data cloud platform. Which means:
- Take into account our design principles and guidelines and our target operating model
- Assess impact to other product and services we provide.
- Design a scalable capability and pass our internal boards for approval to assure quality, security, and compliance restrictions.
- Define an scenario and implement the capability as a framework in our organization
- Collaborate with stakeholders (architects, product managers, consumers of our frameworks, i.e., the local entities). You will be part of a task force who will be responsible for further continuous improvements, maintaining and provide support of this capability as an Internal framework.
For this assignment we need experienced profiles who can be pragmatic, are able to interact with different levels of stakeholders. They should be able to elevate the maturity of our team to bring them to higher level of efficiency and productivity.
Profile
Senior software engineer with technical design skills.
Document analysis and propose scalable secure design,
Elevate our testing strategy towards automated end to end
story testing. Review output of colleagues engineers and
pump up the confidence and quality of the output.
Needed Qualifications
AWS, Python, PySpark
