Job Vacancy Data Engineer

Job descriptionJob Title: Data Engineer
Grade: Band 2
Contract type: Permanent
Location: London
Salary: £61,000 per annum plus civil service pension scheme. Higher ranges may be available for exceptional candidates.
Hours: Flexible working and part time hours will be considered.

Closing date for applications is 23:59pm on Sunday the 3rd November 2024.

Nationality Requirement:
   •   UK Nationals
   •   Nationals of Commonwealth countries who have the right to work in the UK
   •   Nationals from the EU, EEA or Switzerland with (or eligible for) status under the European Union Settlement Scheme (EUSS)

Please note, we are not able to sponsor work visas or accept temporary visas as we are looking to hire on a permanent basis.

Introduction:

The Data Engineer is a role within the NAO's Digital Services (DS) function. With responsibility for the design, delivery and maintenance of robust data analytics solutions, the role looks to ensure optimal performance, scalability and reliability. They will work closely with business stakeholder, data scientists and other digital functions to understand requirements and deliver effective data solutions.

The post holder will develop and construct data products and services and integrate them into systems and business processes. While not limited to these, this role will initially focus on the further development and maintenance of:
   •   Existing Data Analytics platforms (AIMS & Data Service) that serve the organisation's two core service lines; Financial Audit and Value-for-Money.
   •   The Enterprise Data Warehouse and Business Intelligence solution for the purposes of enterprise reporting and corporate analysis.

This role reports into the Head of Data Services.

This role requires regular attendance at the NAO's office either in Victoria, London, or at the office in Newcastle.ResponsibilitiesResponsibilities of the role:
Data Architecture
   •   Construct, maintain and troubleshoot data architecture supporting analytic solutions.
   •   Support the selection of appropriate technology for our needs
   •   Ensure documentation of data meets standards (e.g. source-to-target mappings)
   •   Assist in the production of Data Analytics solution design & architecture documents.
   •   Assist in defining standards and best practices for data architecture and related technologies from an enterprise perspective.

Data Infrastructure
   •   Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.
   •   Build, maintain, and deploy data products for analytics and data science teams on cloud platforms (e.g. Azure).
   •   Support work on database management, providing input on issues and solving programming challenges.
   •   Implement data security measures and ensure compliance with industry regulations and internal policies.

Data flows/pipelines
   •   Design, build and maintain batch or real-time data pipelines in production, re-engineering manual flows and optimising code to ensure processes perform optimally.
   •   Develop ETL (extract, transform, load) processes to help extract and manipulate data from multiple sources.
   •   Write ETL/ELT scripts and code to ensure the ETL/ELT process performs optimally (T-SQL stored procedures, Azure ADF/Synapse Pipelines, Azure Databricks/Synapse Notebooks)
   •   Automate data workflows such as data ingestion, aggregation, and ETL/ELT processing.
   •   Implement data flows to connect operational systems, data for analytics and business intelligence (BI) systems
   •   Recognise opportunities to reuse existing data flows

Data Monitoring and Reporting
   •   Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures, leveraging data controls to maintain these for allocated areas of ownership.
   •   Monitor data systems performance and implement optimization strategies.
   •   Build accessible data for analysis
   •   Develop business intelligence reports that can be reused

General
   •   Explore and develop new ways of working with data
   •   Collaborate with Business Architects and Analysts to gather and analyse business requirements, objectives and desired features for the NAO BI platform.
   •   Communicating with end-users, support groups and third-party suppliers on problems & technical concerns to ensure an effective support and maintenance of the NAO's Data Analytics environments.
   •   To provide 3rd line support for fault diagnosis and fixing data analytics services incidents and problems ensuring an effective service is provided in accordance with agreed service levels.
   •   Analyse and estimate feasibility, time, and efforts for the technical aspects of a solution, update and manage work to align with expected processes.Skills requiredKey skills / competencies required
The skill sets listed also include the corresponding skill level (awareness, working, practitioner, expert):
   •   Communicating between the technical and non-technical: You can communicate effectively with technical and non-technical stakeholders, advocating for the team externally and manage differing perspectives. You can support discussions within a multi-disciplinary team, with potentially difficult dynamics. (Skill level: Working)
   •   Data analysis and synthesis: You can understand and help teams to apply a range of techniques for data profiling. You can source system analysis from a complex single source. You can bring together multiple data sources in a conformed model for analysis. (Skill level: Practitioner)
   •   Data development process: You can design, build and test complex or large-scale data products based on feeds from multiple systems, using a range of different storage technologies, access methods or both. You can create repeatable and reusable products. (Skill level: Practitioner)
   •   Data innovation: You can understand the impact on the organisation of emerging trends in data tools, analysis techniques and data usage. (Skill level: Working)
   •   Data integration design: You can deliver data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future proof. (Skill level: Working)
   •   Data modelling: You can explain the concepts and principles of data modelling, understanding industry-recognised data modelling standards and patterns. You can produce, maintain and update relevant data models for an organisation's specific needs. You can reverse-engineer data models from a live system. (Skill level: Working)
   •   Information security: You know how to maintain the security, confidentiality, and integrity of information through compliance with relevant legislation and regulations. You can design, implement, and operate controls and management strategies to allow for this. (Skill level: Practitioner)
   •   Metadata management: You can design an appropriate metadata repository and suggest changes to improve current metadata repositories. You can understand a range of tools for storing and working with metadata and advise less experienced members in the team about metadata management. (Skill level: Practitioner)
   •   Problem resolution (data): You can respond to problems in databases, data processes, data products and services as they occur. You can initiate actions, monitor services and identify trends to resolve problems. (Skill level: Working)
   •   Programming and build (data engineering): You can use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. You can collaborate with others to review specifications where appropriate. (Skill level: Practitioner)
   •   Technical understanding: You can understand the core technical concepts related to the role and apply them with guidance. (Skill level: Working)
   •   Testing: You can review requirements and specifications and define test conditions across appropriate testing types (e.g. technical, integration, accessibility). You can identify issues and risks associated with work. You can analyse and report test activities and results. (Skill level: Working)
   •   Turning business problems into data design: You can design data architecture that deals with problems spanning different business areas, identifying links between problems to devise common solutions. You can work across multiple subject areas, or a single large or complex subject area, producing appropriate patterns. (Skill level: Practitioner)Educational requirementsTechnical Requirements:

Technologies - Experience using MS Teams, Azure DevOps, Azure-Based Data Analytics technologies (see below), Power BI, Office 365, ServiceNow.

Technical Knowledge: Developing, designing, and prototyping, testing, training and implementing practical business solutions using the following technologies:
   •   Azure data analytics stack including products such as Azure Data Factory, Synapse Analytics, Azure Databricks, Azure Analysis Services and Azure SQL DB
   •   Fabric-compatible technologies, including Serverless SQL, Synapse Pipelines, and Semantic Models
   •   SQL-based Data Warehouses, associated Analysis Services OLAP Cubes & Tabular Models.
   •   Functional knowledge & experience of ITSM processes.
   •   Azure Cloud Infrastructure: Azure Portal, ARM, Terraform.