Data Engineer

6 months

£500 - £550 inside IR35

London - Hybrid 1-2 days in the office

On of the UKs top universities is looking for a skilled Data Engineer to join them on a contract basis.

To be considered for the role you will have strong knowledge of Azure services such as Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure SQL Database, Azure Stream Analytics, etc.

Skills:

• Expertise in Azure services: Azure Data Lake Storage, Data Factory, Synapse Analytics, Databricks, SQL Database, Stream Analytics, etc.
• Proficient in Python or Scala with Spark/PySpark experience.
• Skilled in working with both relational and NoSQL databases.
• Extensive experience in creating data pipelines, understanding design principles, data modeling, ETL procedures, and the entire data production process.
• Familiar with unit/integration testing and data quality frameworks such as Deequ, Great Expectations, or Delta Live.
• Knowledgeable in data modeling (medallion and Kimball architectures), data schemas (Avro, JSON), and data serialization (Parquet, ORC, Avro).
• Advanced use of Azure Data Factory for automating data workflows.
• Expertise in SQL and T-SQL within Azure SQL Database.
• Proficient in designing data warehousing solutions using Azure Synapse Analytics.
• Skilled in data processing and analytics with Azure Databricks.
• Deep understanding of Apache Spark for large-scale data processing.
• Expertise in setting up and optimizing Azure Blob Storage and Data Lake Storage.
• Proficient with Azure Resource Manager (ARM) for deploying, managing, and monitoring resources.
• Skilled in using Azure Portal and CLI for resource management.
• Familiar with Azure Monitor and Log Analytics for monitoring and logging.
• Experienced in data migration, profiling, and quality enhancement.
• Proficient in Agile (Scrum) methodologies.
• Understanding of CI/CD processes and tools, including Git.