We’re looking for an experienced Data Engineering to join our amazing FinTech department and work in a complex and challenging data environment, developing exciting features, new pipeline architecture as well as optimizing the data flows. The ideal candidate is an experienced independent data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. We are looking for someone to lead the E2E scope, not just to work on it.
You will lead and develop advanced ETL/ELT procedures and high-scale, real-time, streaming applications, design data solutions, and implement production data pipelines for real time stream processing, optimized for scalability, modularity, quality, and uptime. You will work with technologies consisting of Python/Java, Kafka, Spark, AirFlow and many more in a fast-paced, high-growth company.
- Create and maintain optimal data pipeline architecture for multiple data sources
- Lead end to end development of data models, from gathering the data sources and developing ETL or stream processes to producing front end data model solutions
- Design and model data warehouses on platforms such as MY-SQL, BigQuery/RedShift and other high-performance platforms
- Re-design infrastructure for greater scalability and lower latency
- Monitor and maintain production streaming and batch applications
- Take full ownership of designing, building, testing and deploying data products
- Experience building and optimizing ‘big data’ pipelines, architectures and data sets on cloud native environments
- Ability to work independently and learn new technologies
- 2+ years of experience building processes supporting data transformation, data structures, metadata, dependency and workload management
- Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ stores.
- Proficiency with Python, SQL, Java or Scala.
- Experience in data streaming tools and processes (Kafka, AirFlow, ELK, Spark etc.)
- Experience with big data solutions like Athena, SnowFlake, Redshift, Redis, Aerospike, Impala, Presto, Bigquery, Hive, etc.
- Experience in data modelling, BI solutions development (DWH, ETL) and SW development practices – Agile, CI/CD, TDD.
- Experience in real time analytics
- Experience with GCP and BigQuery
- Experience with building automated validation processes to ensure data integrity