Dathena is a Swiss and Singaporean company developing data governance software based on machine learning algorithms. Dathena is the most complete and accurate data governance platform and is the only solution to easily classify and manage data with this level of accuracy while helping companies to comply with regulations. Using cutting edge machine learning technology, Dathena sift through all unstructured data allowing to regain full control on company’s information.
We are looking for a talented Senior Big Data Engineer that will help us improve data ingestion and processing performance by analysing current Spark applications to design and implement efficient architecture solutions.
Your primary focus will be to build distributed Spark streaming pipelines with Kafka
Design and implement streaming pipelines for a large application
Benchmark different solution approaches and analyze performances
Optimise code and resource usage
Present solutions approaches and architecture choices
Maintain high-performance and data integrity of critical database (NoSQL & SQL type)
Provide project technical support and expertise
Support new projects/integrations working with R&D team
Skills and qualifications
5+ years experience in working on large projects in a Spark environment
Strong experience in Kafka integration with Spark Streaming
Experience in dealing with large HBase databases
Excellent functional programming skills in Scala
Knowledge of cloud computing infrastructure (e.g. Amazon Web Services EC2, Elastic MapReduce) and considerations for scalable, distributed systems.
Location: Singapore R&D Office
The company's founder created Dathena because he was not able to find an existing solution that would fit the needs
of the top Tier 1 Bank he worked for.
Dathena is the first and only data governance platform developed by real users for real users.
Our platform was built with state-of-the-art technologies including machine learning and artificial intelligence capabilities.