Data Engineer

at DeepCrawl in London

DeepCrawl is the world’s best crawler-as-a-service platform. We’re a small(ish) company, but we’ve got an awesome client list. The world’s largest brands and agencies use our service to make their websites easier to find and use. Our ultimate aim is to become the tool of choice for marketers across the world — letting them crawl and analyse all of their websites to identify key insights.

We’ve built our stack on AWS and leading open source technologies: Ruby, PostgreSQL, MongoDB, Node, AngularJS. We are looking for talented engineers to work with us and help make our crawl infrastructure even more awesome, as well as build and launch new features.

Role Description:

You will join our development team to focus on the data platform behind our product. Working in an agile environment, you will improve and expand the data processing architecture, helping it scale and perform more complex data analysis.


  • Improve the data processing and storage pipeline for our large scale web crawler, advising on architectural changes to improve modularity, testability and efficiency
  • Evaluate new data technologies, build prototypes and benchmarks and propose improvements to the system
  • Proactively plan for capacity, gathering information about usage and data growth to keep systems operating smoothly
  • Make ongoing optimisations to the pipeline to improve processing speed and reduce costs
  • Understand the Product team’s plans for new features and consider their impact on the data pipeline development roadmap; also contribute to the feature roadmap based on your knowledge of technology

We are looking for people with a passion for technology and innovation, as well as a keenness to always learn and develop.


  • 4 or more years of professional development experience, including 2 or more years working with data at scale
  • Expertise in designing and optimising modular and testable code, using TDD and benchmarking
  • Experience building systems that process and store large volumes of data efficiently
  • Experience using PostgreSQL at scale
  • Experience with MongoDB
  • A keen interest in learning new programming languages
  • Experience using Ruby, or an ability to learn it quickly
  • An ability to drive features to completion through your own initiative — finding or requesting resources, asking the right people the right questions
  • An ability to focus on the needs of the product and the company as well as the technology
  • As a bonus, some experience of data science and exploration of large data sets would be beneficial


  • Make a big impact in a small, quickly growing company
  • Help build a product used by a long list of global companies
  • Learn from building challenging features
  • Work with a small, highly motivated team

« Go back to category
Is this job ad fake? Report it!   
Published at 23 Nov 2016
Expires on 23 Jan 2017