Online tyre comparison

Problem / Goal
The customer wanted an online product comparison tool for motorists to choose the best deals from multiple retailers on their new tyres and MOTs. Consequently, data needed to be gathered from multiple sources: while some retailers provide their own APIs, most only have websites, sometimes deliberately hindering the work of any scrapers. The tool also needed to be scalable, allowing new retailers to be added progressively.
Solution
We designed and created a database for storing products and product prototypes equipped with procedures for updating, browsing (including full text search) and automatic mapping. The latter was a challenge, as it involved matching the same products from various retailers, even when they had dissimilar or missing descriptions.

The tool includes crawlers that find products, and scrapers that update prices and other information semi-regularly. Schedules and rules for updating prices were carefully prepared, so that scrapers could avoid being detected and blocked by generating as little traffic as possible while working.

Finally, Rabbit MQ enhanced the live update process by reducing both the latency and number of internal requests.

Results
The resulting API provides data through a variety of queries to a front-end website developed by a third party. Search functionalities include searching by tyre specifications, license plate number, or car make and model. Results are sorted by price and filtered by distance from a user-given postcode.

Team

Kamila Marcinek
Database Developer
Radosław Szuban
Infrastructure Engineer
Maciej Weryk
Software Developer
Norbert Tu-Van
Software Developer