Chances are, if you have shopped online, you’ve searched for the item you want to buy, placed that item in your digital cart, paid for that item, and had it delivered in record time. Each of those steps you took to enjoy your shiny, new item wouldn’t be possible without the help of search engines. Search engines help us find products, help merchants confirm your order, and ship it on time. How do we initially get all this data from slower, traditional databases into fast search engines?

In this talk, Conor Landry focuses on how Shopify indexes product, customer, order, and merchant data from MySQL to Elasticsearch in near real-time and how to reindex over 50 terabytes of data in less than 24 hours and the roadblocks we’ve encountered. Conor describes the challenges faced when handling data which is critical to the livelihoods of small business owners and well-known brands as well as strategies used by Shopify when scaling a search indexation system for the long term.