Almost every app seems to need background processing. In ruby, I think sidekiq is the best system for doing this. As your system grows you need to start paying attention to how your organize your sidekiq jobs. I walk through some progressively more complicated setups.
A couple years ago we were working on a reporting project that needed data that wasn’t really kept online. A user would select a number of streams and a time range. Relevant log data would be downloaded from S3, indexed in SOLR, then queried, sent to the client, and displayed. Depending on which stream and how much time they asked for, this might take anywhere from a few seconds to a few hours. Of course, once it was loaded we could make subsequent queries very quickly. Obviously we couldn’t just have the browser stop and wait for the request to complete.
Our solution will surprise you!
A few weeks ago Spredfast held its once or twice yearly
hackweek. The whole engineering department drops everything,
self-organizes into small teams, and works on something cool they want
to work on. Otto and I used this opportunity to jump in and try to get to know elixir and phoenix better.
Powered by Typo – Thème Frédéric de Villamil | Photo Glenn