Join the Queues Using Messaging and SendGrid

Join the Queues Using Messaging and SendGrid

When building applications there are often many tasks you require your user to perform that can take up quite a lot of time. Processing avatar images, preparing large files for download or importing data from a series of connected accounts are all tasks that, whilst they might be essential, should feel quick and seamless for the user.



Handling tasks such as these inside application logic may be fine at first, but at a certain scale, if you're making users wait whilst their account information imports because you've got several processes tied up doing it, they're going to start considering their options elsewhere and hey buddy, those options don't include you.

One of the ways to achieve this is to pass off this work to some background processes, often called 'workers,' that decide what they're doing based on information they get from a message queue.

How message queues work

This calls for a diagram (explanation below):



In their most simple form a message queue behaves like so:

1. Queue has a message written to it by your application. Messages are often a small payload of JSON or XML.
2. Messages persist until they are picked up and actioned by another script, this could be running as a CRON job.
3. Once complete the message should be removed from the queue.

Do you ever write yourself reminders for later on a notepad just so they're out of the way but you won't forget them? That's exactly the same as using a message queue in software. Think of it in that way and there are a tonne of practical uses for them.

Writing and retrieving messages

For these examples I'm going to use HP's Cloud Messaging Service because it's the most recent cloud based queue I've come across. Don't worry if you're not using that though, there are many similarities between queues and messaging-as-a-service (MSGaaS) providers in terms of RESTful operation so you'll find these translate to pretty much any other service.

The following examples use CURL

Create a new queue:

curl -X PUT -H "X-Auth-Token: HPAuth_4f4fcc24e4b04e2d592e7d79" -H "Content-Type: application/json" https://region-a.geo-1.messaging.hpcloudsvc.com/v1.1/1234567890/queues/my_queue

Above we're sending a PUT request that will create a new queue called 'my_queue' for us to begin writing messages to.

You should expect a common response code from your messaging service when sending requests, in this case we're looking for 201 - Created.

Using CURL it would look like this:

HTTP/1.1 201 Created Date: Tue, 23 Jul 2013 15:45:09 GMT Location: /v1.1/1234567890/queues/my_new_queue

Write a message to the queue

Messages are sent to the queue using POST, like so:

curl -X POST -H "X-Auth-Token: HPAuth_4f4fcc24e4b04e2d592e7d79" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"task": "import_checkins", "foursquare_id": "5434534"}' https://region-a.geo-1.messaging.hpcloudsvc.com/v1.1/1234567890/queues/my_queue/messages

Above, we pass the same authentication headers as before, only this time to the 'messages' endpoint of our newly created 'my_queue'. Additionally we're sending over some JSON as our message packet that defines a task to be completed (import_checkins) and an additional ID to use when performing that task.

Getting a task from the queue

Let's say our service imports check-in data from Foursquare for a user on a daily basis and then sends it to them as a daily digest email via SendGrid. We would have a CRON job that runs every 24 hours calling a script called import_checkins.rb. This script would be reading from the message queue, grabbing tasks marked as 'import_checkins' and getting check-in data for the FourSquare ID specified.

A simple GET request can handle this for us:

curl -X GET -H "X-Auth-Token: HPAuth_4f4fcc24e4b04e2d592e7d79" -H "Accept: application/json" -H "Content-Type: application/json" https://region-a.geo-1.messaging.hpcloudsvc.com/v1.1/1234567890/queues/my_queue/messages

If there was a message waiting on the queue, the response would be:

HTTP/1.1 200 OK Content-Type: application/json Date: Tue, 23 Jul 2013 16:01:32 GMT

{ "id": "1", "task": "import_checkins", "foursquare_id": "5434534" }

Each task would be processed by looping through each of the returned messages. Once the script completes its work, it's good form to tidy up afterwards so it should delete the message from the queue. In the case of HP's Cloud Messaging Service, once messages are retrieved they are also removed leaving the latest message at the HEAD of the response. Nifty.

Note: If you're using other queues you may need to issue a DELETE request to remove a message from the queue once it's been dealt with

Uses with SendGrid

If you're sending email that doesn't need to get to the user right away; a daily digest, activities, job listings, something that sits between transactional and newsletter style mail - using a message queue to process that backlog for users could be a great way to go.

Sure, you could absolutely achieve this just by hitting your database, but consider how lightweight and out of the way a message queue might feel in comparison if all you need do is write out a task and an ID in JSON to a queue for processing later.

I also think that there's great scope for working with our Events Notifications endpoint here as well. Particularly if you want to store that information but not process it right away.

Storing events as messages would look like this:

curl -X POST -H "X-Auth-Token: HPAuth_4f4fcc24e4b04e2d592e7d79" -H "Accept: application/json" -H "Content-Type: application/json" -d '{"email":"foo@bar.com","timestamp":1322000095,"unique_arg":"my unique arg","event":"delivered"}' https://region-a.geo-1.messaging.hpcloudsvc.com/v1.1/1234567890/queues/my_queue/messages

Writing these to a queue in real time will be much faster and less process intensive than trying to receive and process them in real time.

My recommendation is that as you scale out your application, consider breaking out lengthy tasks or lower priority pieces of work out into queues and deal with them later. Think of it this way, if it's small and doesn't need to be done right now, queue it up. Your servers and more importantly, your users will thank you for it

Photo by Xiaojun Deng