We Are Hiring

Process SendGrid Events in Real-time with TempoDB


Posted on

tempodbIf you’re a SendGrid customer you will no doubt have taken a look around your statistics dashboard a few times. It’s great isn’t it? Being able to get such a granular view of what’s happening with the email you send through us.

I’m fascinated by statistics and analytics so when I stumbled upon Techstars Cloud graduate company TempoDB recently I had to check them out.

What is TempoDB?

TempoDB is a database designed specifically for storing time series data. This data could be anything really but it’s better used for little pieces of information like thermostat temperatures, network latency or heart rates. Perfect for storing data over extended periods of time, or where the storage requirement is very, very frequent.

There is so much potential here, particularly for the new generation of ‘Internet of Things’ connected devices that will be emerging over the next 10 years; a supposed 50 billion.

How it relates to SendGrid

Hmm, which part of our system can give you data over periods of time? If you don’t know, you haven’t dug deep enough into our awesome docs.

The Event API can spit out a tonne of data to wherever you tell it. We show it to you on your Statistics Dashboard, but there’s no reason why you couldn’t also store the same data elsewhere for further, different, or customised analysis. TempoDB is perfect for this.

Digging the ‘Event Tunnel’

To test it out, I wrote a small server using Node.js that would tunnel data from the SendGrid Event API through to a TempoDB database as the events come through in real time. The full code is below, but here are the steps it takes:

  • Includes Node.js HTTP and QueryString libraries (Lines 1-2)
  • Includes the TempoDB Node.js library (Line 3)
  • Grabs whatever the Event API sends us and write it out to a time series on our TempoDB (Line 11-27)
  • Listens to HTTP server listens for POST requests. If it gets one it hands the data off to be stored on TempoDB. If the request is not POST then we just kill it, 405 style. (Lines 30-55)

The full code, hosted on Github:

That’s all it takes to pipe your SendGrid events, like clicks, opens or unsubscribes, into TempoDB.

Try it out

If you just want to keep a count of the events coming out of your account in a store then you could deploy this code right now, either to Heroku or Nodejitsu or directly to your own server. You might want to tweak it a little once you’ve seen it in action to better suit your specific need.

Once you have the server running somewhere publicly accessible, you’ll need to enable the Event Notification app and change the ‘POST Event URL’ setting to the new URL for the TempoDB tunnel… and that’s it – watch the data flow!

Then what?

That’s really up to you. I would suggest just leaving it initially to build up a decent picture of data you’ll need a few days. After that it’s well worth exploring some of the visualisation options TempoDB offer in their dashboard to give you an idea of what can be done.

TempoDB visualization

In part two of this look at TempoDB, I’ll show you how to extract and analyse some of the data into a beautiful (maybe) daily digest email. In the meantime, why not take a look at your own SendGrid stats? If you’re not already digging in, you’re really missing out.

Tags: , , ,


Martyn Davies is a Developer Evangelist at SendGrid and a creative developer based in London. He has worked in technology for over 14 years with a background in both the music industry and technology. A serial hackathon organiser, mentor and startup advisor, you’ll find him presenting, demoing, hacking and chatting at hack days, conferences and meetups in the UK & Europe on a regular basis.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>