Code Challenge – Storing your incoming emails into Orchestrate.io

For todays Code Challenge post I wanted to try out Orchestrate.io. Orchestrate has an interesting value proposition that’s very similar to what we do at SendGrid. Orchestrate essentially provides a “Database as a Service.” Data storage, just like email infrastructure, is one of those items that is easy to do poorly and very hard to do well.

Orchestrate understands that different types of databases are optimized for different types of storage and retrieval. For example:

• MongoDB, CouchDb, and Redis: Key/Value
• Elasticsearch and Apache Solr: Full Text Search
• Neo4j and OrientDB: Graph
• MongoDB with 2dsphere indexes:  GeoSpatial

According to Orchestrate, there is no individual database that can handle all of the query types that provide a rich user experience. The current alternative is to run multiple databases with synchronization to give the best experience. This, however, becomes a cumbersome solution to implement.

Using Orchestrate for SendGrid Webhook Persistence

One of my favorite features of SendGrid is our webhooks. The SendGrid Event Webhook can provide your application with data such as individual email notifications on dropped, delivered, bounced, open, clicked, unsubscribed, and more events. The SendGrid Parse Webhook can let your application receive incoming email and taken any action.

When your application consumes this data, Orchestrate works great for storage and retrieval.

Getting Started with a Parse Webhook/Orchestrate Example

Step 1:

• Create your Orchestrate account.

• Create your SendGrid account.

Step 2:

Setup your Parse Webhook on SendGrid

1.) Login into to your dashboard and click on the Developers link:

2.) Click on Parse incoming Email:

3.) Specify your hostname:

The “hostname” can be any domain that has its MX records pointed to SendGrid. For more info on how to do that, visit the Parse Webhook docs page. We also have a special domain called bymail.in. You can add any_sub_domain.bymail.in for the hostname. This means that you can have any email address, for instance: kunal@something.bymail.in. Any email that arrives there will flow through SendGrid. All the individual components of that email will get parsed out and posted to the url your provided.

Step 3:

Once you have an account on Orchestrate, just create an application from the Orchestrate Dashboard. Follow their great documentation here: https://orchestrate.io/docs

Step 4:

Create an application that can get the incoming emails from the SendGrid webhook and store that into your Orchestrate application. Below I have some sample code for an application written in python that stores my incoming emails in Orchestrate:

from flask import Flask, request
from porc import Client

app = Flask(__name__)
client = Client("your_orchestrate_api+key")

@app.route ('/incoming', methods = ['POST'])
def nextweb():

   subject = request.form['subject']
   sender = request.form['from']
   recip = request.form['to']
   body = request.form ['text']

   #Store in Orchestrate
   response = client.put('email', sender, {
   "To": recip,
   "Subject": subject,
   "Body":body
   })
   response.raise_for_status()


   return "OK"


if __name__ =='__main__':
   app.run()

Step 5:

Success! If you want to see what you have stored you can either go to the dashboard and search your data in whichever method you want. You can also write code to retrieve data through your app as well.

Over the next few weeks, I’m going to continue writing about different technologies, so check back every Monday, Wednesday, and Friday for new tech posts! You can also visit my intro post that will have an updated list of everything I’ve written.