If you ever want to get a Twilio SendGrid expert talking, ask them about the Event Webhook. Then get comfortable because they’ll have a lot to say!
Twilio SendGrid’s Event Webhook is a bit of a darling. Businesses love this powerful tool because it gives teams unlimited access to real-time data on email campaigns. And our onboarding and deliverability experts love it because good data lays the foundation for building (and sustaining) strong email program performance.
In a recent conversation, Justin Foulk, Email Managed Services Manager, interviewed Dustin Berg, Technical Onboarding Engineer, to find out:
Justin:We have a lot of customers that use the Event Webhook. It's a really powerful tool. Can you explain what the Event Webhook is for anyone who might not know the tool?
Dustin: Think of the Event Webhook as a type of API driven by events rather than requests. Usually, when you trigger an email, you make an API call, or in other words, a request to Twilio SendGrid. But our Event Webhook actually makes the requests to a server endpoint that you stand up and posts the delivery and engagement event data to a URL of your choosing. Then, you can ingest that data into your data warehouse and build customized analytic reporting on your email program.
Justin: That sounds awesome and really effective for senders. Is the Event Webhook real time, delayed, or batched?
Dustin: That's a great question. One of the big advantages and why Onboarding Engineers push our customers to utilize the Event Webhook is that the data we post is in near real time. Additionally, we batch and can push tons of data to you as it unfolds in real time.
The way it works within our system is that events post within 30 seconds, or when going through batching, the batch reaches a size of 768 kilobytes. This is also per server, so if you're a really high-volume email sender, you can expect tens or hundreds of posts per second to your endpoint.
Justin: Awesome, thank you for that insight. Obviously, we never want this to happen and it rarely does, but what happens if the Event Webhook endpoint goes down? Do we retry? Is there a delay? What does that look like in terms of downtime?
Dustin: We understand that things happen, so we have a solution in our Event Webhook API to account for that. If your endpoint is undergoing maintenance—it does go down, and you need to get it to back up—we have retry logic built into our system. What really happens is our system expects a 200-type response (e.g., a success status response) from your webhook endpoint. When we don’t get that 200-type response, we retry that specific post, then queue up additional posts for up to 24 hours. If the Event Webhook URL that you've constructed isn’t up within 24 hours, we start to delete those from the queue.
Justin: A lot of our customers are very familiar with the standard dashboards and how Twilio SendGrid slices and dices the data in the user interface. How does the Event Webhook compare and contrast with the Stats dashboard in the platform?
Dustin: We offer 2 really great dashboards within the console that are native for some of the more nontechnical users. The Event Webhook combines those 2 dashboards, so you can create an entirely new robust engine that can run and collect the data for your events.
One of the key differences between the webhook and the 2 solutions that we offer within the console is due to the way that aggregate dashboards work—sometimes, you have some database lag. With the Event Webhook, we push this data to you in real time automatically.
Justin: I know a lot of folks may be wondering, how do I set it up? How can I use a webhook in my main account and enable it for my various other accounts?
Dustin: In your console, select the Settings drop-down on the left side, then click Mail Settings.
When you arrive at Mail Settings, click the pencil icon next to Event Webhook. Then, we can start to configure the webhook.
A lot of the actual construction of the endpoint itself and connecting it to your data warehouse happens external to Twilio SendGrid. But once it's ready to integrate, and you're ready to start testing, you'll post the URL you've constructed.
Then, you select all the data that you want us to ingest. You also have the ability to select the different types of events you want. So if you want to collect everything, go ahead and click Select All. Or, if you want to pick and choose, you can. For example, select Delivery Data if you send only transactional emails and want to ensure the emails get from point A to point B. You can really get customized on what you want.
Once you're ready to go, click Enabled and Save. Then we start to fire off those events to your webhook.
Justin: Speaking to the customers that really want to make sure bad data or fake data doesn’t infiltrate their webhook endpoints, do we have any security methods in place enabling the webhook? If so, what would those be?
Dustin: The most out-of-the-box solution that we have available is that we sign all our events with a user agent—and you can utilize the Twilio SendGrid Event API for that. Now, that's obviously not the greatest solution for a lot of our customers really concerned about security. So we now support both ECDSA-signed webhooks and OAuth 2.0 integrations, where you can ensure that the posts ingested into your data warehouse come from Twilio SendGrid.
Justin: If somebody really wants to get customized with what they aggregate and how they build the dashboards, what are some examples of tools they can use to ingest web data?
Dustin: What's really cool about our Event Webhook is that you can consume the data however you choose to do so. We've got native integrations with Twilio Segment that you could utilize to collect data on your email. I've also seen customers set up an endpoint that connects and talks directly to a Google Sheet. Then again, I’ve seen other big tools like Splunk and additional data-warehouse-type, third-party solutions. Or, you could stand one up yourself and integrate into a customer relationship management system as well. The possibilities are really endless, and that's what makes the Event Webhook such a robust tool.
Justin: If we have customers that have multiple Twilio SendGrid subuser accounts and want to stand up the webhook functionality, how do we set up the Twilio SendGrid Webhook in the various different accounts or services that somebody might have?
Dustin: The Event Webhooks are one to one with each account, so you can have one webhook endpoint connected to each account whether that's the parent account or subusers. You can utilize the same endpoint across all your subusers and connect it to one centralized point. Or, you could start to build different distribution centers that you could push the data to on a per-user basis. Additionally, you can leverage what we call custom arguments, where you can start to flag specific pieces of metadata that we can pass through the webhook parameters as well. The big key here is that this is a one-to-one ratio, where you can only have one endpoint per account, and not a global, trickled-down setting. You won't be able to set up the parent account and apply your webhook to subusers—you have to set the webhook within each account specifically.
One of the great things about Twilio SendGrid is that we build nearly everything off of our APIs. Whatever you can set up in the console, you can additionally set up via the API to do it programmatically for many users.
Justin: Let's talk about the data piece, the details around the data, and what folks can get back. What can I control in terms of the events getting pushed back to me?
Dustin: Each event will have a specific set of parameters that we pass back to you. These aren’t customizable—these are the standard, no matter which events you choose. That said, you can select and get really granular with the different types of engagement or delivery events that you want us to post to your endpoint. You can collect and consume all data entries or select delivery-only engagement, and so on. But again, the parameters that we pass within each of those different events is the standard.
One question that we get asked about a lot is the new Apple Mail Privacy Protection (MPP) policy that’ll start to inflate the open events because now there isn’t a great way to track if these are real opens or not. Because Twilio SendGrid is a really agile company, we accounted for that and implemented new parameters to account for MPP. So what you could see now with all our open events is we’ll have a specific Twilio SendGrid machine open parameter, a boolean value of true or false that you can start to detect if these are machine opens or not.
Justin: So what is the standard format of the ingested data?
Dustin: All the data will come back in JSON format. Sometimes, you might just get a single processed event as one object, while other times, if we batch the data together, you could expect to see an increase in different events that could post to your endpoint. So that's something to keep in mind when you construct the integration—that is, to ensure your connection can accept an increase in data. But it’ll all be in JSON, so you can easily build around that to ingest it.
Justin: Leading into my final question, obviously, we talked about the customizability of all this data, with users being able to really create dashboards around what they value as a sender. Do you have a common table or dashboard that senders like to use and find valuable in understanding how their mail streams perform from a data perspective?
Dustin: So below is a sample customized dashboard that you can build from the data that we push to your data warehouse—it’s a customized report on stats by domain. This may or may not suit your needs, but this is something that you can build with the data that we push to you. In this dashboard, we get processed events, delivered events, and percentages. With this data, you can build out unique open percentages and get hard bounces and soft bounces (otherwise known as blocks) within the Twilio SendGrid user interface.
You can really start to get some customized reporting and build this to suit your needs with the events you receive from our webhook.
A big thanks to Justin and Dustin for this insightful conversation on the power of the Twilio SendGrid Event Webhook. If you’re inspired to integrate with our Event Webhook but need guidance, Twilio Professional Services is ready to help
