Content Comparison

Introduction

Most often, they were using long polling. In short, polling entails the application periodically sending requests to the server for status updates on any remotely occurring events. Bandwidth issues aside, the other problem with this approach is that it doesn’t reliably provide updates in real time, unless the client application fetches the API with extremely high frequency.

What is a webhook ?

A webhook is an HTTP request, triggered by an event in a source system and sent to a destination system, often with a payload of data. Webhooks are automated, in other words they are automatically sent out when their event is fired in the source system.

What are webhooks used for ?

Used to communicate the occurrence of an event in one system to another system, and they often share data about the event.

Let's say you subscribe to a streaming service. The streaming service wants to send you an email at the beginning of each month when it charges your credit card.

The streaming service can subscribe to the banking service (the source) to send a webhook when a credit card is charged (event trigger) to their emailing service (the destination). When the event is processed, it will send you a notification each time your card is charged.

The banking system webhooks will include information about the charge (event data), which the emailing service uses to construct a suitable message for you, the customer.

How webhooks work ?

1. Webhook request process
   
   Common in SaaS platforms like GitHub, Shopify, Twilio etc… because they support different types of events based on the activities that happen within them. To receive webhook requests, you have to register for one or more of the events (also known as topics) for which the platform offers a webhook. A webhook request will be sent to a destination endpoint (URL). It can be your application, register the URL as the Webhook URL for that event.

   Once the webhook registration for an event is complete, you will receive webhook requests at the destination URL you provided each time the event occurs.
   

2. Consuming a webhook
   
   After the registration, we can receive the webhook requests : regular http requests, webhook payloads are serialized form-encoded JSON or XML format, one way-communication system, can be distributed to multiple destinations that require information, request can be sent more than once making it idempotent.

When to use Webhook ? Comparing to WebSocket, Pub/Sub, Polling and API

Webhook vs WebSocket

WebSockets are used to facilitate two-way real-time communication between two networked systems. WebSockets achieve this by keeping a socket open on both the client and the server for the duration of the conversation.

This opens a duplex communication stream enabling both the client and server to pass information between one another with no significant latency added.

Useful for chat app and data visualization (need to reflect real-time data values).

Webhook vs Pub/Sub

Communication system for distributing messages between a set of publishers (message producers) and subscribers (message consumers). A pub/sub system buffers messages from producers and routes them to subscribers through the use of dedicated channels known as topics. Publishers publish messages to topics and subscribers express interest by subscribing to the topics.

Useful for e-commerce site.

Webhook vs Polling

Polling involves periodically making requests to a system to check for new events or data. If new data is found, a response is returned with the new data in its payload. If no new data is available, nothing is returned.

Polling is used to query systems for new information and can be set up as an automated cron job that runs at certain intervals.

Useful when we don’t need real-time updates.

Webhook vs API

An API is an application's gateway to the world. It is the interface that allows other systems to interact with the application. The application can expose as much as it wants and keep its remaining parts abstracted.

APIs are usually a collection of endpoints that clients can call to communicate with the system. This communication is done via request methods such as GET or POST, and may carry some information related to the action the client is willing to perform.

Scaling Webhooks

Scaling with Async processing of Webhooks

Most of the time, webhooks can be handled easily. But if your server receives too many requests at once, it can become overloaded. Add all the other unrelated jobs your server handles to that ballooning volume of requests, and you may have a service outage on your hands.

Async processing should implemented in 3 steps : ingestion, queueing and processing.

Ingestion

Our only goal here is to expose a simple HTTP POST endpoint. The endpoint should be built to utilize as few resources as possible, which will ensure it remains scalable. Use Serverless services (Azure function, Google Cloud Functions, etc.).

Queuing

However you design your queue, make sure it won't crumble under the load. Most queues are limited by the number of concurrent connections, or throughput in GB/s – you want to make sure that your queueing system can keep pace with any unexpected spikes in volume. See different patterns of queue.

Processing

Once you have your webhooks in a queue, you can safely process them. Just be sure to do so at a rate that is reasonable for your infrastructure and use case. Different queues take different approaches to processing – but, generally speaking, you will want a set of worker services, each pulling from the queue at its own pace.

Retry on ingestion and processing failures

Implement a retry mechanism in our design.

Scaling with Message Queue

Webhook with a message broker (Kafka, …). The main function of a message broker is to buffer and distribute messages from message producers to message consumers. Message routing to consumers is done by using routing rules defined when the message is sent to the broker. The broker uses these rules to place the message in a queue where the message is buffered before it is later picked up by one or more consumers subscribed to the queue.

Most (if not all) message brokers do not support the HTTP protocol used in sending webhooks. Therefore, to use a message broker to ingest webhooks, you need to proxy the webhook requests through an API gateway. The role of the API gateway is to act as a protocol conversion middleware, transforming webhook HTTP requests into the format that the message broker supports (e.g. AMQP and STOMP). Because producers send webhooks using the secure version of HTTP (HTTPS), gateways are often tasked with the responsibility of TLS termination.

Webhook Security

A webhook is a fast, simple, and efficient HTTP communication mechanism for integrating remote applications. This simplicity and efficiency, while very exciting, involves a very critical trade-off: security. Webhooks were not built to be secure out-of-the-box, and the entire security burden falls on the developer.

Concerns

• The webhook communication mechanism does not have a native way of identifying the source or destination of a webhook.

• Webhooks use the HTTP protocol by default. The HTTP protocol communicates data in plain text, which means that data is transmitted openly from one application to another. Webhooks use the protocol as-is with no extra layer of protection added for webhook data transmission.

• Webhook consumers have no idea which data they are to expect and yet do not have a way to verify that it is the right data.

• While producers can specifically target webhook consumers, there is no way for consumers to restrict the sources they are receiving webhooks from.

• A webhook, on the consumer side of things, has no way of detecting if it is being repeated. a webhook request can be duplicated.

Webhook Authentication strategies

Verification (bearer) token and custom headers

Involves the passing of a secret string in the request header. This string is given to the webhook producer after successful authentication and used to verify each webhook request.

Token authentication

One standard method of this approach is token authentication. This is an OAuth verification protocol that allows users to verify their identity on a web application, receive an access token and verify each request to the web app with the access token. The authenticated request uses the Authorization header to send the token using the format below:

Authorization: Bearer {token}

Webhook consumers can use this strategy to authenticate webhook requests arriving at the webhook URL.

Custom headers

A lesser-used method of authentication involving a secret string being passed from a webhook producer to a consumer is the use of custom headers. Webhook providers like Okta give you the ability to define custom HTTP headers when setting up your webhook.

These headers are sent along with your webhook and they are used by the webhook consumer to authenticate webhook requests, and reject webhooks that do not contain the custom headers.

Signature verification