Webhooks are a lightweight, open, and effective way to work with events generated by 3rd party systems that you may want to integrate with.
Indeed one of the integration options you have as a developer integrating into Marketplacer is to subscribe to webhooks and listen for events including, but not limited to:
- Product Creation
- Order Updates
- Shipment notifications
However, despite the benefits of using webhooks to integrate, one drawback that you’ll encounter is their inability to be routed directly to your local development machine.
In this article we discuss 3 ways that allow you to work within that (understandable) constraint.
A typical Webhook set up
Just to clarify the issue a little further, when setting up a webhook on a 3rd party system you typically provide information on:
- The “event” that you want to receive, e.g. new order created
- The data to be sent, e.g. Order Id, Delivery Address, Order Amount etc.
- Where you want the data sent to (the HTTP endpoint destination)
It is this last point where you can become a little unstuck as a developer wanting to work with these events on your local machine — what destination do you configure in order to get events delivered to localhost?
While there isn’t really a webhook standard, for the most part webhooks are typically implemented as HTTP POST requests with a JSON body payload sent to the consuming system (in this case your local machine).
As an example, if I attempt to configure a webhook on Marketplacer to send Invoice events to a localhost endpoint, I’d get the following response:
And while you may get a slightly different error on other platforms, the fundamental issue remains, the 3rd party platform cannot route webhook events to localhost…
The remainder of this article discusses how you can work within this constraint.
1. Fake it
This one is a bit of a cheat actually, but it can be surprisingly useful…
To use this approach all you have to do is use a HTTP API client like Postman or Insomnia and simply make direct HTTP POST requests to your localhost development endpoint, essentially “faking” a real webhook request. (Clients like Postman and Insomnia running locally on your development machine can make requests to localhost destinations).
In order to make this “fake” request as realistic as possible, you’ll need to ensure that you replicate it as close to the original webhook from the source system as you can. To do that you will need to:
- Generate a real webhook request
- Receive that webhook request somewhere (E.g. webhook.site)
- Examine the body payload and headers
- Replicate that body and headers with your HTTP Client (e.g. Postman)
While this method may be considered a bit of a cheat, it can prove really useful in the early stages of development even if you have other methods available to you.
Some of the reasons for this are:
- You don’t actually have to trigger a webhook on the source system which can sometimes be time consuming.
- You can easily manipulate the headers and body payload directly, again without having to generate an actual webhook event…
While this option is useful in the early (and possibly later) stages of development, it doesn’t provide you with a true end to end integration experience. In that case, you’d need to look at 1 of the other options…
2. Use an ingress app
The 2nd method requires the use of an ingress, or tunnelling app, with the most well known of these being Ngrok.
Ngrok (and similar apps such as Localtunnel), provide you with a routable url that maps down to a localhost endpoint on your dev machine. This means that you can provide the routable url to the system generating webhooks, (e.g. Marketplacer), and those webhooks will eventually make their way down to your local machine, and more specifically the app you’re building.
The advantages of the this solution are:
- Very easy to set up (Ngrok & Localtunnel are anyway)
- Ngrok provides a decent free plan
- You get “real” webhooks delivered to your app
One of the potential downsides to this approach is of course the security considerations you may have in opening up a tunnel from your local machine to what is essentially a public endpoint. Depending on the industry you work in, or the organisation that you work for, using something like Ngrok may not even be an option available to you given the misgivings about potential security threats.
For a more detailed guide on setting up Ngrok, please refer to our companion article over on Medium.
If the security concerns surrounding this method prohibit you from employing it, then there is another approach you can take…
3. Use a proxy
There are a few variations of this pattern, but it essentially relies upon you having access to a routable endpoint somewhere (the “proxy”) which receives (and stores) the webhook traffic from the source system. You then need to retrieve the stored webhook messages down to your local machine, and finally route them to your local app.
As you can imagine there are a number of ways that this could be achieved but I use the following architecture:
This approach relies upon 3 components:
- An Azure HTTP Function
- An Azure Service Bus
- Service Bus subscriber
We talk a bit more about these below.
Azure HTTP Function
This acts as a routable HTTP endpoint that can be used to directly receive webhooks from the source system, in this case Marketplacer. When the Azure Function receives a webhook event, it then places it immediately on to the Service Bus.
Azure Service Bus
The Azure Service Bus acts as both:
- The persistent (or semi-persistent) store for the webhook messages
- The method of delivering those messages to a (local) subscriber
In relation to the persistence of the messages, the Service Bus provides this but only for a relatively short period of time (default is 14 days).
Service Bus Subscriber
This can be any app that can subscribe to our Azure Service Bus and read the messages from it. In this particular case this app is a .NET console app that triggers every time a message is placed onto the relevant topic of the Service bus. This agent then forwards these messages to whatever localhost endpoint it’s been configured to.
This approach may seem like using a sledge-hammer to crack a nut, but it does work and has the following advantages:
- Corresponds to more closely to something you’re likely to see in an enterprise production environment
- The use of a Service Bus brings with it all its inherent advantages including, but not limited to: security, routing, message persistence & event driven delivery.
The downsides of this approach are:
- Complexity. There are a few moving parts which makes the whole thing harder to configure, monitor and fault find than using something like Ngrok
- Cost. While the actual cost of this solution with a few hundred or even a few thousand webhooks would be minimal, it does still come with a potential cost attached to it.
In conclusion
0All 3 options are viable to get you up and running with developing with webhooks locally, and like most things in life you will pick the one that meets your own requirements. However, for sheer simplicity and effectiveness, the ingress option is hard to beat in a local development context.
