TLDR; I have worked around my issues with webhooks. Although I would argue they are a workaround, due to the lack of configuration you can apply to a webhook, this post is more about feedback and understanding why things are the way they are.
Before I talk about the issues I faced when working with webhooks, I’d like to give you a bit of background as to why I’ve been working with webhooks.
My setup
- prod: Ghost version 3.26.1 running in Cloud Run
- dev: Ghost version 3.26.1 running locally using Node 12; database is a clone of my prod database
- API service: Python 3.7 service using Flask
Background
For a while, I’ve been thinking about how I can integrate webhooks into something. I wanted to take a look at integrating search into Ghost via Algolia. So, I’ve written a small API service that can accept JSON data from Ghost’s webhooks.
I wanted to receive JSON data from the webhooks, extract specific values and either create a new entry or update an existing entry in Algolia.
I also wanted this service to be flexible, so I could integrate into other services if required.
Problems
Parsing key/value pairs
I’ve had a look around the forum, and I can’t find an exact answer as to why there are a few key/pair values are invalid when data is sent via webhooks. It isn’t enjoyable because I believe the JSON data should be able to be parsed without having to manipulate it.
Below are the offending keys with their values. A period represents the nested key.
- post.current.mobiledoc: “{\“version\”:\“0.3.1\”,\“atoms\”:,\“cards\”:[[\“markdown\”,{\“markdown\”:\“Testing.\\n\\nTesting some more.\”}]],\“markups\”:,\“sections\”:[[10,0],[1,\“p\”,]]}”
- post.current.authors[0].accessibility: “{\“nightShift\”:true,\“whatsNew\”:{\“lastSeenDate\”:\“2020-06-29T16: 11: 36.000+00: 00\”}}”
- post.current.authors[0].tour: “[\“getting-started\”,\“using-the-editor\”,\“featured-post\”,\“upload-a-theme\”]”
- post.current.primar_author.accessibility: “{\“nightShift\”:true,\“whatsNew\”:{\“lastSeenDate\”:\“2020-06-29T16: 11: 36.000+00: 00\”}}”
- post.current.primar_author.tour: “[\“getting-started\”,\“using-the-editor\”,\“featured-post\”,\“upload-a-theme\”]”
post.previous.mobiledoc: “{\“version\”:\“0.3.1\”,\“atoms\”:,\“cards\”:[[\“markdown\”,{\“markdown\”:\“Testing.\\n\\nTesting some more\”}]],\“markups\”:,\“sections\”:[[10,0],[1,\“p\”,]]}”
Solution
This was easy, but I don’t exactly like it in my code. I decided to initially parse as much JSON I could and then remove any invalid characters in the JSON that could not be parsed.
Lack of authentication variables in webhooks
Again, I’ve had a look around the forum and other places, but I couldn’t find an exact answer on this. As I wanted to integrate webhooks into Algolia, via my API service, I wanted to add some form of authorisation to each request. The most straightforward answer is to add an Authorization header using basic auth.
However, after looking at this, and reading a few posts later, I thought the only way to authenticate my requests would be through the URL in the webhook.
Solution
Since I didn’t want to add basic auth directly into the URL, I decided I should add query strings to the URL.
As I already decided to put my API service into Cloud Run, naturally, I used Cloud Endpoints to handle the authentication for each request to my service. I added the query string ‘key’ to each webhook and inputted the API key that I set up in GCP.
For those that don’t know, Cloud Endpoints acts as a proxy to your service, decoupling any authentication required for your service. You can still add additional checks in your service, as I did (and as noted below).
Additionally, I thought if my API key in GCP was compromised, I should try to have another layer of auth in my service. Since I had a Ghost content API key going to waste, I decided to incorporate another query string into my webhook using this key.
What did I learn from this?
Webhooks are not flexible and are a little challenging to work with.
Ghost Team - Please take this post as feedback from my experience using webhooks.
Thanks.