Implement a server to receive webhooks events

Introduction

We are providing you with a complete tutorial designed to let you create a webhook server microservice to receive signals generated by vehicles in this system.

After subscribing to a webhook event, this server automatically responds to our challenge and starts receiving the signals.

🚧

Warning

This tutorial draws upon the notions explained in the Receiving signals guide's page.
If you don't remember what a signal is, please visit the Signals guide's page.

🚧

Warning

To be able to easily follow this tutorial, at least a basic level of computer science knowledge is required.
The configuration of the server is language and framework independent.
The code examples are written in Typescript.

❗️

Alert

If you want to skip the tutorial and see some use cases and examples, you can visit our Webhooks listener cookbook's page.

Create a repository

We will start from an empty project optionally versioned with git.
This step is a straightforward one, you can also skip directly to the Build a web server section.
This section is optional, but we strongly recommend reading it to obtain the code versioning and have a full readable history of what you are building.

Install git

First, you need to install git on your laptop.
Open a terminal pressing Ctrl+Alt+T, and launch:

Check if the installation was successful through:

The first thing you should do when you install git is to set your username and email address.
This part is fundamental. Your data are immutably baked into the commits you will create, and every git commit will use them.

You need to go through these commands only if you pass the --global option, git will always use that information for anything you do on that system.

Remember to replace username with your username and [email protected] with the email address you used to register on Github.

📘

Info

If you do not have a git account yet, please sign up on github.

Create the repo

You are now ready to create your first repo on github.

Click on the top right button +, then press New repository.
It will appear as a new page with some configuration to set. The only mandatory field is Repository Name.
You can use the name you prefer. We will use listener-demo as a demo name for this tutorial.
Now click on the green button Create repository.

👍

Congrats!

You have just created your project repo.

Now there’s a quick set up to be done.

Before executing the GitHub page's suggested commands, let's create a directory where we will work with git.
Open a terminal pressing Ctrl+Alt+T and launch:

As directory-name you can choose the name you prefer. It will be the project's root.
Let us create a directory where the project will be located and where we will execute commands under “... or create a new repository on the command line”:

📘

Info

In the second-last command, you have to change username with your personal GitHub username.
The last command will ask for your username and password: use the ones that you chose to register/log in on Github.

Finally, you can refresh the GitHub page to see your Code repo with the only file created so far: README.md.

Build a web server

The action you will accomplish in this second step is to build a simple hello world server.
With the knowledge acquired so far, we'll develop the requests on the endpoint that we will call, from now on, the listener.

We will use:

• NodeJS as runtime,
• npm as the package manager,
• Typescript as the programming language,
• Fastify as the web framework.
  We will install these dependencies simultaneously.

To start working on this, you need to install npm, so open a terminal pressing Ctrl+Alt+T and execute:

Check if the installation was successful through:

Now, let's use the framework!

Getting started

Go to the root of the project and follow these steps.

1. General set up

• Create a new npm project,
• install Fastify,
• install Typescript and Node.js types as peer dependencies.

2. Script set up

Add the following lines to the scripts section of the package.json.

{
	"scripts": {
		"build": "tsc -p tsconfig.json",
		"start": "node index.js",
	}
}

3. Typescript configuration

Initialize the Typescript configuration file through:

🚧

Warning

Most recent typescript configurations needs tsc instead of typescript

4. Initialize

Create an index.ts file. This file will contain the server's code.

5. Fastify set up

Add the following code block to your file:

import fastify from 'fastify'

const server = fastify()

server.get('/hello', async (request, reply) => {
  return 'World\n'
})

server.listen(8080, (err, address) => {
  if (err) {
    console.error(err)
    process.exit(1)
  }
  console.log(`Server listening at ${address}`)
})

6. Compile

To compile index.ts into index.js, which can be executed using Node.js, digit:

7. Fastify run

You need to run the Fastify server.

Now, you should see on your terminal:

8. Test the server

Open a new terminal and try out your server using:

It should return World.

👍

Congrats!

You now have a working Typescript Fastify server!
By default, the type system assumes you are using an HTTP server. Later on, we will explain how to specify route schemas and a lot more.

Update remote repo

🚧

Remember

If you are using git it is important not to forget to update the repo.

Before uploading our new files remotely, we should ask ourselves some questions.

The answer is no because we want to have a clean and readable repo.

The answer is a useful file, .gitignore, as we can move in it all those files that we don't want to visualize in the remote repo.

e.g.

• the directory node_modules,
• the javascript file generated index.js.

Create a file called .gitignore and add the following code:

# Dependency directory
node_modules

# Exclude typescript generated files
index.js

You are finally ready to push our changes remotely.
Open a terminal pressing Ctrl+Alt+T and:

1. Add

Add all the files to the staging area.

2. Commit

Commit the changes and comment with -m option.

3. Push

Push it remotely.

On your Github account, you can check if your local changes are remotely synchronized.
Now you should be able to easily clone your Hello world server.

Implement listener endpoint

In this section, we’ll finally go deeper into the development of our endpoint.
We will dive into implementing generic types for route schemas and the dynamic properties located on the route level request object.

There are two requests to handle: a GET and a POST.

📘

Info

If you don't remember what we are talking about, please visit the Receiving signal guide's page.

GET /listener

If you skipped the last section, you can clone the repo with the following commands on the terminal and obtain a web server already built:

To run the server, please follow the instructions on this README.md file.

The GET request is the first call we will receive after the subscription.

📘

Info

You will receive the GET request only one time.

It consists of a request with three query parameters:

• hub.mode will always be subscribe. No challenge will be needed in the unsubscribe case because no request is received.
• hub.challenge is a random string generated by the system.
  Your system should return this value in a string/text format.
• hub.topic has to follow the rules described in the Receiving signal guide's page.
  The structure is the following:
  vehicle:$vehicleId:generic:$signalName

🚧

Warning

The $signalType can only be generic.

$SignalName may be one of the following:

• online,
• position,
• distance_covered,
• autonomy_percentage,
• autonomy_meters,
• * can be used to target more than one element.

📘

Info

If you don't remember what a signal is, please visit the Signals guide's page

Query parameters are a defined set of parameters attached at the end of a URL.
They are extensions of the URL used to define specific content or actions based on the data being passed.

e.g.
If our URL is http://example.com an example of request is:
https://example.com?hub.mode=subscribe&hub.topic=vehicle:'UUID':generic:position&hub.challenge=thisIsARandomString

Our goal is to validate these parameters and reply with the hub.challenge value.

To define these properties, we use JSON schema.

JSON schema

It is not mandatory to use JSON schema to define properties, but it is suitable to define TypeScript interfaces.
We chose to use them because JSON schema is a standard for representing JSON data shapes and provides a superset of the basic features of the TypeScript type system for runtime validation.

Open a terminal pressing Ctrl+Alt+T.

1. Install

Install the json-schema-to-typescript module.

2. Create the container

Create a new folder called schemas and add a querystring.json file.
Copy and paste the following schema definition into the file:

{
    "title": "Querystring Schema",
    "type": "object",
    "properties": {
        "hub.mode": { "type": "string" },
        "hub.topic": { "type": "string" },
        "hub.challenge": { "type": "string" }
    },
    "additionalProperties": false,
    "required": ["hub.mode", "hub.topic", "hub.challenge"]
}

3. Compile the schemas

Add a compile-schemas script to the package.json.
Note that json2ts is a CLI utility included in json-schema-to-typescript.
Schemas are the input path, types are the output path.

{
  "scripts": {
    "compile-schemas": "json2ts -i schemas -o types",
  }
}

4. Run

A new file should be now available in the types directory.

5. Resolve JSON modules

You need to add the following line to the compilerOptions of tsconfig.json to build safely when you import schemas.

{
  "compilerOptions": {
    "resolveJsonModule": true,
}

GET request

Once schemas are defined, we are ready to update index.ts to handle the GET request.

1. Import

Import JSON schemas and the generated interfaces on top of index.ts.
It might seem redundant, but you need to import both the schema files and the generated interfaces.

// import json schemas as normal
import QuerystringSchema from "./schemas/querystring.json";

// import the generated interfaces
import { QuerystringSchema as QuerystringSchemaInterface } from "./types/querystring";

2. Build functions

Now you are going to build some useful functions to:

• validate the topic,
• list the valid receivable signals,
• log the error on the response and the terminal.

const topicErrorMessage = "Topic Validation Error";
const modeErrorMessage = "hub.mode must be 'subscribe'";

type Signal =
    | "online"
    | "position"
    | "distance_covered"
    | "autonomy_percentage"
    | "autonomy_meters"
    | "*";
const signal_name = new Set<Signal>([
    "online",
    "position",
    "distance_covered",
    "autonomy_percentage",
    "autonomy_meters",
    "*",
]);
function isSignal(signal: string): signal is Signal {
    return signal_name.has(signal as Signal);
}

let splitted: string[];
function isTopicValid(topic: string): boolean {
    splitted = topic.split(":");
    return (
        splitted.length === 4 &&
        splitted[0] === "vehicle" &&
        splitted[2] === "generic" &&
        isSignal(splitted[3])
    );
}

We can split the code into three blocks:

• two constant messages for the reply in case of error,
• definition of type Signal to validate the last word of the topic,
• the validation topic function that checks the length of the parameter and its values.

3. Define the listener API

Using the query interface, you can define the new /listener API route and pass it as generics.
The shorthand route methods, e.g. .get, accept a generic object RequestGenericInterface.

It contains four named properties:

• Body,
• Querystring,
• Params,
• Headers.

The interface will be passed down through the route method, into the route method handler request instance.

The generic interface is also available inside route level hook methods, adding a preValidation hook.

Add this code to the index.ts file.

server.get<{
    Querystring: QuerystringSchemaInterface;
}>(
    "/listener",
    {
        schema: {
            querystring: QuerystringSchema,
        },
        preValidation: async (request, reply) => {
            const {
                "hub.mode": mode,
                "hub.topic": topic,
            } = request.query;
            if (!isTopicValid(topic)) {
                console.error(topicErrorMessage);
                throw new Error(topicErrorMessage);
            }
            if (mode !== "subscribe") {
                console.error(modeErrorMessage);
                throw new Error(modeErrorMessage);
            }
        },
    },
    async (request, reply) => {
        console.log(" ");
        console.log(
            "Hello, you have subscribed to get information about the topic: " +
                request.query["hub.topic"],
        );
        console.log("The challenge string is: " + request.query["hub.challenge"]);
        return request.query["hub.challenge"];
    },
);

As you can see, we have logged:

• The error in case of a not valid hub.mode or hub.topic,
• A message of successful subscription: "Hello, you have subscribed to get information about the topic".

And replied:

• wIth a 500 error code in case of a validation error,
• with a 200 error code and hub.challenge in string/text format as response.

👍

Congrats!

You have developed the GET listener API.

Update remote repo

Before updating the remote repository, we should add another folder we generated that is types.
Add the following line to .gitignore file.

# Exclude tsc generated files
types/

Now we are ready to push our changes remotely.
Open a terminal pressing Ctrl+Alt+T and digit:

POST /listener

If you skipped the last section, you can clone the repo with the following commands on the terminal and obtain a GET already built:

To run the server, please follow the instructions on this README.md file.

The POST request is the call we will receive after completing the challenge with the GET request.
In this request, we have to handle the headers and the body. We are going to make two schemas.

In the header, there is only one essential and required parameter X-Hub-Signature.
The message signature is generated from the secret provided by the client at subscription time.
It uses the HMAC-SHA256 algorithm from the JSON body.

In the body, there are two parameters:

• topic. The same we have already seen.

🚧

Warning

If you subscribe to all the signals (so ‘*’ as the last word of the topic), you will not receive all the signals in one request but only one signal per request.

• payload. It will consist of:
  • timestamp, a number,
  • deliveryTimestamp, a number,
  • data. It depends on the signal's information that we will receive.

📘

Info

If you want to dive deeper into the signals, visit the Signals guide's page.

e.g.

Body request:

{
	"topic": "vehicle:26c1097a-45d7-4719-a195-595c252a16f7:generic:online",
	"payload": {
		"timestamp": 1610723307931,
		"data": {
			"online": true
		}
	}
}

JSON schema

📘

Info

We suggest you read the JSON schema section of the GET /listener chapter before you proceed on.

1. Header

Add the headers.json file in the schemas folder.
Copy and paste the following schema into the headers.json file.

{
    "title": "Headers Schema",
    "type": "object",
    "properties": {
        "X-Idempotency-Key": { "type": "string"},
        "X-Hub-Signature": { "type": "string"},
        "User-Agent": { "type": "string"},
        "Content-Type": { "type": "string"}
    },
    "additionalProperties": false,
    "required": ["X-Hub-Signature"]
}

2. Body

Add the body.json file in the schemas folder.
Copy and paste the following schema into the body.json file.

{
    "title": "Body Schema",
    "type": "object",
    "properties": {
        "topic": { "type": "string"},
        "payload": { 
            "type": "object",
            "properties": {
                "timestamp": { "type": "number" },
                "deliveryTimestamp": { "type": "number" },
                "data": {
                    "type": "object",
                    "properties": {
                        "latitude": { "type": "number" },
                        "longitude": { "type": "number" },
                        "meters": { "type": "number" },
                        "online": { "type": "boolean" },
                        "percentage": { "type": "number" }
                    }
                }
            },
            "required": ["timestamp", "data"]
        }
    },
    "additionalProperties": false,
    "required": ["topic", "payload"]
}

3. Build

Launch this command on the terminal to generate the interfaces on types directory.

POST request

We are finally ready to update index.ts in order to handle POST requests.

First, let's analyze the headers.

In the headers, the only required field is the signature, so we need to validate it.

The following block of code generates the signature.
You have to add those lines on your index.ts file.

import * as crypto from "crypto";

const generateSignature = (message: string, secret: string, algorithm: string): string => {
    if (!secret) {
        return "";
    }
    const hmac = crypto.createHmac(algorithm, secret);
    hmac.update(message, "utf8");
    return `${algorithm}=${hmac.digest("hex")}`;
};

📘

Info

To get more information, visit the Receiving signals guide's page.

Now we just have to call the function declared above when we validate X-Hub-Signature parameter.

📘

Info

If you want to get more information on HMAC protocol, please visit this page.

Our function needs a message, a secret, and an algorithm.

• The message is the body of the request.
• The secret is the hub.secret chosen at subscription time.
  At this point, we make the assumption that the secret is equal to "A secret of your choice" like in the example in the Receiving signals guide's introduction.
• The algorithm is SHA256, and it can be passed as a string so it will be "sha256".

📘

Info

The function will return a string composed of the algorithm's concatenation on the HMAC signature with = and the real HCMAC signature.

e.g.
sha256=44a464597367490c07d43c6c17eac1f545d203973e98d3a5865c68a62672e158

🚧

Warning

Both the secret and the message have to be encoded in utf-8.

There is no concern with the message parameter, but we have to get the body converted in utf-8 format.
To accomplish this goal, we will use fastify-raw-body module.

📘

Info

To get more information, please visit this page.

Open a terminal and install fastify-raw-body.

Add these lines into the index.ts file.

import fastifyRawBody from "fastify-raw-body";

server.register(require("fastify-raw-body"), {
    field: "rawBody",
    global: false,
    encoding: "utf-8",
    runFirst: true,
});

Now that we can generate the signature, let's see how to handle the body.

Once defined the body's schema, we can see that only the topic parameter has to be validated.
If you read the last part of the GET request section, you can see that we already built the function to validate the topic and its relative signals.

We have everything to define the POST request to the /listener endpoint.

1. Import

Import Schemas and interfaces at the top of index.ts.

// import json schemas as normal
import HeadersSchema from "./schemas/headers.json";
import BodySchema from "./schemas/body.json";

// import the generated interfaces
import { HeadersSchema as HeadersSchemaInterface } from "./types/headers";
import { BodySchema as BodySchemaInterface } from "./types/body";

2. Set

Add those lines to the index.ts to set the POST request.

const signatureErrorMessage = "Signature is not valid";
const SECRET = "A secret of your choice";

enum Algorithm {
    sha256 = "sha256",
}

server.post<{
    Headers: HeadersSchemaInterface;
    Body: BodySchemaInterface;
}>(
    "/listener",
    {
        schema: {
            headers: HeadersSchema,
            body: BodySchema,
        },
        config: {
            rawBody: true,
        },
        preValidation: async (request, reply) => {
            const x_hub_signature = request.headers["x-hub-signature"];
            const signature = generateSignature(
                request.rawBody as string,
                SECRET,
                Algorithm.sha256,
            );
            if (!isTopicValid(request.body.topic)) {
                console.error(topicErrorMessage);
                throw new Error(topicErrorMessage);
            }
            if (x_hub_signature !== signature) {
                console.error(signatureErrorMessage);
                throw new Error(signatureErrorMessage);
            }
        },
    },
    async (request, reply) => {
        console.log(" ");
        console.log("SIGNATURE:         ", request.headers["x-hub-signature"]);
        console.log("VEHICLE:           ", splitted[1]);
        console.log("SIGNAL TYPE:       ", splitted[2]);
        console.log("SIGNAL NAME:       ", splitted[3]);
        if (splitted[3] === "position") {
            console.log("VALUE:             ");
            console.log("   Latitude:       ", request.body.payload.data.latitude);
            console.log("   Longitude:      ", request.body.payload.data.longitude);
        }
        if (splitted[3] === "autonomy_percentage") {
            console.log("VALUE:             ", request.body.payload.data.percentage);
        }
        if (splitted[3] === "autonomy_meters") {
            console.log("VALUE:             ", request.body.payload.data.meters);
        }
        if (splitted[3] === "distance_covered") {
            console.log("VALUE:             ", request.body.payload.data.meters);
        }
        if (splitted[3] === "online") {
            console.log("VALUE:             ", request.body.payload.data.online);
        }
        return {};
    },
);

Here are a few comments on this code.

• We have headers and body schema.
• We added config: { rawBody: true} to enable fastify-raw-body.
• We compute the signature in the prevalidation, calling the function explained before. We validate it with the topic parameter throwing an error, with a 500 code error, in case of mismatch.
• The response is an empty close bracket with a 200 status code, logging the signal's information to the console.

Update remote repo

Now we are ready to push our changes remotely, open the terminal and digit:

👍

Congrats!

Your project is ready to run a subscription.

Secret

If you skipped the last section, you can clone the repo with the following commands on the terminal and obtain a complete listener endpoint already built:

To run the server, please follow the instructions on this README.md file.

We will show you how to configure your secret in Typescript.

1. Install

First, you need to install the dotenv module.

2. Create

Create a file named .env. Add the following line on that file:

SECRET = "A secret of your choice"

3. Import

Import dotenv at the top of index.ts.

import * as dotenv from "dotenv";

4. Update

To complete this part you have to update the SECRET variable already defined in the last section.

dotenv.config();

export const SECRET = process.env.SECRET as string;

📘

Info

dotenv.config() has to be called after the instantiation of Fastify (const server = fastify()).

The last thing you have to do, is to run this line.

👍

Congrats!

You can run the server on the terminal with a different SECRET configured.