Express is a popular unopinionated web framework, written in JavaScript and hosted within the Node.js runtime environment.

Node

Node (or more formally Node.js) is an open-source, cross-platform runtime environment that allows developers to create all kinds of server-side tools and applications in JavaScript.

Express

Express is the most popular Node.js web framework, and is the underlying library for a number of other popular Node.js frameworks. It provides mechanisms to:

• Write handlers for requests with different HTTP verbs at different URL paths (routes).

• Integrate with “view” rendering engines in order to generate responses by inserting data into templates.

• Set common web application settings like the port to use for connecting, and the location of templates that are used for rendering the response.

• Add additional request processing “middleware” at any point within the request handling pipeline.

Is Express opinionated?

Web frameworks often refer to themselves as “opinionated” or “unopinionated”.

Opinionated frameworks are those with opinions about the “right way” to handle any particular task. They often support rapid development in a particular domain (solving problems of a particular type) because the right way to do anything is usually well-understood and well-documented. However they can be less flexible at solving problems outside their main domain, and tend to offer fewer choices for what components and approaches they can use.

Unopinionated frameworks, by contrast, have far fewer restrictions on the best way to glue components together to achieve a goal, or even what components should be used. They make it easier for developers to use the most suitable tools to complete a particular task, albeit at the cost that you need to find those components yourself.

Express is unopinionated. You can insert almost any compatible middleware you like into the request handling chain, in almost any order you like. You can structure the app in one file or multiple files, and using any directory structure. You may sometimes feel that you have too many choices!

Helloworld Express

const express = require("express");
const app = express();
const port = 3000;

app.get("/", function (req, res) {
  res.send("Hello World!");
});

app.listen(port, function () {
  console.log(`Example app listening on port ${port}!`);
});

The first two lines require() (import) the express module and create an Express application. This object, which is traditionally named app, has methods for routing HTTP requests, configuring middleware, rendering HTML views, registering a template engine, and modifying application settings that control how the application behaves (e.g. the environment mode, whether route definitions are case sensitive, etc.)

The middle part of the code (the three lines starting with app.get) shows a route definition. The app.get() method specifies a callback function that will be invoked whenever there is an HTTP GET request with a path (‘/’) relative to the site root. The callback function takes a request and a response object as arguments, and calls send() on the response to return the string “Hello World!”

The final block starts up the server on a specified port (‘3000’) and prints a log comment to the console. With the server running, you could go to localhost:3000 in your browser to see the example response returned.

Import and create modules

A module is a JavaScript library/file that you can import into other code using Node’s require() function. Express itself is a module, as are the middleware and database libraries that we use in our Express applications.

To make objects available outside of a module you just need to expose them as additional properties on the exports object. For example, the square.js module below is a file that exports area() and perimeter() methods:

exports.area = function (width) {
  return width * width;
};
exports.perimeter = function (width) {
  return 4 * width;
};

We can import this module using require(), and then call the exported method(s) as shown:

const square = require("./square"); // Here we require() the name of the file without the (optional) .js file extension
console.log(`The area of a square with a width of 4 is ${square.area(4)}`);

If you want to export a complete object in one assignment instead of building it one property at a time, assign it to module.exports as shown below (you can also do this to make the root of the exports object a constructor or other function):

module.exports = {
  area(width) {
    return width * width;
  },

  perimeter(width) {
    return 4 * width;
  },
};

Use asynchronous APIs

JavaScript code frequently uses asynchronous rather than synchronous APIs for operations that may take some time to complete. A synchronous API is one in which each operation must complete before the next operation can start.

By contrast, an asynchronous API is one in which the API will start an operation and immediately return (before the operation is complete). Once the operation finishes, the API will use some mechanism to perform additional operations. For example, the code below will print out “Second, First” because even though setTimeout() method is called first, and returns immediately, the operation doesn’t complete for several seconds.

setTimeout(function () {
  console.log("First");
}, 3000);
console.log("Second");

Create route handlers

In our Hello World Express example (see above), we defined a (callback) route handler function for HTTP GET requests to the site root (‘/’).

Note: You can use any argument names you like in the callback functions; when the callback is invoked the first argument will always be the request and the second will always be the response. It makes sense to name them such that you can identify the object you’re working with in the body of the callback.

Routes allow you to match particular patterns of characters in a URL, and extract some values from the URL and pass them as parameters to the route handler (as attributes of the request object passed as a parameter).

Often it is useful to group route handlers for a particular part of a site together and access them using a common route-prefix (e.g. a site with a Wiki might have all wiki-related routes in one file and have them accessed with a route prefix of /wiki/). In Express this is achieved by using the express.Router object. For example, we can create our wiki route in a module named wiki.js, and then export the Router object, as shown below:

// wiki.js - Wiki route module

const express = require("express");
const router = express.Router();

// Home page route
router.get("/", function (req, res) {
  res.send("Wiki home page");
});

// About page route
router.get("/about", function (req, res) {
  res.send("About this wiki");
});

module.exports = router;

Note: Adding routes to the Router object is just like adding routes to the app object (as shown previously).

To use the router in our main app file we would then require() the route module (wiki.js), then call use() on the Express application to add the Router to the middleware handling path. The two routes will then be accessible from /wiki/ and /wiki/about/.

const wiki = require("./wiki.js");
// …
app.use("/wiki", wiki);

Use middleware

Middleware is used extensively in Express apps, for tasks from serving static files to error handling, to compressing HTTP responses.

To use third party middleware you first need to install it into your app using npm. For example, to install the morgan HTTP request logger middleware, you’d do this:

npm install morgan

Note: Middleware and routing functions are called in the order that they are declared. For some middleware the order is important (for example if session middleware depends on cookie middleware, then the cookie handler must be added first). It is almost always the case that middleware is called before setting routes, or your route handlers will not have access to functionality added by your middleware.

The only difference between a middleware function and a route handler callback is that middleware functions have a third argument next, which middleware functions are expected to call if they are not that which completes the request cycle (when the middleware function is called, this contains the next function that must be called).

You can add a middleware function to the processing chain for all responses with app.use(), or for a specific HTTP verb using the associated method: app.get(), app.post(), etc. Routes are specified in the same way for both cases, though the route is optional when calling app.use().

const express = require("express");
const app = express();

// An example middleware function
const a_middleware_function = function (req, res, next) {
  // Perform some operations
  next(); // Call next() so Express will call the next middleware function in the chain.
};

// Function added with use() for all routes and verbs
app.use(a_middleware_function);

// Function added with use() for a specific route
app.use("/someroute", a_middleware_function);

// A middleware function added for a specific HTTP verb and route
app.get("/", a_middleware_function);

app.listen(3000);

Serve static files

You can use the express.static middleware to serve static files, including your images, CSS and JavaScript (static() is the only middleware function that is actually part of Express). For example, you would use the line below to serve images, CSS files, and JavaScript files from a directory named ‘public’ at the same level as where you call node:

app.use(express.static("public"));

Any files in the public directory are served by adding their filename (relative to the base “public” directory) to the base URL. So for example:

http://localhost:3000/images/dog.jpg
http://localhost:3000/css/style.css
http://localhost:3000/js/app.js
http://localhost:3000/about.html

You can call static() multiple times to serve multiple directories. If a file cannot be found by one middleware function then it will be passed on to the subsequent middleware (the order that middleware is called is based on your declaration order).

app.use(express.static("public"));
app.use(express.static("media"));

You can also create a virtual prefix for your static URLs, rather than having the files added to the base URL. For example, here we specify a mount path so that the files are loaded with the prefix “/media”:

app.use("/media", express.static("public"));

Handling errors

Errors are handled by one or more special middleware functions that have four arguments, instead of the usual three: (err, req, res, next). For example:

app.use(function (err, req, res, next) {
  console.error(err.stack);
  res.status(500).send("Something broke!");
});

These can return any content required, but must be called after all other app.use() and routes calls so that they are the last middleware in the request handling process!

Express comes with a built-in error handler, which takes care of any remaining errors that might be encountered in the app. This default error-handling middleware function is added at the end of the middleware function stack. If you pass an error to next() and you do not handle it in an error handler, it will be handled by the built-in error handler; the error will be written to the client with the stack trace.

Use databases

Express apps can use any database mechanism supported by Node (Express itself doesn’t define any specific additional behavior/requirements for database management). There are many options, including PostgreSQL, MySQL, Redis, SQLite, MongoDB, etc.

This works with older versions of MongoDB version ~ 2.2.33:

const MongoClient = require("mongodb").MongoClient;

MongoClient.connect("mongodb://localhost:27017/animals", (err, db) => {
  if (err) throw err;

  db.collection("mammals")
    .find()
    .toArray((err, result) => {
      if (err) throw err;

      console.log(result);
    });
});

For MongoDB version 3.0 and up:

const MongoClient = require("mongodb").MongoClient;

MongoClient.connect("mongodb://localhost:27017/animals", (err, client) => {
  if (err) throw err;

  const db = client.db("animals");
  db.collection("mammals")
    .find()
    .toArray((err, result) => {
      if (err) throw err;
      console.log(result);
      client.close();
    });
});

Another popular approach is to access your database indirectly, via an Object Relational Mapper (“ORM”). In this approach you define your data as “objects” or “models” and the ORM maps these through to the underlying database format. This approach has the benefit that as a developer you can continue to think in terms of JavaScript objects rather than database semantics, and that there is an obvious place to perform validation and checking of incoming data.

Render data (views)

Template engines (also referred to as “view engines” in Express) allow you to specify the structure of an output document in a template, using placeholders for data that will be filled in when a page is generated. Templates are often used to create HTML, but can also create other types of documents.

Express has support for a number of template engines, notably Pug (formerly “Jade”), Mustache, and EJS. Each has its own strengths for addressing particular use cases (relative comparisons can easily be found via Internet search). The Express application generator uses Jade as its default, but it also supports several others.

In your application settings code you set the template engine to use and the location where Express should look for templates using the ‘views’ and ‘view engine’ settings, as shown below (you will also have to install the package containing your template library too!)

const express = require("express");
const path = require("path");
const app = express();

// Set directory to contain the templates ('views')
app.set("views", path.join(__dirname, "views"));

// Set view engine to use, in this case 'some_template_engine_name'
app.set("view engine", "some_template_engine_name");

The appearance of the template will depend on what engine you use. Assuming that you have a template file named “index." that contains placeholders for data variables named 'title' and "message", you would call Response.render() in a route handler function to create and send the HTML response:

app.get("/", function (req, res) {
  res.render("index", { title: "About dogs", message: "Dogs rock!" });
});

File structure

Express makes no assumptions in terms of structure or what components you use. Routes, views, static files, and other application-specific logic can live in any number of files with any directory structure. While it is perfectly possible to have the whole Express application in one file, typically it makes sense to split your application into files based on function (e.g. account management, blogs, discussion boards) and architectural problem domain (e.g. model, view or controller if you happen to be using an MVC architecture).

In a later topic we’ll use the Express Application Generator, which creates a modular app skeleton that we can easily extend for creating web applications.

Set up a Node development environment

Add dependencies

The following steps show how you can use npm to download a package, save it into the project dependencies, and then require it in a Node application.

Use the npm init command to create a package.json file for your application. This command prompts you for a number of things, including the name and version of your application and the name of the initial entry point file (by default this is index.js). For now, just accept the defaults:

npm init

If you display the package.json file (cat package.json), you will see the defaults that you accepted, ending with the license.

{
  "name": "myapp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Now install Express in the myapp directory and save it in the dependencies list of your package.json file:

 npm install express
The dependencies section of your package.json will now appear at the end of the package.json file and will include Express.

{
  "name": "myapp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  }
}

You can start the server by calling node with the script in your command prompt:

node index.js

Development dependencies

If a dependency is only used during development, you should instead save it as a “development dependency” (so that your package users don’t have to install it in production). For example, to use the popular JavaScript Linting tool ESLint you would call npm as shown:

npm install eslint --save-dev

The following entry would then be added to your application’s package.json:

  "devDependencies": {
    "eslint": "^7.10.0"
  }

Running tasks

In addition to defining and fetching dependencies you can also define named scripts in your package.json files and call npm to execute them with the run-script command. This approach is commonly used to automate running tests and parts of the development or build toolchain (e.g., running tools to minify JavaScript, shrink images, LINT/analyze your code, etc.).

For example, to define a script to run the eslint development dependency that we specified in the previous section we might add the following script block to our package.json file (assuming that our application source is in a folder /src/js):

"scripts": {
  // …
  "lint": "eslint src/js"
  // …
}

To explain a little further, eslint src/js is a command that we could enter in our terminal/command line to run eslint on JavaScript files contained in the src/js directory inside our app directory. Including the above inside our app’s package.json file provides a shortcut for this command — lint.

We would then be able to run eslint using npm by calling:

npm run-script lint
# OR (using the alias)
npm run lint

Instal the Express Application Generator

The Express Application Generator tool generates an Express application “skeleton”. Install the generator using npm as shown:

npm install express-generator -g

Note: You may need to prefix this line with sudo on Ubuntu or macOS. The -g flag installs the tool globally so that you can call it from anywhere.

To create an Express app named “helloworld” with the default settings, navigate to where you want to create it and run the app as shown:

express helloworld

Note: Unless you’re using an old nodejs version (< 8.2.0), you could alternatively skip the installation and run express-generator with npx. This has the same effect as installing and then running express-generator but does not install the package on your system:

npx express-generator helloworld

The generator will create the new Express app in a sub folder of your current location, displaying build progress on the console. On completion, the tool will display the commands you need to enter to install the Node dependencies and start the app.

The new app will have a package.json file in its root directory. You can open this to see what dependencies are installed, including Express and the template library Jade:

{
  "name": "helloworld",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "start": "node ./bin/www"
  },
  "dependencies": {
    "cookie-parser": "~1.4.4",
    "debug": "~2.6.9",
    "express": "~4.16.1",
    "http-errors": "~1.6.3",
    "jade": "~1.11.0",
    "morgan": "~1.9.1"
  }
}

Install all the dependencies for the helloworld app using npm as shown:

cd helloworld
npm install

Then run the app (the commands are slightly different for Windows and Linux/macOS), as shown below:

# Run helloworld on Windows with Command Prompt
SET DEBUG=helloworld:* & npm start

# Run helloworld on Windows with PowerShell
SET DEBUG=helloworld:* | npm start

# Run helloworld on Linux/macOS
DEBUG=helloworld:* npm start

The DEBUG command creates useful logging, resulting in an output like the following:

>SET DEBUG=helloworld:* & npm start

> helloworld@0.0.0 start D:\GitHub\expresstests\helloworld
> node ./bin/www

  helloworld:server Listening on port 3000 +0ms

Create a skeleton website

Note: The Express Application Generator is not the only generator for Express applications, and the generated project is not the only viable way to structure your files and directories. The generated site does however have a modular structure that is easy to extend and understand.

Note: The Express Application Generator declares most variables using var. We have changed most of these to const (and a few to let) in the tutorial, because we want to demonstrate modern JavaScript practice.

What view engine should I use?

The Express Application Generator allows you to configure a number of popular view/templating engines, including EJS, Hbs, Pug (Jade), Twig, and Vash, although it chooses Jade by default if you don’t specify a view option. Express itself can also support a large number of other templating languages out of the box.

Generally speaking, you should select a templating engine that delivers all the functionality you need and allows you to be productive sooner — or in other words, in the same way that you choose any other component! Some of the things to consider when comparing template engines:

• Time to productivity — If your team already has experience with a templating language then it is likely they will be productive faster using that language. If not, then you should consider the relative learning curve for candidate templating engines.

• Popularity and activity — Review the popularity of the engine and whether it has an active community. It is important to be able to get support when problems arise throughout the lifetime of the website.

• Style — Some template engines use specific markup to indicate inserted content within “ordinary” HTML, while others construct the HTML using a different syntax (for example, using indentation and block names).

• Performance/rendering time.

• Features — you should consider whether the engines you look at have the following features available:

  • Layout inheritance: Allows you to define a base template and then “inherit” just the parts of it that you want to be different for a particular page. This is typically a better approach than building templates by including a number of required components or building a template from scratch each time.

  • “Include” support: Allows you to build up templates by including other templates.

  • Concise variable and loop control syntax.

  • Ability to filter variable values at template level, such as making variables upper-case, or formatting a date value.

  • Ability to generate output formats other than HTML, such as JSON or XML.

  • Support for asynchronous operations and streaming.

  • Client-side features. If a templating engine can be used on the client this allows the possibility of having all or most of the rendering done client-side.

  For this project, we’ll use the Pug templating engine (this is the recently-renamed Jade engine), as this is one of the most popular Express/JavaScript templating languages and is supported out of the box by the generator.

What CSS stylesheet engine should I use?

The Express Application Generator allows you to create a project that is configured to use the most common CSS stylesheet engines: LESS, SASS, Compass, Stylus.

Note: CSS has some limitations that make certain tasks difficult. CSS stylesheet engines allow you to use more powerful syntax for defining your CSS and then compile the definition into plain-old CSS for browsers to use.

As with templating engines, you should use the stylesheet engine that will allow your team to be most productive. For this project, we’ll use vanilla CSS (the default) as our CSS requirements are not sufficiently complicated to justify using anything else.

Create the project

For the sample Local Library app we’re going to build, we’ll create a project named express-locallibrary-tutorial using the Pug template library and no CSS engine.

First, navigate to where you want to create the project and then run the Express Application Generator in the command prompt as shown:

express express-locallibrary-tutorial --view=pug

The generator will create (and list) the project’s files.

   create : express-locallibrary-tutorial\
   create : express-locallibrary-tutorial\public\
   create : express-locallibrary-tutorial\public\javascripts\
   create : express-locallibrary-tutorial\public\images\
   create : express-locallibrary-tutorial\public\stylesheets\
   create : express-locallibrary-tutorial\public\stylesheets\style.css
   create : express-locallibrary-tutorial\routes\
   create : express-locallibrary-tutorial\routes\index.js
   create : express-locallibrary-tutorial\routes\users.js
   create : express-locallibrary-tutorial\views\
   create : express-locallibrary-tutorial\views\error.pug
   create : express-locallibrary-tutorial\views\index.pug
   create : express-locallibrary-tutorial\views\layout.pug
   create : express-locallibrary-tutorial\app.js
   create : express-locallibrary-tutorial\package.json
   create : express-locallibrary-tutorial\bin\
   create : express-locallibrary-tutorial\bin\www

   change directory:
     > cd express-locallibrary-tutorial

   install dependencies:
     > npm install

   run the app (Bash (Linux or macOS))
     > DEBUG=express-locallibrary-tutorial:* npm start

   run the app (PowerShell (Windows))
     > $ENV:DEBUG = "express-locallibrary-tutorial:*"; npm start

   run the app (Command Prompt (Windows)):
     > SET DEBUG=express-locallibrary-tutorial:* & npm start

Note: The generator-created files define all variables as var. Open all of the generated files and change the var declarations to const before you continue (the remainder of the tutorial assumes that you have done so).

Enable server restart on file changes

Any changes you make to your Express website are currently not visible until you restart the server. It quickly becomes very irritating to have to stop and restart your server every time you make a change, so it is worth taking the time to automate restarting the server when needed.

A convenient tool for this purpose is nodemon. This is usually installed globally (as it is a “tool”).

npm install -g nodemon

Because the tool isn’t installed globally we can’t launch it from the command line (unless we add it to the path) but we can call it from an npm script because npm knows all about the installed packages. Find the scripts section of your package.json. Initially, it will contain one line, which begins with “start”. Update it by putting a comma at the end of that line, and adding the “devstart” and “serverstart” lines:

On Linux and macOS, the scripts section will look like this:

  "scripts": {
    "start": "node ./bin/www",
    "devstart": "nodemon ./bin/www",
    "serverstart": "DEBUG=express-locallibrary-tutorial:* npm run devstart"
  },

On Windows, the “serverstart” value would instead look like this (if using the command prompt):

"serverstart": "SET DEBUG=express-locallibrary-tutorial:* & npm run devstart"

We can now start the server in almost exactly the same way as previously, but using the devstart command.

Note: Now if you edit any file in the project the server will restart (or you can restart it by typing rs on the command prompt at any time). You will still need to reload the browser to refresh the page.

We now have to call “npm run " rather than just npm start, because "start" is actually an npm command that is mapped to the named script. We could have replaced the command in the start script but we only want to use nodemon during development, so it makes sense to create a new script command.

The serverstart command added to the scripts in the package.json above is a very good example. Using this approach means you no longer have to type a long command to start the server. Note that the particular command added to the script works for macOS or Linux only.

www file

The file /bin/www is the application entry point! The very first thing this does is require() the “real” application entry point (app.js, in the project root) that sets up and returns the express() application object. require() is the CommonJS way to import JavaScript code, JSON, and other files into the current file. Here we specify app.js module using a relative path and omit the optional (.js) file extension.

#!/usr/bin/env node

/**
 * Module dependencies.
 */

const app = require("../app");

app.js

This file creates an express application object (named app, by convention), sets up the application with various settings and middleware, and then exports the app from the module. The code below shows just the parts of the file that create and export the app object:

const express = require("express");
const app = express();
// …
module.exports = app;

Back in the www entry point file above, it is this module.exports object that is supplied to the caller when this file is imported.

Let’s work through the app.js file in detail. First, we import some useful node libraries into the file using require(), including http-errors, express, morgan and cookie-parser that we previously downloaded for our application using npm; and path, which is a core Node library for parsing file and directory paths.

const createError = require("http-errors");
const express = require("express");
const path = require("path");
const cookieParser = require("cookie-parser");
const logger = require("morgan");

Then we require() modules from our routes directory. These modules/files contain code for handling particular sets of related “routes” (URL paths). When we extend the skeleton application, for example to list all books in the library, we will add a new file for dealing with book-related routes.

const indexRouter = require("./routes/index");
const usersRouter = require("./routes/users");

Next, we create the app object using our imported express module, and then use it to set up the view (template) engine. There are two parts to setting up the engine. First, we set the ‘views’ value to specify the folder where the templates will be stored (in this case the subfolder /views). Then we set the ‘view engine’ value to specify the template library (in this case “pug”).

const app = express();

// view engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "pug");

The next set of functions call app.use() to add the middleware libraries that we imported above into the request handling chain. For example, express.json() and express.urlencoded() are needed to populate req.body with the form fields. After these libraries we also use the express.static middleware, which makes Express serve all the static files in the /public directory in the project root.

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());

app.use(express.static(path.join(__dirname, "public")));

Now that all the other middleware is set up, we add our (previously imported) route-handling code to the request handling chain. The imported code will define particular routes for the different parts of the site:

app.use("/", indexRouter);
app.use("/users", usersRouter);

The last middleware in the file adds handler methods for errors and HTTP 404 responses.

// catch 404 and forward to error handler
app.use((req, res, next) => {
  next(createError(404));
});

// error handler
app.use((err, req, res, next) => {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get("env") === "development" ? err : {};

  // render the error page
  res.status(err.status || 500);
  res.render("error");
});

Routes

The route file /routes/users.js is shown below (route files share a similar structure, so we don’t need to also show index.js). First, it loads the express module and uses it to get an express.Router object. Then it specifies a route on that object and lastly exports the router from the module (this is what allows the file to be imported into app.js).

const express = require("express");
const router = express.Router();

/* GET users listing. */
router.get("/", (req, res, next) => {
  res.send("respond with a resource");
});

module.exports = router;

The route defines a callback that will be invoked whenever an HTTP GET request with the correct pattern is detected. The matching pattern is the route specified when the module is imported (‘/users’) plus whatever is defined in this file (‘/’). In other words, this route will be used when a URL of /users/ is received.

One thing of interest above is that the callback function has the third argument ‘next’, and is hence a middleware function rather than a simple route callback. While the code doesn’t currently use the next argument, it may be useful in the future if you want to add multiple route handlers to the ‘/’ route path.

Views (templates)

The views (templates) are stored in the /views directory (as specified in app.js) and are given the file extension .pug. The method Response.render() is used to render a specified template along with the values of named variables passed in an object, and then send the result as a response. In the code below from /routes/index.js you can see how that route renders a response using the template “index” passing the template variable “title”.

/* GET home page. */
router.get("/", (req, res, next) => {
  res.render("index", { title: "Express" });
});

The corresponding template for the above route is given below (index.pug). We’ll talk more about the syntax later. All you need to know for now is that the title variable (with value ‘Express’) is inserted where specified in the template.

extends layout

block content
  h1= title
  p Welcome to #{title}

Use a Database (with Mongoose)

What is the best way to interact with a database?

There are two common approaches for interacting with a database:

• Using the databases’ native query language, such as SQL.

• Using an Object Relational Mapper (“ORM”). An ORM represents the website’s data as JavaScript objects, which are then mapped to the underlying database. Some ORMs are tied to a specific database, while others provide a database-agnostic backend.

The very best performance can be gained by using SQL, or whatever query language is supported by the database. ODM’s are often slower because they use translation code to map between objects and the database format, which may not use the most efficient database queries (this is particularly true if the ODM supports different database backends, and must make greater compromises in terms of what database features are supported).

The benefit of using an ORM is that programmers can continue to think in terms of JavaScript objects rather than database semantics — this is particularly true if you need to work with different databases (on either the same or different websites). They also provide an obvious place to perform data validation.

Note: Using ODM/ORMs often results in lower costs for development and maintenance! Unless you’re very familiar with the native query language or performance is paramount, you should strongly consider using an ODM.

Use Mongoose and MongoDB for the LocalLibrary

For the Local Library example (and the rest of this topic) we’re going to use the Mongoose ODM to access our library data. Mongoose acts as a front end to MongoDB, an open source NoSQL database that uses a document-oriented data model. A “collection” of “documents” in a MongoDB database is analogous to a “table” of “rows” in a relational database.

This ODM and database combination is extremely popular in the Node community, partially because the document storage and query system looks very much like JSON, and is hence familiar to JavaScript developers.

Database APIs are asynchronous

Database methods to create, find, update, or delete records are asynchronous. What this means is that the methods return immediately, and the code to handle the success or failure of the method runs at a later time when the operation completes. Other code can execute while the server is waiting for the database operation to complete, so the server can remain responsive to other requests.

JavaScript has a number of mechanisms for supporting asynchronous behavior. Historically JavaScript relied heavily on passing callback functions to asynchronous methods to handle the success and error cases. In modern JavaScript callbacks have largely been replaced by Promises. Promises are objects that are (immediately) returned by an asynchronous method that represent its future state. When the operation completes, the promise object is “settled”, and resolves an object that represents the result of the operation or an error.

There are two main ways you can use promises to run code when a promise is settled, and we highly recommend that you read How to use promises for a high level overview of both approaches. In this tutorial, we’ll primarily be using await to wait on promise completion within an async function, because this leads to more readable and understandable asynchronous code.

The way this approach works is that you use the async function keyword to mark a function as asynchronous, and then inside that function apply await to any method that returns a promise. When the asynchronous function is executed its operation is paused at the first await method until the promise settles. From the perspective of the surrounding code the asynchronous function then returns and the code after it is able to run. Later when the promise settles, the await method inside the asynchronous function returns with the result, or an error is thrown if the promise was rejected. The code in the asynchronous function then executes until either another await is encountered, at which point it will pause again, or until all the code in the function has been run.

You can see how this works in the example below. myFunction() is an asynchronous function that is called within a try…catch block. When myFunction() is run, code execution is paused at methodThatReturnsPromise() until the promise resolves, at which point the code continues to aFunctionThatReturnsPromise() and waits again. The code in the catch block runs if an error is thrown in the asynchronous function, and this will happen if the promise returned by either of the methods is rejected.

async function myFunction {
  // ...
  await someObject.methodThatReturnsPromise();
  // ...
  await aFunctionThatReturnsPromise();
  // ...
}

try {
  // ...
  myFunction();
  // ...
} catch (e) {
 // error handling code
}

The asynchronous methods above are run in sequence. If the methods don’t depend on each other then you can run them in parallel and finish the whole operation more quickly. This is done using the Promise.all() method, which takes an iterable of promises as input and returns a single Promise. This returned promise fulfills when all of the input’s promises fulfill, with an array of the fulfillment values. It rejects when any of the input’s promises rejects, with this first rejection reason.

The code below shows how this works. First, we have two functions that return promises. We await on both of them to complete using the promise returned by Promise.all(). Once they both complete await returns and the results array is populated, the function then continues to the next await, and waits until the promise returned by anotherFunctionThatReturnsPromise() is settled. You would call the myFunction() in a try…catch block to catch any errors.

async function myFunction {
  // ...
  const [resultFunction1, resultFunction2] = await Promise.all([
     functionThatReturnsPromise1(),
     functionThatReturnsPromise2()
  ]);
  // ...
  await anotherFunctionThatReturnsPromise(resultFunction1);
}

Mongoose primer

This section provides an overview of how to connect Mongoose to a MongoDB database, how to define a schema and a model, and how to make basic queries.

Connect to MongoDB

Mongoose requires a connection to a MongoDB database. You can require() and connect to a locally hosted database with mongoose.connect() as shown below (for the tutorial we’ll instead connect to an internet-hosted database).

// Import the mongoose module
const mongoose = require("mongoose");

// Set `strictQuery: false` to globally opt into filtering by properties that aren't in the schema
// Included because it removes preparatory warnings for Mongoose 7.
// See: https://mongoosejs.com/docs/migrating_to_6.html#strictquery-is-removed-and-replaced-by-strict
mongoose.set("strictQuery", false);

// Define the database URL to connect to.
const mongoDB = "mongodb://127.0.0.1/my_database";

// Wait for database to connect, logging an error if there is a problem
main().catch((err) => console.log(err));
async function main() {
  await mongoose.connect(mongoDB);
}

You can get the default Connection object with mongoose.connection. If you need to create additional connections you can use mongoose.createConnection(). This takes the same form of database URI (with host, database, port, options, etc.) as connect() and returns a Connection object. Note that createConnection() returns immediately; if you need to wait on the connection to be established you can call it with asPromise() to return a promise (mongoose.createConnection(mongoDB).asPromise()).

Define and create models

Models are defined using the Schema interface. The Schema allows you to define the fields stored in each document along with their validation requirements and default values. In addition, you can define static and instance helper methods to make it easier to work with your data types, and also virtual properties that you can use like any other field, but which aren’t actually stored in the database (we’ll discuss a bit further below).

Schemas are then “compiled” into models using the mongoose.model() method. Once you have a model you can use it to find, create, update, and delete objects of the given type.

Note: Each model maps to a collection of documents in the MongoDB database. The documents will contain the fields/schema types defined in the model Schema.

Define schemas

The code fragment below shows how you might define a simple schema. First you require() mongoose, then use the Schema constructor to create a new schema instance, defining the various fields inside it in the constructor’s object parameter.

// Require Mongoose
const mongoose = require("mongoose");

// Define a schema
const Schema = mongoose.Schema;

const SomeModelSchema = new Schema({
  a_string: String,
  a_date: Date,
});

Create a model

Models are created from schemas using the mongoose.model() method:

// Define schema
const Schema = mongoose.Schema;

const SomeModelSchema = new Schema({
  a_string: String,
  a_date: Date,
});

// Compile model from schema
const SomeModel = mongoose.model("SomeModel", SomeModelSchema);

Schema types (fields)

A schema can have an arbitrary number of fields — each one represents a field in the documents stored in MongoDB. An example schema showing many of the common field types and how they are declared is shown below.

const schema = new Schema({
  name: String,
  binary: Buffer,
  living: Boolean,
  updated: { type: Date, default: Date.now() },
  age: { type: Number, min: 18, max: 65, required: true },
  mixed: Schema.Types.Mixed,
  _someId: Schema.Types.ObjectId,
  array: [],
  ofString: [String], // You can also have an array of each of the other types too.
  nested: { stuff: { type: String, lowercase: true, trim: true } },
});

Most of the SchemaTypes (the descriptors after “type:” or after field names) are self-explanatory. The exceptions are:

• ObjectId: Represents specific instances of a model in the database. For example, a book might use this to represent its author object. This will actually contain the unique ID (_id) for the specified object. We can use the populate() method to pull in the associated information when needed.

• Mixed: An arbitrary schema type.

• []: An array of items. You can perform JavaScript array operations on these models (push, pop, unshift, etc.). The examples above show an array of objects without a specified type and an array of String objects, but you can have an array of any type of object.

The code also shows both ways of declaring a field:

• Field name and type as a key-value pair (i.e. as done with fields name, binary and living).

• Field name followed by an object defining the type, and any other options for the field. Options include things like:

  • default values.

  • built-in validators (e.g. max/min values) and custom validation functions.

  • Whether the field is required

  • Whether String fields should automatically be set to lowercase, uppercase, or trimmed (e.g. { type: String, lowercase: true, trim: true })

Validation

Mongoose provides built-in and custom validators, and synchronous and asynchronous validators. It allows you to specify both the acceptable range of values and the error message for validation failure in all cases.

The built-in validators include:

• All SchemaTypes have the built-in required validator. This is used to specify whether the field must be supplied in order to save a document.

• Numbers have min and max validators.

• Strings have:

  • enum: specifies the set of allowed values for the field.

  • match: specifies a regular expression that the string must match.

  • maxLength and minLength for the string.

The example below (slightly modified from the Mongoose documents) shows how you can specify some of the validator types and error messages:

const breakfastSchema = new Schema({
  eggs: {
    type: Number,
    min: [6, "Too few eggs"],
    max: 12,
    required: [true, "Why no eggs?"],
  },
  drink: {
    type: String,
    enum: ["Coffee", "Tea", "Water"],
  },
});

Virtual properties

Virtual properties are document properties that you can get and set but that do not get persisted to MongoDB. The getters are useful for formatting or combining fields, while setters are useful for de-composing a single value into multiple values for storage. The example in the documentation constructs (and deconstructs) a full name virtual property from a first and last name field, which is easier and cleaner than constructing a full name every time one is used in a template.

Note: We will use a virtual property in the library to define a unique URL for each model record using a path and the record’s _id value.

Methods and query helpers

A schema can also have instance methods, static methods, and query helpers. The instance and static methods are similar, but with the obvious difference that an instance method is associated with a particular record and has access to the current object. Query helpers allow you to extend mongoose’s chainable query builder API (for example, allowing you to add a query “byName” in addition to the find(), findOne() and findById() methods).

Use models

Once you’ve created a schema you can use it to create models. The model represents a collection of documents in the database that you can search, while the model’s instances represent individual documents that you can save and retrieve.

Create and modify documents

To create a record you can define an instance of the model and then call save() on it. The examples below assume SomeModel is a model (with a single field name) that we have created from our schema.

// Create an instance of model SomeModel
const awesome_instance = new SomeModel({ name: "awesome" });

// Save the new model instance asynchronously
await awesome_instance.save();

You can also use create() to define the model instance at the same time as you save it. Below we create just one, but you can create multiple instances by passing in an array of objects.

await SomeModel.create({ name: "also_awesome" });

Every model has an associated connection (this will be the default connection when you use mongoose.model()). You create a new connection and call .model() on it to create the documents on a different database.

You can access the fields in this new record using the dot syntax, and change the values. You have to call save() or update() to store modified values back to the database.

// Access model field values using dot notation
console.log(awesome_instance.name); //should log 'also_awesome'

// Change record by modifying the fields, then calling save().
awesome_instance.name = "New cool name";
await awesome_instance.save();

Search for records

You can search for records using query methods, specifying the query conditions as a JSON document. The code fragment below shows how you might find all athletes in a database that play tennis, returning just the fields for athlete name and age. Here we just specify one matching field (sport) but you can add more criteria, specify regular expression criteria, or remove the conditions altogether to return all athletes.

const Athlete = mongoose.model("Athlete", yourSchema);

// find all athletes who play tennis, returning the 'name' and 'age' fields
const tennisPlayers = await Athlete.find(
  { sport: "Tennis" },
  "name age",
).exec();

Query APIs, such as find(), return a variable of type Query. You can use a query object to build up a query in parts before executing it with the exec() method. exec() executes the query and returns a promise that you can await on for the result.

// find all athletes that play tennis
const query = Athlete.find({ sport: "Tennis" });

// selecting the 'name' and 'age' fields
query.select("name age");

// limit our results to 5 items
query.limit(5);

// sort by age
query.sort({ age: -1 });

// execute the query at a later time
query.exec();

Above we’ve defined the query conditions in the find() method. We can also do this using a where() function, and we can chain all the parts of our query together using the dot operator (.) rather than adding them separately. The code fragment below is the same as our query above, with an additional condition for the age.

Athlete.find()
  .where("sport")
  .equals("Tennis")
  .where("age")
  .gt(17)
  .lt(50) // Additional where query
  .limit(5)
  .sort({ age: -1 })
  .select("name age")
  .exec();

The find() method gets all matching records, but often you just want to get one match. The following methods query for a single record:

• findById(): Finds the document with the specified id (every document has a unique id).

• findOne(): Finds a single document that matches the specified criteria.

• findByIdAndDelete(), findByIdAndUpdate(), findOneAndRemove(), findOneAndUpdate(): Finds a single document by id or criteria and either updates or removes it. These are useful convenience functions for updating and removing records.

Working with related documents — population

You can create references from one document/model instance to another using the ObjectId schema field, or from one document to many using an array of ObjectIds. The field stores the id of the related model. If you need the actual content of the associated document, you can use the populate() method in a query to replace the id with the actual data.

For example, the following schema defines authors and stories. Each author can have multiple stories, which we represent as an array of ObjectId. Each story can have a single author. The ref property tells the schema which model can be assigned to this field.

const mongoose = require("mongoose");

const Schema = mongoose.Schema;

const authorSchema = new Schema({
  name: String,
  stories: [{ type: Schema.Types.ObjectId, ref: "Story" }],
});

const storySchema = new Schema({
  author: { type: Schema.Types.ObjectId, ref: "Author" },
  title: String,
});

const Story = mongoose.model("Story", storySchema);
const Author = mongoose.model("Author", authorSchema);

We can save our references to the related document by assigning the _id value. Below we create an author, then a story, and assign the author id to our story’s author field.

const bob = new Author({ name: "Bob Smith" });

await bob.save();

// Bob now exists, so lets create a story
const story = new Story({
  title: "Bob goes sledding",
  author: bob._id, // assign the _id from our author Bob. This ID is created by default!
});

await story.save();

Note: One great benefit of this style of programming is that we don’t have to complicate the main path of our code with error checking. If any of the save() operations fail, the promise will reject and an error will be thrown. Our error handling code deals with that separately (usually in a catch() block), so the intent of our code is very clear.

Our story document now has an author referenced by the author document’s ID. In order to get the author information in the story results we use populate(), as shown below.

Story.findOne({ title: "Bob goes sledding" })
  .populate("author") // Replace the author id with actual author information in results
  .exec();

Note: Astute readers will have noted that we added an author to our story, but we didn’t do anything to add our story to our author’s stories array. How then can we get all stories by a particular author? One way would be to add our story to the stories array, but this would result in us having two places where the information relating authors and stories needs to be maintained.

A better way is to get the _id of our author, then use find() to search for this in the author field across all stories.

Story.find({ author: bob._id }).exec();  

One schema/model per file

While you can create schemas and models using any file structure you like, we highly recommend defining each model schema in its own module (file), then exporting the method to create the model. This is shown below:

// File: ./models/somemodel.js

// Require Mongoose
const mongoose = require("mongoose");

// Define a schema
const Schema = mongoose.Schema;

const SomeModelSchema = new Schema({
  a_string: String,
  a_date: Date,
});

// Export function to create "SomeModel" model class
module.exports = mongoose.model("SomeModel", SomeModelSchema);

You can then require and use the model immediately in other files. Below we show how you might use it to get all instances of the model.

// Create a SomeModel model just by requiring the module
const SomeModel = require("../models/somemodel");

// Use the SomeModel object (model) to find all SomeModel records
const modelInstances = await SomeModel.find().exec();

Connect to MongoDB

Open /app.js (in the root of your project) and copy the following text below where you declare the Express application object (after the line const app = express();). Replace the database URL string (‘insert_your_database_url_here’) with the location URL representing your own database (i.e. using the information from MongoDB Atlas).

// Set up mongoose connection
const mongoose = require("mongoose");
mongoose.set("strictQuery", false);
const mongoDB = "insert_your_database_url_here";

main().catch((err) => console.log(err));
async function main() {
  await mongoose.connect(mongoDB);
}

As discussed in the Mongoose primer above, this code creates the default connection to the database and reports any errors to the console.

Note that hard-coding database credentials in source code as shown above is not recommended. We do it here because it shows the core connection code, and because during development there is no significant risk that leaking these details will expose or corrupt sensitive information. We’ll show you how to do this more safely when deploying to production!

Define the LocalLibrary Schema

We will define a separate module for each model, as discussed above. Start by creating a folder for our models in the project root (/models) and then create separate files for each of the models:

/express-locallibrary-tutorial  // the project root
  /models
    author.js
    book.js
    bookinstance.js
    genre.js

Author model

Copy the Author schema code shown below and paste it into your ./models/author.js file. The schema defines an author as having String SchemaTypes for the first and family names (required, with a maximum of 100 characters), and Date fields for the dates of birth and death.

const mongoose = require("mongoose");

const Schema = mongoose.Schema;

const AuthorSchema = new Schema({
  first_name: { type: String, required: true, maxLength: 100 },
  family_name: { type: String, required: true, maxLength: 100 },
  date_of_birth: { type: Date },
  date_of_death: { type: Date },
});

// Virtual for author's full name
AuthorSchema.virtual("name").get(function () {
  // To avoid errors in cases where an author does not have either a family name or first name
  // We want to make sure we handle the exception by returning an empty string for that case
  let fullname = "";
  if (this.first_name && this.family_name) {
    fullname = `${this.family_name}, ${this.first_name}`;
  }

  return fullname;
});

// Virtual for author's URL
AuthorSchema.virtual("url").get(function () {
  // We don't use an arrow function as we'll need the this object
  return `/catalog/author/${this._id}`;
});

// Export model
module.exports = mongoose.model("Author", AuthorSchema);

We’ve also declared a virtual for the AuthorSchema named “url” that returns the absolute URL required to get a particular instance of the model — we’ll use the property in our templates whenever we need to get a link to a particular author.

Hoisting is a concept where a variable or function is lifted to the top of its global or local scope before the whole code is executed. This makes it possible for such a variable/function to be accessed before initialization.

All functions and variables in JavaScript are hoisted, but only declared functions can be accessed before initialization.

If you want to create constructors, retain the normal behavior of this or have your functions hoisted, then arrow functions are not the right approach.

Book model

Copy the Book schema code shown below and paste it into your ./models/book.js file. Most of this is similar to the author model — we’ve declared a schema with a number of string fields and a virtual for getting the URL of specific book records, and we’ve exported the model.

const mongoose = require("mongoose");

const Schema = mongoose.Schema;

const BookSchema = new Schema({
  title: { type: String, required: true },
  author: { type: Schema.Types.ObjectId, ref: "Author", required: true },
  summary: { type: String, required: true },
  isbn: { type: String, required: true },
  genre: [{ type: Schema.Types.ObjectId, ref: "Genre" }],
});

// Virtual for book's URL
BookSchema.virtual("url").get(function () {
  // We don't use an arrow function as we'll need the this object
  return `/catalog/book/${this._id}`;
});

// Export model
module.exports = mongoose.model("Book", BookSchema);

The main difference here is that we’ve created two references to other models:

• author is a reference to a single Author model object, and is required.

• genre is a reference to an array of Genre model objects. We haven’t declared this object yet!

BookInstance model

Finally, copy the BookInstance schema code shown below and paste it into your ./models/bookinstance.js file. The BookInstance represents a specific copy of a book that someone might borrow and includes information about whether the copy is available, on what date it is expected back, and “imprint” (or version) details.

const mongoose = require("mongoose");

const Schema = mongoose.Schema;

const BookInstanceSchema = new Schema({
  book: { type: Schema.Types.ObjectId, ref: "Book", required: true }, // reference to the associated book
  imprint: { type: String, required: true },
  status: {
    type: String,
    required: true,
    enum: ["Available", "Maintenance", "Loaned", "Reserved"],
    default: "Maintenance",
  },
  due_back: { type: Date, default: Date.now },
});

// Virtual for bookinstance's URL
BookInstanceSchema.virtual("url").get(function () {
  // We don't use an arrow function as we'll need the this object
  return `/catalog/bookinstance/${this._id}`;
});

// Export model
module.exports = mongoose.model("BookInstance", BookInstanceSchema);

The new things we show here are the field options:

• enum: This allows us to set the allowed values of a string. In this case, we use it to specify the availability status of our books (using an enum means that we can prevent mis-spellings and arbitrary values for our status).

• default: We use default to set the default status for newly created book instances to “Maintenance” and the default due_back date to now (note how you can call the Date function when setting the date!).

Genre model

The definition will be very similar to the other models:

• The model should have a String SchemaType called name to describe the genre.

• This name should be required and have between 3 and 100 characters.

• Declare a virtual for the genre’s URL, named url.

• Export the model.

Testing — create some items

That’s it. We now have all models for the site set up!

In order to test the models (and to create some example books and other items that we can use in our next articles) we’ll now run an independent script to create items of each type:

1. Download (or otherwise create) the file populatedb.js inside your express-locallibrary-tutorial directory (in the same level as package.json).

2. Run the script using node in your command prompt, passing in the URL of your MongoDB database (the same one you replaced the insert_your_database_url_here placeholder with, inside app.js earlier):

node populatedb <your MongoDB url>

Note: On Windows you need to wrap the database URL inside double (“). On other operating systems you may need single (‘) quotation marks.

1. The script should run through to completion, displaying items as it creates them in the terminal.

Routes and controllers

We can now write the code to present that information to users. The first thing we need to do is determine what information we want to be able to display in our pages, and then define appropriate URLs for returning those resources. Then we’re going to need to create the routes (URL handlers) and views (templates) to display those pages.

The diagram below is provided as a reminder of the main flow of data and things that need to be implemented when handling an HTTP request/response. In addition to the views and routes the diagram shows “controllers” — functions that separate out the code to route requests from the code that actually processes requests.

As we’ve already created the models, the main things we’ll need to create are:

• “Routes” to forward the supported requests (and any information encoded in request URLs) to the appropriate controller functions.

• Controller functions to get the requested data from the models, create an HTML page displaying the data, and return it to the user to view in the browser.

• Views (templates) used by the controllers to render the data.

Routes primer

A route is a section of Express code that associates an HTTP verb (GET, POST, PUT, DELETE, etc.), a URL path/pattern, and a function that is called to handle that pattern.

There are several ways to create routes. For this tutorial we’re going to use the express.Router middleware as it allows us to group the route handlers for a particular part of a site together and access them using a common route-prefix. We’ll keep all our library-related routes in a “catalog” module, and, if we add routes for handling user accounts or other functions, we can keep them grouped separately.

Route functions

Our module above defines a couple of typical route functions. The “about” route (reproduced below) is defined using the Router.get() method, which responds only to HTTP GET requests. The first argument to this method is the URL path while the second is a callback function that will be invoked if an HTTP GET request with the path is received.

router.get("/about", function (req, res) {
  res.send("About this wiki");
});

The callback takes three arguments (usually named as shown: req, res, next), that will contain the HTTP Request object, HTTP response, and the next function in the middleware chain.

Note: Router functions are Express middleware, which means that they must either complete (respond to) the request or call the next function in the chain. In the case above we complete the request using send(), so the next argument is not used (and we choose not to specify it).

The router function above takes a single callback, but you can specify as many callback arguments as you want, or an array of callback functions. Each function is part of the middleware chain, and will be called in the order it is added to the chain (unless a preceding function completes the request).

HTTP verbs

The example routes above use the Router.get() method to respond to HTTP GET requests with a certain path.

The Router also provides route methods for all the other HTTP verbs, that are mostly used in exactly the same way: post(), put(), delete(), options(), trace(), copy(), lock(), mkcol(), move(), purge(), propfind(), proppatch(), unlock(), report(), mkactivity(), checkout(), merge(), m-search(), notify(), subscribe(), unsubscribe(), patch(), search(), and connect().

For example, the code below behaves just like the previous /about route, but only responds to HTTP POST requests.

router.post("/about", (req, res) => {
  res.send("About this wiki");
});

Route paths

The route paths define the endpoints at which requests can be made. The examples we’ve seen so far have just been strings, and are used exactly as written: ‘/’, ‘/about’, ‘/book’, ‘/any-random.path’.

Route paths can also be string patterns. String patterns use a form of regular expression syntax to define patterns of endpoints that will be matched. The syntax is listed below (note that the hyphen (-) and the dot (.) are interpreted literally by string-based paths):

• ? : The endpoint must have 0 or 1 of the preceding character (or group), e.g. a route path of ‘/ab?cd’ will match endpoints acd or abcd.

• • : The endpoint must have 1 or more of the preceding character (or group), e.g. a route path of ‘/ab+cd’ will match endpoints abcd, abbcd, abbbcd, and so on.

• * : The endpoint may have an arbitrary string where the * character is placed. E.g. a route path of ‘/ab*cd’ will match endpoints abcd, abXcd, abSOMErandomTEXTcd, and so on.

• () : Grouping match on a set of characters to perform another operation on, e.g. ‘/ab(cd)?e’ will perform a ?-match on the group (cd) — it will match abe and abcde.

The route paths can also be JavaScript regular expressions. For example, the route path below will match catfish and dogfish, but not catflap, catfishhead, and so on. Note that the path for a regular expression uses regular expression syntax (it is not a quoted string as in the previous cases).

app.get(/.*fish$/, function (req, res) {
  // …
});

Route parameters

Route parameters are named URL segments used to capture values at specific positions in the URL. The named segments are prefixed with a colon and then the name (E.g., /:your_parameter_name/). The captured values are stored in the req.params object using the parameter names as keys (E.g., req.params.your_parameter_name).

So for example, consider a URL encoded to contain information about users and books: http://localhost:3000/users/34/books/8989. We can extract this information as shown below, with the userId and bookId path parameters:

app.get("/users/:userId/books/:bookId", (req, res) => {
  // Access userId via: req.params.userId
  // Access bookId via: req.params.bookId
  res.send(req.params);
});

The names of route parameters must be made up of “word characters” (A-Z, a-z, 0-9, and _).

Note: The URL /book/create will be matched by a route like /book/:bookId (because :bookId is a placeholder for any string, therefore create matches). The first route that matches an incoming URL will be used, so if you want to process /book/create URLs specifically, their route handler must be defined before your /book/:bookId route.

Handling errors in the route functions

The route functions shown earlier all have arguments req and res, which represent the request and response, respectively. Route functions are also called with a third argument next, which can be used to pass errors to the Express middleware chain.

The code below shows how this works, using the example of a database query that takes a callback function, and returns either an error err or some results. If err is returned, next is called with err as the value in its first parameter (eventually the error propagates to our global error handling code). On success the desired data is returned and then used in the response.

router.get("/about", (req, res, next) => {
  About.find({}).exec((err, queryResults) => {
    if (err) {
      return next(err);
    }
    //Successful, so render
    res.render("about_view", { title: "About", list: queryResults });
  });
});

Handling exceptions in route functions

The previous section shows how Express expects route functions to return errors. The framework is designed for use with asynchronous functions that take a callback function (with an error and result argument), which is called when the operation completes. That’s a problem because later on we will be making Mongoose database queries that use Promise-based APIs, and which may throw exceptions in our route functions (rather than returning errors in a callback).

In order for the framework to properly handle exceptions, they must be caught, and then forwarded as errors as shown in the previous section.

Note: Express 5, which is currently in beta, is expected to handle JavaScript exceptions natively.

Re-imagining the simple example from the previous section with About.find().exec() as a database query that returns a promise, we might write the route function inside a try…catch block like this:

exports.get("/about", async function (req, res, next) {
  try {
    const successfulResult = await About.find({}).exec();
    res.render("about_view", { title: "About", list: successfulResult });
  } catch (error) {
    return next(error);
  }
});

That’s quite a lot of boilerplate code to add to every function. Instead, for this tutorial we’ll use the express-async-handler module. This defines a wrapper function that hides the try…catch block and the code to forward the error. The same example is now very simple, because we only need to write code for the case where we assume success:

// Import the module
const asyncHandler = require("express-async-handler");

exports.get(
  "/about",
  asyncHandler(async (req, res, next) => {
    const successfulResult = await About.find({}).exec();
    res.render("about_view", { title: "About", list: successfulResult });
  }),
);

Routes needed for the LocalLibrary

The URLs that we’re ultimately going to need for our pages are listed below, where object is replaced by the name of each of our models (book, bookinstance, genre, author), objects is the plural of object, and id is the unique instance field (_id) that is given to each Mongoose model instance by default.

• catalog/ — The home/index page.

• catalog// — The list of all books, bookinstances, genres, or authors (e.g. /catalog/books/, /catalog/genres/, etc.)

• catalog/

• catalog/

• catalog/

• catalog/

Create the route-handler callback functions

Before we define our routes, we’ll first create all the dummy/skeleton callback functions that they will invoke. The callbacks will be stored in separate “controller” modules for Book, BookInstance, Genre, and Author (you can use any file/module structure, but this seems an appropriate granularity for this project).

Start by creating a folder for our controllers in the project root (/controllers) and then create separate controller files/modules for handling each of the models:

/express-locallibrary-tutorial  //the project root
  /controllers
    authorController.js
    bookController.js
    bookinstanceController.js
    genreController.js

Author controller

Open the /controllers/authorController.js file and type in the following code:

const Author = require("../models/author");
const asyncHandler = require("express-async-handler");

// Display list of all Authors.
exports.author_list = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author list");
});

// Display detail page for a specific Author.
exports.author_detail = asyncHandler(async (req, res, next) => {
  res.send(`NOT IMPLEMENTED: Author detail: ${req.params.id}`);
});

// Display Author create form on GET.
exports.author_create_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author create GET");
});

// Handle Author create on POST.
exports.author_create_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author create POST");
});

// Display Author delete form on GET.
exports.author_delete_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author delete GET");
});

// Handle Author delete on POST.
exports.author_delete_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author delete POST");
});

// Display Author update form on GET.
exports.author_update_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author update GET");
});

// Handle Author update on POST.
exports.author_update_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Author update POST");
});

The module first requires the Author model that we’ll later be using to access and update our data, and the asyncHandler wrapper we’ll use to catch any exceptions thrown in our route handler functions. It then exports functions for each of the URLs we wish to handle. Note that the create, update and delete operations use forms, and hence also have additional methods for handling form post requests — we’ll discuss those methods in the “forms article” later on.

BookInstance controller

Open the /controllers/bookinstanceController.js file and copy in the following code (this follows an identical pattern to the Author controller module):

Genre controller

Open the /controllers/genreController.js file and copy in the following text (this follows an identical pattern to the Author and BookInstance files):

Book controller

Open the /controllers/bookController.js file and copy in the following code. This follows the same pattern as the other controller modules, but additionally has an index() function for displaying the site welcome page:

const Book = require("../models/book");
const asyncHandler = require("express-async-handler");

exports.index = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Site Home Page");
});

// Display list of all books.
exports.book_list = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book list");
});

// Display detail page for a specific book.
exports.book_detail = asyncHandler(async (req, res, next) => {
  res.send(`NOT IMPLEMENTED: Book detail: ${req.params.id}`);
});

// Display book create form on GET.
exports.book_create_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book create GET");
});

// Handle book create on POST.
exports.book_create_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book create POST");
});

// Display book delete form on GET.
exports.book_delete_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book delete GET");
});

// Handle book delete on POST.
exports.book_delete_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book delete POST");
});

// Display book update form on GET.
exports.book_update_get = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book update GET");
});

// Handle book update on POST.
exports.book_update_post = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Book update POST");
});

Create the catalog route module

Next we create routes for all the URLs needed by the LocalLibrary website, which will call the controller functions we defined in the previous sections.

The skeleton already has a ./routes folder containing routes for the index and users. Create another route file — catalog.js — inside this folder, as shown.

/express-locallibrary-tutorial //the project root
  /routes
    index.js
    users.js
    catalog.js

Open /routes/catalog.js and copy in the code below:

const express = require("express");
const router = express.Router();

// Require controller modules.
const book_controller = require("../controllers/bookController");
const author_controller = require("../controllers/authorController");
const genre_controller = require("../controllers/genreController");
const book_instance_controller = require("../controllers/bookinstanceController");

/// BOOK ROUTES ///

// GET catalog home page.
router.get("/", book_controller.index);

// GET request for creating a Book. NOTE This must come before routes that display Book (uses id).
router.get("/book/create", book_controller.book_create_get);

// POST request for creating Book.
router.post("/book/create", book_controller.book_create_post);

// GET request to delete Book.
router.get("/book/:id/delete", book_controller.book_delete_get);

// POST request to delete Book.
router.post("/book/:id/delete", book_controller.book_delete_post);

// GET request to update Book.
router.get("/book/:id/update", book_controller.book_update_get);

// POST request to update Book.
router.post("/book/:id/update", book_controller.book_update_post);

// GET request for one Book.
router.get("/book/:id", book_controller.book_detail);

// GET request for list of all Book items.
router.get("/books", book_controller.book_list);

/// AUTHOR ROUTES ///

// GET request for creating Author. NOTE This must come before route for id (i.e. display author).
router.get("/author/create", author_controller.author_create_get);

// POST request for creating Author.
router.post("/author/create", author_controller.author_create_post);

// GET request to delete Author.
router.get("/author/:id/delete", author_controller.author_delete_get);

// POST request to delete Author.
router.post("/author/:id/delete", author_controller.author_delete_post);

// GET request to update Author.
router.get("/author/:id/update", author_controller.author_update_get);

// POST request to update Author.
router.post("/author/:id/update", author_controller.author_update_post);

// GET request for one Author.
router.get("/author/:id", author_controller.author_detail);

// GET request for list of all Authors.
router.get("/authors", author_controller.author_list);

/// GENRE ROUTES ///

// GET request for creating a Genre. NOTE This must come before route that displays Genre (uses id).
router.get("/genre/create", genre_controller.genre_create_get);

//POST request for creating Genre.
router.post("/genre/create", genre_controller.genre_create_post);

// GET request to delete Genre.
router.get("/genre/:id/delete", genre_controller.genre_delete_get);

// POST request to delete Genre.
router.post("/genre/:id/delete", genre_controller.genre_delete_post);

// GET request to update Genre.
router.get("/genre/:id/update", genre_controller.genre_update_get);

// POST request to update Genre.
router.post("/genre/:id/update", genre_controller.genre_update_post);

// GET request for one Genre.
router.get("/genre/:id", genre_controller.genre_detail);

// GET request for list of all Genre.
router.get("/genres", genre_controller.genre_list);

/// BOOKINSTANCE ROUTES ///

// GET request for creating a BookInstance. NOTE This must come before route that displays BookInstance (uses id).
router.get(
  "/bookinstance/create",
  book_instance_controller.bookinstance_create_get,
);

// POST request for creating BookInstance.
router.post(
  "/bookinstance/create",
  book_instance_controller.bookinstance_create_post,
);

// GET request to delete BookInstance.
router.get(
  "/bookinstance/:id/delete",
  book_instance_controller.bookinstance_delete_get,
);

// POST request to delete BookInstance.
router.post(
  "/bookinstance/:id/delete",
  book_instance_controller.bookinstance_delete_post,
);

// GET request to update BookInstance.
router.get(
  "/bookinstance/:id/update",
  book_instance_controller.bookinstance_update_get,
);

// POST request to update BookInstance.
router.post(
  "/bookinstance/:id/update",
  book_instance_controller.bookinstance_update_post,
);

// GET request for one BookInstance.
router.get("/bookinstance/:id", book_instance_controller.bookinstance_detail);

// GET request for list of all BookInstance.
router.get("/bookinstances", book_instance_controller.bookinstance_list);

module.exports = router;

Update the index route module

We’ve set up all our new routes, but we still have a route to the original page. Let’s instead redirect this to the new index page that we’ve created at the path ‘/catalog’.

Open /routes/index.js and replace the existing route with the function below.

// GET home page.
router.get("/", function (req, res) {
  res.redirect("/catalog");
});

Note: This is our first use of the redirect() response method. This redirects to the specified page, by default sending HTTP status code “302 Found”. You can change the status code returned if needed, and supply either absolute or relative paths.

Update app.js

The last step is to add the routes to the middleware chain. We do this in app.js.

Open app.js and require the catalog route below the other routes (add the third line shown below, underneath the other two that should be already present in the file):

var indexRouter = require("./routes/index");
var usersRouter = require("./routes/users");
const catalogRouter = require("./routes/catalog"); //Import routes for "catalog" area of site

Next, add the catalog route to the middleware stack below the other routes (add the third line shown below, underneath the other two that should be already present in the file):

app.use("/", indexRouter);
app.use("/users", usersRouter);
app.use("/catalog", catalogRouter); // Add catalog routes to middleware chain.

Note: We have added our catalog module at a path ‘/catalog’. This is prepended to all of the paths defined in the catalog module. So for example, to access a list of books, the URL will be: /catalog/books/.

That’s it. We should now have routes and skeleton functions enabled for all the URLs that we will eventually support on the LocalLibrary website.

Display library data

Template primer

A template is a text file defining the structure or layout of an output file, with placeholders used to represent where data will be inserted when the template is rendered (in Express, templates are referred to as views).

Express template choices

Express can be used with many different template rendering engines. In this tutorial we use Pug (formerly known as Jade) for our templates. This is the most popular Node template language, and describes itself as a “clean, whitespace-sensitive syntax for writing HTML, heavily influenced by Haml”.

Different template languages use different approaches for defining layout and marking placeholders for data—some use HTML to define the layout while others use different markup formats that can be transpiled to HTML. Pug is of the second type; it uses a representation of HTML where the first word in any line usually represents an HTML element, and indentation on subsequent lines is used to represent nesting. The result is a page definition that translates directly to HTML, but is more concise and arguably easier to read.

Note: A downside of using Pug is that it is sensitive to indentation and whitespace (if you add an extra space in the wrong place you may get an unhelpful error code). Once you have your templates in place, however, they are very easy to read and maintain.

Template configuration

The LocalLibrary was configured to use Pug when we created the skeleton website. You should see the pug module included as a dependency in the website’s package.json file, and the following configuration settings in the app.js file. The settings tell us that we’re using pug as the view engine, and that Express should search for templates in the /views subdirectory.

// View engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "pug");

If you look in the views directory you will see the .pug files for the project’s default views. These include the view for the home page (index.pug) and base template (layout.pug) that we will need to replace with our own content.

/express-locallibrary-tutorial  //the project root
  /views
    error.pug
    index.pug
    layout.pug

Template syntax

The example template file below shows off many of Pug’s most useful features.

The first thing to notice is that the file maps the structure of a typical HTML file, with the first word in (almost) every line being an HTML element, and indentation being used to indicate nested elements. So for example, the body element is inside an html element, and paragraph elements (p) are within the body element, etc. Non-nested elements (e.g. individual paragraphs) are on separate lines.

doctype html
html(lang="en")
  head
    title= title
    script(type='text/javascript').
  body
    h1= title

    p This is a line with #[em some emphasis] and #[strong strong text] markup.
    p This line has un-escaped data: !{'<em> is emphasized</em>'} and escaped data: #{'<em> is not emphasized</em>'}.
      | This line follows on.
    p= 'Evaluated and <em>escaped expression</em>:' + title

    <!-- You can add HTML comments directly -->
    // You can add single line JavaScript comments and they are generated to HTML comments
    //- Introducing a single line JavaScript comment with "//-" ensures the comment isn't rendered to HTML

    p A line with a link
      a(href='/catalog/authors') Some link text
      |  and some extra text.

    #container.col
      if title
        p A variable named "title" exists.
      else
        p A variable named "title" does not exist.
      p.
        Pug is a terse and simple template language with a
        strong focus on performance and powerful features.

    h2 Generate a list

    ul
      each val in [1, 2, 3, 4, 5]
        li= val

Element attributes are defined in parentheses after their associated element. Inside the parentheses, the attributes are defined in comma- or whitespace- separated lists of the pairs of attribute names and attribute values, for example:

• script(type=’text/javascript’), link(rel=’stylesheet’, href=’/stylesheets/style.css’)

• meta(name=’viewport’ content=’width=device-width initial-scale=1’)

The values of all attributes are escaped (e.g. characters like > are converted to their HTML code equivalents like >) to prevent JavaScript injection or cross-site scripting attacks.

If a tag is followed by the equals sign, the following text is treated as a JavaScript expression. So for example, in the first line below, the content of the h1 tag will be variable title (either defined in the file or passed into the template from Express). In the second line the paragraph content is a text string concatenated with the title variable. In both cases the default behavior is to escape the line.

h1= title
p= 'Evaluated and <em>escaped expression</em>:' + title

Note: In Pug templates, a variable that is used but not passed in from your Express code (or defined locally) is “undefined”. If you used this template without passing in a title variable the tags would be created but would contain an empty string. If you use undefined variables in conditional statements then they evaluate to false. Other template languages may require that variables used in the template must be defined.

If there is no equals symbol after the tag then the content is treated as plain text. Within the plain text you can insert escaped and unescaped data using the #{} and !{} syntax respectively, as shown below. You can also add raw HTML within the plain text.

p This is a line with #[em some emphasis] and #[strong strong text] markup.
p This line has an un-escaped string: !{'<em> is emphasized</em>'}, an escaped string: #{'<em> is not emphasized</em>'}, and escaped variables: #{title}.

Note: You will almost always want to escape data from users (via the #{} syntax). Data that can be trusted (e.g. generated counts of records, etc.) may be displayed without escaping the values.

a(href='http://someurl/') Link text
| Plain text

Pug allows you to perform conditional operations using if, else, else if and unless — for example:

if title
  p A variable named "title" exists
else
  p A variable named "title" does not exist

while syntax. In the code fragment below we’ve looped through an array to display a list of variables (note the use of the ‘li=’ to evaluate the “val” as a variable below. The value you iterate across can also be passed into the template as a variable!

ul
  each val in [1, 2, 3, 4, 5]
    li= val

The syntax also supports comments (that can be rendered in the output—or not—as you choose), mixins to create reusable blocks of code, case statements, and many other features.

Extending templates

Across a site, it is usual for all pages to have a common structure, including standard HTML markup for the head, footer, navigation, etc. Rather than forcing developers to duplicate this “boilerplate” in every page, Pug allows you to declare a base template and then extend it, replacing just the bits that are different for each specific page.

For example, the base template layout.pug created in our skeleton project looks like this:

doctype html
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
  body
    block content

The block tag is used to mark up sections of content that may be replaced in a derived template (if the block is not redefined then its implementation in the base class is used).

The default index.pug (created for our skeleton project) shows how we override the base template. The extends tag identifies the base template to use, and then we use block section_name to indicate the new content of the section that we will override.

extends layout

block content
  h1= title
  p Welcome to #{title}

LocalLibrary base template

Now that we understand how to extend templates using Pug, let’s start by creating a base template for the project. This will have a sidebar with links for the pages we hope to create across the tutorial articles (e.g. to display and create books, genres, authors, etc.) and a main content area that we’ll override in each of our individual pages.

Open /views/layout.pug and replace the content with the code below.

doctype html
html(lang='en')
  head
    title= title
    meta(charset='utf-8')
    meta(name='viewport', content='width=device-width, initial-scale=1')
    link(rel="stylesheet", href="https://cdn.jsdelivr.net/npm/bootstrap@4.6.2/dist/css/bootstrap.min.css", integrity="sha384-xOolHFLEh07PJGoPkLv1IbcEPTNtaed2xpHsD9ESMhqIYd0nLMwNLD69Npy4HI+N", crossorigin="anonymous")
    script(src="https://code.jquery.com/jquery-3.5.1.slim.min.js", integrity="sha384-DfXdz2htPH0lsSSs5nCTpuj/zy4C+OGpamoFVy38MVBnE+IbbVYUew+OrCXaRkfj", crossorigin="anonymous")
    script(src="https://cdn.jsdelivr.net/npm/bootstrap@4.6.2/dist/js/bootstrap.min.js", integrity="sha384-+sLIOodYLS7CIrQpBjl+C7nPvqq+FbNUBDunl/OZv93DB7Ln/533i8e/mZXLi/P+", crossorigin="anonymous")
    link(rel='stylesheet', href='/stylesheets/style.css')
  body
    div(class='container-fluid')
      div(class='row')
        div(class='col-sm-2')
          block sidebar
            ul(class='sidebar-nav')
              li
                a(href='/catalog') Home
              li
                a(href='/catalog/books') All books
              li
                a(href='/catalog/authors') All authors
              li
                a(href='/catalog/genres') All genres
              li
                a(href='/catalog/bookinstances') All book-instances
              li
                hr
              li
                a(href='/catalog/author/create') Create new author
              li
                a(href='/catalog/genre/create') Create new genre
              li
                a(href='/catalog/book/create') Create new book
              li
                a(href='/catalog/bookinstance/create') Create new book instance (copy)

        div(class='col-sm-10')
          block content

The template uses (and includes) JavaScript and CSS from Bootstrap to improve the layout and presentation of the HTML page. Using Bootstrap or another client-side web framework is a quick way to create an attractive page that can scale well on different browser sizes, and it also allows us to deal with the page presentation without having to get into any of the details—we just want to focus on the server-side code here!

The base template also references a local CSS file (style.css) that provides a little additional styling. Open /public/stylesheets/style.css and replace its content with the following CSS code:

.sidebar-nav {
  margin-top: 20px;
  padding: 0;
  list-style: none;
}

Home page

The first page we’ll create will be the website home page, which is accessible from either the site (/) or catalog (catalog/) root. This will display some static text describing the site, along with dynamically calculated “counts” of different record types in the database.

We’ve already created a route for the home page. In order to complete the page we need to update our controller function to fetch “counts” of records from the database, and create a view (template) that we can use to render the page.

Route

We created our index page routes in a previous tutorial. As a reminder, all the route functions are defined in /routes/catalog.js:

// GET catalog home page.
router.get("/", book_controller.index); //This actually maps to /catalog/ because we import the route with a /catalog prefix

The book controller index function passed as a parameter (book_controller.index) has a “placeholder” implementation defined in /controllers/bookController.js:

exports.index = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Site Home Page");
});

It is this controller function that we extend to get information from our models and then render it using a template (view).

Controller

The index controller function needs to fetch information about how many Book, BookInstance (all), BookInstance (available), Author, and Genre records we have in the database, render this data in a template to create an HTML page, and then return it in an HTTP response.

Open /controllers/bookController.js. Near the top of the file you should see the exported index() function.

const Book = require("../models/book");
const asyncHandler = require("express-async-handler");

exports.index = asyncHandler(async (req, res, next) => {
  res.send("NOT IMPLEMENTED: Site Home Page");
});

Replace all the code above with the following code fragment. The first thing this does is import (require()) all the models. We need to do this because we’ll be using them to get our counts of documents. The code also requires “express-async-handler”, which provides a wrapper to catch exceptions thrown in route handler functions.

const Book = require("../models/book");
const Author = require("../models/author");
const Genre = require("../models/genre");
const BookInstance = require("../models/bookinstance");

const asyncHandler = require("express-async-handler");

exports.index = asyncHandler(async (req, res, next) => {
  // Get details of books, book instances, authors and genre counts (in parallel)
  const [
    numBooks,
    numBookInstances,
    numAvailableBookInstances,
    numAuthors,
    numGenres,
  ] = await Promise.all([
    Book.countDocuments({}).exec(),
    BookInstance.countDocuments({}).exec(),
    BookInstance.countDocuments({ status: "Available" }).exec(),
    Author.countDocuments({}).exec(),
    Genre.countDocuments({}).exec(),
  ]);

  res.render("index", {
    title: "Local Library Home",
    book_count: numBooks,
    book_instance_count: numBookInstances,
    book_instance_available_count: numAvailableBookInstances,
    author_count: numAuthors,
    genre_count: numGenres,
  });
});

We use the countDocuments() method to get the number of instances of each model. This method is called on a model, with an optional set of conditions to match against, and returns a Query object. The query can be executed by calling exec(), which returns a Promise that is either fulfilled with a result, or rejected if there is a database error.

Because the queries for document counts are independent of each other we use Promise.all() to run them in parallel. The method returns a new promise that we await for completion (execution pauses within this function at await). When all the queries complete, the promise returned by all() fulfills, continuing execution of the route handler function, and populating the array with the results of the database queries.

We then call res.render(), specifying a view (template) named ‘index’ and objects mapping the results of the database queries to the view template. The data is supplied as key-value pairs, and can be accessed in the template using the key.

View

Open /views/index.pug and replace its content with the text below.

extends layout

block content
  h1= title
  p Welcome to #[em LocalLibrary], a very basic Express website developed as a tutorial example on the Mozilla Developer Network.

  h2 Dynamic content

  p The library has the following record counts:

  ul
    li #[strong Books:] !{book_count}
    li #[strong Copies:] !{book_instance_count}
    li #[strong Copies available:] !{book_instance_available_count}
    li #[strong Authors:] !{author_count}
    li #[strong Genres:] !{genre_count}

The view is straightforward. We extend the layout.pug base template, overriding the block named ‘content’. The first h1 heading will be the escaped text for the title variable that was passed into the render() function—note the use of the ‘h1=’ so that the following text is treated as a JavaScript expression. We then include a paragraph introducing the LocalLibrary.

Under the Dynamic content heading we list the number of copies of each model. Note that the template values for the data are the keys that were specified when render() was called in the route handler function.

Note: We didn’t escape the count values (i.e. we used the !{} syntax) because the count values are calculated. If the information was supplied by end-users then we’d escape the variable for display.

Book list page

Controller

The book list controller function needs to get a list of all Book objects in the database, sort them, and then pass these to the template for rendering.

Open /controllers/bookController.js. Find the exported book_list() controller method and replace it with the following code.

// Display list of all books.
exports.book_list = asyncHandler(async (req, res, next) => {
  const allBooks = await Book.find({}, "title author")
    .sort({ title: 1 })
    .populate("author")
    .exec();

  res.render("book_list", { title: "Book List", book_list: allBooks });
});

The route handler calls the find() function on the Book model, selecting to return only the title and author as we don’t need the other fields (it will also return the _id and virtual fields), and sorting the results by the title alphabetically using the sort() method. We also call populate() on Book, specifying the author field—this will replace the stored book author id with the full author details. exec() is then daisy-chained on the end in order to execute the query and return a promise.

The route handler uses await to wait on the promise, pausing execution until it is settled. If the promise is fulfilled, the results of the query are saved to the allBooks variable and the handler continues execution.

The final part of the route handler calls render(), specifying the book_list (.pug) template and passing values for the title and book_list into the template.

View

Create /views/book_list.pug and copy in the text below.

extends layout

block content
  h1= title
  if book_list.length
    ul
      each book in book_list
        li
          a(href=book.url) #{book.title}
          |  (#{book.author.name})

  else
    p There are no books.

The view extends the layout.pug base template and overrides the block named ‘content’. It displays the title we passed in from the controller (via the render() method) and iterates through the book_list variable using the each-in syntax. A list item is created for each book displaying the book title as a link to the book’s detail page followed by the author name. If there are no books in the book_list then the else clause is executed, and displays the text ‘There are no books’.

BookInstance list page

Controller

The BookInstance list controller function needs to get a list of all book instances, populate the associated book information, and then pass the list to the template for rendering.

Open /controllers/bookinstanceController.js. Find the exported bookinstance_list() controller method and replace it with the following code.

// Display list of all BookInstances.
exports.bookinstance_list = asyncHandler(async (req, res, next) => {
  const allBookInstances = await BookInstance.find().populate("book").exec();

  res.render("bookinstance_list", {
    title: "Book Instance List",
    bookinstance_list: allBookInstances,
  });
});

The route handler calls the find() function on the BookInstance model, and then daisy-chains a call to populate() with the book field—this will replace the book id stored for each BookInstance with a full Book document. exec() is then daisy-chained on the end in order to execute the query and return a promise.

The route handler uses await to wait on the promise, pausing execution until it is settled. If the promise is fulfilled, the results of the query are saved to the allBookInstances variable, and the route handler continues execution.

The last part of the code calls render(), specifying the bookinstance_list (.pug) template and passing values for the title and bookinstance_list into the template.

View

Create /views/bookinstance_list.pug and copy in the text below.

extends layout

block content
  h1= title

  if bookinstance_list.length
    ul
      each val in bookinstance_list
        li
          a(href=val.url) #{val.book.title} : #{val.imprint} - 
          if val.status=='Available'
            span.text-success #{val.status}
          else if val.status=='Maintenance'
            span.text-danger #{val.status}
          else
            span.text-warning #{val.status}
          if val.status!='Available'
            span  (Due: #{val.due_back} )

  else
    p There are no book copies in this library.

This view is much the same as all the others. It extends the layout, replacing the content block, displays the title passed in from the controller, and iterates through all the book copies in bookinstance_list. For each copy we display its status (color coded) and if the book is not available, its expected return date. One new feature is introduced—we can use dot notation after a tag to assign a class. So span.text-success will be compiled to <span class="text-success"> (and might also be written in Pug as span(class=”text-success”)).

Date formatting using luxon

The default rendering of dates from our models is very ugly: Mon Apr 10 2020 15:49:58 GMT+1100 (AUS Eastern Daylight Time). In this section we’ll show how you can update the BookInstance List page from the previous section to present the due_date field in a more friendly format: Apr 10th, 2023.

The approach we will use is to create a virtual property in our BookInstance model that returns the formatted date. We’ll do the actual formatting using luxon, a powerful, modern, and friendly library for parsing, validating, manipulating, formatting and localising dates.

Note: It is possible to use luxon to format the strings directly in our Pug templates, or we could format the string in a number of other places. Using a virtual property allows us to get the formatted date in exactly the same way as we get the due_date currently.

Create the virtual property

1. Open ./models/bookinstance.js.

2. At the top of the page, import luxon.

const { DateTime } = require("luxon");

Add the virtual property due_back_formatted just after the URL property.

BookInstanceSchema.virtual("due_back_formatted").get(function () {
  return DateTime.fromJSDate(this.due_back).toLocaleString(DateTime.DATE_MED);
});

Update the view

Open /views/bookinstance_list.pug and replace due_back with due_back_formatted.

      if val.status != 'Available'
        //span  (Due: #{val.due_back} )
        span  (Due: #{val.due_back_formatted} )

Author list page and Genre list page challenge

Controller

The author list controller function needs to get a list of all Author instances, and then pass these to the template for rendering.

Open /controllers/authorController.js. Find the exported author_list() controller method near the top of the file and replace it with the following code.

// Display list of all Authors.
exports.author_list = asyncHandler(async (req, res, next) => {
  const allAuthors = await Author.find().sort({ family_name: 1 }).exec();
  res.render("author_list", {
    title: "Author List",
    author_list: allAuthors,
  });
});

The route controller function follows the same pattern as for the other list pages. It defines a query on the Author model, using the find() function to get all authors, and the sort() method to sort them by family_name in alphabetic order. exec() is daisy-chained on the end in order to execute the query and return a promise that the function can await.

Once the promise is fulfilled the route handler renders the author_list(.pug) template, passing the page title and the list of authors (allAuthors) using template keys.

View

Create /views/author_list.pug and replace its content with the text below.

extends layout

block content
  h1= title

  if author_list.length
    ul
      each author in author_list
        li
          a(href=author.url) #{author.name}
          |  (#{author.date_of_birth} - #{author.date_of_death})
  else
    p There are no authors.

Note: The appearance of the author lifespan dates is ugly! You can improve this using the same approach as we used for the BookInstance list (adding the virtual property for the lifespan to the Author model).

However, as the author may not be dead or may have missing birth/death data, in this case we need to ignore missing dates or references to nonexistent properties. One way to deal with this is to return either a formatted date, or a blank string, depending on whether the property is defined. For example:

return this.date_of_birth ? DateTime.fromJSDate(this.date_of_birth).toLocaleString(DateTime.DATE_MED) : ‘’;

Genre detail page

The genre detail page needs to display the information for a particular genre instance, using its automatically generated _id field value as the identifier. The ID of the required genre record is encoded at the end of the URL and extracted automatically based on the route definition (/genre/:id). It is then accessed within the controller via the request parameters: req.params.id.

The page should display the genre name and a list of all books in the genre with links to each book’s details page.

Controller

Open /controllers/genreController.js and require the Book module at the top of the file (the file should already require() the Genre module and “express-async-handler”).

const Book = require("../models/book");

Find the exported genre_detail() controller method and replace it with the following code.

// Display detail page for a specific Genre.
exports.genre_detail = asyncHandler(async (req, res, next) => {
  // Get details of genre and all associated books (in parallel)
  const [genre, booksInGenre] = await Promise.all([
    Genre.findById(req.params.id).exec(),
    Book.find({ genre: req.params.id }, "title summary").exec(),
  ]);
  if (genre === null) {
    // No results.
    const err = new Error("Genre not found");
    err.status = 404;
    return next(err);
  }

  res.render("genre_detail", {
    title: "Genre Detail",
    genre: genre,
    genre_books: booksInGenre,
  });
});

We first use Genre.findById() to get Genre information for a specific ID, and Book.find() to get all books records that have that same associated genre ID. Because the two requests do not depend on each other, we use Promise.all() to run the database queries in parallel (this same approach for running queries in parallel was demonstrated in the home page).

We await on the returned promise, and once it settles we check the results. If the genre does not exist in the database (i.e. it may have been deleted) then findById() will return successfully with no results. In this case we want to display a “not found” page, so we create an Error object and pass it to the next middleware function in the chain.

If the genre is found, then we call render() to display the view. The view template is genre_detail (.pug). The values for the title, genre and booksInGenre are passed into the template using the corresponding keys (title, genre and genre_books).

View

Create /views/genre_detail.pug and fill it with the text below:

extends layout

block content

  h1 Genre: #{genre.name}

  div(style='margin-left:20px;margin-top:20px')

    h2(style='font-size: 1.5rem;') Books
    if genre_books.length
      dl
        each book in genre_books
          dt
            a(href=book.url) #{book.title}
          dd #{book.summary}
    else
      p This genre has no books.

The view is very similar to all our other templates. The main difference is that we don’t use the title passed in for the first heading (though it is used in the underlying layout.pug template to set the page title).

Book detail page

The Book detail page needs to display the information for a specific Book (identified using its automatically generated _id field value), along with information about each associated copy in the library (BookInstance). Wherever we display an author, genre, or book instance, these should be linked to the associated detail page for that item.

Controller

Open /controllers/bookController.js. Find the exported book_detail() controller method and replace it with the following code.

// Display detail page for a specific book.
exports.book_detail = asyncHandler(async (req, res, next) => {
  // Get details of books, book instances for specific book
  const [book, bookInstances] = await Promise.all([
    Book.findById(req.params.id).populate("author").populate("genre").exec(),
    BookInstance.find({ book: req.params.id }).exec(),
  ]);

  if (book === null) {
    // No results.
    const err = new Error("Book not found");
    err.status = 404;
    return next(err);
  }

  res.render("book_detail", {
    title: book.title,
    book: book,
    book_instances: bookInstances,
  });
});

The approach is exactly the same as described for the Genre detail page. The route controller function uses Promise.all() to query the specified Book and its associated copies (BookInstance) in parallel. If no matching book is found an Error object is returned with a “404: Not Found” error. If the book is found, then the retrieved database information is rendered using the “book_detail” template. Since the key ‘title’ is used to give name to the webpage (as defined in the header in ‘layout.pug’), this time we are passing results.book.title while rendering the webpage.

View

Create /views/book_detail.pug and add the text below.

extends layout

block content
  h1 Title: #{book.title}

  p #[strong Author: ]
    a(href=book.author.url) #{book.author.name}
  p #[strong Summary:] #{book.summary}
  p #[strong ISBN:] #{book.isbn}
  p #[strong Genre: ]
    each val, index in book.genre
      a(href=val.url) #{val.name}
      if index < book.genre.length - 1
        |,&nbsp;

  div(style='margin-left:20px;margin-top:20px')
    h2(style='font-size: 1.5rem;') Copies

    each val in book_instances
      hr
      if val.status=='Available'
        p.text-success #{val.status}
      else if val.status=='Maintenance'
        p.text-danger #{val.status}
      else
        p.text-warning #{val.status}
      p #[strong Imprint:] #{val.imprint}
      if val.status!='Available'
        p #[strong Due back:] #{val.due_back}
      p #[strong Id: ]
        a(href=val.url) #{val._id}

    else
      p There are no copies of this book in the library.

Almost everything in this template has been demonstrated in previous sections.

Note: The list of genres associated with the book is implemented in the template as below. This adds a comma and a non breaking space after every genre associated with the book except for the last one.

  p #[strong Genre: ]
    each val, index in book.genre
      a(href=val.url) #{val.name}
      if index < book.genre.length - 1
        `|,&nbsp;`

Author detail page

The author detail page needs to display the information about the specified Author, identified using their (automatically generated) _id field value, along with a list of all the Book objects associated with that Author.

Controller

Open /controllers/authorController.js.

Add the following lines to the top of the file to require() the Book module needed by the author detail page (other modules such as “express-async-handler” should already be present).

const Book = require("../models/book");

Find the exported author_detail() controller method and replace it with the following code.

// Display detail page for a specific Author.
exports.author_detail = asyncHandler(async (req, res, next) => {
  // Get details of author and all their books (in parallel)
  const [author, allBooksByAuthor] = await Promise.all([
    Author.findById(req.params.id).exec(),
    Book.find({ author: req.params.id }, "title summary").exec(),
  ]);

  if (author === null) {
    // No results.
    const err = new Error("Author not found");
    err.status = 404;
    return next(err);
  }

  res.render("author_detail", {
    title: "Author Detail",
    author: author,
    author_books: allBooksByAuthor,
  });
});

The approach is exactly the same as described for the Genre detail page. The route controller function uses Promise.all() to query the specified Author and their associated Book instances in parallel. If no matching author is found an Error object is sent to the Express error handling middleware. If the author is found then the retrieved database information is rendered using the “author_detail” template.

View

Create /views/author_detail.pug and copy in the following text.

extends layout

block content

  h1 Author: #{author.name}
  p #{author.date_of_birth} - #{author.date_of_death}

  div(style='margin-left:20px;margin-top:20px')

    h2(style='font-size: 1.5rem;') Books
    if author_books.length
      dl
        each book in author_books
          dt
            a(href=book.url) #{book.title}
          dd #{book.summary}
    else
      p This author has no books.

BookInstance detail page

The BookInstance detail page needs to display the information for each BookInstance, identified using its (automatically generated) _id field value. This will include the Book name (as a link to the Book detail page) along with other information in the record.

Controller

Open /controllers/bookinstanceController.js. Find the exported bookinstance_detail() controller method and replace it with the following code.

// Display detail page for a specific BookInstance.
exports.bookinstance_detail = asyncHandler(async (req, res, next) => {
  const bookInstance = await BookInstance.findById(req.params.id)
    .populate("book")
    .exec();

  if (bookInstance === null) {
    // No results.
    const err = new Error("Book copy not found");
    err.status = 404;
    return next(err);
  }

  res.render("bookinstance_detail", {
    title: "Book:",
    bookinstance: bookInstance,
  });
});

The implementation is very similar to that used for the other model detail pages. The route controller function calls BookInstance.findById() with the ID of a specific book instance extracted from the URL (using the route), and accessed within the controller via the request parameters: req.params.id. It then calls populate() to get the details of the associated Book. If a matching BookInstance is not found an error is sent to the Express middleware. Otherwise the returned data is rendered using the bookinstance_detail.pug view.

View

Create /views/bookinstance_detail.pug and copy in the content below.

extends layout

block content

  h1 ID: #{bookinstance._id}

  p #[strong Title: ]
    a(href=bookinstance.book.url) #{bookinstance.book.title}
  p #[strong Imprint:] #{bookinstance.imprint}

  p #[strong Status: ]
    if bookinstance.status=='Available'
      span.text-success #{bookinstance.status}
    else if bookinstance.status=='Maintenance'
      span.text-danger #{bookinstance.status}
    else
      span.text-warning #{bookinstance.status}

  if bookinstance.status!='Available'
    p #[strong Due back:] #{bookinstance.due_back}

Working with forms

Overview

An HTML Form is a group of one or more fields/widgets on a web page that can be used to collect information from users for submission to a server. Forms are a flexible mechanism for collecting user input because there are suitable form inputs available for entering many different types of data—text boxes, checkboxes, radio buttons, date pickers, etc. Forms are also a relatively secure way of sharing data with the server, as they allow us to send data in POST requests with cross-site request forgery protection.

Working with forms can be complicated! Developers need to write HTML for the form, validate and properly sanitize entered data on the server (and possibly also in the browser), repost the form with error messages to inform users of any invalid fields, handle the data when it has successfully been submitted, and finally respond to the user in some way to indicate success.

HTML Forms

First a brief overview of HTML Forms. Consider a simple HTML form, with a single text field for entering the name of some “team”, and its associated label:

The form is defined in HTML as a collection of elements inside <form>…</form> tags, containing at least one input element of type=”submit”.

<form action="/team_name_url/" method="post">
  <label for="team_name">Enter name: </label>
  <input
    id="team_name"
    type="text"
    name="name_field"
    value="Default name for team." />
  <input type="submit" value="OK" />
</form>

While here we have included just one (text) field for entering the team name, a form may contain any number of other input elements and their associated labels. The field’s type attribute defines what sort of widget will be displayed. The name and id of the field are used to identify the field in JavaScript/CSS/HTML, while value defines the initial value for the field when it is first displayed. The matching team label is specified using the label tag (see “Enter name” above), with a for field containing the id value of the associated input.

The submit input will be displayed as a button (by default)—this can be pressed by the user to upload the data contained by the other input elements to the server (in this case, just the team_name). The form attributes define the HTTP method used to send the data and the destination of the data on the server (action):

• action: The resource/URL where data is to be sent for processing when the form is submitted. If this is not set (or set to an empty string), then the form will be submitted back to the current page URL.

• method: The HTTP method used to send the data: POST or GET.

  • The POST method should always be used if the data is going to result in a change to the server’s database, because this can be made more resistant to cross-site forgery request attacks.

  • The GET method should only be used for forms that don’t change user data (e.g. a search form). It is recommended for when you want to be able to bookmark or share the URL.

Form handling process

Form handling uses all of the same techniques that we learned for displaying information about our models: the route sends our request to a controller function which performs any database actions required, including reading data from the models, then generates and returns an HTML page. What makes things more complicated is that the server also needs to be able to process the data provided by the user, and redisplay the form with error information if there are any problems.

A process flowchart for processing form requests is shown below, starting with a request for a page containing a form (shown in green):

As shown in the diagram above, the main things that form handling code needs to do are:

1. Display the default form the first time it is requested by the user.

• The form may contain blank fields (e.g. if you’re creating a new record), or it may be pre-populated with initial values (e.g. if you are changing a record, or have useful default initial values).

1. Receive data submitted by the user, usually in an HTTP POST request.

2. Validate and sanitize the data.

3. If any data is invalid, re-display the form—this time with any user populated values and error messages for the problem fields.

4. If all data is valid, perform required actions (e.g. save the data in the database, send a notification email, return the result of a search, upload a file, etc.)

5. Once all actions are complete, redirect the user to another page.

Often form handling code is implemented using a GET route for the initial display of the form and a POST route to the same path for handling validation and processing of form data. This is the approach that will be used in this tutorial.

Express itself doesn’t provide any specific support for form handling operations, but it can use middleware to process POST and GET parameters from the form, and to validate/sanitize their values.

Validation and sanitization

Before the data from a form is stored it must be validated and sanitized:

• Validation checks that entered values are appropriate for each field (are in the right range, format, etc.) and that values have been supplied for all required fields.

• Sanitization removes/replaces characters in the data that might potentially be used to send malicious content to the server.

For this tutorial, we’ll be using the popular express-validator module to perform both validation and sanitization of our form data.

Installation

Install the module by running the following command in the root of the project.

npm install express-validator

Using express-validator

To use the validator in our controllers, we specify the particular functions we want to import from the express-validator module, as shown below:

const { body, validationResult } = require("express-validator");

There are many functions available, allowing you to check and sanitize data from request parameters, body, headers, cookies, etc., or all of them at once. For this tutorial, we’ll primarily be using body and validationResult (as “required” above).

The functions are defined as below:

• body(fields, message): Specifies a set of fields in the request body (a POST parameter) to validate and/or sanitize along with an optional error message that can be displayed if it fails the tests. The validation and sanitize criteria are daisy-chained to the body() method. For example, the line below first defines that we’re checking the “name” field and that a validation error will set an error message “Empty name”. We then call the sanitization method trim() to remove whitespace from the start and end of the string, and then isLength() to check the resulting string isn’t empty. Finally, we call escape() to remove HTML characters from the variable that might be used in JavaScript cross-site scripting attacks.

[
  // …
  body("name", "Empty name").trim().isLength({ min: 1 }).escape(),
  // …
];

This test checks that the age field is a valid date and uses optional() to specify that null and empty strings will not fail validation.

[
  // …
  body("age", "Invalid age")
    .optional({ values: "falsy" })
    .isISO8601()
    .toDate(),
  // …
];

You can also daisy chain different validators, and add messages that are displayed if the preceding validators are false.

[
  // …
  body("name")
    .trim()
    .isLength({ min: 1 })
    .withMessage("Name empty.")
    .isAlpha()
    .withMessage("Name must be alphabet letters."),
  // …
];

• validationResult(req): Runs the validation, making errors available in the form of a validation result object. This is invoked in a separate callback, as shown below:

asyncHandler(async (req, res, next) => {
  // Extract the validation errors from a request.
  const errors = validationResult(req);

  if (!errors.isEmpty()) {
    // There are errors. Render form again with sanitized values/errors messages.
    // Error messages can be returned in an array using `errors.array()`.
  } else {
    // Data from form is valid.
  }
});

We use the validation result’s isEmpty() method to check if there were errors, and its array() method to get the set of error messages. See the Handling validation section for more information.

The validation and sanitization chains are middleware that should be passed to the Express route handler (we do this indirectly, via the controller). When the middleware runs, each validator/sanitizer is run in the order specified.

We’ll cover some real examples when we implement the LocalLibrary forms below.

Form design

Many of the models in the library are related/dependent—for example, a Book requires an Author, and may also have one or more Genres. This raises the question of how we should handle the case where a user wishes to:

• Create an object when its related objects do not yet exist (for example, a book where the author object hasn’t been defined).

• Delete an object that is still being used by another object (so for example, deleting a Genre that is still being used by a Book).

For this project we will simplify the implementation by stating that a form can only:

• Create an object using objects that already exist (so users will have to create any required Author and Genre instances before attempting to create any Book objects).

• Delete an object if it is not referenced by other objects (so for example, you won’t be able to delete a Book until all associated BookInstance objects have been deleted).

Note: A more flexible implementation might allow you to create the dependent objects when creating a new object, and delete any object at any time (for example, by deleting dependent objects, or by removing references to the deleted object from the database).

Routes

In order to implement our form handling code, we will need two routes that have the same URL pattern. The first (GET) route is used to display a new empty form for creating the object. The second route (POST) is used for validating data entered by the user, and then saving the information and redirecting to the detail page (if the data is valid) or redisplaying the form with errors (if the data is invalid).

We have already created the routes for all our model’s create pages in /routes/catalog.js (in a previous tutorial). For example, the genre routes are shown below:

// GET request for creating a Genre. NOTE This must come before route that displays Genre (uses id).
router.get("/genre/create", genre_controller.genre_create_get);

// POST request for creating Genre.
router.post("/genre/create", genre_controller.genre_create_post);

Create genre form

This sub article shows how we define our page to create Genre objects (this is a good place to start because the Genre has only one field, its name, and no dependencies). Like any other pages, we need to set up routes, controllers, and views.

Import validation and sanitization methods

To use the express-validator in our controllers we have to require the functions we want to use from the ‘express-validator’ module.

Open /controllers/genreController.js, and add the following line at the top of the file, before any route handler functions:

const { body, validationResult } = require("express-validator");

Note: This syntax allows us to use body and validationResult as the associated middleware functions, as you will see in the post route section below. It is equivalent to:

const validator = require("express-validator");
const body = validator.body;
const validationResult = validator.validationResult;

Controller—get route

Find the exported genre_create_get() controller method and replace it with the following code. This renders the genre_form.pug view, passing a title variable.

// Display Genre create form on GET.
exports.genre_create_get = (req, res, next) => {
  res.render("genre_form", { title: "Create Genre" });
};

Note that this replaces the placeholder asynchronous handler that we added in the Express Tutorial Part 4: Routes and controllers with a “normal” express route handler function. We don’t need the asyncHandler() wrapper for this route, because it doesn’t contain any code that can throw an exception.

Controller—post route

Find the exported genre_create_post() controller method and replace it with the following code.

// Handle Genre create on POST.
exports.genre_create_post = [
  // Validate and sanitize the name field.
  body("name", "Genre name must contain at least 3 characters")
    .trim()
    .isLength({ min: 3 })
    .escape(),

  // Process request after validation and sanitization.
  asyncHandler(async (req, res, next) => {
    // Extract the validation errors from a request.
    const errors = validationResult(req);

    // Create a genre object with escaped and trimmed data.
    const genre = new Genre({ name: req.body.name });

    if (!errors.isEmpty()) {
      // There are errors. Render the form again with sanitized values/error messages.
      res.render("genre_form", {
        title: "Create Genre",
        genre: genre,
        errors: errors.array(),
      });
      return;
    } else {
      // Data from form is valid.
      // Check if Genre with same name already exists.
      const genreExists = await Genre.findOne({ name: req.body.name })
        .collation({ locale: "en", strength: 2 })
        .exec();
      if (genreExists) {
        // Genre exists, redirect to its detail page.
        res.redirect(genreExists.url);
      } else {
        await genre.save();
        // New genre saved. Redirect to genre detail page.
        res.redirect(genre.url);
      }
    }
  }),
];

The first thing to note is that instead of being a single middleware function (with arguments (req, res, next)) the controller specifies an array of middleware functions. The array is passed to the router function and each method is called in order.

Note: This approach is needed, because the validators are middleware functions.

The first method in the array defines a body validator (body()) that validates and sanitizes the field. This uses trim() to remove any trailing/leading whitespace, checks that the name field is not empty, and then uses escape() to remove any dangerous HTML characters).

After specifying the validators we create a middleware function to extract any validation errors. We use isEmpty() to check whether there are any errors in the validation result. If there are then we render the form again, passing in our sanitized genre object and the array of error messages (errors.array()).

If the genre name data is valid then we perform a case-insensitive search to see if a Genre with the same name already exists (as we don’t want to create duplicate or near duplicate records that vary only in letter case, such as: “Fantasy”, “fantasy”, “FaNtAsY”, and so on). To ignore letter case and accents when searching we chain the collation() method, specifying the locale of ‘en’ and strength of 2 (for more information see the MongoDB Collation topic).

If a Genre with a matching name already exists we redirect to its detail page. If not, we save the new Genre and redirect to its detail page. Note that here we await on the result of the database query, following the same pattern as in other route handlers.

View

The same view is rendered in both the GET and POST controllers/routes when we create a new Genre (and later on it is also used when we update a Genre). In the GET case the form is empty, and we just pass a title variable. In the POST case the user has previously entered invalid data—in the genre variable we pass back a sanitized version of the entered data and in the errors variable we pass back an array of error messages. The code below shows the controller code for rendering the template in both cases.

// Render the GET route
res.render("genre_form", { title: "Create Genre" });

// Render the POST route
res.render("genre_form", {
  title: "Create Genre",
  genre,
  errors: errors.array(),
});

Create /views/genre_form.pug and copy in the text below.

extends layout

block content

  h1 #{title}

  form(method='POST')
    div.form-group
      label(for='name') Genre:
      input#name.form-control(type='text', placeholder='Fantasy, Poetry etc.' name='name' required value=(undefined===genre ? '' : genre.name) )
    button.btn.btn-primary(type='submit') Submit

  if errors
    ul
      for error in errors
        li!= error.msg

Much of this template will be familiar from our previous tutorials. First, we extend the layout.pug base template and override the block named ‘content’. We then have a heading with the title we passed in from the controller (via the render() method).

Next, we have the pug code for our HTML form that uses method=”POST” to send the data to the server, and because the action is an empty string, will send the data to the same URL as the page.

The form defines a single required field of type “text” called “name”. The default value of the field depends on whether the genre variable is defined. If called from the GET route it will be empty as this is a new form. If called from a POST route it will contain the (invalid) value originally entered by the user.

The last part of the page is the error code. This prints a list of errors, if the error variable has been defined (in other words, this section will not appear when the template is rendered on the GET route).

Note: This is just one way to render the errors. You can also get the names of the affected fields from the error variable, and use these to control where the error messages are rendered, whether to apply custom CSS, etc.

Note: Our validation uses trim() to ensure that whitespace is not accepted as a genre name. We also validate that the field is not empty on the client side by adding the boolean attribute required to the field definition in the form:

input#name.form-control(type='text', placeholder='Fantasy, Poetry etc.' name='name' required value=(undefined===genre ? '' : genre.name) )

Create Author form

Import validation and sanitization methods

As with the genre form, to use express-validator we have to require the functions we want to use.

Open /controllers/authorController.js, and add the following line at the top of the file (above the route functions):

const { body, validationResult } = require("express-validator");

Controller—get route

Find the exported author_create_get() controller method and replace it with the following code. This renders the author_form.pug view, passing a title variable.

// Display Author create form on GET.
exports.author_create_get = (req, res, next) => {
  res.render("author_form", { title: "Create Author" });
};

Controller—post route

Find the exported author_create_post() controller method, and replace it with the following code.

// Handle Author create on POST.
exports.author_create_post = [
  // Validate and sanitize fields.
  body("first_name")
    .trim()
    .isLength({ min: 1 })
    .escape()
    .withMessage("First name must be specified.")
    .isAlphanumeric()
    .withMessage("First name has non-alphanumeric characters."),
  body("family_name")
    .trim()
    .isLength({ min: 1 })
    .escape()
    .withMessage("Family name must be specified.")
    .isAlphanumeric()
    .withMessage("Family name has non-alphanumeric characters."),
  body("date_of_birth", "Invalid date of birth")
    .optional({ values: "falsy" })
    .isISO8601()
    .toDate(),
  body("date_of_death", "Invalid date of death")
    .optional({ values: "falsy" })
    .isISO8601()
    .toDate(),

  // Process request after validation and sanitization.
  asyncHandler(async (req, res, next) => {
    // Extract the validation errors from a request.
    const errors = validationResult(req);

    // Create Author object with escaped and trimmed data
    const author = new Author({
      first_name: req.body.first_name,
      family_name: req.body.family_name,
      date_of_birth: req.body.date_of_birth,
      date_of_death: req.body.date_of_death,
    });

    if (!errors.isEmpty()) {
      // There are errors. Render form again with sanitized values/errors messages.
      res.render("author_form", {
        title: "Create Author",
        author: author,
        errors: errors.array(),
      });
      return;
    } else {
      // Data from form is valid.

      // Save author.
      await author.save();
      // Redirect to new author record.
      res.redirect(author.url);
    }
  }),
];

Warning: Never validate names using isAlphanumeric() (as we have done above) as there are many names that use other character sets. We do it here in order to demonstrate how the validator is used, and how it can be daisy-chained with other validators and error reporting.

The structure and behavior of this code is almost exactly the same as for creating a Genre object. First we validate and sanitize the data. If the data is invalid then we re-display the form along with the data that was originally entered by the user and a list of error messages. If the data is valid then we save the new author record and redirect the user to the author detail page.

Unlike with the Genre post handler, we don’t check whether the Author object already exists before saving it. Arguably we should, though as it is now we can have multiple authors with the same name.

The validation code demonstrates several new features:

• We can daisy chain validators, using withMessage() to specify the error message to display if the previous validation method fails. This makes it very easy to provide specific error messages without lots of code duplication.

• We can use the optional() function to run a subsequent validation only if a field has been entered (this allows us to validate optional fields). For example, below we check that the optional date of birth is an ISO8601-compliant date (the { values: “falsy” } object passed means that we’ll accept either an empty string or null as an empty value).

• Parameters are received from the request as strings. We can use toDate() (or toBoolean()) to cast these to the proper JavaScript types (as shown at the end of the validator chain above).

View

Create /views/author_form.pug and copy in the text below.

extends layout

block content
  h1=title

  form(method='POST')
    div.form-group
      label(for='first_name') First Name:
      input#first_name.form-control(type='text', placeholder='First name (Christian)' name='first_name' required value=(undefined===author ? '' : author.first_name) )
      label(for='family_name') Family Name:
      input#family_name.form-control(type='text', placeholder='Family name (Surname)' name='family_name' required value=(undefined===author ? '' : author.family_name))
    div.form-group
      label(for='date_of_birth') Date of birth:
      input#date_of_birth.form-control(type='date' name='date_of_birth' value=(undefined===author ? '' : author.date_of_birth) )
    button.btn.btn-primary(type='submit') Submit

  if errors
    ul
      for error in errors
        li!= error.msg

The structure and behavior for this view is exactly the same as for the genre_form.pug template, so we won’t describe it again.

Note: Some browsers don’t support the input type=”date”, so you won’t get the datepicker widget or the default dd/mm/yyyy placeholder, but will instead get an empty plain text field. One workaround is to explicitly add the attribute placeholder=’dd/mm/yyyy’ so that on less capable browsers you will still get information about the desired text format.

Note: If you experiment with various input formats for the dates, you may find that the format yyyy-mm-dd misbehaves. This is because JavaScript treats date strings as including the time of 0 hours, but additionally treats date strings in that format (the ISO 8601 standard) as including the time 0 hours UTC, rather than the local time. If your time zone is west of UTC, the date display, being local, will be one day before the date you entered. This is one of several complexities (such as multi-word family names and multi-author books) that we are not addressing here.

Create Book form

This subarticle shows how to define a page/form to create Book objects. This is a little more complicated than the equivalent Author or Genre pages because we need to get and display available Author and Genre records in our Book form.

Import validation and sanitization methods

Open /controllers/bookController.js, and add the following line at the top of the file (before the route functions):

const { body, validationResult } = require("express-validator");

Controller—get route

Find the exported book_create_get() controller method and replace it with the following code:

// Display book create form on GET.
exports.book_create_get = asyncHandler(async (req, res, next) => {
  // Get all authors and genres, which we can use for adding to our book.
  const [allAuthors, allGenres] = await Promise.all([
    Author.find().sort({ family_name: 1 }).exec(),
    Genre.find().sort({ name: 1 }).exec(),
  ]);

  res.render("book_form", {
    title: "Create Book",
    authors: allAuthors,
    genres: allGenres,
  });
});

This uses await on the result of Promise.all() to get all Author and Genre objects in parallel (the same approach used in Express Tutorial Part 5: Displaying library data). These are then passed to the view book_form.pug as variables named authors and genres (along with the page title).

Controller—post route

Find the exported book_create_post() controller method and replace it with the following code.

// Handle book create on POST.
exports.book_create_post = [
  // Convert the genre to an array.
  (req, res, next) => {
    if (!Array.isArray(req.body.genre)) {
      req.body.genre =
        typeof req.body.genre === "undefined" ? [] : [req.body.genre];
    }
    next();
  },

  // Validate and sanitize fields.
  body("title", "Title must not be empty.")
    .trim()
    .isLength({ min: 1 })
    .escape(),
  body("author", "Author must not be empty.")
    .trim()
    .isLength({ min: 1 })
    .escape(),
  body("summary", "Summary must not be empty.")
    .trim()
    .isLength({ min: 1 })
    .escape(),
  body("isbn", "ISBN must not be empty").trim().isLength({ min: 1 }).escape(),
  body("genre.*").escape(),
  // Process request after validation and sanitization.

  asyncHandler(async (req, res, next) => {
    // Extract the validation errors from a request.
    const errors = validationResult(req);

    // Create a Book object with escaped and trimmed data.
    const book = new Book({
      title: req.body.title,
      author: req.body.author,
      summary: req.body.summary,
      isbn: req.body.isbn,
      genre: req.body.genre,
    });

    if (!errors.isEmpty()) {
      // There are errors. Render form again with sanitized values/error messages.

      // Get all authors and genres for form.
      const [allAuthors, allGenres] = await Promise.all([
        Author.find().sort({ family_name: 1 }).exec(),
        Genre.find().sort({ name: 1 }).exec(),
      ]);

      // Mark our selected genres as checked.
      for (const genre of allGenres) {
        if (book.genre.includes(genre._id)) {
          genre.checked = "true";
        }
      }
      res.render("book_form", {
        title: "Create Book",
        authors: allAuthors,
        genres: allGenres,
        book: book,
        errors: errors.array(),
      });
    } else {
      // Data from form is valid. Save book.
      await book.save();
      res.redirect(book.url);
    }
  }),
];

The structure and behavior of this code is almost exactly the same as the post route functions for the Genre and Author forms. First we validate and sanitize the data. If the data is invalid then we re-display the form along with the data that was originally entered by the user and a list of error messages. If the data is valid, we then save the new Book record and redirect the user to the book detail page.

The main difference with respect to the other form handling code is how we sanitize the genre information. The form returns an array of Genre items (while for other fields it returns a string). In order to validate the information we first convert the request to an array (required for the next step).

[
  // Convert the genre to an array.
  (req, res, next) => {
    if (!Array.isArray(req.body.genre)) {
      req.body.genre =
        typeof req.body.genre === "undefined" ? [] : [req.body.genre];
    }
    next();
  },
  // …
];

We then use a wildcard (*) in the sanitizer to individually validate each of the genre array entries. The code below shows how - this translates to “sanitize every item below key genre”.

[
  // …
  body("genre.*").escape(),
  // …
];

The final difference with respect to the other form handling code is that we need to pass in all existing genres and authors to the form. In order to mark the genres that were checked by the user we iterate through all the genres and add the checked=”true” parameter to those that were in our post data (as reproduced in the code fragment below).

// Mark our selected genres as checked.
for (const genre of allGenres) {
  if (book.genre.includes(genre._id)) {
    genre.checked = "true";
  }
}

View

Create /views/book_form.pug and copy in the text below.

extends layout

block content
  h1= title

  form(method='POST')
    div.form-group
      label(for='title') Title:
      input#title.form-control(type='text', placeholder='Name of book' name='title' required value=(undefined===book ? '' : book.title) )
    div.form-group
      label(for='author') Author:
      select#author.form-control(name='author' required)
        option(value='') --Please select an author--
        for author in authors
          if book
            if author._id.toString()===book.author._id.toString()
              option(value=author._id selected) #{author.name}
            else
              option(value=author._id) #{author.name}
          else
            option(value=author._id) #{author.name}
    div.form-group
      label(for='summary') Summary:
      textarea#summary.form-control(placeholder='Summary' name='summary' required)= undefined===book ? '' : book.summary
    div.form-group
      label(for='isbn') ISBN:
      input#isbn.form-control(type='text', placeholder='ISBN13' name='isbn' value=(undefined===book ? '' : book.isbn) required)
    div.form-group
      label Genre:
      div
        for genre in genres
          div(style='display: inline; padding-right:10px;')
            if genre.checked
              input.checkbox-input(type='checkbox', name='genre', id=genre._id, value=genre._id, checked)
            else
              input.checkbox-input(type='checkbox', name='genre', id=genre._id, value=genre._id)
            label(for=genre._id)  #{genre.name}
    button.btn.btn-primary(type='submit') Submit

  if errors
    ul
      for error in errors
        li!= error.msg

The view structure and behavior is almost the same as for the genre_form.pug template.

The main differences are in how we implement the selection-type fields: Author and Genre.

• The set of genres are displayed as checkboxes, and use the checked value we set in the controller to determine whether or not the box should be selected.

• The set of authors are displayed as a single-selection alphabetically ordered drop-down list (the list passed to the template is already sorted, so we don’t need to do that in the template). If the user has previously selected a book author (i.e. when fixing invalid field values after initial form submission, or when updating book details) the author will be re-selected when the form is displayed. Here we determine what author to select by comparing the id of the current author option with the value previously entered by the user (passed in via the book variable).

Note: If there is an error in the submitted form, then, when the form is to be re-rendered, the new book author’s id and the existing books’s authors ids are of type Schema.Types.ObjectId. So to compare them we must convert them to strings first.

Create BookInstance form

This subarticle shows how to define a page/form to create BookInstance objects. This is very much like the form we used to create Book objects.

Import validation and sanitization methods

Open /controllers/bookinstanceController.js, and add the following lines at the top of the file:

const { body, validationResult } = require("express-validator");

Controller—get route

At the top of the file, require the Book module (needed because each BookInstance is associated with a particular Book).

const Book = require("../models/book");

Find the exported bookinstance_create_get() controller method and replace it with the following code.

// Display BookInstance create form on GET.
exports.bookinstance_create_get = asyncHandler(async (req, res, next) => {
  const allBooks = await Book.find({}, "title").sort({ title: 1 }).exec();

  res.render("bookinstance_form", {
    title: "Create BookInstance",
    book_list: allBooks,
  });
});

The controller gets a sorted list of all books (allBooks) and passes it via book_list to the view bookinstance_form.pug (along with a title). Note that no book has been selected when we first display this form, so we don’t pass the selected_book variable to render(). Because of this, selected_book will have a value of undefined in the template.

Controller—post route

Find the exported bookinstance_create_post() controller method and replace it with the following code.

// Handle BookInstance create on POST.
exports.bookinstance_create_post = [
  // Validate and sanitize fields.
  body("book", "Book must be specified").trim().isLength({ min: 1 }).escape(),
  body("imprint", "Imprint must be specified")
    .trim()
    .isLength({ min: 1 })
    .escape(),
  body("status").escape(),
  body("due_back", "Invalid date")
    .optional({ values: "falsy" })
    .isISO8601()
    .toDate(),

  // Process request after validation and sanitization.
  asyncHandler(async (req, res, next) => {
    // Extract the validation errors from a request.
    const errors = validationResult(req);

    // Create a BookInstance object with escaped and trimmed data.
    const bookInstance = new BookInstance({
      book: req.body.book,
      imprint: req.body.imprint,
      status: req.body.status,
      due_back: req.body.due_back,
    });

    if (!errors.isEmpty()) {
      // There are errors.
      // Render form again with sanitized values and error messages.
      const allBooks = await Book.find({}, "title").sort({ title: 1 }).exec();

      res.render("bookinstance_form", {
        title: "Create BookInstance",
        book_list: allBooks,
        selected_book: bookInstance.book._id,
        errors: errors.array(),
        bookinstance: bookInstance,
      });
      return;
    } else {
      // Data from form is valid
      await bookInstance.save();
      res.redirect(bookInstance.url);
    }
  }),
];

The structure and behavior of this code is the same as for creating our other objects. First we validate and sanitize the data. If the data is invalid, we then re-display the form along with the data that was originally entered by the user and a list of error messages. If the data is valid, we save the new BookInstance record and redirect the user to the detail page.

View

Create /views/bookinstance_form.pug and copy in the text below.

extends layout

block content
  h1=title

  form(method='POST')
    div.form-group
      label(for='book') Book:
      select#book.form-control(name='book' required)
        option(value='') --Please select a book--
        for book in book_list
          if selected_book==book._id.toString()
            option(value=book._id, selected) #{book.title}
          else
            option(value=book._id) #{book.title}

    div.form-group
      label(for='imprint') Imprint:
      input#imprint.form-control(type='text' placeholder='Publisher and date information' name='imprint' required value=(undefined===bookinstance ? '' : bookinstance.imprint) )
    div.form-group
      label(for='due_back') Date when book available:
      input#due_back.form-control(type='date' name='due_back' value=(undefined===bookinstance ? '' : bookinstance.due_back_yyyy_mm_dd))

    div.form-group
      label(for='status') Status:
      select#status.form-control(name='status' required)
        option(value='') --Please select a status--
        each val in ['Maintenance', 'Available', 'Loaned', 'Reserved']
          if undefined===bookinstance || bookinstance.status!=val
            option(value=val)= val
          else
            option(value=val selected)= val

    button.btn.btn-primary(type='submit') Submit

  if errors
    ul
      for error in errors
        li!= error.msg

Note: The above template hard-codes the Status values (Maintenance, Available, etc.) and does not “remember” the user’s entered values. Should you so wish, consider reimplementing the list, passing in option data from the controller and setting the selected value when the form is re-displayed.

The view structure and behavior is almost the same as for the book_form.pug template, so we won’t go over it in detail. The one thing to note is the line where we set the “due back” date to bookinstance.due_back_yyyy_mm_dd if we are populating the date input for an existing instance.

input#due_back.form-control(type='date', name='due_back' value=(undefined===bookinstance ? '' : bookinstance.due_back_yyyy_mm_dd))

The date value has to be set in the format YYYY-MM-DD because this is expected by elements with type=”date”, however the date is not stored in this format so we have to convert it before setting the value in the control. The due_back_yyyy_mm_dd() method is added to the BookInstance model in the next section.

Model—virtual due_back_yyyy_mm_dd() method

Open the file where you defined the BookInstanceSchema model (models/bookinstance.js). Add the due_back_yyyy_mm_dd() virtual function shown below (after the due_back_formatted() virtual function):

BookInstanceSchema.virtual("due_back_yyyy_mm_dd").get(function () {
  return DateTime.fromJSDate(this.due_back).toISODate(); // format 'YYYY-MM-DD'
});

Delete Author form

This subarticle shows how to define a page to delete Author objects.

As discussed in the form design section, our strategy will be to only allow deletion of objects that are not referenced by other objects (in this case that means we won’t allow an Author to be deleted if it is referenced by a Book). In terms of implementation this means that the form needs to confirm that there are no associated books before the author is deleted. If there are associated books, it should display them, and state that they must be deleted before the Author object can be deleted.

Controller—get route

Open /controllers/authorController.js. Find the exported author_delete_get() controller method and replace it with the following code.

// Display Author delete form on GET.
exports.author_delete_get = asyncHandler(async (req, res, next) => {
  // Get details of author and all their books (in parallel)
  const [author, allBooksByAuthor] = await Promise.all([
    Author.findById(req.params.id).exec(),
    Book.find({ author: req.params.id }, "title summary").exec(),
  ]);

  if (author === null) {
    // No results.
    res.redirect("/catalog/authors");
  }

  res.render("author_delete", {
    title: "Delete Author",
    author: author,
    author_books: allBooksByAuthor,
  });
});

The controller gets the id of the Author instance to be deleted from the URL parameter (req.params.id). It uses await on the promise returned by Promise.all() to asynchronously wait on the specified author record and all associated books (in parallel). When both operations have completed it renders the author_delete.pug view, passing variables for the title, author, and author_books.

Note: If findById() returns no results the author is not in the database. In this case there is nothing to delete, so we immediately redirect to the list of all authors.