← Home

Trying out npm and yarn workspaces

I’m doing most of my work with CDK at the moment, to deploy Serverless projects. These projects often consist of multiple projects in the same repo - i.e. a monorepo. There’s the CDK project itself, containing the definition of the infrastructure, and one or more projects containing application code that will be deployed as AWS Lambda functions.

For example, a project I’m working on has a cdk directory and a web directory.

Within each directory is a package.json that defines the libraries that are used by the project.

./cdk/package.json
./cdk/index.ts
./web/package.json
./web/index.ts

In the case of a TypeScript or JavaScript CDK project, the dependenices are things like @aws-cdk/aws-lambda-nodejs that are used as part of the deployment process, but aren’t included in any build outputs.

In the case of a TypeScript or JavaScript project being used to create AWS Lambda functions, libraries that are referenced within the package.json need to be bundled into the zip file or Docker container used to create the Lambda function, along with custom code.

In both cases, this means that a CI pipeline will need to install packages before a CDK deployment or packaging can occur.

This can be done by simply running npm install or yarn install in both directories one after the other.

name: GitHub Actions Demo
on: [push]
jobs:
  npm-install:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: '14'
      - name: web - Install npm modules
        run: cd ./web && npm install
      - name: cdk - Install npm modules
        run: cd ./cdk && npm install

Since it takes time to run the installation, it’s tempting to try and run them in parallel using background tasks in bash, but this can cause problems, and take extra time, as there are two processes racing to download (often) the same things to the same npm or yarn cache locations.

(cd ./web && npm install) &
(cd ./cdk && npm install) &
wait

Workspaces

Yarn introduced the concept of workspaces [0] a few years back to solve the problem, and npm introduced support for them in version 7 [1] which was released in Feb 2021, so I thought this would be an easy way to speed up the installation.

You place a package.json file in the root of the repo and npm or yarn uses this to work out which packages to run.

And you end up with a repo structured like this.

./package.json <-- new!
./cdk/package.json
./cdk/index.ts
./web/package.json
./web/index.ts

The package.json in the root has a new “workspaces” section which tells yarn or npm where the packages are.

{
  "name": "workspace",
  "private": true,
  "workspaces": {
    "packages": [
      "web",
      "api"
    ]
  }
}

With this in place, you only need to run npm install or yarn install once, in the root of the directory. It sounds straightforward, but I ran into a few issues.

Github actions NPM support

In Github Actions, Node 14 still uses npm 6 [2] which doesn’t support workspaces. npm 6 doesn’t throw an obvious error, so it can look like it’s working from the root directory, but it doesn’t actually install anything.

Hoisting

When you run yarn install in the root directory, all of the required packages only get added to the node_modules folder in the root directory, so you don’t get web/node_modules and api/node_modules directories.

If you cd into the web directory and run your project, it will work fine because node will look in the parent directory and find the node module there, but if you move the web directory into another directory tree, then it will fail due to missing dependencie.

This behaviour isn’t often what you want in a CI pipeline, because you might need to mount a child directory within your mono repo into a Docker image with the node_modules present, or bundle the directory up in some way.

Fortunately, the yarn team added the nohoist feature [3] which creates node_modules in all of the subdirectories.

It requires a minor update to the package.json.

{
  "name": "workspace",
  "private": true,
  "workspaces": {
    "packages": [
      "web",
      "api"
    ],
    "nohoist": [
      "**"
    ]
  }
}

However, it’s not supported in npm at the moment [4], which cost me a bit of time trying to work out why it didn’t work.

If you’re using npm, you can consider lerna

The npm alternative is more tooling. lerna [5] can execute the multiple npm installation commands for you.

Instead of adding a workspaces section to your package.json, you add a lerna.json file to the root which accomplishes the same goal.

{
    "packages": [
        "web",
        "api"
    ],
    "version": "0.1.0"
}

Then, executing npx lerna bootstrap will populate the node_modules directories in the child paths.

But it’s another tool that people need to know about, versus a simple shell command, so for npm projects, I’ll stick with the shell command for now, until npm workspaces are a little more developed.

(cd web && npm install) && (cd api && npm install)

Conclusion

If you’re using yarn, workspaces seem to be a quick win, but check if your CI pipeline needs that “hoisting” feature.