Olle Lundberg
79f7461709
Piping find to xargs is dangerous as xargs will interpret any characters
defined in $IFS (Input Field Separator), usually <space><tab><newline>,
as separators in its input. This might lead to unintented operations
on files if any of the input to xargs containts any of the characters
defined in $IFS, most often this happens if a file contains a space in
its name. The safest way to execute a find | xargs is to force find to
separate its output with null characters and tell xargs to read null
characters as the delimiter AND also tell xargs to put the argument to
the command its supposed to run in quotation marks:
`find ... -print0 | xargs -I {} -0 rm -rf "{}"`
When running with GNU find you most likely also want to add
--no-run-if-empty or -r for short:
`find ... -print0 | xargs -I {} -0 --no-run-if-empty rm -rf "{}"`
This stops the invocation of xargs if there is no input on stdin, this
is however not portable and will break on BSD/macOS, the portability is
not a concern in this case though as the find | xargs happens in docker.
As you can see this gets unwieldly fast and despite using every
precaution, it's still not safe. When xargs is run with -0 to treat null
characters as the delimiter for its input and a file has a null
character in its name, xargs will treat the null character in the file
name as a delimiter and xargs will exhibit the same behaviour as it did
with spaces in file names.
Ahhh, isn't Unix wonderful? Loose APIs defined as untyped strings...
There is a salvation though! Most find xargs pipes are unnecessary and
can be replaced with built in functionality in find, the -exec flag.
Now, -exec comes in 2 flavours, one that is terminated with \; (the most
commonly used) and one terminated with \+ (the one most people actually
want to use). \; spawns a new invocation per found entry, thus creating
some process creation overhead. \+ instead concatenates the found
entries as arguments to the program we want to run, resulting in less
overhead and usually a faster execution.
Since find is smart enough to be aware of what constitutes an entry
(i.e it doesn't treat the entries as just a bunch of random strings to
read from stdin) it makes the whole invocation of the program, rm in
this case, safe even if it contains characters defines in $IFS or null
characters.
And with this overly elaborate commit message I bring you this 1 line
change.
8.4 KiB
id, title, description
| id | title | description |
|---|---|---|
| deployment-docker | Docker | Documentation on how to deploy Backstage as a Docker image |
This section describes how to build a Backstage App into a deployable Docker image. It is split into three sections, first covering the host build approach, which is recommended due its speed and more efficient and often simpler caching. The second section covers a full multi-stage Docker build, and the last section covers how to split frontend content into a separate image.
Something that goes for all of these docker deployment strategies is that they are stateless, so for a production deployment you will want to set up and connect to an external PostgreSQL instance where the backend plugins can store their state, rather than using SQLite.
Host Build
This section describes how to build a Docker image from a Backstage repo with most of the build happening outside of Docker. This is almost always the faster approach, as the build steps tend to execute faster, and it's possible to have more efficient caching of dependencies on the host, where a single change won't bust the entire cache.
The required steps in the host build are to install dependencies with
yarn install, generate type definitions using yarn tsc, and build all
packages with yarn build.
NOTE: Using
yarn buildto build packages and bundle the backend assumes that you have migrated to usingbackstage-cli backend:bundleas your build script in the backend package.
In a CI workflow it might look something like this:
yarn install --frozen-lockfile
# tsc outputs type definitions to dist-types/ in the repo root, which are then consumed by the build
yarn tsc
# Build all packages and in the end bundle them all up into the packages/backend/dist folder.
yarn build
Once the host build is complete, we are ready to build our image. We use the
following Dockerfile, which is also included when creating a new app with
@backstage/create-app:
# This dockerfile builds an image for the backend package.
# It should be executed with the root of the repo as docker context.
#
# Before building this image, be sure to have run the following commands in the repo root:
#
# yarn install
# yarn tsc
# yarn build
#
# Once the commands have been run, you can build the image using `yarn build-image`
FROM node:14-buster-slim
WORKDIR /app
# Copy repo skeleton first, to avoid unnecessary docker cache invalidation.
# The skeleton contains the package.json of each package in the monorepo,
# and along with yarn.lock and the root package.json, that's enough to run yarn install.
ADD yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./
RUN yarn install --frozen-lockfile --production --network-timeout 300000 && rm -rf "$(yarn cache dir)"
# Then copy the rest of the backend bundle, along with any other files we might want.
ADD packages/backend/dist/bundle.tar.gz app-config.yaml ./
CMD ["node", "packages/backend", "--config", "app-config.yaml"]
For more details on how the backend:bundle command and the skeleton.tar.gz
file works, see the
backend:bundle command docs
The Dockerfile is typically placed at packages/backend/Dockerfile, but needs
to be executed with the root of the repo as the build context, in order to get
access to the root yarn.lock and package.json, along with any other files
that might be needed, such as .npmrc.
In order to speed up the build we can significantly reduce the build context
size using the following .dockerignore in the root of the repo:
.git
node_modules
packages
!packages/backend/dist
plugins
With the project build and the .dockerignore and Dockerfile in place, we are
now ready to build the final image. Assuming we're at the root of the repo, we
execute the build like this:
docker image build . -f packages/backend/Dockerfile --tag backstage
To try out the image locally you can run the following:
docker run -it -p 7000:7000 backstage
You should then start to get logs in your terminal, and then you can open your
browser at http://localhost:7000
Multistage Build
This section describes how to set up a multi-stage Docker build that builds the entire project within Docker. This is typically slower than a host build, but is sometimes desired because Docker in Docker is not available in the build environment, or due to other requirements.
The build is split into three different stages, where the first stage finds all
of the package.jsons that are relevant for the initial install step enabling
us to cache the initial yarn install that installs all dependencies. The
second stage executes the build itself, and is similar to the steps we execute
on the host in the host build. The third and final stage then packages it all
together into the final image, and is similar to the Dockerfile of the host
build.
The following Dockerfile executes the multi-stage build and should be added to
the repo root:
# Stage 1 - Create yarn install skeleton layer
FROM node:14-buster-slim AS packages
WORKDIR /app
COPY package.json yarn.lock ./
COPY packages packages
COPY plugins plugins
RUN find packages \! -name "package.json" -mindepth 2 -maxdepth 2 -exec rm -rf {} \+
# Stage 2 - Install dependencies and build packages
FROM node:14-buster-slim AS build
WORKDIR /app
COPY --from=packages /app .
RUN yarn install --frozen-lockfile --network-timeout 600000 && rm -rf "$(yarn cache dir)"
COPY . .
RUN yarn tsc
RUN yarn --cwd packages/backend backstage-cli backend:bundle --build-dependencies
# Stage 3 - Build the actual backend image and install production dependencies
FROM node:14-buster-slim
WORKDIR /app
# Copy the install dependencies from the build stage and context
COPY --from=build /app/yarn.lock /app/package.json /app/packages/backend/dist/skeleton.tar.gz ./
RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz
RUN yarn install --frozen-lockfile --production --network-timeout 600000 && rm -rf "$(yarn cache dir)"
# Copy the built packages from the build stage
COPY --from=build /app/packages/backend/dist/bundle.tar.gz .
RUN tar xzf bundle.tar.gz && rm bundle.tar.gz
# Copy any other files that we need at runtime
COPY app-config.yaml ./
CMD ["node", "packages/backend", "--config", "app-config.yaml"]
Note that a newly created Backstage app will typically not have a plugins/
folder, so you will want to comment that line out. This build also does not work
in the main repo, since the backstage-cli which is used for the build doesn't
end up being properly installed.
To speed up the build when not running in a fresh clone of the repo you should
set up a .dockerignore. This one is different than the host build one, because
we want to have access to the source code of all packages for the build, but can
ignore any existing build output or dependencies:
node_modules
packages/*/dist
packages/*/node_modules
plugins/*/dist
plugins/*/node_modules
Once you have added both the Dockerfile and .dockerignore to the root of
your project, run the following to build the container under a specified tag.
docker image build -t backstage .
To try out the image locally you can run the following:
docker run -it -p 7000:7000 backstage
You should then start to get logs in your terminal, and then you can open your
browser at http://localhost:7000
Separate Frontend
It is sometimes desirable to serve the frontend separately from the backend,
either from a separate image or for example a static file serving provider. The
first step in doing so is to remove the app-backend plugin from the backend
package, which is done as follows:
- Delete
packages/backend/src/plugins/app.ts - Remove the following lines from
packages/backend/src/index.ts:import app from './plugins/app'; // ... const appEnv = useHotMemoize(module, () => createEnv('app')); // ... .addRouter('', await app(appEnv)); - Remove the
@backstage/plugin-app-backendand the app package dependency (e.g.app) frompackages/backend/packages.json. If you don't remove the app package dependency the app will still be built and bundled with the backend.
Once the app-backend is removed from the backend, you can use your favorite
static file serving method for serving the frontend. An example of how to set up
an NGINX image is available in the
contrib folder in the main repo