Merge pull request #7355 from backstage/feat/blam/nunjucks-template
Add support for `scaffolder.backstage.io/v1beta3` Software Templates
This commit is contained in:
Ben Lambert
@@ -0,0 +1,5 @@
|
||||
---
|
||||
'@backstage/plugin-scaffolder-backend': patch
|
||||
---
|
||||
|
||||
Introduce the new `scaffolder.backstage.io/v1beta3` template kind with nunjucks support 🥋
|
||||
@@ -620,9 +620,6 @@ The following describes the following entity kind:
|
||||
| `apiVersion` | `backstage.io/v1beta2` |
|
||||
| `kind` | `Template` |
|
||||
|
||||
If you're looking for docs on `v1alpha1` you can find them
|
||||
[here](../software-templates/legacy.md)
|
||||
|
||||
A template definition describes both the parameters that are rendered in the
|
||||
frontend part of the scaffolding wizard, and the steps that are executed when
|
||||
scaffolding that component.
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
---
|
||||
id: template-legacy
|
||||
title: Writing Templates (Legacy)
|
||||
# prettier-ignore
|
||||
description: Old documentation describing the backstage.io/v1alpha1 format of the Template Schema
|
||||
---
|
||||
|
||||
## Kind: Template
|
||||
|
||||
Describes the following entity kind:
|
||||
|
||||
| Field | Value |
|
||||
| ------------ | ----------------------- |
|
||||
| `apiVersion` | `backstage.io/v1alpha1` |
|
||||
| `kind` | `Template` |
|
||||
|
||||
A Template describes a skeleton for use with the Scaffolder. It is used for
|
||||
describing what templating library is supported, and also for documenting the
|
||||
variables that the template requires using
|
||||
[JSON Forms Schema](https://jsonforms.io/).
|
||||
|
||||
Descriptor files for this kind may look as follows.
|
||||
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Template
|
||||
metadata:
|
||||
name: react-ssr-template
|
||||
title: React SSR Template
|
||||
description:
|
||||
Next.js application skeleton for creating isomorphic web applications.
|
||||
tags:
|
||||
- recommended
|
||||
- react
|
||||
spec:
|
||||
owner: web@example.com
|
||||
templater: cookiecutter
|
||||
type: website
|
||||
path: '.'
|
||||
schema:
|
||||
required:
|
||||
- component_id
|
||||
- description
|
||||
properties:
|
||||
component_id:
|
||||
title: Name
|
||||
type: string
|
||||
description: Unique name of the component
|
||||
description:
|
||||
title: Description
|
||||
type: string
|
||||
description: Description of the component
|
||||
```
|
||||
|
||||
In addition to the [common envelope metadata](#common-to-all-kinds-the-metadata)
|
||||
shape, this kind has the following structure.
|
||||
|
||||
### `apiVersion` and `kind` [required]
|
||||
|
||||
Exactly equal to `backstage.io/v1alpha1` and `Template`, respectively.
|
||||
|
||||
### `metadata.title` [required]
|
||||
|
||||
The nice display name for the template as a string, e.g. `React SSR Template`.
|
||||
This field is required as is used to reference the template to the user instead
|
||||
of the `metadata.name` field.
|
||||
|
||||
### `metadata.tags` [optional]
|
||||
|
||||
A list of strings that can be associated with the template, e.g.
|
||||
`['recommended', 'react']`.
|
||||
|
||||
This list will also be used in the frontend to display to the user so you can
|
||||
potentially search and group templates by these tags.
|
||||
|
||||
### `spec.type` [optional]
|
||||
|
||||
The type of component as a string, e.g. `website`. This field is optional but
|
||||
recommended.
|
||||
|
||||
The software catalog accepts any type value, but an organization should take
|
||||
great care to establish a proper taxonomy for these. Tools including Backstage
|
||||
itself may read this field and behave differently depending on its value. For
|
||||
example, a website type component may present tooling in the Backstage interface
|
||||
that is specific to just websites.
|
||||
|
||||
The current set of well-known and common values for this field is:
|
||||
|
||||
- `service` - a backend service, typically exposing an API
|
||||
- `website` - a website
|
||||
- `library` - a software library, such as an npm module or a Java library
|
||||
|
||||
### `spec.templater` [required]
|
||||
|
||||
The templating library that is supported by the template skeleton as a string,
|
||||
e.g `cookiecutter`.
|
||||
|
||||
Different skeletons will use different templating syntax, so it's common that
|
||||
the template will need to be run with a particular piece of software.
|
||||
|
||||
This key will be used to identify the correct templater which is registered into
|
||||
the `TemplatersBuilder`.
|
||||
|
||||
The values which are available by default are:
|
||||
|
||||
- `cookiecutter` - [cookiecutter](https://github.com/cookiecutter/cookiecutter).
|
||||
|
||||
### `spec.path` [optional]
|
||||
|
||||
The string location where the templater should be run if it is not on the same
|
||||
level as the `template.yaml` definition, e.g. `./cookiecutter/skeleton`.
|
||||
|
||||
This will set the `cwd` when running the templater to the folder path that you
|
||||
specify relative to the `template.yaml` definition.
|
||||
|
||||
This is also particularly useful when you have multiple template definitions in
|
||||
the same repository but only a single `template.yaml` registered in backstage.
|
||||
@@ -1,333 +0,0 @@
|
||||
---
|
||||
id: migrating-from-v1alpha1-to-v1beta2
|
||||
title: Migrating to v1beta2 templates
|
||||
# prettier-ignore
|
||||
description: How to move your old templates from v1alpha1 to the more declarative v1beta2
|
||||
---
|
||||
|
||||
# What's new?
|
||||
|
||||
Previously, the scaffolder was very restricted in what you could do when
|
||||
creating new software components from templates. There were three scaffolding
|
||||
steps which was pretty hard to extend and add new functionality to, difficult to
|
||||
re-use logic between templates. There used to be a fixed pipeline of
|
||||
`preparers`, `templaters`, and `publishers`, which were defined by the backend
|
||||
and needed to be run for each template. This is now changed, to give the
|
||||
template total control over what should be executed as part of the templating
|
||||
run. This makes templates a little more declarative as you can now register
|
||||
different `actions` or `functions` with the `scaffolder-backend` which you then
|
||||
can decide how, and in what order, to run using the template definition YAML
|
||||
file.
|
||||
|
||||
We've also made some improvements, and added some helpers to work with
|
||||
cookiecutter. The skeleton for a template can now be stored in a different place
|
||||
to where your entity definition is: previously you needed to have your
|
||||
`template.yaml` next to the skeleton source (`{{cookiecutter.component_id}}`
|
||||
directory), but now that's not the case. Part of the changes with the `v1beta2`
|
||||
syntax is that you can grab your template source from any repository, and re-use
|
||||
them between templates.
|
||||
|
||||
We've also renamed the `schema` property to `parameters` as this makes more
|
||||
sense when using them as parameters to the actions or steps that you've setup
|
||||
for your templates. There's the added benefit that you can now assign an array
|
||||
to the `parameters` property, which will then give you multiple steps in the UI,
|
||||
so you can split apart your input parameters and group them as needed rather
|
||||
than having one long list of input fields.
|
||||
|
||||
## The `parameters` property
|
||||
|
||||
The `schema` key has now been renamed to `parameters` with a few more features.
|
||||
You can pass an array now to break apart the input form into different steps in
|
||||
the UI. You can also specify `ui:schema` fields that are passed along to
|
||||
[`react-jsonschema-form`](https://rjsf-team.github.io/react-jsonschema-form/)
|
||||
inline with the JSON schema.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
parameters:
|
||||
- title: Fill in some steps
|
||||
required:
|
||||
- name
|
||||
properties:
|
||||
name:
|
||||
title: Name
|
||||
type: string
|
||||
description: Unique name of the component
|
||||
ui:autofocus: true
|
||||
ui:options:
|
||||
rows: 5
|
||||
```
|
||||
|
||||
## The `steps` property
|
||||
|
||||
`v1beta2` template syntax introduces the new `steps` property, which is an array
|
||||
of `actions` that the scaffolder will run in combination with the user input
|
||||
that is declared in the `schema`. Actions look like the following:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
steps:
|
||||
- id: publish # a unique id for the step, can be anything you like
|
||||
name: Publish # a user friendly name for the step, this is what is shown in the frontend
|
||||
action: publish:github # the action ID that has been registered with the scaffolder-backend
|
||||
input: # parameters that are passed as input to the action handler function
|
||||
allowedHosts: ['github.com']
|
||||
description: 'This is {{ parameters.name }}' # handlebars templating is supported with the values from the parameters section in the same file.
|
||||
repoUrl: '{{ parameters.repoUrl }}'
|
||||
```
|
||||
|
||||
# Migrating a `v1alpha1` template
|
||||
|
||||
## The template definition (.yaml)
|
||||
|
||||
### `parameters`
|
||||
|
||||
Because of the changes to invert the control to the `template.yaml` definition
|
||||
for running the workflow, we need to adjust the `schema` property and we also
|
||||
now need to define what the template is actually going to do as part of the
|
||||
template run.
|
||||
|
||||
A simple migration would move the following yaml:
|
||||
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1alpha1
|
||||
kind: Template
|
||||
metadata:
|
||||
name: react-ssr-template
|
||||
title: React SSR Template
|
||||
description: Create a website powered with Next.js
|
||||
tags:
|
||||
- recommended
|
||||
- react
|
||||
spec:
|
||||
owner: web@example.com
|
||||
templater: cookiecutter
|
||||
type: website
|
||||
path: '.'
|
||||
schema:
|
||||
required:
|
||||
- component_id
|
||||
- description
|
||||
properties:
|
||||
component_id:
|
||||
title: Name
|
||||
type: string
|
||||
description: Unique name of the component
|
||||
description:
|
||||
title: Description
|
||||
type: string
|
||||
description: Help others understand what this website is for.
|
||||
```
|
||||
|
||||
To something that looks like the following:
|
||||
|
||||
```yaml
|
||||
apiVersion: backstage.io/v1beta2
|
||||
kind: Template
|
||||
metadata:
|
||||
name: react-ssr-template
|
||||
title: React SSR Template
|
||||
description: Create a website powered with Next.js
|
||||
tags:
|
||||
- recommended
|
||||
- react
|
||||
spec:
|
||||
owner: web@example.com
|
||||
type: website
|
||||
parameters:
|
||||
- title: Add some input
|
||||
required:
|
||||
- component_id
|
||||
- description
|
||||
properties:
|
||||
component_id:
|
||||
title: Name
|
||||
type: string
|
||||
description: Unique name of the component
|
||||
description:
|
||||
title: Description
|
||||
type: string
|
||||
description: Help others understand what this website is for.
|
||||
- title: Some more additional info that was previously provided automatically
|
||||
required:
|
||||
- owner
|
||||
- repoUrl
|
||||
properties:
|
||||
owner:
|
||||
title: Owner
|
||||
type: string
|
||||
description: Owner of the component
|
||||
ui:field: OwnerPicker
|
||||
ui:options:
|
||||
allowedKinds:
|
||||
- Group
|
||||
- title: Choose a location
|
||||
repoUrl:
|
||||
title: Repository Location
|
||||
type: string
|
||||
ui:field: RepoUrlPicker
|
||||
ui:options:
|
||||
allowedHosts:
|
||||
- github.com
|
||||
```
|
||||
|
||||
There are a few things to note here. On the `alpha` version, the second step of
|
||||
the template flow in the frontend was provided by Backstage for free, so we used
|
||||
to collect the user input for the `owner` field and the `repositoryUrl` that you
|
||||
were going to publish to. Now because `actions` can have any workflow they like,
|
||||
it doesn't make sense to still provide these fields for every scaffolding
|
||||
workflow, as you might not need these anymore. That's why we now manually add
|
||||
those fields back into the template parameters that are shown to the user:
|
||||
|
||||
```yaml
|
||||
- title: Some more additional info that was previously provided automatically
|
||||
required:
|
||||
- owner
|
||||
- repoUrl
|
||||
properties:
|
||||
owner:
|
||||
title: Owner
|
||||
type: string
|
||||
description: Owner of the component
|
||||
ui:field: OwnerPicker
|
||||
ui:options:
|
||||
allowedKinds:
|
||||
- Group
|
||||
- title: Choose a location
|
||||
repoUrl:
|
||||
title: Repository Location
|
||||
type: string
|
||||
ui:field: RepoUrlPicker
|
||||
ui:options:
|
||||
allowedHosts:
|
||||
- github.com
|
||||
```
|
||||
|
||||
Maybe you also don't need to publish to `github.com`, you should replace this
|
||||
with your VCS provider URL that is listed in your `integrations` config instead.
|
||||
|
||||
### `steps`
|
||||
|
||||
So now we should have all the required information that we need from the user in
|
||||
a much more extensible way. We now need to tell the scaffolder what to do with
|
||||
these parameters and what to do with the user input.
|
||||
|
||||
We've made templating using `cookiecutter` a little simpler. You don't need to
|
||||
store the `cookiecutter` skeleton in the same directory as the `template.yaml`
|
||||
definition, it can live wherever you like - maybe a shared repository somewhere
|
||||
so you can re-use the skeletons but apply different actions for different
|
||||
templates depending on your use case.
|
||||
|
||||
We also no longer need to have a directory called
|
||||
`{{cookiecutter.component_id}}`. This is because now we can't ensure that
|
||||
`component_id` will be a parameter that is provided from the frontend, this
|
||||
could break `cookiecutter`. If your directory structure used to look like this:
|
||||
|
||||
```
|
||||
my-awesome-template
|
||||
-> {{cookiecutter.component_id}}
|
||||
-> file.txt
|
||||
-> some_more_files.ts
|
||||
-> hooks
|
||||
-> post_gen_project.sh
|
||||
-> template.yaml
|
||||
```
|
||||
|
||||
We now recommend that you move to the following structure:
|
||||
|
||||
```
|
||||
my-awesome-template
|
||||
-> skeleton
|
||||
-> file.txt
|
||||
-> some_more_files.ts
|
||||
-> template.yaml
|
||||
```
|
||||
|
||||
This migration renames the skeleton folder to something more semantic, and also
|
||||
drops support for `cookiecutter` hooks. We've dropped support for `cookiecutter`
|
||||
hooks for now, as hopefully everything that is stored in these hooks can be
|
||||
moved to `actions` instead, and for security reasons, it's more secure to run
|
||||
trusted code that you ship with Backstage as an action rather than some script
|
||||
that can be pulled in from anywhere which doesn't get vetted first. It's a
|
||||
pretty big security risk that those scripts will be run on Backstage instances
|
||||
inside your infrastructure, especially `.sh` files.
|
||||
|
||||
If you really need hooks and can't find a suitable solution by using actions
|
||||
please reach out to us through a ticket and we'll see what we can do to assist
|
||||
:)
|
||||
|
||||
You'll notice that we removed the `templater` property from the `spec`
|
||||
definition in the template `yaml`, so there's no way to define that this is a
|
||||
`cookiecutter` `templater`.
|
||||
|
||||
We've created a built-in action that you can use which will when run, go grab a
|
||||
directory from anywhere and run `cookiecutter` on top of it, and then extract
|
||||
the contents into the working directory for the scaffolder.
|
||||
|
||||
Adding the `steps` for a simple template should look something like the
|
||||
following:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
steps:
|
||||
# this action will go use cookiecutter to template some files into the working directory
|
||||
- id: template # an ID for the templating step
|
||||
name: Create skeleton # A user friendly name for the action
|
||||
action: fetch:cookiecutter
|
||||
input:
|
||||
url: ./skeleton # this is the directory for your skeleton files.
|
||||
# If it's located next to the `template.yaml` then you can use a relative path,
|
||||
# otherwise you can use absolute URLs that point at the VCS: https://github.com/backstage/backstage/tree/master/some_folder_somewhere
|
||||
values:
|
||||
# for each value that you need to pass to cookiecutter, they should be listed here and set in this values object.
|
||||
# You can use the handlebars templating syntax to pull them from the input parameters listed in the same file
|
||||
name: '{{ parameters.name }}'
|
||||
owner: '{{ parameters.owner }}'
|
||||
destination: '{{ parseRepoUrl parameters.repoUrl }}'
|
||||
|
||||
# this action is for publishing the working directory to the VCS
|
||||
- id: publish
|
||||
name: Publish
|
||||
action: publish:github
|
||||
input:
|
||||
allowedHosts: ['github.com']
|
||||
description: 'This is {{ parameters.name }}'
|
||||
repoUrl: '{{ parameters.repoUrl }}'
|
||||
|
||||
# this action will then register the created component in Backstage
|
||||
- id: register
|
||||
name: Register
|
||||
action: catalog:register
|
||||
input:
|
||||
repoContentsUrl: '{{ steps.publish.output.repoContentsUrl }}'
|
||||
catalogInfoPath: '/catalog-info.yaml'
|
||||
```
|
||||
|
||||
### `output`
|
||||
|
||||
Steps can output values, and so can the template itself. This is good for
|
||||
returning values to the frontend, so we can make the buttons like
|
||||
`Go to catalog` and `Go to repo` work correctly. You can add the following to
|
||||
your `template.yaml` to make sure you return the right values from the steps:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
output:
|
||||
remoteUrl: '{{ steps.publish.output.remoteUrl }}'
|
||||
entityRef: '{{ steps.register.output.entityRef }}'
|
||||
```
|
||||
|
||||
Or you can return a `links` array with text and a URL explicitly:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
output:
|
||||
links:
|
||||
- url: '{{steps.publish.output.remoteUrl}}'
|
||||
title: 'Go to Repo'
|
||||
```
|
||||
|
||||
## Questions?
|
||||
|
||||
If you have any questions or feedback, please reach out to us on GitHub or
|
||||
Discord and we will do our best to help!
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
id: migrating-from-v1beta2-to-v1beta3
|
||||
title: Migrating to v1beta3 templates
|
||||
# prettier-ignore
|
||||
description: How to migrate your existing templates to beta3 syntax
|
||||
---
|
||||
|
||||
# What's new?
|
||||
|
||||
Well then, here we are! 🚀
|
||||
|
||||
Backstage has had many forms of templating languages throughout different
|
||||
plugins and different systems. We've had `cookiecutter` syntax in templates, and
|
||||
we also had `handlebars` templating in the `kind: Template`. Then we wanted to
|
||||
remove the additional dependency on `cookiecutter` for Software Templates out of
|
||||
the box, so we introduced `nunjucks` as an alternative in `fetch:template`
|
||||
action which is based on the `jinja2` syntax so they're pretty similar. In an
|
||||
effort to reduce confusion and unify on to one templating language, we're
|
||||
officially deprecating support for `handlebars` templating in the
|
||||
`kind: Template` entities with `apiVersion` `scaffolder.backstage.io/v1beta3`
|
||||
and moving to using `nunjucks` instead.
|
||||
|
||||
This provides us a lot of built in `filters` (`handlebars` helpers), that as
|
||||
Template authors will give you much more flexibility out of the box, and also
|
||||
open up sharing of filters in the Entity and the actual `skeleton` too, and
|
||||
removing the slight differences between the two languages.
|
||||
|
||||
We've also removed a lot of the built in helpers that we shipped with
|
||||
`handlebars`, as they're now supported as first class citizens by either
|
||||
`nunjucks` or the new `scaffolder` when using `scaffolder.backstage.io/v1beta3`
|
||||
`apiVersion`
|
||||
|
||||
The migration path is pretty simple, and we've removed some of the pain points
|
||||
from writing the `handlebars` templates too. Let's go through what's new and how
|
||||
to upgrade.
|
||||
|
||||
## `backstage.io/v1beta2` -> `scaffolder.backstage.io/v1beta3`
|
||||
|
||||
The most important change is that you'll need to switch over the `apiVersion` in
|
||||
your templates to the new one.
|
||||
|
||||
```diff
|
||||
kind: Template
|
||||
- apiVersion: backstage.io/v1beta2
|
||||
+ apiVersion: scaffolder.backstage.io/v1beta3
|
||||
```
|
||||
|
||||
## `${{ }}` instead of `"{{ }}"`
|
||||
|
||||
One really big readability issue and cause for confusion was the fact that with
|
||||
`handlebars` and `yaml` you always had to wrap your templating strings in quotes
|
||||
in `yaml` so that it didn't try to parse it as a `json` object and fail. This
|
||||
was pretty annoying, as it also meant that all things look like strings. Now
|
||||
that's no longer the case, you can now remove the `""` and take advantage of
|
||||
writing nice `yaml` files that just work.
|
||||
|
||||
```diff
|
||||
spec:
|
||||
steps:
|
||||
input:
|
||||
allowedHosts: ['github.com']
|
||||
- description: 'This is {{ parameters.name }}'
|
||||
+ description: This is ${{ parameters.name }}
|
||||
- repoUrl: '{{ parameters.repoUrl }}'
|
||||
+ repoUrl: ${{ parameters.repoUrl }}
|
||||
```
|
||||
|
||||
## No more `eq` or `not` helpers
|
||||
|
||||
These helpers are no longer needed with the more expressive `api` that
|
||||
`nunjucks` provides. You can simply use the built-in `nunjucks` and `jinja2`
|
||||
style operators.
|
||||
|
||||
```diff
|
||||
spec:
|
||||
steps:
|
||||
input:
|
||||
- if: '{{ eq parameters.value "backstage" }}'
|
||||
+ if: ${{ parameters.value === "backstage" }}
|
||||
...
|
||||
```
|
||||
|
||||
And then for the `not`
|
||||
|
||||
```diff
|
||||
spec:
|
||||
steps:
|
||||
input:
|
||||
- if: '{{ not parameters.value "backstage" }}'
|
||||
+ if: ${{ parameters.value !== "backstage" }}
|
||||
...
|
||||
```
|
||||
|
||||
Much better right? ✨
|
||||
|
||||
## No more `json` helper
|
||||
|
||||
This helper is no longer needed, as we've added support for complex values and
|
||||
supporting the additional primitive values now rather than everything being a
|
||||
`string`. This means that now that you can pass around `parameters` and it
|
||||
should all work as expected and keep the type that has been declared in the
|
||||
input schema.
|
||||
|
||||
```diff
|
||||
spec:
|
||||
parameters:
|
||||
test:
|
||||
type: number
|
||||
name: Test Number
|
||||
address:
|
||||
type: object
|
||||
required:
|
||||
- line1
|
||||
properties:
|
||||
line1:🙏
|
||||
type: string
|
||||
name: Line 1
|
||||
line2:
|
||||
type: string
|
||||
name: Line 2
|
||||
|
||||
steps:
|
||||
- id: test step
|
||||
action: run:something
|
||||
input:
|
||||
- address: '{{ json parameters.address }}'
|
||||
+ address: ${{ parameters.address }}
|
||||
- number: '{{ parameters.number }}'
|
||||
+ number: ${{ parameters.number }} # this will now make sure that the type of number is a number 🙏
|
||||
```
|
||||
|
||||
## `parseRepoUrl` is now a `filter`
|
||||
|
||||
All calls to `parseRepoUrl` are now a `jinja2` `filter`, which means you'll need
|
||||
to update the syntax.
|
||||
|
||||
```diff
|
||||
spec:
|
||||
steps:
|
||||
input:
|
||||
- repoUrl: '{{ parseRepoUrl parameters.repoUrl }}'
|
||||
+ repoUrl: ${{ parameters.repoUrl | parseRepoUrl }}
|
||||
...
|
||||
```
|
||||
|
||||
Now we have complex value support here too, expect that this `filter` will go
|
||||
away in future versions and the `RepoUrlPicker` will return an object so
|
||||
`parameters.repoUrl` will already be a
|
||||
`{ host: string; owner: string; repo: string }` 🚀
|
||||
|
||||
### Summary
|
||||
|
||||
Of course, we're always available on [discord](https://discord.gg/MUpMjP2) if
|
||||
you're stuck or something's not working as expected. You can also
|
||||
[raise an issue](https://github.com/backstage/backstage/issues/new/choose) with
|
||||
feedback or bugs!
|
||||
@@ -78,8 +78,7 @@
|
||||
"features/software-templates/builtin-actions",
|
||||
"features/software-templates/writing-custom-actions",
|
||||
"features/software-templates/writing-custom-field-extensions",
|
||||
"features/software-templates/template-legacy",
|
||||
"features/software-templates/migrating-from-v1alpha1-to-v1beta2"
|
||||
"features/software-templates/migrating-from-v1beta2-to-v1beta3"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
apiVersion: scaffolder.backstage.io/v1beta3
|
||||
kind: Template
|
||||
metadata:
|
||||
name: test-v1beta3
|
||||
title: Test v1beta3
|
||||
description: Test V1 Beta 3 Demo Templates
|
||||
spec:
|
||||
type: website
|
||||
owner: team-a
|
||||
parameters:
|
||||
- name: Enter some stuff
|
||||
description: Enter some stuff
|
||||
properties:
|
||||
inputString:
|
||||
type: string
|
||||
title: string input test
|
||||
inputObject:
|
||||
type: object
|
||||
title: object input test
|
||||
description: a little nested thing never hurt anyone right?
|
||||
properties:
|
||||
first:
|
||||
type: string
|
||||
title: first
|
||||
second:
|
||||
type: number
|
||||
title: second
|
||||
steps:
|
||||
- id: debug
|
||||
if: ${{ true === true }}
|
||||
name: Debug
|
||||
action: debug:log
|
||||
input:
|
||||
message: ${{ parameters.inputString }}
|
||||
extra: ${{ parameters.inputObject }}
|
||||
@@ -39,10 +39,15 @@ export function createDebugLogAction() {
|
||||
title: 'List all files in the workspace, if true.',
|
||||
type: 'boolean',
|
||||
},
|
||||
extra: {
|
||||
title: 'Extra info',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
ctx.logger.info(JSON.stringify(ctx.input, null, 2));
|
||||
|
||||
if (ctx.input?.message) {
|
||||
ctx.logStream.write(ctx.input.message);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/*
|
||||
* Copyright 2021 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
import os from 'os';
|
||||
import { getVoidLogger } from '@backstage/backend-common';
|
||||
import { DefaultWorkflowRunner } from './DefaultWorkflowRunner';
|
||||
import { TemplateActionRegistry } from '../actions';
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { ConfigReader } from '@backstage/config';
|
||||
import { Task, TaskSpec } from './types';
|
||||
|
||||
describe('DefaultWorkflowRunner', () => {
|
||||
const workingDirectory = os.tmpdir();
|
||||
const logger = getVoidLogger();
|
||||
let actionRegistry = new TemplateActionRegistry();
|
||||
let runner: DefaultWorkflowRunner;
|
||||
let fakeActionHandler: jest.Mock;
|
||||
|
||||
const integrations = ScmIntegrations.fromConfig(
|
||||
new ConfigReader({
|
||||
integrations: {
|
||||
github: [{ host: 'github.com', token: 'token' }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const createMockTaskWithSpec = (spec: TaskSpec): Task => ({
|
||||
spec,
|
||||
complete: async () => {},
|
||||
done: false,
|
||||
emitLog: async () => {},
|
||||
getWorkspaceName: () => Promise.resolve('test-workspace'),
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
jest.resetAllMocks();
|
||||
actionRegistry = new TemplateActionRegistry();
|
||||
fakeActionHandler = jest.fn();
|
||||
|
||||
actionRegistry.register({
|
||||
id: 'jest-mock-action',
|
||||
description: 'Mock action for testing',
|
||||
handler: fakeActionHandler,
|
||||
});
|
||||
|
||||
actionRegistry.register({
|
||||
id: 'jest-validated-action',
|
||||
description: 'Mock action for testing',
|
||||
handler: fakeActionHandler,
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['foo'],
|
||||
properties: {
|
||||
foo: {
|
||||
type: 'number',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
actionRegistry.register({
|
||||
id: 'output-action',
|
||||
description: 'Mock action for testing',
|
||||
handler: async ctx => {
|
||||
ctx.output('mock', 'backstage');
|
||||
ctx.output('shouldRun', true);
|
||||
},
|
||||
});
|
||||
|
||||
runner = new DefaultWorkflowRunner({
|
||||
actionRegistry,
|
||||
integrations,
|
||||
workingDirectory,
|
||||
logger,
|
||||
});
|
||||
});
|
||||
|
||||
it('should throw an error if the action does not exist', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
parameters: {},
|
||||
output: {},
|
||||
steps: [{ id: 'test', name: 'name', action: 'does-not-exist' }],
|
||||
});
|
||||
|
||||
await expect(runner.execute(task)).rejects.toThrowError(
|
||||
"Template action with ID 'does-not-exist' is not registered.",
|
||||
);
|
||||
});
|
||||
|
||||
describe('validation', () => {
|
||||
it('should throw an error if the action has a schema and the input does not match', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
parameters: {},
|
||||
output: {},
|
||||
steps: [{ id: 'test', name: 'name', action: 'jest-validated-action' }],
|
||||
});
|
||||
|
||||
await expect(runner.execute(task)).rejects.toThrowError(
|
||||
/Invalid input passed to action jest-validated-action, instance requires property \"foo\"/,
|
||||
);
|
||||
});
|
||||
|
||||
it('should run the action when the validation passes', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
parameters: {},
|
||||
output: {},
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-validated-action',
|
||||
input: { foo: 1 },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('conditionals', () => {
|
||||
it('should execute steps conditionally', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'output-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'output-action',
|
||||
if: '${{ steps.test.output.shouldRun }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '${{ steps.conditional.output.mock }}',
|
||||
},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('backstage');
|
||||
});
|
||||
|
||||
it('should skips steps conditionally', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'output-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'output-action',
|
||||
if: '${{ not steps.test.output.shouldRun}}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '${{ steps.conditional.output.mock }}',
|
||||
},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBeUndefined();
|
||||
});
|
||||
|
||||
it('should skips steps using the negating equals operator', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'output-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'output-action',
|
||||
if: '${{ steps.test.output.mock !== "backstage"}}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '${{ steps.conditional.output.mock }}',
|
||||
},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('templating', () => {
|
||||
it('should template the input to an action', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-mock-action',
|
||||
input: {
|
||||
foo: '${{parameters.input | lower }}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {},
|
||||
parameters: {
|
||||
input: 'BACKSTAGE',
|
||||
},
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ input: { foo: 'backstage' } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('should keep the original types for the input and not parse things that arent meant to be parsed', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-mock-action',
|
||||
input: {
|
||||
number: '${{parameters.number}}',
|
||||
string: '${{parameters.string}}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {},
|
||||
parameters: {
|
||||
number: 0,
|
||||
string: '1',
|
||||
},
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ input: { number: 0, string: '1' } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('should template complex values into the action', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-mock-action',
|
||||
input: {
|
||||
foo: '${{parameters.complex}}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {},
|
||||
parameters: {
|
||||
complex: { bar: 'BACKSTAGE' },
|
||||
},
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ input: { foo: { bar: 'BACKSTAGE' } } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('supports really complex structures', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-mock-action',
|
||||
input: {
|
||||
foo: '${{parameters.complex.baz.something}}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {},
|
||||
parameters: {
|
||||
complex: {
|
||||
bar: 'BACKSTAGE',
|
||||
baz: { something: 'nested', here: 'yas' },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ input: { foo: 'nested' } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('supports numbers as first class too', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'jest-mock-action',
|
||||
input: {
|
||||
foo: '${{parameters.complex.baz.number}}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {},
|
||||
parameters: {
|
||||
complex: {
|
||||
bar: 'BACKSTAGE',
|
||||
baz: { number: 1 },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
await runner.execute(task);
|
||||
|
||||
expect(fakeActionHandler).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ input: { foo: 1 } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('should template the output from simple actions', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'output-action',
|
||||
input: {},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
foo: '${{steps.test.output.mock | upper}}',
|
||||
},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.foo).toEqual('BACKSTAGE');
|
||||
});
|
||||
});
|
||||
|
||||
describe('filters', () => {
|
||||
it('provides the parseRepoUrl filter', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [
|
||||
{
|
||||
id: 'test',
|
||||
name: 'name',
|
||||
action: 'output-action',
|
||||
input: {},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
foo: '${{ parameters.repoUrl | parseRepoUrl }}',
|
||||
},
|
||||
parameters: {
|
||||
repoUrl: 'github.com?repo=repo&owner=owner',
|
||||
},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.foo).toEqual({
|
||||
host: 'github.com',
|
||||
owner: 'owner',
|
||||
repo: 'repo',
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,272 @@
|
||||
/*
|
||||
* Copyright 2021 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import {
|
||||
Task,
|
||||
TaskSpec,
|
||||
TaskSpecV1beta3,
|
||||
TaskStep,
|
||||
WorkflowResponse,
|
||||
WorkflowRunner,
|
||||
} from './types';
|
||||
import * as winston from 'winston';
|
||||
import nunjucks from 'nunjucks';
|
||||
import fs from 'fs-extra';
|
||||
import path from 'path';
|
||||
import { JsonObject, JsonValue } from '@backstage/config';
|
||||
import { InputError } from '@backstage/errors';
|
||||
import { PassThrough } from 'stream';
|
||||
import { isTruthy } from './helper';
|
||||
import { validate as validateJsonSchema } from 'jsonschema';
|
||||
import { parseRepoUrl } from '../actions/builtin/publish/util';
|
||||
import { TemplateActionRegistry } from '../actions';
|
||||
|
||||
type Options = {
|
||||
workingDirectory: string;
|
||||
actionRegistry: TemplateActionRegistry;
|
||||
integrations: ScmIntegrations;
|
||||
logger: winston.Logger;
|
||||
};
|
||||
|
||||
type TemplateContext = {
|
||||
parameters: JsonObject;
|
||||
steps: {
|
||||
[stepName: string]: { output: { [outputName: string]: JsonValue } };
|
||||
};
|
||||
};
|
||||
|
||||
const isValidTaskSpec = (taskSpec: TaskSpec): taskSpec is TaskSpecV1beta3 => {
|
||||
return taskSpec.apiVersion === 'scaffolder.backstage.io/v1beta3';
|
||||
};
|
||||
|
||||
const createStepLogger = ({ task, step }: { task: Task; step: TaskStep }) => {
|
||||
const metadata = { stepId: step.id };
|
||||
const taskLogger = winston.createLogger({
|
||||
level: process.env.LOG_LEVEL || 'info',
|
||||
format: winston.format.combine(
|
||||
winston.format.colorize(),
|
||||
winston.format.timestamp(),
|
||||
winston.format.simple(),
|
||||
),
|
||||
defaultMeta: {},
|
||||
});
|
||||
|
||||
const streamLogger = new PassThrough();
|
||||
streamLogger.on('data', async data => {
|
||||
const message = data.toString().trim();
|
||||
if (message?.length > 1) {
|
||||
await task.emitLog(message, metadata);
|
||||
}
|
||||
});
|
||||
|
||||
taskLogger.add(new winston.transports.Stream({ stream: streamLogger }));
|
||||
|
||||
return { taskLogger, streamLogger };
|
||||
};
|
||||
|
||||
export class DefaultWorkflowRunner implements WorkflowRunner {
|
||||
private readonly nunjucks: nunjucks.Environment;
|
||||
|
||||
private readonly nunjucksOptions: nunjucks.ConfigureOptions = {
|
||||
autoescape: false,
|
||||
tags: {
|
||||
variableStart: '${{',
|
||||
variableEnd: '}}',
|
||||
},
|
||||
};
|
||||
|
||||
constructor(private readonly options: Options) {
|
||||
this.nunjucks = nunjucks.configure(this.nunjucksOptions);
|
||||
|
||||
// TODO(blam): let's work out how we can deprecate these.
|
||||
// We shouldn't really need to be exposing these now we can deal with
|
||||
// objects in the params block.
|
||||
// Maybe we can expose a new RepoUrlPicker with secrets for V3 that provides an object already.
|
||||
this.nunjucks.addFilter('parseRepoUrl', repoUrl => {
|
||||
return parseRepoUrl(repoUrl, this.options.integrations);
|
||||
});
|
||||
|
||||
this.nunjucks.addFilter('projectSlug', repoUrl => {
|
||||
const { owner, repo } = parseRepoUrl(repoUrl, this.options.integrations);
|
||||
return `${owner}/${repo}`;
|
||||
});
|
||||
}
|
||||
|
||||
private isSingleTemplateString(input: string) {
|
||||
const { parser, nodes } = require('nunjucks');
|
||||
const parsed = parser.parse(input, {}, this.nunjucksOptions);
|
||||
return (
|
||||
parsed.children.length === 1 &&
|
||||
!(parsed.children[0] instanceof nodes.TemplateData)
|
||||
);
|
||||
}
|
||||
|
||||
private render<T>(input: T, context: TemplateContext): T {
|
||||
return JSON.parse(JSON.stringify(input), (_key, value) => {
|
||||
try {
|
||||
if (typeof value === 'string') {
|
||||
try {
|
||||
if (this.isSingleTemplateString(value)) {
|
||||
// Lets convert ${{ parameters.bob }} to ${{ (parameters.bob) | dump }} so we can keep the input type
|
||||
const wrappedDumped = value.replace(
|
||||
/\${{(.+)}}/g,
|
||||
'${{ ( $1 ) | dump }}',
|
||||
);
|
||||
|
||||
// Run the templating
|
||||
const templated = this.nunjucks.renderString(
|
||||
wrappedDumped,
|
||||
context,
|
||||
);
|
||||
|
||||
// If there's an empty string returned, then it's undefined
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// Reparse the dumped string
|
||||
return JSON.parse(templated);
|
||||
}
|
||||
} catch (ex) {
|
||||
this.options.logger.error(
|
||||
`Failed to parse template string: ${value} with error ${ex.message}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Fallback to default behaviour
|
||||
const templated = this.nunjucks.renderString(value, context);
|
||||
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return templated;
|
||||
}
|
||||
} catch {
|
||||
return value;
|
||||
}
|
||||
return value;
|
||||
});
|
||||
}
|
||||
|
||||
async execute(task: Task): Promise<WorkflowResponse> {
|
||||
if (!isValidTaskSpec(task.spec)) {
|
||||
throw new InputError(
|
||||
'Wrong template version executed with the workflow engine',
|
||||
);
|
||||
}
|
||||
const workspacePath = path.join(
|
||||
this.options.workingDirectory,
|
||||
await task.getWorkspaceName(),
|
||||
);
|
||||
try {
|
||||
await fs.ensureDir(workspacePath);
|
||||
await task.emitLog(
|
||||
`Starting up task with ${task.spec.steps.length} steps`,
|
||||
);
|
||||
|
||||
const context: TemplateContext = {
|
||||
parameters: task.spec.parameters,
|
||||
steps: {},
|
||||
};
|
||||
|
||||
for (const step of task.spec.steps) {
|
||||
try {
|
||||
if (step.if) {
|
||||
const ifResult = await this.render(step.if, context);
|
||||
if (!isTruthy(ifResult)) {
|
||||
await task.emitLog(
|
||||
`Skipping step ${step.id} because it's if condition was false`,
|
||||
{ stepId: step.id, status: 'skipped' },
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
await task.emitLog(`Beginning step ${step.name}`, {
|
||||
stepId: step.id,
|
||||
status: 'processing',
|
||||
});
|
||||
|
||||
const action = this.options.actionRegistry.get(step.action);
|
||||
const { taskLogger, streamLogger } = createStepLogger({ task, step });
|
||||
|
||||
const input = (step.input && this.render(step.input, context)) ?? {};
|
||||
|
||||
if (action.schema?.input) {
|
||||
const validateResult = validateJsonSchema(
|
||||
input,
|
||||
action.schema.input,
|
||||
);
|
||||
if (!validateResult.valid) {
|
||||
const errors = validateResult.errors.join(', ');
|
||||
throw new InputError(
|
||||
`Invalid input passed to action ${action.id}, ${errors}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const tmpDirs = new Array<string>();
|
||||
const stepOutput: { [outputName: string]: JsonValue } = {};
|
||||
|
||||
await action.handler({
|
||||
baseUrl: task.spec.baseUrl,
|
||||
input,
|
||||
logger: taskLogger,
|
||||
logStream: streamLogger,
|
||||
workspacePath,
|
||||
createTemporaryDirectory: async () => {
|
||||
const tmpDir = await fs.mkdtemp(
|
||||
`${workspacePath}_step-${step.id}-`,
|
||||
);
|
||||
tmpDirs.push(tmpDir);
|
||||
return tmpDir;
|
||||
},
|
||||
output(name: string, value: JsonValue) {
|
||||
stepOutput[name] = value;
|
||||
},
|
||||
});
|
||||
|
||||
// Remove all temporary directories that were created when executing the action
|
||||
for (const tmpDir of tmpDirs) {
|
||||
await fs.remove(tmpDir);
|
||||
}
|
||||
|
||||
context.steps[step.id] = { output: stepOutput };
|
||||
|
||||
await task.emitLog(`Finished step ${step.name}`, {
|
||||
stepId: step.id,
|
||||
status: 'completed',
|
||||
});
|
||||
} catch (err) {
|
||||
await task.emitLog(String(err.stack), {
|
||||
stepId: step.id,
|
||||
status: 'failed',
|
||||
});
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
const output = this.render(task.spec.output, context);
|
||||
|
||||
return { output };
|
||||
} finally {
|
||||
if (workspacePath) {
|
||||
await fs.remove(workspacePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,400 @@
|
||||
/*
|
||||
* Copyright 2021 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
import { createTemplateAction, TemplateActionRegistry } from '../actions';
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { ConfigReader } from '@backstage/config';
|
||||
import { getVoidLogger } from '@backstage/backend-common';
|
||||
import { LegacyWorkflowRunner } from './LegacyWorkflowRunner';
|
||||
import os from 'os';
|
||||
import { Task, TaskSpec } from './types';
|
||||
import { RepoSpec } from '../actions/builtin/publish/util';
|
||||
|
||||
describe('LegacyWorkflowRunner', () => {
|
||||
let runner: LegacyWorkflowRunner;
|
||||
const workingDirectory = os.tmpdir();
|
||||
const logger = getVoidLogger();
|
||||
let actionRegistry = new TemplateActionRegistry();
|
||||
|
||||
const integrations = ScmIntegrations.fromConfig(
|
||||
new ConfigReader({
|
||||
integrations: {
|
||||
github: [{ host: 'github.com', token: 'token' }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const createMockTaskWithSpec = (spec: TaskSpec): Task => ({
|
||||
spec,
|
||||
complete: async () => {},
|
||||
done: false,
|
||||
emitLog: async () => {},
|
||||
getWorkspaceName: () => Promise.resolve('test-workspace'),
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
actionRegistry = new TemplateActionRegistry();
|
||||
actionRegistry.register({
|
||||
id: 'test-action',
|
||||
handler: async ctx => {
|
||||
ctx.output('testOutput', 'mockOutputData');
|
||||
ctx.output('badOutput', false);
|
||||
},
|
||||
});
|
||||
|
||||
runner = new LegacyWorkflowRunner({
|
||||
actionRegistry,
|
||||
integrations,
|
||||
workingDirectory,
|
||||
logger,
|
||||
});
|
||||
});
|
||||
|
||||
it('should fail when the action does not exist', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [{ id: 'test', name: 'test', action: 'not-found-action' }],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
await expect(() => runner.execute(task)).rejects.toThrow(
|
||||
/Template action with ID 'not-found-action' is not registered/,
|
||||
);
|
||||
});
|
||||
|
||||
describe('templating', () => {
|
||||
it('should template the output', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [{ id: 'test', name: 'test', action: 'test-action' }],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('mockOutputData');
|
||||
});
|
||||
|
||||
it('should template the input', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
name: string;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['name'],
|
||||
properties: {
|
||||
name: {
|
||||
title: 'name',
|
||||
description: 'Enter name',
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
if (ctx.input.name !== 'mockOutputData') {
|
||||
throw new Error(
|
||||
`expected name to be "mockOutputData" got ${ctx.input.name}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
name: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('mockOutputData');
|
||||
});
|
||||
});
|
||||
|
||||
describe('conditionals', () => {
|
||||
it('should execute steps conditionally', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('mockOutputData');
|
||||
});
|
||||
|
||||
it('should execute steps conditionally with eq helper', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ eq steps.test.output.testOutput "mockOutputData" }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('mockOutputData');
|
||||
});
|
||||
|
||||
it('should skip test conditionally', async () => {
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ steps.test.output.badOutput }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('parsing', () => {
|
||||
it('should parse strings as objects if possible', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
address: { line1: string };
|
||||
list: string[];
|
||||
address2: string;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['address'],
|
||||
properties: {
|
||||
address: {
|
||||
title: 'address',
|
||||
description: 'Enter name',
|
||||
type: 'object',
|
||||
properties: {
|
||||
line1: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
address2: {
|
||||
type: 'string',
|
||||
},
|
||||
list: {
|
||||
type: 'array',
|
||||
items: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
if (ctx.input.list.length !== 1) {
|
||||
throw new Error(
|
||||
`expected list to have length "1" got ${ctx.input.list.length}`,
|
||||
);
|
||||
}
|
||||
if (ctx.input.address.line1 !== 'line 1') {
|
||||
throw new Error(
|
||||
`expected address.line1 to be "line 1" got ${ctx.input.address.line1}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (ctx.input.address2 !== '{"not valid"}') {
|
||||
throw new Error(
|
||||
`expected address2 to be "{"not valid"}" got ${ctx.input.address2}`,
|
||||
);
|
||||
}
|
||||
ctx.output('address', ctx.input.address.line1);
|
||||
},
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
address: JSON.stringify({ line1: 'line 1' }),
|
||||
list: JSON.stringify(['hey!']),
|
||||
address2: '{"not valid"}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.test-input.output.address }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.result).toBe('line 1');
|
||||
});
|
||||
|
||||
it('should provide a parseRepoUrl helper', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
destination: RepoSpec;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['destination'],
|
||||
properties: {
|
||||
destination: {
|
||||
title: 'destination',
|
||||
type: 'object',
|
||||
properties: {
|
||||
repo: {
|
||||
type: 'string',
|
||||
},
|
||||
host: {
|
||||
type: 'string',
|
||||
},
|
||||
owner: {
|
||||
type: 'string',
|
||||
},
|
||||
organization: {
|
||||
type: 'string',
|
||||
},
|
||||
workspace: {
|
||||
type: 'string',
|
||||
},
|
||||
project: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
ctx.output('host', ctx.input.destination.host);
|
||||
ctx.output('repo', ctx.input.destination.repo);
|
||||
|
||||
if (ctx.input.destination.owner) {
|
||||
ctx.output('owner', ctx.input.destination.owner);
|
||||
}
|
||||
|
||||
if (ctx.input.destination.host !== 'github.com') {
|
||||
throw new Error(
|
||||
`expected host to be "github.com" got ${ctx.input.destination.host}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (ctx.input.destination.repo !== 'repo') {
|
||||
throw new Error(
|
||||
`expected repo to be "repo" got ${ctx.input.destination.repo}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ctx.input.destination.owner &&
|
||||
ctx.input.destination.owner !== 'owner'
|
||||
) {
|
||||
throw new Error(
|
||||
`expected repo to be "owner" got ${ctx.input.destination.owner}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const task = createMockTaskWithSpec({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
destination: '{{ parseRepoUrl parameters.repoUrl }}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
host: '{{ steps.test-input.output.host }}',
|
||||
repo: '{{ steps.test-input.output.repo }}',
|
||||
owner: '{{ steps.test-input.output.owner }}',
|
||||
},
|
||||
values: {
|
||||
repoUrl: 'github.com?repo=repo&owner=owner',
|
||||
},
|
||||
});
|
||||
|
||||
const { output } = await runner.execute(task);
|
||||
|
||||
expect(output.host).toBe('github.com');
|
||||
expect(output.repo).toBe('repo');
|
||||
expect(output.owner).toBe('owner');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,309 @@
|
||||
/*
|
||||
* Copyright 2021 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
import {
|
||||
Task,
|
||||
WorkflowRunner,
|
||||
WorkflowResponse,
|
||||
TaskSpecV1beta2,
|
||||
TaskSpec,
|
||||
} from './types';
|
||||
import * as Handlebars from 'handlebars';
|
||||
import { TemplateActionRegistry } from '..';
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { parseRepoUrl } from '../actions/builtin/publish/util';
|
||||
import { isTruthy } from './helper';
|
||||
import { PassThrough } from 'stream';
|
||||
import * as winston from 'winston';
|
||||
import { Logger } from 'winston';
|
||||
import path from 'path';
|
||||
import fs from 'fs-extra';
|
||||
import { validate as validateJsonSchema } from 'jsonschema';
|
||||
import { JsonObject, JsonValue } from '@backstage/config';
|
||||
import { InputError } from '@backstage/errors';
|
||||
|
||||
type Options = {
|
||||
workingDirectory: string;
|
||||
actionRegistry: TemplateActionRegistry;
|
||||
integrations: ScmIntegrations;
|
||||
logger: Logger;
|
||||
};
|
||||
|
||||
const isValidTaskSpec = (taskSpec: TaskSpec): taskSpec is TaskSpecV1beta2 =>
|
||||
taskSpec.apiVersion === 'backstage.io/v1beta2';
|
||||
/**
|
||||
* This is the legacy workflow runner, which supports handlebars. This entire implementation will be replaced
|
||||
* with the default workflow runner interface in the future so this entire thing can go bye bye.
|
||||
*/
|
||||
export class LegacyWorkflowRunner implements WorkflowRunner {
|
||||
private readonly handlebars: typeof Handlebars;
|
||||
|
||||
constructor(private readonly options: Options) {
|
||||
this.handlebars = Handlebars.create();
|
||||
|
||||
// TODO(blam): this should be a public facing API but it's a little
|
||||
// scary right now, so we're going to lock it off like the component API is
|
||||
// in the frontend until we can work out a nice way to do it.
|
||||
this.handlebars.registerHelper('parseRepoUrl', repoUrl => {
|
||||
return JSON.stringify(parseRepoUrl(repoUrl, this.options.integrations));
|
||||
});
|
||||
|
||||
this.handlebars.registerHelper('projectSlug', repoUrl => {
|
||||
const { owner, repo } = parseRepoUrl(repoUrl, this.options.integrations);
|
||||
return `${owner}/${repo}`;
|
||||
});
|
||||
|
||||
this.handlebars.registerHelper('json', obj => JSON.stringify(obj));
|
||||
|
||||
this.handlebars.registerHelper('not', value => !isTruthy(value));
|
||||
|
||||
this.handlebars.registerHelper('eq', (a, b) => a === b);
|
||||
}
|
||||
|
||||
async execute(task: Task): Promise<WorkflowResponse> {
|
||||
if (!isValidTaskSpec(task.spec)) {
|
||||
throw new InputError(`Task spec is not a valid v1beta2 task spec`);
|
||||
}
|
||||
|
||||
const { actionRegistry } = this.options;
|
||||
|
||||
const workspacePath = path.join(
|
||||
this.options.workingDirectory,
|
||||
await task.getWorkspaceName(),
|
||||
);
|
||||
try {
|
||||
await fs.ensureDir(workspacePath);
|
||||
await task.emitLog(
|
||||
`Starting up task with ${task.spec.steps.length} steps`,
|
||||
);
|
||||
|
||||
const templateCtx: {
|
||||
parameters: JsonObject;
|
||||
steps: {
|
||||
[stepName: string]: { output: { [outputName: string]: JsonValue } };
|
||||
};
|
||||
} = { parameters: task.spec.values, steps: {} };
|
||||
|
||||
for (const step of task.spec.steps) {
|
||||
const metadata = { stepId: step.id };
|
||||
try {
|
||||
const taskLogger = winston.createLogger({
|
||||
level: process.env.LOG_LEVEL || 'info',
|
||||
format: winston.format.combine(
|
||||
winston.format.colorize(),
|
||||
winston.format.timestamp(),
|
||||
winston.format.simple(),
|
||||
),
|
||||
defaultMeta: {},
|
||||
});
|
||||
|
||||
const stream = new PassThrough();
|
||||
stream.on('data', async data => {
|
||||
const message = data.toString().trim();
|
||||
if (message?.length > 1) {
|
||||
await task.emitLog(message, metadata);
|
||||
}
|
||||
});
|
||||
|
||||
taskLogger.add(new winston.transports.Stream({ stream }));
|
||||
|
||||
if (step.if !== undefined) {
|
||||
// Support passing values like false to disable steps
|
||||
let skip = !step.if;
|
||||
|
||||
// Evaluate strings as handlebar templates
|
||||
if (typeof step.if === 'string') {
|
||||
const condition = JSON.parse(
|
||||
JSON.stringify(step.if),
|
||||
(_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it's just an empty string, treat it as undefined
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
|
||||
return value;
|
||||
},
|
||||
);
|
||||
|
||||
skip = !isTruthy(condition);
|
||||
}
|
||||
|
||||
if (skip) {
|
||||
await task.emitLog(`Skipped step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'skipped',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
await task.emitLog(`Beginning step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'processing',
|
||||
});
|
||||
|
||||
const action = actionRegistry.get(step.action);
|
||||
if (!action) {
|
||||
throw new Error(`Action '${step.action}' does not exist`);
|
||||
}
|
||||
|
||||
const input =
|
||||
step.input &&
|
||||
JSON.parse(JSON.stringify(step.input), (_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it smells like a JSON object then give it a parse as an object and if it fails return the string
|
||||
if (
|
||||
(templated.startsWith('"') && templated.endsWith('"')) ||
|
||||
(templated.startsWith('{') && templated.endsWith('}')) ||
|
||||
(templated.startsWith('[') && templated.endsWith(']'))
|
||||
) {
|
||||
try {
|
||||
// Don't recursively JSON parse the values of this string.
|
||||
// Shouldn't need to, don't want to encourage the use of returning handlebars from somewhere else
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
return templated;
|
||||
}
|
||||
|
||||
return value;
|
||||
});
|
||||
|
||||
if (action.schema?.input) {
|
||||
const validateResult = validateJsonSchema(
|
||||
input,
|
||||
action.schema.input,
|
||||
);
|
||||
if (!validateResult.valid) {
|
||||
const errors = validateResult.errors.join(', ');
|
||||
throw new InputError(
|
||||
`Invalid input passed to action ${action.id}, ${errors}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const stepOutputs: { [name: string]: JsonValue } = {};
|
||||
|
||||
// Keep track of all tmp dirs that are created by the action so we can remove them after
|
||||
const tmpDirs = new Array<string>();
|
||||
|
||||
this.options.logger.debug(`Running ${action.id} with input`, {
|
||||
input: JSON.stringify(input, null, 2),
|
||||
});
|
||||
|
||||
await action.handler({
|
||||
baseUrl: task.spec.baseUrl,
|
||||
logger: taskLogger,
|
||||
logStream: stream,
|
||||
input,
|
||||
token: task.secrets?.token,
|
||||
workspacePath,
|
||||
async createTemporaryDirectory() {
|
||||
const tmpDir = await fs.mkdtemp(
|
||||
`${workspacePath}_step-${step.id}-`,
|
||||
);
|
||||
tmpDirs.push(tmpDir);
|
||||
return tmpDir;
|
||||
},
|
||||
output(name: string, value: JsonValue) {
|
||||
stepOutputs[name] = value;
|
||||
},
|
||||
});
|
||||
|
||||
// Remove all temporary directories that were created when executing the action
|
||||
for (const tmpDir of tmpDirs) {
|
||||
await fs.remove(tmpDir);
|
||||
}
|
||||
|
||||
templateCtx.steps[step.id] = { output: stepOutputs };
|
||||
|
||||
await task.emitLog(`Finished step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'completed',
|
||||
});
|
||||
} catch (error) {
|
||||
await task.emitLog(String(error.stack), {
|
||||
...metadata,
|
||||
status: 'failed',
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
const output = JSON.parse(
|
||||
JSON.stringify(task.spec.output),
|
||||
(_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it's just an empty string, treat it as undefined
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// If it smells like a JSON object then give it a parse as an object and if it fails return the string
|
||||
if (
|
||||
(templated.startsWith('"') && templated.endsWith('"')) ||
|
||||
(templated.startsWith('{') && templated.endsWith('}')) ||
|
||||
(templated.startsWith('[') && templated.endsWith(']'))
|
||||
) {
|
||||
try {
|
||||
// Don't recursively JSON parse the values of this string.
|
||||
// Shouldn't need to, don't want to encourage the use of returning handlebars from somewhere else
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
return templated;
|
||||
}
|
||||
return value;
|
||||
},
|
||||
);
|
||||
|
||||
return { output };
|
||||
} finally {
|
||||
if (workspacePath) {
|
||||
await fs.remove(workspacePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,15 +14,13 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import os from 'os';
|
||||
import { getVoidLogger, DatabaseManager } from '@backstage/backend-common';
|
||||
import { ConfigReader, JsonObject } from '@backstage/config';
|
||||
import { createTemplateAction, TemplateActionRegistry } from '../actions';
|
||||
import { RepoSpec } from '../actions/builtin/publish/util';
|
||||
import { ConfigReader } from '@backstage/config';
|
||||
import { DatabaseTaskStore } from './DatabaseTaskStore';
|
||||
import { StorageTaskBroker } from './StorageTaskBroker';
|
||||
import { TaskWorker } from './TaskWorker';
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { WorkflowRunner } from './types';
|
||||
import { LegacyWorkflowRunner } from './LegacyWorkflowRunner';
|
||||
|
||||
async function createStore(): Promise<DatabaseTaskStore> {
|
||||
const manager = DatabaseManager.fromConfig(
|
||||
@@ -40,138 +38,95 @@ async function createStore(): Promise<DatabaseTaskStore> {
|
||||
|
||||
describe('TaskWorker', () => {
|
||||
let storage: DatabaseTaskStore;
|
||||
let actionRegistry = new TemplateActionRegistry();
|
||||
const workflowRunner: WorkflowRunner = {
|
||||
execute: jest.fn(),
|
||||
} as unknown as WorkflowRunner;
|
||||
|
||||
const integrations = ScmIntegrations.fromConfig(
|
||||
new ConfigReader({
|
||||
integrations: {
|
||||
github: [{ host: 'github.com', token: 'token' }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
const legacyWorkflowRunner: LegacyWorkflowRunner = {
|
||||
execute: jest.fn(),
|
||||
} as unknown as LegacyWorkflowRunner;
|
||||
|
||||
beforeAll(async () => {
|
||||
storage = await createStore();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
actionRegistry = new TemplateActionRegistry();
|
||||
actionRegistry.register({
|
||||
id: 'test-action',
|
||||
handler: async ctx => {
|
||||
ctx.output('testOutput', 'winning');
|
||||
ctx.output('badOutput', false);
|
||||
},
|
||||
});
|
||||
jest.resetAllMocks();
|
||||
});
|
||||
|
||||
const logger = getVoidLogger();
|
||||
|
||||
it('should fail when action does not exist', async () => {
|
||||
it('should call the legacy workflow runner when the apiVersion is not beta3', async () => {
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
runners: {
|
||||
legacyWorkflowRunner,
|
||||
workflowRunner,
|
||||
},
|
||||
});
|
||||
const { taskId } = await broker.dispatch({
|
||||
|
||||
await broker.dispatch({
|
||||
apiVersion: 'backstage.io/v1beta2',
|
||||
steps: [{ id: 'test', name: 'test', action: 'not-found-action' }],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
|
||||
expect((event?.body?.error as JsonObject)?.message).toBe(
|
||||
"Template action with ID 'not-found-action' is not registered.",
|
||||
);
|
||||
expect(legacyWorkflowRunner.execute).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should template output', async () => {
|
||||
it('should call the default workflow runner when the apiVersion is beta3', async () => {
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
runners: {
|
||||
legacyWorkflowRunner,
|
||||
workflowRunner,
|
||||
},
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [{ id: 'test', name: 'test', action: 'test-action' }],
|
||||
await broker.dispatch({
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [{ id: 'test', name: 'test', action: 'not-found-action' }],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
expect((event?.body?.output as JsonObject).result).toBe('winning');
|
||||
expect(workflowRunner.execute).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should template input', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
name: string;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['name'],
|
||||
properties: {
|
||||
name: {
|
||||
title: 'name',
|
||||
description: 'Enter name',
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
if (ctx.input.name !== 'winning') {
|
||||
throw new Error(
|
||||
`expected name to be "winning" got ${ctx.input.name}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
it('should save the output to the task', async () => {
|
||||
(workflowRunner.execute as jest.Mock).mockResolvedValue({
|
||||
output: { testOutput: 'testmockoutput' },
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
runners: {
|
||||
legacyWorkflowRunner,
|
||||
workflowRunner,
|
||||
},
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
name: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
},
|
||||
],
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3',
|
||||
steps: [{ id: 'test', name: 'test', action: 'not-found-action' }],
|
||||
output: {
|
||||
result: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
parameters: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
@@ -179,312 +134,6 @@ describe('TaskWorker', () => {
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
expect((event?.body?.output as JsonObject).result).toBe('winning');
|
||||
});
|
||||
|
||||
it('should execute steps conditionally', async () => {
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ steps.test.output.testOutput }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
expect((event?.body?.output as JsonObject).result).toBe('winning');
|
||||
});
|
||||
|
||||
it('should execute steps conditionally with eq helper', async () => {
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ eq steps.test.output.testOutput "winning" }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
expect((event?.body?.output as JsonObject).result).toBe('winning');
|
||||
});
|
||||
|
||||
it('should skip steps conditionally', async () => {
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{ id: 'test', name: 'test', action: 'test-action' },
|
||||
{
|
||||
id: 'conditional',
|
||||
name: 'conditional',
|
||||
action: 'test-action',
|
||||
if: '{{ steps.test.output.badOutput }}',
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.conditional.output.testOutput }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
expect((event?.body?.output as JsonObject).result).toBeUndefined();
|
||||
});
|
||||
|
||||
it('should parse strings as objects if possible', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
address: { line1: string };
|
||||
list: string[];
|
||||
address2: string;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['address'],
|
||||
properties: {
|
||||
address: {
|
||||
title: 'address',
|
||||
description: 'Enter name',
|
||||
type: 'object',
|
||||
properties: {
|
||||
line1: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
address2: {
|
||||
type: 'string',
|
||||
},
|
||||
list: {
|
||||
type: 'array',
|
||||
items: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
if (ctx.input.list.length !== 1) {
|
||||
throw new Error(
|
||||
`expected list to have length "1" got ${ctx.input.list.length}`,
|
||||
);
|
||||
}
|
||||
if (ctx.input.address.line1 !== 'line 1') {
|
||||
throw new Error(
|
||||
`expected address.line1 to be "line 1" got ${ctx.input.address.line1}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (ctx.input.address2 !== '{"not valid"}') {
|
||||
throw new Error(
|
||||
`expected address2 to be "{"not valid"}" got ${ctx.input.address2}`,
|
||||
);
|
||||
}
|
||||
ctx.output('address', ctx.input.address.line1);
|
||||
},
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
address: JSON.stringify({ line1: 'line 1' }),
|
||||
list: JSON.stringify(['hey!']),
|
||||
address2: '{"not valid"}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
result: '{{ steps.test-input.output.address }}',
|
||||
},
|
||||
values: {},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
|
||||
expect((event?.body?.output as JsonObject).result).toBe('line 1');
|
||||
});
|
||||
|
||||
// TODO(blam): Can delete this test when we make the helpers a public API
|
||||
it('should provide a repoUrlParse helper for the templates', async () => {
|
||||
const inputAction = createTemplateAction<{
|
||||
destination: RepoSpec;
|
||||
}>({
|
||||
id: 'test-input',
|
||||
schema: {
|
||||
input: {
|
||||
type: 'object',
|
||||
required: ['destination'],
|
||||
properties: {
|
||||
destination: {
|
||||
title: 'destination',
|
||||
type: 'object',
|
||||
properties: {
|
||||
repo: {
|
||||
type: 'string',
|
||||
},
|
||||
host: {
|
||||
type: 'string',
|
||||
},
|
||||
owner: {
|
||||
type: 'string',
|
||||
},
|
||||
organization: {
|
||||
type: 'string',
|
||||
},
|
||||
workspace: {
|
||||
type: 'string',
|
||||
},
|
||||
project: {
|
||||
type: 'string',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
async handler(ctx) {
|
||||
ctx.output('host', ctx.input.destination.host);
|
||||
ctx.output('repo', ctx.input.destination.repo);
|
||||
|
||||
if (ctx.input.destination.owner) {
|
||||
ctx.output('owner', ctx.input.destination.owner);
|
||||
}
|
||||
|
||||
if (ctx.input.destination.host !== 'github.com') {
|
||||
throw new Error(
|
||||
`expected host to be "github.com" got ${ctx.input.destination.host}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (ctx.input.destination.repo !== 'repo') {
|
||||
throw new Error(
|
||||
`expected repo to be "repo" got ${ctx.input.destination.repo}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ctx.input.destination.owner &&
|
||||
ctx.input.destination.owner !== 'owner'
|
||||
) {
|
||||
throw new Error(
|
||||
`expected repo to be "owner" got ${ctx.input.destination.owner}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
});
|
||||
actionRegistry.register(inputAction);
|
||||
|
||||
const broker = new StorageTaskBroker(storage, logger);
|
||||
const taskWorker = new TaskWorker({
|
||||
logger,
|
||||
workingDirectory: os.tmpdir(),
|
||||
actionRegistry,
|
||||
taskBroker: broker,
|
||||
integrations,
|
||||
});
|
||||
|
||||
const { taskId } = await broker.dispatch({
|
||||
steps: [
|
||||
{
|
||||
id: 'test-input',
|
||||
name: 'test-input',
|
||||
action: 'test-input',
|
||||
input: {
|
||||
destination: '{{ parseRepoUrl parameters.repoUrl }}',
|
||||
},
|
||||
},
|
||||
],
|
||||
output: {
|
||||
host: '{{ steps.test-input.output.host }}',
|
||||
repo: '{{ steps.test-input.output.repo }}',
|
||||
owner: '{{ steps.test-input.output.owner }}',
|
||||
},
|
||||
values: {
|
||||
repoUrl: 'github.com?repo=repo&owner=owner',
|
||||
},
|
||||
});
|
||||
|
||||
const task = await broker.claim();
|
||||
await taskWorker.runOneTask(task);
|
||||
|
||||
const { events } = await storage.listEvents({ taskId });
|
||||
const event = events.find(e => e.type === 'completion');
|
||||
|
||||
expect((event?.body?.output as JsonObject).host).toBe('github.com');
|
||||
expect((event?.body?.output as JsonObject).repo).toBe('repo');
|
||||
expect((event?.body?.output as JsonObject).owner).toBe('owner');
|
||||
expect(event?.body.output).toEqual({ testOutput: 'testmockoutput' });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -13,55 +13,19 @@
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { JsonObject, JsonValue } from '@backstage/config';
|
||||
import { InputError } from '@backstage/errors';
|
||||
import fs from 'fs-extra';
|
||||
import * as Handlebars from 'handlebars';
|
||||
import { validate as validateJsonSchema } from 'jsonschema';
|
||||
import path from 'path';
|
||||
import { PassThrough } from 'stream';
|
||||
import * as winston from 'winston';
|
||||
import { Logger } from 'winston';
|
||||
import { parseRepoUrl } from '../actions/builtin/publish/util';
|
||||
import { TemplateActionRegistry } from '../actions/TemplateActionRegistry';
|
||||
import { isTruthy } from './helper';
|
||||
import { Task, TaskBroker } from './types';
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { Task, TaskBroker, WorkflowRunner } from './types';
|
||||
import { LegacyWorkflowRunner } from './LegacyWorkflowRunner';
|
||||
|
||||
type Options = {
|
||||
logger: Logger;
|
||||
taskBroker: TaskBroker;
|
||||
workingDirectory: string;
|
||||
actionRegistry: TemplateActionRegistry;
|
||||
integrations: ScmIntegrations;
|
||||
runners: {
|
||||
legacyWorkflowRunner: LegacyWorkflowRunner;
|
||||
workflowRunner: WorkflowRunner;
|
||||
};
|
||||
};
|
||||
|
||||
export class TaskWorker {
|
||||
private readonly handlebars: typeof Handlebars;
|
||||
|
||||
constructor(private readonly options: Options) {
|
||||
this.handlebars = Handlebars.create();
|
||||
|
||||
// TODO(blam): this should be a public facing API but it's a little
|
||||
// scary right now, so we're going to lock it off like the component API is
|
||||
// in the frontend until we can work out a nice way to do it.
|
||||
this.handlebars.registerHelper('parseRepoUrl', repoUrl => {
|
||||
return JSON.stringify(parseRepoUrl(repoUrl, options.integrations));
|
||||
});
|
||||
|
||||
this.handlebars.registerHelper('projectSlug', repoUrl => {
|
||||
const { owner, repo } = parseRepoUrl(repoUrl, options.integrations);
|
||||
return `${owner}/${repo}`;
|
||||
});
|
||||
|
||||
this.handlebars.registerHelper('json', obj => JSON.stringify(obj));
|
||||
|
||||
this.handlebars.registerHelper('not', value => !isTruthy(value));
|
||||
|
||||
this.handlebars.registerHelper('eq', (a, b) => a === b);
|
||||
}
|
||||
|
||||
constructor(private readonly options: Options) {}
|
||||
start() {
|
||||
(async () => {
|
||||
for (;;) {
|
||||
@@ -72,238 +36,17 @@ export class TaskWorker {
|
||||
}
|
||||
|
||||
async runOneTask(task: Task) {
|
||||
let workspacePath: string | undefined = undefined;
|
||||
try {
|
||||
const { actionRegistry } = this.options;
|
||||
|
||||
workspacePath = path.join(
|
||||
this.options.workingDirectory,
|
||||
await task.getWorkspaceName(),
|
||||
);
|
||||
await fs.ensureDir(workspacePath);
|
||||
await task.emitLog(
|
||||
`Starting up task with ${task.spec.steps.length} steps`,
|
||||
);
|
||||
|
||||
const templateCtx: {
|
||||
parameters: JsonObject;
|
||||
steps: {
|
||||
[stepName: string]: { output: { [outputName: string]: JsonValue } };
|
||||
};
|
||||
} = { parameters: task.spec.values, steps: {} };
|
||||
|
||||
for (const step of task.spec.steps) {
|
||||
const metadata = { stepId: step.id };
|
||||
try {
|
||||
const taskLogger = winston.createLogger({
|
||||
level: process.env.LOG_LEVEL || 'info',
|
||||
format: winston.format.combine(
|
||||
winston.format.colorize(),
|
||||
winston.format.timestamp(),
|
||||
winston.format.simple(),
|
||||
),
|
||||
defaultMeta: {},
|
||||
});
|
||||
|
||||
const stream = new PassThrough();
|
||||
stream.on('data', async data => {
|
||||
const message = data.toString().trim();
|
||||
if (message?.length > 1) {
|
||||
await task.emitLog(message, metadata);
|
||||
}
|
||||
});
|
||||
|
||||
taskLogger.add(new winston.transports.Stream({ stream }));
|
||||
|
||||
if (step.if !== undefined) {
|
||||
// Support passing values like false to disable steps
|
||||
let skip = !step.if;
|
||||
|
||||
// Evaluate strings as handlebar templates
|
||||
if (typeof step.if === 'string') {
|
||||
const condition = JSON.parse(
|
||||
JSON.stringify(step.if),
|
||||
(_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it's just an empty string, treat it as undefined
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
|
||||
return value;
|
||||
},
|
||||
);
|
||||
|
||||
skip = !isTruthy(condition);
|
||||
}
|
||||
|
||||
if (skip) {
|
||||
await task.emitLog(`Skipped step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'skipped',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
await task.emitLog(`Beginning step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'processing',
|
||||
});
|
||||
|
||||
const action = actionRegistry.get(step.action);
|
||||
if (!action) {
|
||||
throw new Error(`Action '${step.action}' does not exist`);
|
||||
}
|
||||
|
||||
const input =
|
||||
step.input &&
|
||||
JSON.parse(JSON.stringify(step.input), (_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it smells like a JSON object then give it a parse as an object and if it fails return the string
|
||||
if (
|
||||
(templated.startsWith('"') && templated.endsWith('"')) ||
|
||||
(templated.startsWith('{') && templated.endsWith('}')) ||
|
||||
(templated.startsWith('[') && templated.endsWith(']'))
|
||||
) {
|
||||
try {
|
||||
// Don't recursively JSON parse the values of this string.
|
||||
// Shouldn't need to, don't want to encourage the use of returning handlebars from somewhere else
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
return templated;
|
||||
}
|
||||
|
||||
return value;
|
||||
});
|
||||
|
||||
if (action.schema?.input) {
|
||||
const validateResult = validateJsonSchema(
|
||||
input,
|
||||
action.schema.input,
|
||||
);
|
||||
if (!validateResult.valid) {
|
||||
const errors = validateResult.errors.join(', ');
|
||||
throw new InputError(
|
||||
`Invalid input passed to action ${action.id}, ${errors}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const stepOutputs: { [name: string]: JsonValue } = {};
|
||||
|
||||
// Keep track of all tmp dirs that are created by the action so we can remove them after
|
||||
const tmpDirs = new Array<string>();
|
||||
|
||||
this.options.logger.debug(`Running ${action.id} with input`, {
|
||||
input: JSON.stringify(input, null, 2),
|
||||
});
|
||||
|
||||
await action.handler({
|
||||
baseUrl: task.spec.baseUrl,
|
||||
logger: taskLogger,
|
||||
logStream: stream,
|
||||
input,
|
||||
token: task.secrets?.token,
|
||||
workspacePath,
|
||||
async createTemporaryDirectory() {
|
||||
const tmpDir = await fs.mkdtemp(
|
||||
`${workspacePath}_step-${step.id}-`,
|
||||
);
|
||||
tmpDirs.push(tmpDir);
|
||||
return tmpDir;
|
||||
},
|
||||
output(name: string, value: JsonValue) {
|
||||
stepOutputs[name] = value;
|
||||
},
|
||||
});
|
||||
|
||||
// Remove all temporary directories that were created when executing the action
|
||||
for (const tmpDir of tmpDirs) {
|
||||
await fs.remove(tmpDir);
|
||||
}
|
||||
|
||||
templateCtx.steps[step.id] = { output: stepOutputs };
|
||||
|
||||
await task.emitLog(`Finished step ${step.name}`, {
|
||||
...metadata,
|
||||
status: 'completed',
|
||||
});
|
||||
} catch (error) {
|
||||
await task.emitLog(String(error.stack), {
|
||||
...metadata,
|
||||
status: 'failed',
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
const output = JSON.parse(
|
||||
JSON.stringify(task.spec.output),
|
||||
(_key, value) => {
|
||||
if (typeof value === 'string') {
|
||||
const templated = this.handlebars.compile(value, {
|
||||
noEscape: true,
|
||||
data: false,
|
||||
preventIndent: true,
|
||||
})(templateCtx);
|
||||
|
||||
// If it's just an empty string, treat it as undefined
|
||||
if (templated === '') {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// If it smells like a JSON object then give it a parse as an object and if it fails return the string
|
||||
if (
|
||||
(templated.startsWith('"') && templated.endsWith('"')) ||
|
||||
(templated.startsWith('{') && templated.endsWith('}')) ||
|
||||
(templated.startsWith('[') && templated.endsWith(']'))
|
||||
) {
|
||||
try {
|
||||
// Don't recursively JSON parse the values of this string.
|
||||
// Shouldn't need to, don't want to encourage the use of returning handlebars from somewhere else
|
||||
return JSON.parse(templated);
|
||||
} catch {
|
||||
return templated;
|
||||
}
|
||||
}
|
||||
return templated;
|
||||
}
|
||||
return value;
|
||||
},
|
||||
);
|
||||
const { output } =
|
||||
task.spec.apiVersion === 'scaffolder.backstage.io/v1beta3'
|
||||
? await this.options.runners.workflowRunner.execute(task)
|
||||
: await this.options.runners.legacyWorkflowRunner.execute(task);
|
||||
|
||||
await task.complete('completed', { output });
|
||||
} catch (error) {
|
||||
await task.complete('failed', {
|
||||
error: { name: error.name, message: error.message },
|
||||
});
|
||||
} finally {
|
||||
if (workspacePath) {
|
||||
await fs.remove(workspacePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,7 +43,8 @@ export type DbTaskEventRow = {
|
||||
createdAt: string;
|
||||
};
|
||||
|
||||
export type TaskSpec = {
|
||||
export interface TaskSpecV1beta2 {
|
||||
apiVersion: 'backstage.io/v1beta2';
|
||||
baseUrl?: string;
|
||||
values: JsonObject;
|
||||
steps: Array<{
|
||||
@@ -54,7 +55,24 @@ export type TaskSpec = {
|
||||
if?: string | boolean;
|
||||
}>;
|
||||
output: { [name: string]: string };
|
||||
};
|
||||
}
|
||||
|
||||
export interface TaskStep {
|
||||
id: string;
|
||||
name: string;
|
||||
action: string;
|
||||
input?: JsonObject;
|
||||
if?: string | boolean;
|
||||
}
|
||||
export interface TaskSpecV1beta3 {
|
||||
apiVersion: 'scaffolder.backstage.io/v1beta3';
|
||||
baseUrl?: string;
|
||||
parameters: JsonObject;
|
||||
steps: TaskStep[];
|
||||
output: { [name: string]: JsonValue };
|
||||
}
|
||||
|
||||
export type TaskSpec = TaskSpecV1beta2 | TaskSpecV1beta3;
|
||||
|
||||
export type TaskSecrets = {
|
||||
token: string | undefined;
|
||||
@@ -122,3 +140,8 @@ export interface TaskStore {
|
||||
after,
|
||||
}: TaskStoreGetEventsOptions): Promise<{ events: DbTaskEventRow[] }>;
|
||||
}
|
||||
|
||||
export type WorkflowResponse = { output: { [key: string]: JsonValue } };
|
||||
export interface WorkflowRunner {
|
||||
execute(task: Task): Promise<WorkflowResponse>;
|
||||
}
|
||||
|
||||
@@ -35,9 +35,14 @@ import {
|
||||
import { InputError, NotFoundError } from '@backstage/errors';
|
||||
import { CatalogApi } from '@backstage/catalog-client';
|
||||
import { TemplateEntityV1beta2, Entity } from '@backstage/catalog-model';
|
||||
import { TemplateEntityV1beta3 } from '@backstage/plugin-scaffolder-common';
|
||||
|
||||
import { ScmIntegrations } from '@backstage/integration';
|
||||
import { TemplateAction } from '../scaffolder/actions';
|
||||
import { createBuiltinActions } from '../scaffolder/actions/builtin/createBuiltinActions';
|
||||
import { LegacyWorkflowRunner } from '../scaffolder/tasks/LegacyWorkflowRunner';
|
||||
import { DefaultWorkflowRunner } from '../scaffolder/tasks/DefaultWorkflowRunner';
|
||||
import { TaskSpec } from '../scaffolder/tasks/types';
|
||||
|
||||
export interface RouterOptions {
|
||||
logger: Logger;
|
||||
@@ -50,10 +55,13 @@ export interface RouterOptions {
|
||||
containerRunner: ContainerRunner;
|
||||
}
|
||||
|
||||
function isBeta2Template(
|
||||
entity: TemplateEntityV1beta2,
|
||||
): entity is TemplateEntityV1beta2 {
|
||||
return entity.apiVersion === 'backstage.io/v1beta2';
|
||||
function isSupportedTemplate(
|
||||
entity: TemplateEntityV1beta2 | TemplateEntityV1beta3,
|
||||
) {
|
||||
return (
|
||||
entity.apiVersion === 'backstage.io/v1beta2' ||
|
||||
entity.apiVersion === 'scaffolder.backstage.io/v1beta3'
|
||||
);
|
||||
}
|
||||
|
||||
export async function createRouter(
|
||||
@@ -83,14 +91,28 @@ export async function createRouter(
|
||||
);
|
||||
const taskBroker = new StorageTaskBroker(databaseTaskStore, logger);
|
||||
const actionRegistry = new TemplateActionRegistry();
|
||||
const legacyWorkflowRunner = new LegacyWorkflowRunner({
|
||||
logger,
|
||||
actionRegistry,
|
||||
integrations,
|
||||
workingDirectory,
|
||||
});
|
||||
|
||||
const workflowRunner = new DefaultWorkflowRunner({
|
||||
actionRegistry,
|
||||
integrations,
|
||||
logger,
|
||||
workingDirectory,
|
||||
});
|
||||
|
||||
const workers = [];
|
||||
for (let i = 0; i < (taskWorkers || 1); i++) {
|
||||
const worker = new TaskWorker({
|
||||
logger,
|
||||
taskBroker,
|
||||
actionRegistry,
|
||||
workingDirectory,
|
||||
integrations,
|
||||
runners: {
|
||||
legacyWorkflowRunner,
|
||||
workflowRunner,
|
||||
},
|
||||
});
|
||||
workers.push(worker);
|
||||
}
|
||||
@@ -128,7 +150,7 @@ export async function createRouter(
|
||||
const template = await entityClient.findTemplate(name, {
|
||||
token: getBearerToken(req.headers.authorization),
|
||||
});
|
||||
if (isBeta2Template(template)) {
|
||||
if (isSupportedTemplate(template)) {
|
||||
const parameters = [template.spec.parameters ?? []].flat();
|
||||
res.json({
|
||||
title: template.metadata.title ?? template.metadata.name,
|
||||
@@ -164,9 +186,9 @@ export async function createRouter(
|
||||
token,
|
||||
});
|
||||
|
||||
let taskSpec;
|
||||
let taskSpec: TaskSpec;
|
||||
|
||||
if (isBeta2Template(template)) {
|
||||
if (isSupportedTemplate(template)) {
|
||||
for (const parameters of [template.spec.parameters ?? []].flat()) {
|
||||
const result = validate(values, parameters);
|
||||
|
||||
@@ -178,16 +200,30 @@ export async function createRouter(
|
||||
|
||||
const baseUrl = getEntityBaseUrl(template);
|
||||
|
||||
taskSpec = {
|
||||
baseUrl,
|
||||
values,
|
||||
steps: template.spec.steps.map((step, index) => ({
|
||||
...step,
|
||||
id: step.id ?? `step-${index + 1}`,
|
||||
name: step.name ?? step.action,
|
||||
})),
|
||||
output: template.spec.output ?? {},
|
||||
};
|
||||
taskSpec =
|
||||
template.apiVersion === 'backstage.io/v1beta2'
|
||||
? {
|
||||
apiVersion: template.apiVersion,
|
||||
baseUrl,
|
||||
values,
|
||||
steps: template.spec.steps.map((step, index) => ({
|
||||
...step,
|
||||
id: step.id ?? `step-${index + 1}`,
|
||||
name: step.name ?? step.action,
|
||||
})),
|
||||
output: template.spec.output ?? {},
|
||||
}
|
||||
: {
|
||||
apiVersion: template.apiVersion,
|
||||
baseUrl,
|
||||
parameters: values,
|
||||
steps: template.spec.steps.map((step, index) => ({
|
||||
...step,
|
||||
id: step.id ?? `step-${index + 1}`,
|
||||
name: step.name ?? step.action,
|
||||
})),
|
||||
output: template.spec.output ?? {},
|
||||
};
|
||||
} else {
|
||||
throw new InputError(
|
||||
`Unsupported apiVersion field in schema entity, ${
|
||||
|
||||
Reference in New Issue
Block a user