paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 30 Packages Projects Releases Wiki Activity

feat: add auditor to coreServices

Browse Source 
Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

create eventAuditor and rootEventAuditor mockServices

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rerun api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

apply requested changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

remove rootAuditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run build:api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

misc fixes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

misc fixes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add dynamic plugin auditor service factory

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rerun api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add comments to auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

apply requested changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix tests

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor to the catalog plugin

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix tsc issues

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rename 'args' to 'options'

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor event shape

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add error option in auditor event shape

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rename events

Signed-off-by: Paul Schultz <pschultz@pobox.com>

demo new api?

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update service impl

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add the auditor service to the scaffolder

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update tests

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

revert config secret enumerator changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

clean up auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add documentation

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor to sign in/out

Signed-off-by: Paul Schultz <pschultz@pobox.com>

remove dynamic auditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add winston dep back

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run prettier

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add the auditor service as a default service factory

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix eslint errror

Signed-off-by: Paul Schultz <pschultz@pobox.com>

revert auth changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>
This commit is contained in:
Paul Schultz
2024-08-29 19:16:29 -05:00
parent 7b77cead3f
commit e0ae80edee
40 changed files with 3256 additions and 909 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/catalog-backend/README.md
+19
View File
@@ -87,6 +87,25 @@ yarn start
This will launch both frontend and backend in the same window, populated with
some example entities.
## Audit Events
- **`ancestry-fetch`**: Tracks `GET` requests to the `/entities/by-name/:kind/:namespace/:name/ancestry` endpoint which return the ancestry of an entity.
- **`batch-fetch`**: Tracks `POST` requests to the `/entities/by-refs` endpoint which return a batch of entities.
- **`delete`**: Tracks `DELETE` requests to the `/entities/by-uid/:uid` endpoint which delete an entity. Note: this will not be a permanent deletion and the entity will be restored if the parent location is still present in the catalog.
- **`facet-fetch`**: Tracks `GET` requests to the `/entity-facets` endpoint which return the facets of an entity.
- **`fetch`**: Tracks `GET` requests to the `/entities` endpoint which returns a list of entities.
- **`fetch-by-name`**: Tracks `GET` requests to the `/entities/by-name/:kind/:namespace/:name` endpoint which return an entity matching the specified entity ref.
- **`fetch-by-uid`**: Tracks `GET` requests to the `/entities/by-uid/:uid` endpoint which return an entity matching the specified entity uid.
- **`fetch-by-query`**: Tracks `GET` requests to the `/entities/by-query` endpoint which returns a list of entities matching the specified query.
- **`refresh`**: Tracks `POST` requests to the `/entities/refresh` endpoint which schedules the specified entity to be refreshed.
- **`validate`**: Tracks `POST` requests to the `/entities/validate` endpoint which validates the specified entity.
- **`location-analyze`**: Tracks `POST` requests to the `/locations/analyze` endpoint which analyzes the specified location.
- **`location-create`**: Tracks `POST` requests to the `/locations` endpoint which creates a location.
- **`location-delete`**: Tracks `DELETE` requests to the `/locations/:id` endpoint which deletes a location as well as all child entities associated with it.
- **`location-fetch`**: Tracks `GET` requests to the `/locations` endpoint which returns a list of locations.
- **`location-fetch-by-entity-ref`**: Tracks `GET` requests to the `/locations/by-entity` endpoint which returns a list of locations associated with the specified entity ref.
- **`location-fetch-by-id`**: Tracks `GET` requests to the `/locations/:id` endpoint which returns a location matching the specified location id.
## Links
- [catalog](https://github.com/backstage/backstage/tree/master/plugins/catalog)
plugins/catalog-backend/report.api.md
+2
View File
@@ -11,6 +11,7 @@ import { AnalyzeLocationGenerateEntity as AnalyzeLocationGenerateEntity_2 } from
import { AnalyzeLocationRequest as AnalyzeLocationRequest_2 } from '@backstage/plugin-catalog-common';
import { AnalyzeLocationResponse as AnalyzeLocationResponse_2 } from '@backstage/plugin-catalog-common';
import { AnalyzeOptions as AnalyzeOptions_2 } from '@backstage/plugin-catalog-node';
import { AuditorService } from '@backstage/backend-plugin-api';
import { AuthService } from '@backstage/backend-plugin-api';
import { BackendFeature } from '@backstage/backend-plugin-api';
import { CatalogApi } from '@backstage/catalog-client';
@@ -206,6 +207,7 @@ export type CatalogEnvironment = {
discovery?: DiscoveryService;
auth?: AuthService;
httpAuth?: HttpAuthService;
auditor?: AuditorService;
};
// @public
plugins/catalog-backend/src/service/CatalogBuilder.ts
+61 -56
View File
@@ -36,15 +36,61 @@ import { createHash } from 'crypto';
import { Router } from 'express';
import lodash, { keyBy } from 'lodash';
import {
AuditorService,
AuthService,
DatabaseService,
DiscoveryService,
HttpAuthService,
LoggerService,
PermissionsRegistryService,
PermissionsService,
RootConfigService,
SchedulerService,
UrlReaderService,
} from '@backstage/backend-plugin-api';
import { Config, readDurationFromConfig } from '@backstage/config';
import {
catalogPermissions,
RESOURCE_TYPE_CATALOG_ENTITY,
} from '@backstage/plugin-catalog-common/alpha';
import {
CatalogProcessor,
CatalogProcessorParser,
EntitiesSearchFilter,
EntityProvider,
PlaceholderResolver,
LocationAnalyzer,
PlaceholderResolver,
ScmLocationAnalyzer,
} from '@backstage/plugin-catalog-node';
import { EventBroker, EventsService } from '@backstage/plugin-events-node';
import {
Permission,
PermissionAuthorizer,
PermissionRuleParams,
toPermissionEvaluator,
} from '@backstage/plugin-permission-common';
import {
createConditionTransformer,
createPermissionIntegrationRouter,
PermissionRule,
} from '@backstage/plugin-permission-node';
import { durationToMilliseconds } from '@backstage/types';
import { DefaultCatalogDatabase } from '../database/DefaultCatalogDatabase';
import { DefaultProcessingDatabase } from '../database/DefaultProcessingDatabase';
import { DefaultProviderDatabase } from '../database/DefaultProviderDatabase';
import { applyDatabaseMigrations } from '../database/migrations';
import { DefaultCatalogRulesEnforcer } from '../ingestion/CatalogRules';
import { RepoLocationAnalyzer } from '../ingestion/LocationAnalyzer';
import { permissionRules as catalogPermissionRules } from '../permissions/rules';
import {
CatalogProcessingEngine,
createRandomProcessingInterval,
ProcessingIntervalFunction,
} from '../processing';
import { connectEntityProviders } from '../processing/connectEntityProviders';
import { DefaultCatalogProcessingEngine } from '../processing/DefaultCatalogProcessingEngine';
import { DefaultCatalogProcessingOrchestrator } from '../processing/DefaultCatalogProcessingOrchestrator';
import {
AnnotateLocationEntityProcessor,
BuiltinKindsEntityProcessor,
@@ -53,69 +99,24 @@ import {
PlaceholderProcessor,
UrlReaderProcessor,
} from '../processors';
import { ConfigLocationEntityProvider } from '../providers/ConfigLocationEntityProvider';
import { DefaultLocationStore } from '../providers/DefaultLocationStore';
import { RepoLocationAnalyzer } from '../ingestion/LocationAnalyzer';
import { AuthorizedLocationAnalyzer } from './AuthorizedLocationAnalyzer';
import {
jsonPlaceholderResolver,
textPlaceholderResolver,
yamlPlaceholderResolver,
} from '../processors/PlaceholderProcessor';
import { defaultEntityDataParser } from '../util/parse';
import {
CatalogProcessingEngine,
createRandomProcessingInterval,
ProcessingIntervalFunction,
} from '../processing';
import { DefaultProcessingDatabase } from '../database/DefaultProcessingDatabase';
import { applyDatabaseMigrations } from '../database/migrations';
import { DefaultCatalogProcessingEngine } from '../processing/DefaultCatalogProcessingEngine';
import { DefaultLocationService } from './DefaultLocationService';
import { DefaultEntitiesCatalog } from './DefaultEntitiesCatalog';
import { DefaultCatalogProcessingOrchestrator } from '../processing/DefaultCatalogProcessingOrchestrator';
import { ConfigLocationEntityProvider } from '../providers/ConfigLocationEntityProvider';
import { DefaultLocationStore } from '../providers/DefaultLocationStore';
import { DefaultStitcher } from '../stitching/DefaultStitcher';
import { createRouter } from './createRouter';
import { DefaultRefreshService } from './DefaultRefreshService';
import { AuthorizedRefreshService } from './AuthorizedRefreshService';
import { DefaultCatalogRulesEnforcer } from '../ingestion/CatalogRules';
import { Config, readDurationFromConfig } from '@backstage/config';
import { connectEntityProviders } from '../processing/connectEntityProviders';
import {
Permission,
PermissionAuthorizer,
PermissionRuleParams,
toPermissionEvaluator,
} from '@backstage/plugin-permission-common';
import { permissionRules as catalogPermissionRules } from '../permissions/rules';
import {
createConditionTransformer,
createPermissionIntegrationRouter,
PermissionRule,
} from '@backstage/plugin-permission-node';
import { defaultEntityDataParser } from '../util/parse';
import { AuthorizedEntitiesCatalog } from './AuthorizedEntitiesCatalog';
import { basicEntityFilter } from './request';
import {
catalogPermissions,
RESOURCE_TYPE_CATALOG_ENTITY,
} from '@backstage/plugin-catalog-common/alpha';
import { AuthorizedLocationAnalyzer } from './AuthorizedLocationAnalyzer';
import { AuthorizedLocationService } from './AuthorizedLocationService';
import { DefaultProviderDatabase } from '../database/DefaultProviderDatabase';
import { DefaultCatalogDatabase } from '../database/DefaultCatalogDatabase';
import { EventBroker, EventsService } from '@backstage/plugin-events-node';
import { durationToMilliseconds } from '@backstage/types';
import {
AuthService,
DatabaseService,
DiscoveryService,
HttpAuthService,
LoggerService,
PermissionsService,
RootConfigService,
UrlReaderService,
SchedulerService,
PermissionsRegistryService,
} from '@backstage/backend-plugin-api';
import { AuthorizedRefreshService } from './AuthorizedRefreshService';
import { createRouter } from './createRouter';
import { DefaultEntitiesCatalog } from './DefaultEntitiesCatalog';
import { DefaultLocationService } from './DefaultLocationService';
import { DefaultRefreshService } from './DefaultRefreshService';
import { basicEntityFilter } from './request';
import { entitiesResponseToObjects } from './response';
/**
@@ -136,12 +137,14 @@ export type CatalogEnvironment = {
database: DatabaseService;
config: RootConfigService;
reader: UrlReaderService;
// TODO: Require all services once `backend-legacy` is removed
permissions: PermissionsService | PermissionAuthorizer;
permissionsRegistry?: PermissionsRegistryService;
scheduler?: SchedulerService;
discovery?: DiscoveryService;
auth?: AuthService;
httpAuth?: HttpAuthService;
auditor?: AuditorService;
};
/**
@@ -482,6 +485,7 @@ export class CatalogBuilder {
scheduler,
permissionsRegistry,
discovery = HostDiscovery.fromConfig(config),
auditor,
} = this.env;
const { auth, httpAuth } = createLegacyAuthAdapters({
@@ -653,6 +657,7 @@ export class CatalogBuilder {
auth,
httpAuth,
permissionsService,
auditor,
disableRelationsCompatibility,
});
plugins/catalog-backend/src/service/CatalogPlugin.ts
+18 -15
View File
@@ -17,20 +17,8 @@ import {
coreServices,
createBackendPlugin,
} from '@backstage/backend-plugin-api';
import { eventsServiceRef } from '@backstage/plugin-events-node';
import { Entity, Validators } from '@backstage/catalog-model';
import { CatalogBuilder, CatalogPermissionRuleInput } from './CatalogBuilder';
import {
catalogAnalysisExtensionPoint,
CatalogModelExtensionPoint,
catalogModelExtensionPoint,
CatalogPermissionExtensionPoint,
catalogPermissionExtensionPoint,
CatalogProcessingExtensionPoint,
catalogProcessingExtensionPoint,
CatalogLocationsExtensionPoint,
catalogLocationsExtensionPoint,
} from '@backstage/plugin-catalog-node/alpha';
import { ForwardedError } from '@backstage/errors';
import {
CatalogProcessor,
CatalogProcessorParser,
@@ -39,9 +27,21 @@ import {
PlaceholderResolver,
ScmLocationAnalyzer,
} from '@backstage/plugin-catalog-node';
import { merge } from 'lodash';
import {
catalogAnalysisExtensionPoint,
CatalogLocationsExtensionPoint,
catalogLocationsExtensionPoint,
CatalogModelExtensionPoint,
catalogModelExtensionPoint,
CatalogPermissionExtensionPoint,
catalogPermissionExtensionPoint,
CatalogProcessingExtensionPoint,
catalogProcessingExtensionPoint,
} from '@backstage/plugin-catalog-node/alpha';
import { eventsServiceRef } from '@backstage/plugin-events-node';
import { Permission } from '@backstage/plugin-permission-common';
import { ForwardedError } from '@backstage/errors';
import { merge } from 'lodash';
import { CatalogBuilder, CatalogPermissionRuleInput } from './CatalogBuilder';
class CatalogLocationsExtensionPointImpl
implements CatalogLocationsExtensionPoint
@@ -235,6 +235,7 @@ export const catalogPlugin = createBackendPlugin({
discovery: coreServices.discovery,
auth: coreServices.auth,
httpAuth: coreServices.httpAuth,
auditor: coreServices.auditor,
events: eventsServiceRef,
},
async init({
@@ -250,6 +251,7 @@ export const catalogPlugin = createBackendPlugin({
discovery,
auth,
httpAuth,
auditor,
events,
}) {
const builder = await CatalogBuilder.create({
@@ -263,6 +265,7 @@ export const catalogPlugin = createBackendPlugin({
discovery,
auth,
httpAuth,
auditor,
});
builder.setEventBroker(events);
plugins/catalog-backend/src/service/createRouter.test.ts
+19 -16
View File
@@ -14,8 +14,9 @@
* limitations under the License.
*/
import { ConfigReader } from '@backstage/config';
import { NotFoundError } from '@backstage/errors';
import { MiddlewareFactory } from '@backstage/backend-defaults/rootHttpRouter';
import { wrapServer } from '@backstage/backend-openapi-utils';
import { mockCredentials, mockServices } from '@backstage/backend-test-utils';
import type { Location } from '@backstage/catalog-client';
import {
ANNOTATION_LOCATION,
@@ -23,26 +24,25 @@ import {
Entity,
stringifyEntityRef,
} from '@backstage/catalog-model';
import express from 'express';
import request from 'supertest';
import { Cursor, EntitiesCatalog } from '../catalog/types';
import { LocationInput, LocationService, RefreshService } from './types';
import { basicEntityFilter } from './request';
import { createRouter } from './createRouter';
import { ConfigReader } from '@backstage/config';
import { NotFoundError } from '@backstage/errors';
import { RESOURCE_TYPE_CATALOG_ENTITY } from '@backstage/plugin-catalog-common/alpha';
import { LocationAnalyzer } from '@backstage/plugin-catalog-node';
import { AuthorizeResult } from '@backstage/plugin-permission-common';
import {
createPermissionIntegrationRouter,
createPermissionRule,
} from '@backstage/plugin-permission-node';
import { RESOURCE_TYPE_CATALOG_ENTITY } from '@backstage/plugin-catalog-common/alpha';
import { CatalogProcessingOrchestrator } from '../processing/types';
import { z } from 'zod';
import { decodeCursor, encodeCursor } from './util';
import { wrapServer } from '@backstage/backend-openapi-utils';
import express from 'express';
import { Server } from 'http';
import { mockCredentials, mockServices } from '@backstage/backend-test-utils';
import { LocationAnalyzer } from '@backstage/plugin-catalog-node';
import { MiddlewareFactory } from '@backstage/backend-defaults/rootHttpRouter';
import request from 'supertest';
import { z } from 'zod';
import { Cursor, EntitiesCatalog } from '../catalog/types';
import { CatalogProcessingOrchestrator } from '../processing/types';
import { createRouter } from './createRouter';
import { basicEntityFilter } from './request';
import { LocationInput, LocationService, RefreshService } from './types';
import { decodeCursor, encodeCursor } from './util';
const middleware = MiddlewareFactory.create({
logger: mockServices.logger.mock(),
@@ -91,6 +91,7 @@ describe('createRouter readonly disabled', () => {
httpAuth: mockServices.httpAuth(),
locationAnalyzer,
permissionsService,
auditor: mockServices.auditor.mock(),
});
router.use(middleware.error());
app = await wrapServer(express().use(router));
@@ -971,6 +972,7 @@ describe('createRouter readonly and raw json enabled', () => {
auth: mockServices.auth(),
httpAuth: mockServices.httpAuth(),
permissionsService,
auditor: mockServices.auditor.mock(),
});
router.use(middleware.error());
app = express().use(router);
@@ -1190,6 +1192,7 @@ describe('NextRouter permissioning', () => {
auth: mockServices.auth(),
httpAuth: mockServices.httpAuth(),
permissionsService,
auditor: mockServices.auditor.mock(),
});
app = express().use(router);
});
plugins/catalog-backend/src/service/createRouter.ts
+573 -228
View File
@@ -14,6 +14,14 @@
* limitations under the License.
*/
import {
AuditorService,
AuthService,
HttpAuthService,
LoggerService,
PermissionsService,
SchedulerService,
} from '@backstage/backend-plugin-api';
import {
ANNOTATION_LOCATION,
ANNOTATION_ORIGIN_LOCATION,
@@ -23,12 +31,15 @@ import {
} from '@backstage/catalog-model';
import { Config } from '@backstage/config';
import { InputError, serializeError } from '@backstage/errors';
import { LocationAnalyzer } from '@backstage/plugin-catalog-node';
import express from 'express';
import yn from 'yn';
import { z } from 'zod';
import { Cursor, EntitiesCatalog } from '../catalog/types';
import { CatalogProcessingOrchestrator } from '../processing/types';
import { validateEntityEnvelope } from '../processing/util';
import { createOpenApiRouter } from '../schema/openapi';
import { AuthorizedValidationService } from './AuthorizedValidationService';
import {
basicEntityFilter,
entitiesBatchRequest,
@@ -38,6 +49,12 @@ import {
} from './request';
import { parseEntityFacetParams } from './request/parseEntityFacetParams';
import { parseEntityOrderParams } from './request/parseEntityOrderParams';
import { parseEntityPaginationParams } from './request/parseEntityPaginationParams';
import {
createEntityArrayJsonStream,
writeEntitiesResponse,
writeSingleEntityResponse,
} from './response';
import { LocationService, RefreshService } from './types';
import {
disallowReadonlyMode,
@@ -45,22 +62,6 @@ import {
locationInput,
validateRequestBody,
} from './util';
import { createOpenApiRouter } from '../schema/openapi';
import { parseEntityPaginationParams } from './request/parseEntityPaginationParams';
import {
AuthService,
HttpAuthService,
LoggerService,
SchedulerService,
PermissionsService,
} from '@backstage/backend-plugin-api';
import { LocationAnalyzer } from '@backstage/plugin-catalog-node';
import { AuthorizedValidationService } from './AuthorizedValidationService';
import {
createEntityArrayJsonStream,
writeEntitiesResponse,
writeSingleEntityResponse,
} from './response';
/**
* Options used by {@link createRouter}.
@@ -81,6 +82,8 @@ export interface RouterOptions {
auth: AuthService;
httpAuth: HttpAuthService;
permissionsService: PermissionsService;
// TODO: Require AuditorService once `backend-legacy` is removed
auditor?: AuditorService;
disableRelationsCompatibility?: boolean;
}
@@ -109,6 +112,7 @@ export async function createRouter(
permissionsService,
auth,
httpAuth,
auditor,
disableRelationsCompatibility = false,
} = options;
@@ -119,18 +123,34 @@ export async function createRouter(
}
if (refreshService) {
// TODO: Potentially find a way to track the ancestor that gets refreshed to refresh this entity (as well as the child of that ancestor?)
router.post('/refresh', async (req, res) => {
const { authorizationToken, ...restBody } = req.body;
const credentials = authorizationToken
? await auth.authenticate(authorizationToken)
: await httpAuth.credentials(req);
await refreshService.refresh({
...restBody,
credentials,
const auditorEvent = await auditor?.createEvent({
eventId: 'refresh',
meta: {
entityRef: restBody.entityRef,
},
request: req,
});
res.status(200).end();
try {
const credentials = authorizationToken
? await auth.authenticate(authorizationToken)
: await httpAuth.credentials(req);
await refreshService.refresh({
...restBody,
credentials,
});
await auditorEvent?.success();
res.status(200).end();
} catch (err) {
await auditorEvent?.fail({ error: err });
throw err;
}
});
}
@@ -141,95 +161,115 @@ export async function createRouter(
if (entitiesCatalog) {
router
.get('/entities', async (req, res) => {
const filter = parseEntityFilterParams(req.query);
const fields = parseEntityTransformParams(req.query);
const order = parseEntityOrderParams(req.query);
const pagination = parseEntityPaginationParams(req.query);
const credentials = await httpAuth.credentials(req);
// When pagination parameters are passed in, use the legacy slow path
// that loads all entities into memory
if (pagination || disableRelationsCompatibility !== true) {
const { entities, pageInfo } = await entitiesCatalog.entities({
filter,
fields,
order,
pagination,
credentials,
});
// Add a Link header to the next page
if (pageInfo.hasNextPage) {
const url = new URL(`http://ignored${req.url}`);
url.searchParams.delete('offset');
url.searchParams.set('after', pageInfo.endCursor);
res.setHeader('link', `<${url.pathname}${url.search}>; rel="next"`);
}
await writeEntitiesResponse({
res,
items: entities,
alwaysUseObjectMode: !disableRelationsCompatibility,
});
return;
}
const responseStream = createEntityArrayJsonStream(res);
const limit = 10000;
let cursor: Cursor | undefined;
const auditorEvent = await auditor?.createEvent({
eventId: 'CatalogEntityFetch',
request: req,
});
try {
let currentWrite: Promise<boolean> | undefined = undefined;
do {
const result = await entitiesCatalog.queryEntities(
!cursor
? {
credentials,
fields,
limit,
filter,
orderFields: order,
skipTotalItems: true,
}
: { credentials, fields, limit, cursor },
);
const filter = parseEntityFilterParams(req.query);
const fields = parseEntityTransformParams(req.query);
const order = parseEntityOrderParams(req.query);
const pagination = parseEntityPaginationParams(req.query);
const credentials = await httpAuth.credentials(req);
// Wait for previous write to complete
if (await currentWrite) {
return; // Client closed connection
// When pagination parameters are passed in, use the legacy slow path
// that loads all entities into memory
if (pagination || disableRelationsCompatibility !== true) {
const { entities, pageInfo } = await entitiesCatalog.entities({
filter,
fields,
order,
pagination,
credentials,
});
// Add a Link header to the next page
if (pageInfo.hasNextPage) {
const url = new URL(`http://ignored${req.url}`);
url.searchParams.delete('offset');
url.searchParams.set('after', pageInfo.endCursor);
res.setHeader(
'link',
`<${url.pathname}${url.search}>; rel="next"`,
);
}
if (result.items.entities.length) {
currentWrite = responseStream.send(result.items);
}
await auditorEvent?.success();
cursor = result.pageInfo?.nextCursor;
} while (cursor);
await writeEntitiesResponse({
res,
items: entities,
alwaysUseObjectMode: !disableRelationsCompatibility,
});
return;
}
// Wait for last write to complete
await currentWrite;
const responseStream = createEntityArrayJsonStream(res);
const limit = 10000;
let cursor: Cursor | undefined;
responseStream.complete();
} finally {
responseStream.close();
try {
let currentWrite: Promise<boolean> | undefined = undefined;
do {
const result = await entitiesCatalog.queryEntities(
!cursor
? {
credentials,
fields,
limit,
filter,
orderFields: order,
skipTotalItems: true,
}
: { credentials, fields, limit, cursor },
);
// Wait for previous write to complete
if (await currentWrite) {
return; // Client closed connection
}
if (result.items.entities.length) {
currentWrite = responseStream.send(result.items);
}
cursor = result.pageInfo?.nextCursor;
} while (cursor);
// Wait for last write to complete
await currentWrite;
await auditorEvent?.success();
responseStream.complete();
} finally {
responseStream.close();
}
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/entities/by-query', async (req, res) => {
const { items, pageInfo, totalItems } =
await entitiesCatalog.queryEntities({
limit: req.query.limit,
offset: req.query.offset,
...parseQueryEntitiesParams(req.query),
credentials: await httpAuth.credentials(req),
});
const auditorEvent = await auditor?.createEvent({
eventId: 'fetch-by-query',
request: req,
});
await writeEntitiesResponse({
res,
items,
alwaysUseObjectMode: !disableRelationsCompatibility,
responseWrapper: entities => ({
items: entities,
try {
const { items, pageInfo, totalItems } =
await entitiesCatalog.queryEntities({
limit: req.query.limit,
offset: req.query.offset,
...parseQueryEntitiesParams(req.query),
credentials: await httpAuth.credentials(req),
});
const meta = {
totalItems,
pageInfo: {
...(pageInfo.nextCursor && {
@@ -239,71 +279,217 @@ export async function createRouter(
prevCursor: encodeCursor(pageInfo.prevCursor),
}),
},
}),
});
};
await auditorEvent?.success({
// Let's not log out the entities since this can make the log very big
meta,
});
await writeEntitiesResponse({
res,
items,
alwaysUseObjectMode: !disableRelationsCompatibility,
responseWrapper: entities => ({
items: entities,
...meta,
}),
});
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/entities/by-uid/:uid', async (req, res) => {
const { uid } = req.params;
const { entities } = await entitiesCatalog.entities({
filter: basicEntityFilter({ 'metadata.uid': uid }),
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'fetch-by-uid',
request: req,
meta: {
uid: uid,
},
});
writeSingleEntityResponse(res, entities, `No entity with uid ${uid}`);
try {
const { entities } = await entitiesCatalog.entities({
filter: basicEntityFilter({ 'metadata.uid': uid }),
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success({
meta: {
entities: entities,
},
});
writeSingleEntityResponse(res, entities, `No entity with uid ${uid}`);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.delete('/entities/by-uid/:uid', async (req, res) => {
const { uid } = req.params;
await entitiesCatalog.removeEntityByUid(uid, {
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'delete',
severityLevel: 'medium',
request: req,
meta: {
uid: uid,
},
});
res.status(204).end();
try {
await entitiesCatalog.removeEntityByUid(uid, {
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success();
res.status(204).end();
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/entities/by-name/:kind/:namespace/:name', async (req, res) => {
const { kind, namespace, name } = req.params;
const { items } = await entitiesCatalog.entitiesBatch({
entityRefs: [stringifyEntityRef({ kind, namespace, name })],
credentials: await httpAuth.credentials(req),
const entityRef = stringifyEntityRef({ kind, namespace, name });
const auditorEvent = await auditor?.createEvent({
eventId: 'fetch-by-name',
request: req,
meta: {
entityRef: entityRef,
},
});
writeSingleEntityResponse(
res,
items,
`No entity named '${name}' found, with kind '${kind}' in namespace '${namespace}'`,
);
try {
const { items } = await entitiesCatalog.entitiesBatch({
entityRefs: [stringifyEntityRef({ kind, namespace, name })],
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success();
writeSingleEntityResponse(
res,
items,
`No entity named '${name}' found, with kind '${kind}' in namespace '${namespace}'`,
);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get(
'/entities/by-name/:kind/:namespace/:name/ancestry',
async (req, res) => {
const { kind, namespace, name } = req.params;
const entityRef = stringifyEntityRef({ kind, namespace, name });
const response = await entitiesCatalog.entityAncestry(entityRef, {
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'ancestry-fetch',
request: req,
meta: {
entityRef: entityRef,
},
});
res.status(200).json(response);
try {
const response = await entitiesCatalog.entityAncestry(entityRef, {
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success({
meta: {
rootEntityRef: response.rootEntityRef,
ancestry: response.items.map(ancestryLink => {
return {
entityRef: stringifyEntityRef(ancestryLink.entity),
parentEntityRefs: ancestryLink.parentEntityRefs,
};
}),
},
});
res.status(200).json(response);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
},
)
.post('/entities/by-refs', async (req, res) => {
const request = entitiesBatchRequest(req);
const { items } = await entitiesCatalog.entitiesBatch({
entityRefs: request.entityRefs,
filter: parseEntityFilterParams(req.query),
fields: parseEntityTransformParams(req.query, request.fields),
credentials: await httpAuth.credentials(req),
});
await writeEntitiesResponse({
res,
items,
alwaysUseObjectMode: !disableRelationsCompatibility,
responseWrapper: entities => ({
items: entities,
}),
const auditorEvent = await auditor?.createEvent({
eventId: 'batch-fetch',
request: req,
});
try {
const request = entitiesBatchRequest(req);
const { items } = await entitiesCatalog.entitiesBatch({
entityRefs: request.entityRefs,
filter: parseEntityFilterParams(req.query),
fields: parseEntityTransformParams(req.query, request.fields),
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success({
meta: {
...request,
},
});
await writeEntitiesResponse({
res,
items,
alwaysUseObjectMode: !disableRelationsCompatibility,
responseWrapper: entities => ({
items: entities,
}),
});
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/entity-facets', async (req, res) => {
const response = await entitiesCatalog.facets({
filter: parseEntityFilterParams(req.query),
facets: parseEntityFacetParams(req.query),
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'facet-fetch',
request: req,
});
res.status(200).json(response);
try {
const response = await entitiesCatalog.facets({
filter: parseEntityFilterParams(req.query),
facets: parseEntityFacetParams(req.query),
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success();
res.status(200).json(response);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
});
}
@@ -313,79 +499,212 @@ export async function createRouter(
const location = await validateRequestBody(req, locationInput);
const dryRun = yn(req.query.dryRun, { default: false });
// when in dryRun addLocation is effectively a read operation so we don't
// need to disallow readonly
if (!dryRun) {
disallowReadonlyMode(readonlyEnabled);
}
const output = await locationService.createLocation(location, dryRun, {
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'location-create',
severityLevel: dryRun ? 'low' : 'medium',
request: req,
meta: {
location: location,
isDryRun: dryRun,
},
});
res.status(201).json(output);
try {
// when in dryRun addLocation is effectively a read operation so we don't
// need to disallow readonly
if (!dryRun) {
disallowReadonlyMode(readonlyEnabled);
}
const output = await locationService.createLocation(
location,
dryRun,
{
credentials: await httpAuth.credentials(req),
},
);
await auditorEvent?.success({
meta: {
location: output.location,
},
});
res.status(201).json(output);
} catch (err) {
await auditorEvent?.fail({
error: err,
meta: {
location: location,
isDryRun: dryRun,
},
});
throw err;
}
})
.get('/locations', async (req, res) => {
const locations = await locationService.listLocations({
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'location-fetch',
request: req,
});
res.status(200).json(locations.map(l => ({ data: l })));
try {
const locations = await locationService.listLocations({
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success();
res.status(200).json(locations.map(l => ({ data: l })));
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/locations/:id', async (req, res) => {
const { id } = req.params;
const output = await locationService.getLocation(id, {
credentials: await httpAuth.credentials(req),
const auditorEvent = await auditor?.createEvent({
eventId: 'location-fetch-by-id',
request: req,
meta: {
id: id,
},
});
res.status(200).json(output);
try {
const output = await locationService.getLocation(id, {
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success({
meta: {
output: output,
},
});
res.status(200).json(output);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.delete('/locations/:id', async (req, res) => {
const { id } = req.params;
const auditorEvent = await auditor?.createEvent({
eventId: 'location-delete',
severityLevel: 'medium',
request: req,
meta: {
id: id,
},
});
disallowReadonlyMode(readonlyEnabled);
const { id } = req.params;
await locationService.deleteLocation(id, {
credentials: await httpAuth.credentials(req),
});
res.status(204).end();
try {
await locationService.deleteLocation(id, {
credentials: await httpAuth.credentials(req),
});
await auditorEvent?.success();
res.status(204).end();
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
})
.get('/locations/by-entity/:kind/:namespace/:name', async (req, res) => {
const { kind, namespace, name } = req.params;
const output = await locationService.getLocationByEntity(
{ kind, namespace, name },
{ credentials: await httpAuth.credentials(req) },
);
res.status(200).json(output);
const locationRef = `${kind}:${namespace}/${name}`;
const auditorEvent = await auditor?.createEvent({
eventId: 'location-fetch-by-entity-ref',
request: req,
meta: {
locationRef: locationRef,
},
});
try {
const output = await locationService.getLocationByEntity(
{ kind, namespace, name },
{ credentials: await httpAuth.credentials(req) },
);
await auditorEvent?.success({
meta: {
output: output,
},
});
res.status(200).json(output);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
});
}
if (locationAnalyzer) {
router.post('/analyze-location', async (req, res) => {
const body = await validateRequestBody(
req,
z.object({
const auditorEvent = await auditor?.createEvent({
eventId: 'location-analyze',
request: req,
});
try {
const body = await validateRequestBody(
req,
z.object({
location: locationInput,
catalogFilename: z.string().optional(),
}),
);
const schema = z.object({
location: locationInput,
catalogFilename: z.string().optional(),
}),
);
const schema = z.object({
location: locationInput,
catalogFilename: z.string().optional(),
});
const credentials = await httpAuth.credentials(req);
const parsedBody = schema.parse(body);
try {
const output = await locationAnalyzer.analyzeLocation(
parsedBody,
credentials,
);
res.status(200).json(output);
} catch (err) {
if (
// Catch errors from parse-url library.
err.name === 'Error' &&
'subject_url' in err
) {
throw new InputError('The given location.target is not a URL');
});
const credentials = await httpAuth.credentials(req);
const parsedBody = schema.parse(body);
try {
const output = await locationAnalyzer.analyzeLocation(
parsedBody,
credentials,
);
await auditorEvent?.success({
meta: {
output: output,
},
});
res.status(200).json(output);
} catch (err) {
if (
// Catch errors from parse-url library.
err.name === 'Error' &&
'subject_url' in err
) {
throw new InputError('The given location.target is not a URL');
}
throw err;
}
} catch (err) {
await auditorEvent?.fail({
error: err,
});
throw err;
}
});
@@ -393,55 +712,81 @@ export async function createRouter(
if (orchestrator) {
router.post('/validate-entity', async (req, res) => {
const bodySchema = z.object({
entity: z.unknown(),
location: z.string(),
const auditorEvent = await auditor?.createEvent({
eventId: 'validate',
request: req,
});
let body: z.infer<typeof bodySchema>;
let entity: Entity;
let location: { type: string; target: string };
try {
body = await validateRequestBody(req, bodySchema);
entity = validateEntityEnvelope(body.entity);
location = parseLocationRef(body.location);
if (location.type !== 'url')
throw new TypeError(
`Invalid location ref ${body.location}, only 'url:<target>' is supported, e.g. url:https://host/path`,
);
} catch (err) {
return res.status(400).json({
errors: [serializeError(err)],
const bodySchema = z.object({
entity: z.unknown(),
location: z.string(),
});
}
const credentials = await httpAuth.credentials(req);
const authorizedValidationService = new AuthorizedValidationService(
orchestrator,
permissionsService,
);
const processingResult = await authorizedValidationService.process(
{
entity: {
...entity,
metadata: {
...entity.metadata,
annotations: {
[ANNOTATION_LOCATION]: body.location,
[ANNOTATION_ORIGIN_LOCATION]: body.location,
...entity.metadata.annotations,
let body: z.infer<typeof bodySchema>;
let entity: Entity;
let location: { type: string; target: string };
try {
body = await validateRequestBody(req, bodySchema);
entity = validateEntityEnvelope(body.entity);
location = parseLocationRef(body.location);
if (location.type !== 'url')
throw new TypeError(
`Invalid location ref ${body.location}, only 'url:<target>' is supported, e.g. url:https://host/path`,
);
} catch (err) {
await auditorEvent?.fail({
error: err,
});
return res.status(400).json({
errors: [serializeError(err)],
});
}
const credentials = await httpAuth.credentials(req);
const authorizedValidationService = new AuthorizedValidationService(
orchestrator,
permissionsService,
);
const processingResult = await authorizedValidationService.process(
{
entity: {
...entity,
metadata: {
...entity.metadata,
annotations: {
[ANNOTATION_LOCATION]: body.location,
[ANNOTATION_ORIGIN_LOCATION]: body.location,
...entity.metadata.annotations,
},
},
},
},
},
credentials,
);
credentials,
);
if (!processingResult.ok)
res.status(400).json({
errors: processingResult.errors.map(e => serializeError(e)),
if (!processingResult.ok) {
const errors = processingResult.errors.map(e => serializeError(e));
await auditorEvent?.fail({
errors: errors,
});
res.status(400).json({
errors,
});
}
await auditorEvent?.success();
return res.status(200).end();
} catch (err) {
await auditorEvent?.fail({
error: err,
});
return res.status(200).end();
throw err;
}
});
}
plugins/scaffolder-backend/README.md
+15
View File
@@ -63,3 +63,18 @@ you will not have any templates available to use. These need to be [added to the
To get up and running and try out some templates quickly, you can or copy the
catalog locations from the [create-app template](https://github.com/backstage/backstage/blob/master/packages/create-app/templates/default-app/app-config.yaml.hbs).
## Audit Events
- **`parameter-schema-fetch`**: Tracks`GET` requests to the `/v2/templates/:namespace/:kind/:name/parameter-schema` endpoint which return template parameter schemas
- **`installed-actions-fetch`**: Tracks`GET` requests to the `/v2/actions` endpoint which grabs the list of installed actions
- **`task-creation`**: Tracks`POST` requests to the `/v2/tasks` endpoint which creates tasks that the scaffolder executes
- **`task-list-fetch`**: Tracks`GET` requests to the `/v2/tasks` endpoint which fetches details of all tasks in the scaffolder.
- **`task-fetch`**: Tracks`GET` requests to the `/v2/tasks/:taskId` endpoint which fetches details of a specified task `:taskId`
- **`task-cancellation`**: Tracks`POST` requests to the `/v2/tasks/:taskId/cancel` endpoint which cancels a running task
- **`task-retry`**: Tracks`POST` requests to the `/v2/tasks/:taskId/retry` endpoint which retries a failed task
- **`task-stream`**: Tracks`GET` requests to the `/v2/tasks/:taskId/eventstream` endpoint which returns an event stream of the task logs of task `:taskId`
- **`task-event-fetch`**: Tracks`GET` requests to the `/v2/tasks/:taskId/events` endpoint which returns a snapshot of the task logs of task `:taskId`
- **`task-dry-run`**: Tracks`POST` requests to the `/v2/dry-run` endpoint which creates a dry-run task. All audit logs for events associated with dry runs have the `meta.isDryLog` flag set to `true`.
- **`stale-task-cancellation`**: Tracks automated cancellation of stale tasks
- **`task-execution`**: Tracks the`initiation` and `completion` of a real scaffolder task execution (will not occur during dry runs)
plugins/scaffolder-backend/report.api.md
+5
View File
@@ -6,6 +6,7 @@
/// <reference types="node" />
import { ActionContext as ActionContext_2 } from '@backstage/plugin-scaffolder-node';
import { AuditorService } from '@backstage/backend-plugin-api';
import { AuthService } from '@backstage/backend-plugin-api';
import { AutocompleteHandler } from '@backstage/plugin-scaffolder-node/alpha';
import * as azure from '@backstage/plugin-scaffolder-backend-module-azure';
@@ -407,6 +408,7 @@ export type CreateWorkerOptions = {
integrations: ScmIntegrations;
workingDirectory: string;
logger: Logger;
auditor?: AuditorService;
additionalTemplateFilters?: Record<string, TemplateFilter_2>;
concurrentTasksLimit?: number;
additionalTemplateGlobals?: Record<string, TemplateGlobal_2>;
@@ -541,6 +543,8 @@ export interface RouterOptions {
// (undocumented)
additionalWorkspaceProviders?: Record<string, WorkspaceProvider>;
// (undocumented)
auditor?: AuditorService;
// (undocumented)
auth?: AuthService;
// (undocumented)
autocompleteHandlers?: Record<string, AutocompleteHandler>;
@@ -630,6 +634,7 @@ export class TaskManager implements TaskContext_2 {
auth?: AuthService,
config?: Config,
additionalWorkspaceProviders?: Record<string, WorkspaceProvider>,
auditor?: AuditorService,
): TaskManager;
// (undocumented)
get createdBy(): string | undefined;
plugins/scaffolder-backend/src/ScaffolderPlugin.ts
+6 -3
View File
@@ -14,13 +14,14 @@
* limitations under the License.
*/
import { loggerToWinstonLogger } from '@backstage/backend-common';
import {
coreServices,
createBackendPlugin,
} from '@backstage/backend-plugin-api';
import { loggerToWinstonLogger } from '@backstage/backend-common';
import { ScmIntegrations } from '@backstage/integration';
import { catalogServiceRef } from '@backstage/plugin-catalog-node/alpha';
import { eventsServiceRef } from '@backstage/plugin-events-node';
import {
TaskBroker,
TemplateAction,
@@ -46,12 +47,11 @@ import {
createFetchTemplateAction,
createFetchTemplateFileAction,
createFilesystemDeleteAction,
createFilesystemRenameAction,
createFilesystemReadDirAction,
createFilesystemRenameAction,
createWaitAction,
} from './scaffolder';
import { createRouter } from './service/router';
import { eventsServiceRef } from '@backstage/plugin-events-node';
/**
* Scaffolder plugin
@@ -115,6 +115,7 @@ export const scaffolderPlugin = createBackendPlugin({
discovery: coreServices.discovery,
httpRouter: coreServices.httpRouter,
httpAuth: coreServices.httpAuth,
auditor: coreServices.auditor,
catalogClient: catalogServiceRef,
events: eventsServiceRef,
},
@@ -131,6 +132,7 @@ export const scaffolderPlugin = createBackendPlugin({
catalogClient,
permissions,
events,
auditor,
}) {
const log = loggerToWinstonLogger(logger);
const integrations = ScmIntegrations.fromConfig(config);
@@ -195,6 +197,7 @@ export const scaffolderPlugin = createBackendPlugin({
autocompleteHandlers,
additionalWorkspaceProviders,
events,
auditor,
});
httpRouter.use(router);
},
plugins/scaffolder-backend/src/scaffolder/dryrun/createDryRunner.ts
+16 -11
View File
@@ -14,29 +14,32 @@
* limitations under the License.
*/
import {
AuditorService,
BackstageCredentials,
} from '@backstage/backend-plugin-api';
import type { UserEntity } from '@backstage/catalog-model';
import { ScmIntegrations } from '@backstage/integration';
import { PermissionEvaluator } from '@backstage/plugin-permission-common';
import { TaskSpec, TemplateInfo } from '@backstage/plugin-scaffolder-common';
import { JsonObject } from '@backstage/types';
import { fileURLToPath } from 'url';
import { Logger } from 'winston';
import {
createTemplateAction,
TaskSecrets,
TemplateFilter,
TemplateGlobal,
deserializeDirectoryContents,
SerializedFile,
serializeDirectoryContents,
TaskSecrets,
TemplateFilter,
TemplateGlobal,
} from '@backstage/plugin-scaffolder-node';
import { JsonObject } from '@backstage/types';
import fs from 'fs-extra';
import path from 'path';
import { fileURLToPath } from 'url';
import { v4 as uuid } from 'uuid';
import { Logger } from 'winston';
import { TemplateActionRegistry } from '../actions';
import { NunjucksWorkflowRunner } from '../tasks/NunjucksWorkflowRunner';
import { DecoratedActionsRegistry } from './DecoratedActionsRegistry';
import fs from 'fs-extra';
import { PermissionEvaluator } from '@backstage/plugin-permission-common';
import { BackstageCredentials } from '@backstage/backend-plugin-api';
import type { UserEntity } from '@backstage/catalog-model';
import { v4 as uuid } from 'uuid';
interface DryRunInput {
spec: TaskSpec;
@@ -59,6 +62,7 @@ interface DryRunResult {
/** @internal */
export type TemplateTesterCreateOptions = {
logger: Logger;
auditor?: AuditorService;
integrations: ScmIntegrations;
actionRegistry: TemplateActionRegistry;
workingDirectory: string;
@@ -109,6 +113,7 @@ export function createDryRunner(options: TemplateTesterCreateOptions) {
const abortSignal = new AbortController().signal;
const result = await workflowRunner.execute({
taskId: dryRunId,
spec: {
...input.spec,
steps: [
plugins/scaffolder-backend/src/scaffolder/tasks/NunjucksWorkflowRunner.ts
+39 -31
View File
@@ -14,48 +14,51 @@
* limitations under the License.
*/
import { ScmIntegrations } from '@backstage/integration';
import { TaskTrackType, WorkflowResponse, WorkflowRunner } from './types';
import * as winston from 'winston';
import fs from 'fs-extra';
import path from 'path';
import nunjucks from 'nunjucks';
import { JsonArray, JsonObject, JsonValue } from '@backstage/types';
import { InputError, NotAllowedError, stringifyError } from '@backstage/errors';
import { PassThrough } from 'stream';
import { generateExampleOutput, isTruthy } from './helper';
import { validate as validateJsonSchema } from 'jsonschema';
import { TemplateActionRegistry } from '../actions';
import { metrics } from '@opentelemetry/api';
import {
SecureTemplater,
SecureTemplateRenderer,
} from '../../lib/templating/SecureTemplater';
import { ScmIntegrations } from '@backstage/integration';
import {
TaskRecovery,
TaskSpec,
TaskSpecV1beta3,
TaskStep,
} from '@backstage/plugin-scaffolder-common';
import { JsonArray, JsonObject, JsonValue } from '@backstage/types';
import { metrics } from '@opentelemetry/api';
import fs from 'fs-extra';
import { validate as validateJsonSchema } from 'jsonschema';
import nunjucks from 'nunjucks';
import path from 'path';
import { PassThrough } from 'stream';
import * as winston from 'winston';
import {
TemplateAction,
TemplateFilter,
TemplateGlobal,
TaskContext,
} from '@backstage/plugin-scaffolder-node';
import { createConditionAuthorizer } from '@backstage/plugin-permission-node';
SecureTemplater,
SecureTemplateRenderer,
} from '../../lib/templating/SecureTemplater';
import { TemplateActionRegistry } from '../actions';
import { generateExampleOutput, isTruthy } from './helper';
import { TaskTrackType, WorkflowResponse, WorkflowRunner } from './types';
import { loggerToWinstonLogger } from '@backstage/backend-common';
import type {
AuditorService,
PermissionsService,
} from '@backstage/backend-plugin-api';
import { UserEntity } from '@backstage/catalog-model';
import { createCounterMetric, createHistogramMetric } from '../../util/metrics';
import { createDefaultFilters } from '../../lib/templating/filters';
import {
AuthorizeResult,
PolicyDecision,
} from '@backstage/plugin-permission-common';
import { scaffolderActionRules } from '../../service/rules';
import { createConditionAuthorizer } from '@backstage/plugin-permission-node';
import { actionExecutePermission } from '@backstage/plugin-scaffolder-common/alpha';
import { PermissionsService } from '@backstage/backend-plugin-api';
import { loggerToWinstonLogger } from '@backstage/backend-common';
import {
TaskContext,
TemplateAction,
TemplateFilter,
TemplateGlobal,
} from '@backstage/plugin-scaffolder-node';
import { createDefaultFilters } from '../../lib/templating/filters';
import { scaffolderActionRules } from '../../service/rules';
import { createCounterMetric, createHistogramMetric } from '../../util/metrics';
import { BackstageLoggerTransport, WinstonLogger } from './logger';
type NunjucksWorkflowRunnerOptions = {
@@ -63,6 +66,7 @@ type NunjucksWorkflowRunnerOptions = {
actionRegistry: TemplateActionRegistry;
integrations: ScmIntegrations;
logger: winston.Logger;
auditor?: AuditorService;
additionalTemplateFilters?: Record<string, TemplateFilter>;
additionalTemplateGlobals?: Record<string, TemplateGlobal>;
permissions?: PermissionsService;
@@ -127,7 +131,7 @@ const createStepLogger = ({
// Initially this stream used to be the only way to write to the client logs, but that
// has changed over time, there's not really a need for this anymore.
// You can just create a simple wrapper like the below in your action to write to the main logger.
// This way we also get recactions for free.
// This way we also get redactions for free.
const streamLogger = new PassThrough();
streamLogger.on('data', async data => {
const message = data.toString().trim();
@@ -245,7 +249,9 @@ export class NunjucksWorkflowRunner implements WorkflowRunner {
const stepTrack = await this.tracker.stepStart(task, step);
if (task.cancelSignal.aborted) {
throw new Error(`Step ${step.name} has been cancelled.`);
throw new Error(
`Step ${step.id} (${step.name}) of task ${task.taskId} has been cancelled.`,
);
}
try {
@@ -454,7 +460,9 @@ export class NunjucksWorkflowRunner implements WorkflowRunner {
context.steps[step.id] = { output: stepOutput };
if (task.cancelSignal.aborted) {
throw new Error(`Step ${step.name} has been cancelled.`);
throw new Error(
`Step ${step.id} (${step.name}) of task ${task.taskId} has been cancelled.`,
);
}
await stepTrack.markSuccessful();
plugins/scaffolder-backend/src/scaffolder/tasks/StorageTaskBroker.ts
+50 -19
View File
@@ -14,16 +14,13 @@
* limitations under the License.
*/
import {
AuditorService,
AuthService,
BackstageCredentials,
} from '@backstage/backend-plugin-api';
import { Config } from '@backstage/config';
import { TaskSpec } from '@backstage/plugin-scaffolder-common';
import {
JsonObject,
JsonValue,
Observable,
createDeferred,
} from '@backstage/types';
import { Logger } from 'winston';
import ObservableImpl from 'zen-observable';
import {
SerializedTask,
SerializedTaskEvent,
@@ -34,14 +31,18 @@ import {
TaskSecrets,
TaskStatus,
} from '@backstage/plugin-scaffolder-node';
import { InternalTaskSecrets, TaskStore } from './types';
import { readDuration } from './helper';
import {
AuthService,
BackstageCredentials,
} from '@backstage/backend-plugin-api';
import { DefaultWorkspaceService, WorkspaceService } from './WorkspaceService';
import { WorkspaceProvider } from '@backstage/plugin-scaffolder-node/alpha';
import {
JsonObject,
JsonValue,
Observable,
createDeferred,
} from '@backstage/types';
import { Logger } from 'winston';
import ObservableImpl from 'zen-observable';
import { readDuration } from './helper';
import { InternalTaskSecrets, TaskStore } from './types';
import { DefaultWorkspaceService, WorkspaceService } from './WorkspaceService';
type TaskState = {
checkpoints: {
@@ -74,6 +75,7 @@ export class TaskManager implements TaskContext {
auth?: AuthService,
config?: Config,
additionalWorkspaceProviders?: Record<string, WorkspaceProvider>,
auditor?: AuditorService,
) {
const workspaceService = DefaultWorkspaceService.create(
task,
@@ -89,6 +91,7 @@ export class TaskManager implements TaskContext {
logger,
workspaceService,
auth,
auditor,
);
agent.startTimeout();
return agent;
@@ -102,6 +105,7 @@ export class TaskManager implements TaskContext {
private readonly logger: Logger,
private readonly workspaceService: WorkspaceService,
private readonly auth?: AuthService,
private readonly auditor?: AuditorService,
) {}
get spec() {
@@ -200,6 +204,25 @@ export class TaskManager implements TaskContext {
if (this.heartbeatTimeoutId) {
clearTimeout(this.heartbeatTimeoutId);
}
const auditorEvent = await this.auditor?.createEvent({
eventId: 'task-execution',
severityLevel: 'medium',
meta: {
taskId: this.task.taskId,
taskParameters: this.task.spec.parameters,
},
// The initial event is created in TaskWorker
suppressInitialEvent: true,
});
if (result === 'failed') {
await auditorEvent?.fail({
error: metadata?.error as any,
});
} else {
await auditorEvent?.success();
}
}
private startTimeout() {
@@ -275,6 +298,7 @@ export class StorageTaskBroker implements TaskBroker {
string,
WorkspaceProvider
>,
private readonly auditor?: AuditorService,
) {}
async list(options?: {
@@ -329,10 +353,7 @@ export class StorageTaskBroker implements TaskBroker {
public async recoverTasks(): Promise<void> {
const enabled =
(this.config &&
this.config.getOptionalBoolean(
'scaffolder.EXPERIMENTAL_recoverTasks',
)) ??
this.config?.getOptionalBoolean('scaffolder.EXPERIMENTAL_recoverTasks') ??
false;
if (enabled) {
@@ -374,6 +395,7 @@ export class StorageTaskBroker implements TaskBroker {
this.auth,
this.config,
this.additionalWorkspaceProviders,
this.auditor,
);
}
@@ -449,6 +471,13 @@ export class StorageTaskBroker implements TaskBroker {
const { tasks } = await this.storage.listStaleTasks(options);
await Promise.all(
tasks.map(async task => {
const auditorEvent = await this.auditor?.createEvent({
eventId: 'stale-task-cancellation',
severityLevel: 'medium',
meta: {
taskId: task.taskId,
},
});
try {
await this.storage.completeTask({
taskId: task.taskId,
@@ -458,8 +487,10 @@ export class StorageTaskBroker implements TaskBroker {
'The task was cancelled because the task worker lost connection to the task broker',
},
});
await auditorEvent?.success();
} catch (error) {
this.logger.warn(`Failed to cancel task '${task.taskId}', ${error}`);
await auditorEvent?.fail({ error: error });
}
}),
);
plugins/scaffolder-backend/src/scaffolder/tasks/TaskWorker.ts
+24 -6
View File
@@ -14,20 +14,21 @@
* limitations under the License.
*/
import { WorkflowRunner } from './types';
import { AuditorService } from '@backstage/backend-plugin-api';
import { assertError, stringifyError } from '@backstage/errors';
import { ScmIntegrations } from '@backstage/integration';
import { PermissionEvaluator } from '@backstage/plugin-permission-common';
import {
TaskContext,
TaskBroker,
TaskContext,
TemplateFilter,
TemplateGlobal,
} from '@backstage/plugin-scaffolder-node';
import PQueue from 'p-queue';
import { NunjucksWorkflowRunner } from './NunjucksWorkflowRunner';
import { Logger } from 'winston';
import { TemplateActionRegistry } from '../actions';
import { ScmIntegrations } from '@backstage/integration';
import { assertError, stringifyError } from '@backstage/errors';
import { PermissionEvaluator } from '@backstage/plugin-permission-common';
import { NunjucksWorkflowRunner } from './NunjucksWorkflowRunner';
import { WorkflowRunner } from './types';
/**
* TaskWorkerOptions
@@ -42,6 +43,7 @@ export type TaskWorkerOptions = {
concurrentTasksLimit: number;
permissions?: PermissionEvaluator;
logger?: Logger;
auditor?: AuditorService;
};
/**
@@ -55,6 +57,7 @@ export type CreateWorkerOptions = {
integrations: ScmIntegrations;
workingDirectory: string;
logger: Logger;
auditor?: AuditorService;
additionalTemplateFilters?: Record<string, TemplateFilter>;
/**
* The number of tasks that can be executed at the same time by the worker
@@ -81,11 +84,13 @@ export type CreateWorkerOptions = {
export class TaskWorker {
private taskQueue: PQueue;
private logger: Logger | undefined;
private auditor: AuditorService | undefined;
private stopWorkers: boolean;
private constructor(private readonly options: TaskWorkerOptions) {
this.stopWorkers = false;
this.logger = options.logger;
this.auditor = options.auditor;
this.taskQueue = new PQueue({
concurrency: options.concurrentTasksLimit,
});
@@ -95,6 +100,7 @@ export class TaskWorker {
const {
taskBroker,
logger,
auditor,
actionRegistry,
integrations,
workingDirectory,
@@ -108,6 +114,7 @@ export class TaskWorker {
actionRegistry,
integrations,
logger,
auditor,
workingDirectory,
additionalTemplateFilters,
additionalTemplateGlobals,
@@ -119,6 +126,7 @@ export class TaskWorker {
runners: { workflowRunner },
concurrentTasksLimit,
permissions,
auditor,
});
}
@@ -166,6 +174,16 @@ export class TaskWorker {
}
async runOneTask(task: TaskContext) {
await this.auditor?.createEvent({
eventId: 'task-execution',
severityLevel: 'medium',
meta: {
taskId: task.taskId,
taskParameters: task.spec.parameters,
templateRef: task.spec.templateInfo?.entityRef,
},
});
try {
if (task.spec.apiVersion !== 'scaffolder.backstage.io/v1beta3') {
throw new Error(
plugins/scaffolder-backend/src/service/router.ts
+590 -418
View File
File diff suppressed because it is too large Load Diff
plugins/scaffolder-node/report.api.md
+2
View File
@@ -405,6 +405,8 @@ export interface TaskContext {
// (undocumented)
spec: TaskSpec;
// (undocumented)
taskId?: string;
// (undocumented)
updateCheckpoint?(
options:
| {
plugins/scaffolder-node/src/tasks/types.ts
+1
View File
@@ -110,6 +110,7 @@ export type TaskBrokerDispatchOptions = {
* @public
*/
export interface TaskContext {
taskId?: string;
cancelSignal: AbortSignal;
spec: TaskSpec;
secrets?: TaskSecrets;