introduce AzureDevOpsEntityProvider

Signed-off-by: goenning <me@goenning.net>
This commit is contained in:
goenning
2022-05-18 22:20:45 +01:00
parent b983cde258
commit b8884fd579
15 changed files with 751 additions and 19 deletions
+62
View File
@@ -0,0 +1,62 @@
---
'@backstage/plugin-catalog-backend-module-azure': patch
---
Add a new provider `AzureDevOpsEntityProvider` as replacement for `AzureDevOpsDiscoveryProcessor`.
In order to migrate from the `AzureDevOpsDiscoveryProcessor` you need to apply
the following changes:
**Before:**
```yaml
# app-config.yaml
catalog:
locations:
- type: azure-discovery
target: https://dev.azure.com/myorg/myproject
```
```ts
/* packages/backend/src/plugins/catalog.ts */
import { AzureDevOpsDiscoveryProcessor } from '@backstage/plugin-catalog-backend-module-azure';
const builder = await CatalogBuilder.create(env);
/** ... other processors ... */
builder.addProcessor(new AzureDevOpsDiscoveryProcessor(env.reader));
```
**After:**
```yaml
# app-config.yaml
catalog:
providers:
azureDevOps:
anyProviderId:
organization: myorg
project: myproject
```
```ts
/* packages/backend/src/plugins/catalog.ts */
import { AzureDevOpsEntityProvider } from '@backstage/plugin-catalog-backend-module-azure';
const builder = await CatalogBuilder.create(env);
/** ... other processors and/or providers ... */
builder.addEntityProvider(
...AzureDevOpsEntityProvider.fromConfig(env.config, {
logger: env.logger,
schedule: env.scheduler.createScheduledTaskRunner({
frequency: Duration.fromObject({ minutes: 30 }),
timeout: Duration.fromObject({ minutes: 3 }),
}),
}),
);
```
Visit [https://backstage.io/docs/integrations/azure/discovery](https://backstage.io/docs/integrations/azure/discovery) for more details and options on configuration.
+60 -13
View File
@@ -6,25 +6,78 @@ sidebar_label: Discovery
description: Automatically discovering catalog entities from repositories in an Azure DevOps organization
---
The Azure DevOps integration has a special discovery processor for discovering
catalog entities within an Azure DevOps. The processor will crawl the Azure
The Azure DevOps integration has a special entity provider for discovering
catalog entities within an Azure DevOps. The provider will crawl your Azure
DevOps organization and register entities matching the configured path. This can
be useful as an alternative to static locations or manually adding things to the
catalog.
## Installation
You will have to add the processors in the catalog initialization code of your
backend. They are not installed by default, therefore you have to add a
dependency to `@backstage/plugin-catalog-backend-module-azure` to your backend
package.
At your configuration, you add one or more provider configs:
```yaml
# app-config.yaml
catalog:
providers:
azureDevOps:
yourProviderId: # identifies your dataset / provider independent of config changes
organization: myorg
project: myproject
repository: service-* # this will match all repos starting with service-*
path: /catalog-info.yaml
anotherProviderId: # another identifier
organization: myorg
project: myproject
repository: '*' # this will match all repos starting with service-*
path: /src/*/catalog-info.yaml # this will search for files deep inside the /src folder
yetAotherProviderId: # guess, what? Another one :)
host: selfhostedazure.yourcompany.com
organization: myorg
project: myproject
```
The parameters available are:
- `host:` Leave empty for Cloud hosted, otherwise set to your self-hosted instance host.
- `organization:` Your organization slug. Required.
- `project:` Your project slug. Required.
- `repository:` The repository name. Wildcards are supported as show on the examples above. If not set, all repositories will be searched.
- `path:` Where to find catalog-info.yaml files. Defaults to /catalog-info.yaml.
To use the entity provider, you'll need an Azure integration
[set up](locations.md) with `host` and `token`.
As this provider is not one of the default providers, you will first need to install
the Azure catalog plugin:
```bash
# From your Backstage root directory
yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-azure
```
And then add the processors to your catalog builder:
Once you've done that, you'll also need to add the segment below to `packages/backend/src/plugins/catalog.ts`:
```diff
/* packages/backend/src/plugins/catalog.ts */
+import { AzureDevOpsEntityProvider } from '@backstage/plugin-catalog-backend-module-azure';
const builder = await CatalogBuilder.create(env);
/** ... other processors and/or providers ... */
+builder.addEntityProvider(
+ ...AzureDevOpsEntityProvider.fromConfig(env.config, {
+ logger: env.logger,
+ schedule: env.scheduler.createScheduledTaskRunner({
+ frequency: Duration.fromObject({ minutes: 30 }),
+ timeout: Duration.fromObject({ minutes: 3 }),
+ }),
+ }),
+);
```
## Alternative Processor
As alternative to the entity provider `AzureDevOpsEntityProvider` you can still use the `AzureDevopsDiscoveryProcessor`.
```diff
// In packages/backend/src/plugins/catalog.ts
@@ -37,12 +90,6 @@ And then add the processors to your catalog builder:
+ builder.addProcessor(AzureDevOpsDiscoveryProcessor.fromConfig(env.config, { logger: env.logger }));
```
## Configuration
To use the discovery processor, you'll need a Azure integration
[set up](locations.md) with a `AZURE_TOKEN`. Then you can add a location target
to the catalog configuration:
```yaml
catalog:
locations:
@@ -6,9 +6,12 @@
import { CatalogProcessor } from '@backstage/plugin-catalog-backend';
import { CatalogProcessorEmit } from '@backstage/plugin-catalog-backend';
import { Config } from '@backstage/config';
import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { ScmIntegrationRegistry } from '@backstage/integration';
import { TaskRunner } from '@backstage/backend-tasks';
// @public
export class AzureDevOpsDiscoveryProcessor implements CatalogProcessor {
@@ -32,4 +35,22 @@ export class AzureDevOpsDiscoveryProcessor implements CatalogProcessor {
emit: CatalogProcessorEmit,
): Promise<boolean>;
}
// @public
export class AzureDevOpsEntityProvider implements EntityProvider {
// (undocumented)
connect(connection: EntityProviderConnection): Promise<void>;
// (undocumented)
static fromConfig(
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
},
): AzureDevOpsEntityProvider[];
// (undocumented)
getProviderName(): string;
// (undocumented)
refresh(logger: Logger): Promise<void>;
}
```
+59
View File
@@ -0,0 +1,59 @@
/*
* Copyright 2020 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
interface AzureDevOpsConfig {
/**
* (Optional) The DevOps host; leave empty for `dev.azure.com`, otherwise set to your self-hosted instance host.
* @visibility backend
*/
host: string;
/**
* (Required) Your organization slug.
* @visibility backend
*/
organization: string;
/**
* (Required) Your project slug.
* @visibility backend
*/
project: string;
/**
* (Optional) The repository name. Wildcards are supported as show on the examples above.
* If not set, all repositories will be searched.
* @visibility backend
*/
repository?: string;
/**
* (Optional) Where to find catalog-info.yaml files. Wildcards are supported.
* If not set, defaults to /catalog-info.yaml.
* @visibility backend
*/
path?: string;
}
export interface Config {
catalog?: {
/**
* List of provider-specific options and attributes
*/
providers?: {
/**
* AzureDevopsEntityProvider configuration
*/
azureDevOps?: Record<string, AzureDevOpsConfig>;
};
};
}
@@ -35,6 +35,7 @@
"dependencies": {
"@backstage/backend-common": "^0.13.6-next.1",
"@backstage/catalog-model": "^1.0.3-next.0",
"@backstage/backend-tasks": "^0.3.1",
"@backstage/config": "^1.0.1",
"@backstage/errors": "^1.0.0",
"@backstage/integration": "^1.2.1-next.1",
@@ -42,6 +43,7 @@
"@backstage/types": "^1.0.0",
"lodash": "^4.17.21",
"msw": "^0.42.0",
"uuid": "^8.0.0",
"node-fetch": "^2.6.7",
"winston": "^3.2.1"
},
@@ -51,6 +53,8 @@
"@types/lodash": "^4.14.151"
},
"files": [
"dist"
]
"dist",
"config.d.ts"
],
"configSchema": "config.d.ts"
}
@@ -20,4 +20,5 @@
* @packageDocumentation
*/
export { AzureDevOpsDiscoveryProcessor } from './AzureDevOpsDiscoveryProcessor';
export { AzureDevOpsDiscoveryProcessor } from './processors';
export { AzureDevOpsEntityProvider } from './providers';
@@ -21,9 +21,9 @@ import {
AzureDevOpsDiscoveryProcessor,
parseUrl,
} from './AzureDevOpsDiscoveryProcessor';
import { codeSearch } from './lib';
import { codeSearch } from '../lib';
jest.mock('./lib');
jest.mock('../lib');
const mockCodeSearch = codeSearch as jest.MockedFunction<typeof codeSearch>;
describe('AzureDevOpsDiscoveryProcessor', () => {
@@ -26,7 +26,7 @@ import {
processingResult,
} from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { codeSearch } from './lib';
import { codeSearch } from '../lib';
/**
* Extracts repositories out of an Azure DevOps org.
@@ -0,0 +1,17 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export { AzureDevOpsDiscoveryProcessor } from './AzureDevOpsDiscoveryProcessor';
@@ -0,0 +1,193 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getVoidLogger } from '@backstage/backend-common';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { CodeSearchResultItem } from '../lib';
import { AzureDevOpsEntityProvider } from './AzureDevOpsEntityProvider';
import { codeSearch } from '../lib';
jest.mock('../lib');
const mockCodeSearch = codeSearch as jest.MockedFunction<typeof codeSearch>;
class PersistingTaskRunner implements TaskRunner {
private tasks: TaskInvocationDefinition[] = [];
getTasks() {
return this.tasks;
}
run(task: TaskInvocationDefinition): Promise<void> {
this.tasks.push(task);
return Promise.resolve(undefined);
}
}
const logger = getVoidLogger();
describe('AzureDevOpsEntityProvider', () => {
afterEach(() => {
mockCodeSearch.mockClear();
});
const expectMutation = async (
providerId: string,
providerConfig: object,
codeSearchResults: CodeSearchResultItem[],
expectedBaseUrl: string,
names: Record<string, string>,
integrationConfig?: object,
) => {
const config = new ConfigReader({
integrations: {
azure: integrationConfig ? [integrationConfig] : [],
},
catalog: {
providers: {
azureDevOps: {
[providerId]: providerConfig,
},
},
},
});
mockCodeSearch.mockResolvedValueOnce(codeSearchResults);
const schedule = new PersistingTaskRunner();
const entityProviderConnection: EntityProviderConnection = {
applyMutation: jest.fn(),
};
const provider = AzureDevOpsEntityProvider.fromConfig(config, {
logger,
schedule,
})[0];
expect(provider.getProviderName()).toEqual(
`azureDevOps-provider:${providerId}`,
);
await provider.connect(entityProviderConnection);
const taskDef = schedule.getTasks()[0];
expect(taskDef.id).toEqual(`azureDevOps-provider:${providerId}:refresh`);
await (taskDef.fn as () => Promise<void>)();
const expectedEntities = codeSearchResults.map(item => {
const url = encodeURI(
`${expectedBaseUrl}/_git/${item.repository.name}?path=${item.path}`,
);
return {
entity: {
apiVersion: 'backstage.io/v1alpha1',
kind: 'Location',
metadata: {
annotations: {
'backstage.io/managed-by-location': `url:${url}`,
'backstage.io/managed-by-origin-location': `url:${url}`,
},
name: names[`${item.repository.name}?path=${item.path}`],
},
spec: {
presence: 'required',
target: `${url}`,
type: 'url',
},
},
locationKey: `azureDevOps-provider:${providerId}`,
};
});
expect(entityProviderConnection.applyMutation).toBeCalledWith({
type: 'full',
entities: expectedEntities,
});
};
// eslint-disable-next-line jest/expect-expect
it('no mutation when repos are empty', async () => {
return expectMutation(
'allRepos',
{
organization: 'myorganization',
project: 'myproject',
},
[],
'https://dev.azure.com/myorganization/myproject',
{},
);
});
// eslint-disable-next-line jest/expect-expect
it('single mutation when repos have 1 file found', async () => {
return expectMutation(
'allReposSingleFile',
{
organization: 'myorganization',
project: 'myproject',
},
[
{
fileName: 'catalog-info.yaml',
path: '/catalog-info.yaml',
repository: {
name: 'myrepo',
},
},
],
'https://dev.azure.com/myorganization/myproject',
{
'myrepo?path=/catalog-info.yaml':
'generated-87865246726bb12a8c4fb4f914443f1fbb91648c',
},
);
});
// eslint-disable-next-line jest/expect-expect
it('single mutation when multiple repos have multiple files', async () => {
return expectMutation(
'allReposMultipleFiles',
{
organization: 'myorganization',
project: 'myproject',
},
[
{
fileName: 'catalog-info.yaml',
path: '/catalog-info.yaml',
repository: {
name: 'myrepo',
},
},
{
fileName: 'catalog-info.yaml',
path: '/catalog-info.yaml',
repository: {
name: 'myotherrepo',
},
},
],
'https://dev.azure.com/myorganization/myproject',
{
'myrepo?path=/catalog-info.yaml':
'generated-87865246726bb12a8c4fb4f914443f1fbb91648c',
'myotherrepo?path=/catalog-info.yaml':
'generated-2deccac384c34d0dca37be0ebb4b1c8cf6913fe1',
},
);
});
});
@@ -0,0 +1,167 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { TaskRunner } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { AzureIntegration, ScmIntegrations } from '@backstage/integration';
import {
EntityProvider,
EntityProviderConnection,
LocationSpec,
locationSpecToLocationEntity,
} from '@backstage/plugin-catalog-backend';
import { readAzureDevOpsConfigs } from './config';
import { Logger } from 'winston';
import { AzureDevOpsConfig } from './types';
import * as uuid from 'uuid';
import { codeSearch, CodeSearchResultItem } from '../lib';
/**
* Provider which discovers catalog files within an Azure DevOps repositories.
*
* Use `AzureDevOpsEntityProvider.fromConfig(...)` to create instances.
*
* @public
*/
export class AzureDevOpsEntityProvider implements EntityProvider {
private readonly logger: Logger;
private readonly scheduleFn: () => Promise<void>;
private connection?: EntityProviderConnection;
static fromConfig(
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
},
): AzureDevOpsEntityProvider[] {
const providerConfigs = readAzureDevOpsConfigs(configRoot);
return providerConfigs.map(providerConfig => {
const integration = ScmIntegrations.fromConfig(configRoot).azure.byHost(
providerConfig.host,
);
if (!integration) {
throw new Error(
`There is no Azure integration for host ${providerConfig.host}. Please add a configuration entry for it under integrations.azure`,
);
}
return new AzureDevOpsEntityProvider(
providerConfig,
integration,
options.logger,
options.schedule,
);
});
}
private constructor(
private readonly config: AzureDevOpsConfig,
private readonly integration: AzureIntegration,
logger: Logger,
schedule: TaskRunner,
) {
this.logger = logger.child({
target: this.getProviderName(),
});
this.scheduleFn = this.createScheduleFn(schedule);
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
class: AzureDevOpsEntityProvider.prototype.constructor.name,
taskId,
taskInstanceId: uuid.v4(),
});
try {
await this.refresh(logger);
} catch (error) {
logger.error(error);
}
},
});
};
}
/** {@inheritdoc @backstage/plugin-catalog-backend#EntityProvider.getProviderName} */
getProviderName(): string {
return `azureDevOps-provider:${this.config.id}`;
}
/** {@inheritdoc @backstage/plugin-catalog-backend#EntityProvider.connect} */
async connect(connection: EntityProviderConnection): Promise<void> {
this.connection = connection;
await this.scheduleFn();
}
async refresh(logger: Logger) {
if (!this.connection) {
throw new Error('Not initialized');
}
logger.info('Discovering Azure DevOps catalog files');
const files = await codeSearch(
this.integration.config,
this.config.organization,
this.config.project,
this.config.repository,
this.config.path,
);
logger.info(`Discovered ${files.length} catalog files`);
const locations = files.map(key => this.createLocationSpec(key));
await this.connection.applyMutation({
type: 'full',
entities: locations.map(location => {
return {
locationKey: this.getProviderName(),
entity: locationSpecToLocationEntity({ location }),
};
}),
});
logger.info(
`Committed ${locations.length} locations for AzureDevOps catalog files`,
);
}
private createLocationSpec(file: CodeSearchResultItem): LocationSpec {
return {
type: 'url',
target: this.createObjectUrl(file),
presence: 'required',
};
}
private createObjectUrl(file: CodeSearchResultItem): string {
const baseUrl = `https://${this.config.host}/${this.config.organization}/${this.config.project}`;
return encodeURI(
`${baseUrl}/_git/${file.repository.name}?path=${file.path}`,
);
}
}
@@ -0,0 +1,67 @@
/*
* Copyright 2020 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { ConfigReader } from '@backstage/config';
import { readAzureDevOpsConfigs } from './config';
describe('readAzureDevOpsConfigs', () => {
it('reads all provider configs and set default values', () => {
const provider1 = {
host: 'azure.mycompany.com',
organization: 'mycompany',
project: 'myproject',
};
const provider2 = {
organization: 'mycompany',
project: 'myproject',
};
const provider3 = {
organization: 'mycompany',
project: 'myproject',
repository: 'service-*',
};
const config = {
catalog: {
providers: {
azureDevOps: { provider1, provider2, provider3 },
},
},
};
const actual = readAzureDevOpsConfigs(new ConfigReader(config));
expect(actual).toHaveLength(3);
expect(actual[0]).toEqual({
...provider1,
path: '/catalog-info.yaml',
repository: '*',
id: 'provider1',
});
expect(actual[1]).toEqual({
...provider2,
host: 'dev.azure.com',
path: '/catalog-info.yaml',
repository: '*',
id: 'provider2',
});
expect(actual[2]).toEqual({
...provider3,
host: 'dev.azure.com',
path: '/catalog-info.yaml',
id: 'provider3',
});
});
});
@@ -0,0 +1,53 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Config } from '@backstage/config';
import { AzureDevOpsConfig } from './types';
export function readAzureDevOpsConfigs(config: Config): AzureDevOpsConfig[] {
const configs: AzureDevOpsConfig[] = [];
const providerConfigs = config.getOptionalConfig(
'catalog.providers.azureDevOps',
);
if (!providerConfigs) {
return configs;
}
for (const id of providerConfigs.keys()) {
configs.push(readAzureDevOpsConfig(id, providerConfigs.getConfig(id)));
}
return configs;
}
function readAzureDevOpsConfig(id: string, config: Config): AzureDevOpsConfig {
const organization = config.getString('organization');
const project = config.getString('project');
const host = config.getOptionalString('host') || 'dev.azure.com';
const repository = config.getOptionalString('repository') || '*';
const path = config.getOptionalString('path') || '/catalog-info.yaml';
return {
id,
host,
organization,
project,
repository,
path,
};
}
@@ -0,0 +1,17 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export { AzureDevOpsEntityProvider } from './AzureDevOpsEntityProvider';
@@ -0,0 +1,24 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export type AzureDevOpsConfig = {
id: string;
host: string;
organization: string;
project: string;
repository: string;
path: string;
};