backstage/beps/0005-split-backend-discovery

title, status, authors, owners, project-areas, creation-date

title	status	authors	owners	project-areas	creation-date
Discovery API - Split Backends	provisional	@aramissennyeydd	@aramissennyeydd	core	2024-02-16

BEP: Discovery API - Split Backends

Discussion Issue

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
• Design Details
• Release Plan
• Dependencies
• Alternatives

Summary

The goal of this BEP is to define the architecture that we will be using for an automatic discovery API that handles split backends. While users can use the current target-based config, it is not runtime driven and adding plugins requires a config update. This new system allows existing backends to register with gateway nodes at start time, allows them to unregister before the program exits and catches system errors by timing out plugin registrations that haven't been refreshed recently.

Motivation

Split backends are a consistently difficult space to operate in and design for. There has been a growing desire for the framework to provide a way to get a list of the installed plugins. This was nearly impossible in the old backend, where plugins were hosted on denormalized routes and had non-standard startup sequences. In the new backend, this has become significantly more doable. Moving this forward would unblock a number of cases that require knowledge of your entire Backstage installation, namely a single OpenAPI spec for your instance, checking installed permissions, and DevTools information.

Ideally, this work will also make it easier for adopters to go down the path of split backends.

Goals

1. As an integrator, I can now get a list of currently installed plugins across my deployment.
2. As an administrator, I can use the default discovery API across both frontend and backend for complex situations like split backends, without writing my own.
3. As an administrator, I can add/remove plugins without having to do a full redeployment of all of my Backstage nodes.
4. A

Non-Goals

1. I can't install plugins using the new discovery API.
2. I can't control my deployments using the new discovery API.
3. If I fork the BackendInitialization logic, I may not be able to use this API.

Proposal

Design Details

Glossary

Gateway Node

A node that serves as the primary discovery point. A gateway node will have all of the information necessary to route traffic through your system.

Non-gateway Node

A node that needs to call a Gateway node for routing.

Backend Requests

Backend DiscoveryApi requests will either

1. route to their own instance and can use the existing HostDiscovery implementation, or
2. need to route to a separate instance and will have to go to the gateway node for routing information.

Registration Flow

It is imperative that bad actors not be able to register into this new discovery implementation. For the initial launch, we propose using the service-to-service auth system and sharing keys between backends. This is the current recommended approach to protect this endpoint. In the future, we may revisit this.

New InstanceMetadataService

While we could attach the existing information to the PluginMetadataService, we propose a new service that handles instance-level information. The existing PluginMetadataService should reveal information about the plugin itself, its pluginId, dependencies or similar. The new InstanceMetadataService should give you information about the entire Backstage instance that you're interrogating. At launch, this should include the list of features installed on your instance that can then be aggregated by the discovery API across the gateway nodes. One could imagine this service also having information about instance URLs, health or gateway status.

interface InstanceMetadataService {
  listFeatures: () => BackendFeature[];
  // or
  listFeatures: () => string[]; // list of pluginIds/moduleIds.
}

Frontend Requests

This will leverage the existing DiscoveryService. We propose adding a new method, listPlugins that will return a list of all plugins installed in your Backstage deployment.

interface DiscoveryService {
  ...
  listPlugins: () => Promise<string[]>;
}

All methods will just call the gateway node's HTTP discovery endpoint for the data, see diagram for more information.

Gateway Scaling

The primary concern with having multiple gateway nodes is alignment on what plugins are installed across the instance. For this, we propose a new database that will store,

export interface PluginRegistrations {
  plugin_id: string;
  internal_url: string;
  external_url: string;
  last_check_in: timestamp;
}

As any gateway node could be hit by any given plugin, the implementation should not rely on in-memory values per node. Gateway nodes should read and write from/to the database directly.

The triplet plugin_id, internal_url, external_url should be unique. We may have multiple plugins on multiple URLs either internal or external. Routing in those cases is not covered in this BEP. Horizontally scaled plugins should use external technologies to route requests and handle load balancing. This should be reflected in their backend.baseUrl properties. Instance IP addresses should not be sent or stored in this database.

Check ins

To prevent stale data, we propose implementing "check ins". Each check in will write to the database's last_check_in row, these will be infrequent enough that this isn't a crazy load on the database. These check ins serve 2 purposes,

1. to verify that the gateway node is reachable by the non-gateway node.
2. to verify that the gateway node has the most up to date data from the non-gateway node.

The first is necessary to prevent a case where the non-gateway node loses network access or crashes. In this case, we will use the last_check_in timestamp. If an instance hasn't contacted the gateway plugin in x check-ins (or seconds), we remove that instance's plugins. This will most likely be an optimistic update and will happen at read time for other operations. So the database may still have entries with expired last_check_in rows if the discovery API is not actively used.

The second is necessary to prevent a case where an instance may restart with more/less plugins. The gateway needs a way of knowing that the instance data changed. This is why we propose a "check in" approach over just a heartbeat. The non-gateway plugin should send its list of plugins to the gateway plugin to check if anything has changed since it last checked in. This allows us to be much more specific about when the instance will re-register.

In the case of horizontally scaled plugins, we should be able to keep the current check-ins, but we may revisit if the volume is too high.

Unregistering Plugins

While some service discovery implementation have the ability for plugins to unregister themselves on shutdown or error, this proves difficult for horizontally scaled deployments. As such, we propose using the check ins described above to verify that a plugin is operational.

Release Plan

Dependencies

Alternatives