backstage/beps/0002-dynamic-frontend-plugins

title, status, authors, owners, project-areas, creation-date

title	status	authors	owners	project-areas	creation-date
Dynamic Frontend Plugins	provisional	@Hyperkid123		core	2024-01-17

BEP:

Discussion Issue

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
• Design Details
• Release Plan
• Dependencies
• Alternatives

Summary

The dynamic frontend plugins feature is a way of loading additional frontend plugins at runtime, without the requirement of rebuilding and restarting a running backstage instance.

This system should significantly improve frontend plugin management for backstage instances.

The dynamic plugins leverage the declarative UI system to define what a plugin is and how it should be represented in the browser.

Motivation

Goals

• discover and choose tooling to enable dynamic frontend plugins
• easy way of converting existing frontend plugins to be dynamic
• turning on and off UI plugins at runtime without the requirement of rebuilding or restarting backstage
• manage dynamic plugins declaratively

An additional goal might be exploring the scope of a "on demand" rending and loading. Currently all plugins, or at least some portion of them must be loaded at UI bootstrap. This approach is not fully dynamic and might be considered a waste of resources. Fully optimizing dynamic plugins loading would require additional changes to the new UI system.

Non-Goals

Proposal

Definition of UI dynamic plugin

A dynamic UI plugin (from now just plugin) is a plugin that is not part of the output of a backstage instance build. The plugin and its assets are injected into backstage at runtime. In this case, its injected into the browser at some point during user session.

From the user POV, there is no difference between classic and dynamic plugins.

The difference is known only to maintainers and should be limited to

• build requirements
• integration into backstage

Dynamic loading tool

Traditional plugins are not dynamic out of the box. Additional changes are required during build time to make a plugin dynamic.

The main application (shell) also requires some changes to be able to load, inject, and propagate mandatory context to dynamic plugins.

Over the last few year, Module Federation has become the standard when it comes to dynamically load JS modules browser (and nodejs) environment from remote locations and sharing context between the shell and the remote module.

Although there are other options, like externalizing dependencies, the module federation has proven itself as a robust solution to this particular problem.

Module federation implementation

There are multiple available implementations of module federation.

Historically, module federation was implemented as a Webpack feature. Since then, additional implementations were created. Mainly for rspack and Vite.

Recent changes claim, that all of these module federation implementations should be compatible with each other and there should be no need for locking backstage into a single implementation.

Compatibility between Webpack and rspack should available via the @module-federation/enhanced package. The compatibility is further described here.

The Vite plugin claims webpack compatibility as well. The Vite plugin is not part of the @module-federation organization. The extend of compatibility is at this time unknown.

The level of compatibility has to be tested. Before then, no decision in regards to which implementation(s) will be used should be made. The chosen tool, or their mix, will shape the design and implementation.

That said, packages from the @module-federation/* organization should have higher priority as they are based directly on the module federation concepts and are well supported.

Plugin integration into shell applications

Plugins should be integrated via the new UI system. The system already provides an asynchronous way of loading plugins. The dynamic plugins can be loaded in a similar fashion.

Plugins should be defined declaratively through configuration. Similar to what was described in this RFC and in this issue.

Plugin registry

Because plugins are not available at build time, some sort of registry needs to exist to store the information.

This registry needs to be mutable at runtime (add/remove new plugin metadata) and changes have to be reflected on session refresh.

This is currently an issue as the app config is embedded into JS assets during build time.

Each plugin is required to provide manifest file (metadata) in predefined format. This manifest will be used to inject the plugin assets into the browser.

Plugin discovery

Plugin discovery is a pre-requisite for Plugin registry. This should be responsible for scanning for available plugins and generating/modifying the plugin registry to always keep it up to date.

Design Details

NOTE The details are based on the Janus implementation of dynamic frontend plugins. The implementation leverages Scalprum which is a Webpack based dynamic plugin manager for React applications.

Module federation implementation experiments

Test should consist of trying to run permutations of webpack/Rspack/vite based shell apps/plugins and discover if we can freely choose any tool, or if we should restrict the tooling to just a subset of the available options.

The outcome of initial testing is positive and it is possible to mix and match different build tools and consume different remote modules in a single shell application.

The experimental code can be found in this repository.

The testing so far was done only on very simple modules. Although core React features are working (Context API and hooks), more testings needs to be done in order to declare this approach 100% reliable.

So far a lot of custom code needs to be written to bridge Webpack, Rspack, @module-federation/enhanced with Vite. The first three are compatible out of the box, but Vite requires extra bridge to be able to consume/provide modules with/to other builds.

Plugin manifest

Each plugin should have a manifest file with important metadata. This metadata is used to load the remote assets to browser. The plugin manifest should be part of a build output.

A manifest should have:

• name of plugin
• how can be the init container accessed
• what is the base URL (assets pathname)
• name of the entry script(s)

Scalprum compatible manifest

type PluginManifest = {
  name: string;
  version: string;
  dependencies?: Record<string, string>;
  customProperties?: AnyObject;
  baseURL: string;
  extensions: Extension[];
  loadScripts: string[];
  registrationMethod: 'callback' | 'custom';
  buildHash?: string;
}

{
  "name": "backstage.plugin-github-actions",
  "version": "0.6.6",
  "extensions": [],
  "registrationMethod": "callback", // where container init is available  in browser
  "baseURL": "auto",
  "loadScripts": [
    "backstage.plugin-github-actions.804b91040fcbca6585ce.js"
  ],
  "buildHash": "804b91040fcbca6585ce1bcd4b1f8aa2"
}

registrationMethod

Refers to webpack output.libraryTarget.

Callback refers to jsonp and a the custom is used if other available target configuration has been picked.

It is recommended to use either global or jsonp as these are environment agnostic (browser VS node). jsonp requires additional configurations. The global is preferable due to its simplicity.

baseURL

The baseURL is derived from webpack public path

The public path can also be set to auto to remove the need to specify origin or pathname and resolve the pathname at runtime.

In Scalprum, some manifest post processing is required to load the initial scripts if the auto baseURL is chosen.

Plugin registry

Plugin registry can be fairly simplistic. It can be as simple as JSON file containing list/map of available plugins and their manifests

type RegistryEntry = {
  name: string // plugin name
  manifestLocation: string // path to the manifest resource
}

// object for easy access
type PluginRegistry = {
  [pluginName: string]: RegistryEntry
}

// or as an array
type PluginRegistry = RegistryEntry[]

Example of such registry

// as object
{
  "backstage.plugin-github-actions": {
    "name": "backstage.plugin-github-actions",
    "manifestLocation": "https://foo-bar.com/api/plugin-storage/plugin-manifest.json"
  },
  // ..rest of plugins
}

// as array
[
  {
    "name": "backstage.plugin-github-actions",
    "manifestLocation": "https://foo-bar.com/api/plugin-storage/plugin-manifest.json"
  },
  // ...rest of plugins
]

Scalprum by default lazy loads plugins and manifests. That is because Scalprum initializes plugins only once they are supposed to be rendered in browser.

Because backstage does not require that functionality in initial dynamic plugin implementation and loads all plugins at bootstrap, there is an alternative to embed manifest data into the registry itself.

type RegistryEntry = {
  name: string // plugin name
  pluginManifest: PluginManifest
}

// object for easy access
type PluginRegistry = {
  [pluginName: string]: RegistryEntry
}

// or as an array
type PluginRegistry = RegistryEntry[]

Example of registry with embedded manifests

// as object
{
  "backstage.plugin-github-actions": {
    "name": "backstage.plugin-github-actions",
    "pluginManifest": {
      "name": "backstage.plugin-github-actions",
      "version": "0.6.6",
      "extensions": [],
      "registrationMethod": "callback",
      "baseURL": "auto",
      "loadScripts": [
        "backstage.plugin-github-actions.804b91040fcbca6585ce.js"
      ],
      "buildHash": "804b91040fcbca6585ce1bcd4b1f8aa2"
    }
  },
  // ..rest of plugins
}

// as array
[
  {
    "name": "backstage.plugin-github-actions",
    "pluginManifest": {
      "name": "backstage.plugin-github-actions",
      "version": "0.6.6",
      "extensions": [],
      "registrationMethod": "callback",
      "baseURL": "auto",
      "loadScripts": [
        "backstage.plugin-github-actions.804b91040fcbca6585ce.js"
      ],
      "buildHash": "804b91040fcbca6585ce1bcd4b1f8aa2"
    }
  },
  // ...rest of plugins
]

Webpack build configuration

NOTE This is a sample current configuration in the Janus project. It uses Scalprum webpack based build plugin to generate the output. It does not take the all options in consideration. This section will likely change considerably.

Part of a Scalprum tooling is also dynamic-plugins-sdk. Right now the package is a part of a different project, but that is about to change. More details in the Scalprum roadmap.

Sample plugin configuration using Scalprum SDK

import { DynamicRemotePlugin } from '@openshift/dynamic-plugin-sdk-webpack';

const sharedModules = {
  /**
   * Mandatory singleton packages for sharing
   */
  react: {
    singleton: true,
    requiredVersion: '*',
  },
  'react-dom': {
    singleton: true,
    requiredVersion: '*',
  },
  'react-router-dom': {
    singleton: true,
    requiredVersion: '*',
  },
  'react-router': {
    singleton: true,
    requiredVersion: '*',
  },
  ...
  /**
   * Full list of shared modules in Janus
   * https://github.com/janus-idp/backstage-plugins/blob/87a6b045c7b0f301ebed8b8f99dc1741fa2b044b/packages/cli/src/lib/bundler/scalprumConfig.ts#L16
  */

}

const dynamicPluginPlugin = new DynamicRemotePlugin({
  extensions: [],
  sharedModules,
  entryScriptFilename: `${options.pluginMetadata.name}.[contenthash].js`,
  pluginMetadata: {
    // version cna be used from the package.json version field
    version: '1.0.0',
    /**
     * Name can be easily derived from the plugin name
     * https://github.com/janus-idp/backstage-plugins/blob/87a6b045c7b0f301ebed8b8f99dc1741fa2b044b/packages/cli/src/commands/export-dynamic-plugin/frontend.ts#L40
    */
    name: 'backstage.plugin-github-actions',
    /**
     * Path to the plugin entry point.
     * It can default to the same entry point as in regular build.
     * Plugins can expose multiple modules. We have found that one is sufficient from backstage plugins.
    */
    exposedModules: {
      PluginRoot: './src/index.ts'
    }
  },
});

The DynamicRemotePlugin webpack plugin takes care of the rest, including the manifest generation.

Sample plugin raw webpack configuration

The Scalprum config translates to a following base webpack plugin configuration:

import { container } from 'webpack';

const dynamicPlugin = new container.ModuleFederationPlugin({
  name: 'backstage.plugin-github-actions',
  library: {
    type: 'global',
    /**
     * Some library.type has name limitation
     * for example, if "type": "var" is used, the library.name can contain the "-" character
    */
    name: 'backstage.plugin-github-actions'
  },
  filename: 'backstage.plugin-github-actions.[contenthash].js',
  exposes: {
    PluginRoot: './src/index.ts'
  },
  // list of shared modules like "react", "react-dom", etc
  shared: sharedModules,
});

Additional plugin would have to be written from scratch to generate the plugin manifest file.

Sample shell application webpack configuration

Shell application (backstage) has very simple module federation configuration. It does not require anything special to be able to inject plugins. Main requirement is to provide core shared packages like react and react-dom.

import { container } from 'webpack';

const scalprumPlugin = new container.ModuleFederationPlugin({
  name: 'backstageHost',
  filename: 'backstageHost.[contenthash].js',
  // same share modules list as with plugin config
  shared: [sharedModules],
});

Nothing else is required.

Plugin storage

Where will the dynamic plugin assets be hosted? Module federation does not strictly require the remote assets to be all hosted on the same origin.

Theoretically plugins can be hosted on some "public CDN" which is detached from individual backstage instances.

Assets can be also be hosted in the same way as they have always been.

Plugin discovery

How to notify/send data to browser

Plugin initialization

Currently all plugins have to be initialized at UI bootstrap (page refresh). The new UI async API can be used to initialize the remote assets.

The CreateAppFeatureLoader can be leveraged to initialize the remote container.

const allPluginManifests = {...} // get this from the config

const asyncFeatureLoaders: CreateAppFeatureLoader[] = Object.values(allPluginManifests).map(({ manifest }) => {
  return {
      getLoaderName: () => manifest.name,
      load: (options) => {
        // initialize the remote container, depends on tooling
        const plugin = initDynamicPlugin(manifest)
      }
  }
})

const app = createApp({
  features: [
    ...asyncFeatureLoaders,
    // rest of classic features
  ]
})

Plugin declarative configuration

Janus dynamic plugins ref: https://github.com/janus-idp/backstage-showcase/blob/main/showcase-docs%2Fdynamic-plugins.md#frontend-layout-configuration

TBD, depends heavily on the new UI system

Release Plan

Dependencies

Alternatives