Replace the window-function full-table scan with a two-phase approach
that leverages search_key_value_entity_idx (key, value, entity_id):
Phase 1: GROUP BY entity_id, key, value with HAVING COUNT(*) > 1
resolves as a pure index-only scan (Heap Fetches: 0).
Stores only the duplicate groups in a temp table (~8s for
a 14M-row table with 700k dupes).
Phase 2: CROSS JOIN LATERAL back into search using the same covering
index (Nested Loop + Index Scan). row_number() runs per-group
over the 2-3 matching rows, so there is no global external
sort. A single DELETE removes all extras in one statement
(~16s).
NULL values get a separate UNION ALL arm so the index equality
condition stays usable (value = NULL is always false in SQL).
Benchmarked on a 14.4M-row production-like staging master with
700k injected duplicates, post-VACUUM:
Old (seq scan + external merge sort): ~101s
New (index-only + index scan): ~25s (~4× faster)
Clean second run (no dupes, fast path skips dedup): <50ms.
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Use `=== null ? '\x01' : value` instead of `?? ''` in both buildEntitySearch
and syncSearchRows dedup maps, so that null and empty-string values are
treated as distinct keys. In theory an entity could produce both value=null
and value='' for the same key (e.g. spec.foo: [null, '']), and the old
encoding would silently drop one of the two distinct rows.
Also adds two focused unit tests to buildEntitySearch.test.ts: one covering
deduplication of duplicate array values (e.g. tags: ['java', 'java', 'Java']),
and one confirming that null and empty-string are kept as separate rows.
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The original single-pass CTE window-function DELETE re-scanned every row
on each invocation — including the idempotent "nothing to do" case which
took over 4 minutes on a 14 M-row table, and timed out the DB proxy on
the dirty case (~217 s for 722 k duplicates).
Replace it with two improvements:
1. Fast path: if the UNIQUE index already exists and is valid, skip dedup
entirely. A valid unique index is a proof-of-no-duplicates. This makes
migration startup essentially free for installations that prepare the
index manually beforehand (as documented in the migration comment).
2. Two-phase dedup when dedup is needed: Phase 1 does one full-table scan
to collect all duplicate ctids into a temp table (~60 s on 14 M rows).
Phase 2 drains that temp table in 10 k-row batches via cheap ctid
lookups with no further full-table scans (~5 s for 722 k rows). Total
~65 s dirty vs >217 s+ before, and ~35 ms clean vs 4+ minutes before.
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Prevents a false match if any non-index object (table, sequence, view)
shares a name with one of the search indices.
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Verifies that a conflicting insert with a different original_value casing
updates the stored value, and documents that DO UPDATE requires explicit
conflict columns unlike DO NOTHING.
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the explicit column list from .onConflict() calls in test setup —
ON CONFLICT DO NOTHING without a target is equivalent since (entity_id,
key, value) is the only unique constraint on the search table.
Add a dedicated test in syncSearchRows that directly inserts a duplicate
row and verifies it is silently rejected via the UNIQUE constraint.
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Avoids a window with no coverage on (key, value) during rollback.
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Be explicit that interrupted index builds do not leave partial progress —
each retry starts from scratch. On large tables with short liveness
probe timeouts this repeats indefinitely. Recommend running the SQL
commands manually before deploying.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
The down migration now restores the previous index state (drops new
indices, recreates the old search_key_value_idx and
search_key_original_value_idx). Updated the SQL report to reflect the
new index set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Single knex migration that cleans up duplicate search rows and creates
covering indices including a UNIQUE constraint on (entity_id, key, value).
Each step is idempotent and handles INVALID indices from interrupted runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Hold a shared promise rather than just a resolved value. Concurrent
callers awaiting a fresh count get the same in-flight promise back, so
the underlying query is never overlapped by a duplicate. The TTL is now
the minimum gap between the resolution of one query and the start of
the next, rather than a hard bound on cache age.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
The legacy Prometheus and OpenTelemetry observable gauges previously each
ran the per-kind count query against the search table on every metrics
scrape. With multiple pods and short scrape intervals, identical
sequential scans piled up faster than they completed, contending for
buffers in the database.
Extract a shared helper that wraps a 30-second TTL cache around a single
query, and have both gauges read from it. The query itself moves from
the (large) search table to final_entities, parsing kind out of
entity_ref via per-engine substring functions. The emitted labels and
values are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Replace the `WHERE search.entity_id IN (...)` form with an INNER JOIN
against the filtered final_entities subquery in DefaultEntitiesCatalog#facets.
Results are unchanged; the planner gets more freedom to pick cheaper plans,
which on large catalogs leads to substantial speedups (1.2× to 7×+ in
adversarial testing on a ~13.8M-row search table) and avoids the
materialize-then-spill pattern of the IN form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Use a word-boundary regex (:name\b) instead of a plain string replace
so that a shorter param like :a doesn't corrupt a longer param :ab
when both are present in the route.
Signed-off-by: Patrik Oldsberg <rugvip@backstage.io>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The app/routes redirect config now performs the same :param and *
substitution that the legacy Redirect component did before navigating.
Named params captured by the `from` pattern are replaced in the `to`
string, enabling redirects like /users/:userId → /profile/:userId and
/old-docs → /docs/* (with splat forwarding).
Adds tests for both named-param and splat substitution.
Signed-off-by: Patrik Oldsberg <rugvip@backstage.io>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Net-new package does not need alpha exports per awanlin's review.
Revert alpha.ts to export {}, remove default alias from module/index.ts,
and restore empty report-alpha.api.md.
Signed-off-by: pillaris <pillaris@adobe.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: Add status check functions for scaffolder steps
- Introduced `always()` and `failure()` functions to control step execution after failures.
- Updated documentation to explain usage of new status check functions.
- Enhanced NunjucksWorkflowRunner to process these functions in step conditions.
- Added tests to verify behavior of steps using `always()` and `failure()`.
Signed-off-by: ferin79 <ferinpatel79@gmail.com>
* feat: Enhance status check functions in scaffolder steps
- Updated documentation to clarify usage of status check functions with template expressions.
- Modified tests to reflect changes in syntax for status checks.
- Refactored NunjucksWorkflowRunner to ensure proper handling of status check functions in step conditions.
Signed-off-by: ferin79 <ferinpatel79@gmail.com>
* docs: Clarify usage of status check functions in writing templates
- Removed redundant explanation about truthy conditions after step failure.
- Streamlined the description for better clarity on status check functions.
Signed-off-by: ferin79 <ferinpatel79@gmail.com>
---------
Signed-off-by: ferin79 <ferinpatel79@gmail.com>
Re-export catalogModuleMicrosoftGraphIncrementalEntityProvider as @alpha
default from src/alpha.ts, matching the pattern of catalog-backend-module-msgraph.
Add default export alias in module/index.ts and regenerate report-alpha.api.md.
Signed-off-by: pillaris <pillaris@adobe.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The manually-written empty report-alpha.api.md didn't match what API
Extractor generates for an empty alpha entry point. Use the correct
generated output to pass the CI api-reports check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: pillaris <pillaris@adobe.com>
- Update copyright year from 2024 to 2026 across all new source files
- Clear alpha entry point (no alpha exports needed for a net-new package)
and regenerate report-alpha.api.md accordingly
- Delete CHANGELOG.md (auto-generated by the release process)
- Change changeset bump from patch to minor so the first release is 0.1.0,
and remove the redundant "New package:" title line
- Reset package version to 0.0.0 (release process sets the real version)
- Add incremental ingestion section to docs/integrations/azure/org.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: pillaris <pillaris@adobe.com>
A throwing custom groupTransformer on a child group member would fail
the entire groups-phase burst. Apply the same try/catch + debug/warn
logging pattern already used for user member transformation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: pillaris <pillaris@adobe.com>
- Guard getUserPhotoGated with user.id presence to prevent requesting
users/undefined/photo when Graph omits the id field
- Log a warning when groupIncludeSubGroups is configured, matching the
existing warning for unsupported userGroupMember* options
- Skip spec.children population when groupFilter/groupSearch is active,
since child groups may not pass the filter and would create dangling
references in the catalog
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: pillaris <pillaris@adobe.com>
Update all references to "up to 999 items" to accurately reflect that
users are fetched in pages of 999 while groups use a smaller page of
100. Also document groupIncludeSubGroups as unsupported in the JSDoc
@remarks, README, and comparison table.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: pillaris <pillaris@adobe.com>
Fredrik Adelöw