Skip to content

[critical] fix: four generators remint every uuid on regeneration (uuid4 / index-seeded uuid5) - #1279

Open
elhoim wants to merge 1 commit into
MISP:mainfrom
elhoim:fix/generator-uuid-stability
Open

elhoim wants to merge 1 commit into
MISP:mainfrom
elhoim:fix/generator-uuid-stability

Conversation

@elhoim

@elhoim elhoim commented Aug 30, 2026

Copy link
Copy Markdown
Member

BLUF — Four generators remint every uuid on regeneration, orphaning every inbound related edge

  • Problem — A cluster value's uuid is its permanent identity — MISP instances key on it and every related edge points at it — yet gen_malpedia.py and gen_amitt.py mint uuid4 on every run, plot4ai's generator seeds uuid5 with the enumerate() index, and gen_microsoft_activity_group.py seeds it with the current display name. Regenerating replaces rather than updates entries (3,683 malpedia families, 61 AM!TT entries).
  • Fix — Add a shared tools/galaxy_uuid.py UuidAssigner that derives uuid5 from stable seeds, indexes synonyms so renames keep the old id, and reuses whatever uuid is already committed.
  • Effect — Regeneration updates entries in place and inbound references survive.

Problem

A cluster value's uuid is its permanent identity — MISP instances key on it, and every related edge in every other galaxy points at it by uuid. Four generators mint uuids that do not survive a regeneration:

generator what it did consequence
gen_malpedia.py:38,62 uuid.uuid4() for the cluster and for every malware family regenerating replaces all 3,683 entries with new ids
gen_amitt.py:46,73,94,120,141 uuid.uuid4() throughout same, for all 61 entries
plot4ai/generate_plot4ai_galaxy.py:200 uuid5 seeded with the enumerate() index inserting or reordering one card remints every uuid after it
gen_microsoft_activity_group.py:173 uuid5 over the current display name a Microsoft rename mints a new uuid and orphans the old entry

In each case "regenerating the cluster" does not update entries, it replaces them — and silently breaks every inbound reference.

Demonstrating the plot4ai index dependency:

card 'X' at position 5 : c7882688-6c87-52de-bbb4-245e3c9da250
same card at position 6: 733b77ee-a696-5c2e-bf55-b2214f29770c   <- remints

Fix

A shared tools/galaxy_uuid.py with a UuidAssigner that applies two rules:

  1. Never uuid4, never seed on a position. Derive with uuid5 from something stable — the entry's name, or its upstream framework id (external_id for AM!TT).
  2. Prefer the uuid already committed. Each generator reads the cluster it is about to overwrite and reuses what is there, so even a future revision of the seed cannot cause churn.

Synonyms are indexed too, so an upstream rename that demotes the old name to a synonym keeps the original uuid:

entry 'Amethyst Rain'
  committed uuid            : 043ebe48-3101-5f67-9dd0-de3c9f230031
  after upstream renames it : 043ebe48-3101-5f67-9dd0-de3c9f230031   <- preserved
  old derivation would give : 342b292a-ca80-5b23-a730-314a98e98104   <- orphaned

gen_malpedia.py also got two reproducibility fixes on the same code path: os.walk's filenames are now sorted, and list(set(...)) for refs/synonyms became sorted(set(...)) — set iteration order varies between runs, so the file churned even when the data had not changed.

Verification

These generators need upstream inputs that are not available offline (a cloned Malpedia repository, an AM!TT spreadsheet, network fetches). So rather than running them, the committed values are fed through the exact for_value(...) call each generator now makes, and checked against the uuid already in the file:

generator / cluster                      values  preserved    CHANGED
gen_malpedia.py                            3683       3683          0
gen_amitt.py                                 61         61          0
plot4ai                                     138        138          0
gen_microsoft_activity_group.py             179        179          0

total committed uuids that would change: 0

Each cluster's own top-level uuid is asserted preserved too. All five files compile.

No committed uuid changes. The behavioural change is only for entries that do not exist yet, and for regenerations that previously churned.

🤖 Generated with Claude Code

…ference

A cluster value's uuid is its permanent identity: MISP keys on it and every
`related` edge across the repo points at it. Four generators produced uuids
that do not survive regeneration:

  gen_malpedia.py:38,62      uuid4() for the cluster and all 3683 families
  gen_amitt.py:46,73,94,...  uuid4() throughout
  plot4ai:200                uuid5 seeded with the enumerate() index, so
                             reordering one card reminted every later uuid
  gen_microsoft_activity_group.py:173
                             uuid5 over the display name, so a Microsoft
                             rename orphaned the old entry

Add tools/galaxy_uuid.py with a UuidAssigner that (1) derives with uuid5
from a stable key -- name or upstream id, never a position -- and (2)
prefers the uuid already committed in the cluster being overwritten.
Synonyms are indexed as well, so a rename that demotes the old name to a
synonym keeps the original uuid.

gen_malpedia.py additionally sorts os.walk filenames and replaces
list(set(...)) with sorted(set(...)) for refs/synonyms, so the output stops
churning on set iteration order.

Verified by feeding the committed values through the exact for_value() call
each generator now makes (they need upstream inputs unavailable offline):

  gen_malpedia.py                   3683 values, 3683 preserved, 0 changed
  gen_amitt.py                        61 values,   61 preserved, 0 changed
  plot4ai                            138 values,  138 preserved, 0 changed
  gen_microsoft_activity_group.py    179 values,  179 preserved, 0 changed

No committed uuid changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HZGwPoa8MMfkhCw47rDLA4
@elhoim elhoim changed the title fix: four generators remint every uuid on regeneration (uuid4 / index-seeded uuid5) [critical] fix: four generators remint every uuid on regeneration (uuid4 / index-seeded uuid5) Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant