[critical] fix: fetch_malpedia.sh overwrites clusters/malpedia.json with API error bodies

[elhoim]

BLUF — fetch_malpedia.sh overwrites the curated clusters/malpedia.json with API error bodies.

• Problem — tools/fetch_malpedia.sh runs curl without --fail and mvs the response straight over the curated clusters/malpedia.json, so an HTTP 401, 403 or 500 writes the API error body over the committed cluster. With no set -e the four del_*.py cleanup scripts then run over the wreckage, and a JSON syntax check would not help since API error bodies are valid JSON.
• Fix — Stage the whole update in a mktemp file, require a non-empty values array, run the cleanup scripts there, and mv into place as the final step under set -euo pipefail.
• Effect — A failed fetch now leaves the committed cluster untouched instead of half-processed.

Problem

tools/fetch_malpedia.sh overwrites the committed clusters/malpedia.json with whatever the API returned:

curl -H 'Authorization: apitoken ...' https://malpedia.../api/get/misp >malpedia.json
mv malpedia.json ../clusters/malpedia.json
./del_duplicate_refs.py ../clusters/malpedia.json
./del_duplicate_uuids.py ../clusters/malpedia.json
./del_duplicate_value.py ../clusters/malpedia.json
./del_empty.py ../clusters/malpedia.json

• curl without --fail exits 0 on HTTP 401/403/500 and writes the error body to malpedia.json.
• That body is then mv'd straight over the curated cluster. No check of any kind runs first.
• There is no set -e, so the four cleanup scripts then run over the wreckage — and if one of them raises, the overwrite has already landed and the cleanup is left half-applied.

A JSON-syntax check alone would not be enough: a REST API's error response is usually valid JSON ({"detail": "Invalid token."}), so it would sail through and still destroy the cluster.

Fix

Stage the entire update in a temp file and only replace clusters/malpedia.json once the download, the sanity check and all four cleanup scripts have succeeded. The update is now atomic — a failure at any stage leaves the committed cluster untouched.

• set -euo pipefail
• curl --fail --show-error --silent so an HTTP error is a non-zero exit
• output to a mktemp file with a cleanup trap
• a shape check rather than a syntax check: the response must have a non-empty values array
• the cleanup scripts operate on the temp file; mv into place is the last step

The token line is otherwise untouched — rotating that credential is a separate matter and deliberately not addressed here.

Verification

The guard against realistic failure modes, and against the real cluster:

  HTML error page:              REJECT
  valid-JSON API error:         REJECT
  empty cluster {"values":[]}:  REJECT
  empty body:                   REJECT
  real clusters/malpedia.json:  ACCEPT

And confirming set -e introduces no regression — all four cleanup scripts run clean over the current cluster:

  del_duplicate_refs.py:  OK
  del_duplicate_uuids.py: OK
  del_duplicate_value.py: OK
  del_empty.py:           OK

🤖 Generated with Claude Code