Based on code-path analysis, a remote client can trigger a reachable
assertion in nanocoap_fileserver by sending a CoAP request with a
sufficiently large extended token when nanocoap_token_ext is enabled.

The root cause is that _resp_init() may fail when coap_build_reply()
cannot fit the response header into the response buffer, but its
callers in nanocoap_fileserver continue operating on stale or invalid
response state instead of checking the return value.

Affected code paths:

• sys/net/application_layer/nanocoap/fileserver.c
  • _resp_init() at line 116
  • _calc_szx2() assertion at line 176
  • _get_file() calls _resp_init() at line 240 and ignores its return value
  • _get_directory() calls _resp_init() at line 482 and ignores its return value
• sys/net/application_layer/nanocoap/nanocoap.c
  • coap_build_reply() at line 740
  • buffer-length failure path at line 758

Code-path details:

1. A remote client sends a CoAP GET request with a large token using
   the extended token-length mechanism (RFC 8974 / nanocoap_token_ext).
2. The request token length influences the response header length
   because coap_build_reply() uses coap_get_total_hdr_len(pkt) from the
   request when constructing the reply.
3. If the response buffer is smaller than that header length,
   coap_build_reply() returns -ENOSPC.
4. _resp_init() propagates that failure as -1.
5. _get_file() and _get_directory() do not check the result and
   continue using pdu.
6. The code then reaches _calc_szx2(), which asserts:
   assert(pdu->payload_len > reserve);
7. This appears to create a remotely reachable assertion / denial of
   service condition.

Why this appears realistic:

• CONFIG_GCOAP_PDU_BUF_SIZE defaults to 128 in
  sys/net/application_layer/gcoap/Makefile.include.
• With nanocoap_token_ext enabled, a token around 300 bytes yields a
  response header length around 306 bytes by deterministic arithmetic:
  4-byte base header + 2-byte extended TKL field + 300-byte token.
• That exceeds a 128-byte response buffer before any payload or
  options are added.

Impact:
This appears to be a remote denial of service affecting applications
that expose nanocoap_fileserver and build with nanocoap_token_ext
enabled.

CWE candidates:

• CWE-252: Unchecked Return Value
• CWE-617: Reachable Assertion

Important limitation / disclosure note:
I was unable to obtain a runtime crash log because the RIOT native
target is not supported in my current macOS environment. This report
is therefore based on direct code-path analysis and deterministic
buffer-size arithmetic, not on a captured runtime assertion trace.

Suggested fix direction:

• Check the return value of _resp_init() in _get_file() and
  _get_directory() and abort cleanly on failure.
• Consider replacing the assertion in _calc_szx2() with explicit error
  handling for externally influenced state.
• Audit other nanocoap_fileserver response-building paths for the same
  unchecked _resp_init() pattern.

I’m happy to help clarify the report if needed.