Document macOS telemetry research

Author: spofdamon

MACOS-TELEMETRY-RESEARCH.md

@@ -0,0 +1,323 @@
+# macOS Media and GPU Workload Detection Research
+
+Status: research only; no implementation is proposed by this document.
+
+This document records an investigation into two missing areas of the Apple
+backend:
+
+- hardware video encoder and decoder detection; and
+- graphics-versus-compute detection, especially for individual processes.
+
+The central finding is that macOS exposes useful private activity signals, but
+not enough information to populate nvtop's existing percentage and per-process
+fields accurately. Media sessions and some system-wide activity can be
+detected. Trustworthy encoder/decoder utilization percentages and per-process
+graphics/compute classification cannot currently be derived.
+
+## nvtop's data requirements
+
+The generic nvtop data model expects:
+
+- `encoder_rate` and `decoder_rate` to be device utilization percentages;
+- `encode_usage` and `decode_usage` to be the percentage of the corresponding
+  engine used by a process; and
+- each process to be classified as unknown, graphical, compute, or both.
+
+These contracts are defined in
+[`include/nvtop/extract_gpuinfo_common.h`](include/nvtop/extract_gpuinfo_common.h).
+They matter because activity, power, bandwidth, and engine residency are not
+interchangeable with a capacity-normalized percentage.
+
+If device-level encoder or decoder rates are absent, the common refresh code
+may construct them by summing per-process percentages. This behavior is in
+[`src/extract_gpuinfo.c`](src/extract_gpuinfo.c), so inaccurate per-process
+values would also produce inaccurate device values.
+
+## Research environment and limitations
+
+The live experiments were performed on:
+
+- Apple M5 Max integrated GPU;
+- macOS 26.5.2, build 25F84; and
+- an ordinary, unentitled user process.
+
+Controlled 4K60 H.264 encode and decode workloads were run through FFmpeg's
+VideoToolbox support. Idle and active samples were compared for I/O Registry
+objects and IOReport channels.
+
+The observations establish what is available on this machine. Private class
+names, registry properties, channel names, and permissions may differ across
+M1 through M5 generations and macOS releases.
+
+## Media engine detection
+
+### VideoToolbox sessions in the I/O Registry
+
+Hardware VideoToolbox sessions create private IOKit user clients. The relevant
+classes found during this investigation were:
+
+| User client | Apparent role | Observed information |
+| --- | --- | --- |
+| `AppleAVDUserClient` | Hardware video decoder | Codec, frame size, chroma format, luma bit depth, usage mode, and creation time |
+| `AppleAVE2UserClient` | Hardware video encoder | Creator information; few useful session properties |
+| `AppleProResUserClient` | ProRes media engine | Class present, but not exercised by the H.264 workloads |
+
+An open user client proves that a hardware media session exists. It does not
+prove that the session processed data during the current sampling interval;
+applications can retain idle VideoToolbox sessions.
+
+Apple's public VideoToolbox API lets a client require or request hardware
+acceleration for
+[encoding](https://developer.apple.com/documentation/videotoolbox/kvtvideoencoderspecification_enablehardwareacceleratedvideoencoder?language=objc)
+and
+[decoding](https://developer.apple.com/documentation/videotoolbox/kvtvideodecoderspecification_enablehardwareacceleratedvideodecoder),
+but it does not provide a system-monitor API for engine utilization.
+
+### System-wide activity signals
+
+The M5 Max IOReport legends advertise several media-related signals:
+
+- an `AVE` energy-model channel;
+- `AVE0` and `AVE1` system interconnect bandwidth channels;
+- AVD, AVE, and ProRes performance-floor residency;
+- AVD and ProRes fabric bandwidth histograms; and
+- lower-level media performance counters in the `AMC Stats / Perf Counters`
+  group.
+
+Not all advertised channels were usable by an ordinary process. Subscribing to
+the lower-level AMC performance counters failed without additional privilege
+or entitlement, and the advertised AVD bandwidth histograms remained zero
+during the decoder test.
+
+The controlled tests produced these results:
+
+| Workload | Registry evidence | IOReport evidence |
+| --- | --- | --- |
+| Idle | No test media session | `AVE` energy delta approximately 6 mJ over one second; media floors stayed at their minimum state |
+| 4K60 H.264 hardware encode | `AppleAVE2UserClient` owned by `VTEncoderXPCServ` | `AVE` energy delta approximately 298 mJ over one second; AVE bandwidth and high performance-floor residency increased |
+| 4K60 H.264 hardware decode | `AppleAVDUserClient` owned by `VTDecoderXPCServ` | AVD high performance-floor residency increased; `AVE` energy stayed near idle and AVD bandwidth counters remained unavailable or zero |
+
+These signals are sufficient to infer recent system-wide media activity in
+some cases, but none has a documented maximum capacity. Consequently:
+
+- energy is a power input, not an encoder occupancy percentage;
+- bandwidth is traffic, not media-engine occupancy;
+- performance-floor residency is a requested operating state, not time spent
+  executing work; and
+- session presence is state, not utilization.
+
+Mapping any of these to 0 or 100 percent, or calibrating observed peak activity
+as 100 percent, would create workload- and machine-dependent values.
+
+### IOKit busy state is not hardware utilization
+
+`IOServiceGetBusyState` does not provide a missing execution-time counter.
+Apple documents IOKit busy state as tracking asynchronous service registration,
+matching, and termination work. It propagates through the service tree and is
+not a measure of device-engine occupancy.
+
+See Apple's
+[`IOKitGetBusyState` documentation](https://developer.apple.com/documentation/iokit/1514460-iokitgetbusystate).
+
+### Per-process media attribution
+
+Media user clients were not owned by the application that submitted the work.
+They were owned by shared VideoToolbox XPC helpers:
+
+- `VTEncoderXPCServ` for encoding; and
+- `VTDecoderXPCServ` for decoding.
+
+The active decoder user client reported `ClientPID=0`, and its
+`IOUserClientCreator` identified the XPC helper rather than FFmpeg. Parent PID
+or process-name matching therefore cannot recover the originating application.
+A shared helper may also multiplex sessions belonging to multiple clients.
+
+Endpoint Security exposes a responsible-process audit token, but it is not a
+practical dependency for nvtop. Creating a general Endpoint Security client
+requires an Apple-granted restricted entitlement and user approval through Full
+Disk Access. Even then, the responsible process describes responsibility for
+the helper process; it does not expose ownership of an individual VideoToolbox
+session.
+
+See Apple's documentation for
+[`responsible_audit_token`](https://developer.apple.com/documentation/endpointsecurity/es_process_t/responsible_audit_token)
+and
+[`es_new_client`](https://developer.apple.com/documentation/endpointsecurity/3259700-es_new_client).
+
+Timing correlation, helper ancestry, application names, and a "sole active
+client" assumption are not reliable enough for process attribution.
+
+### Media conclusion
+
+The available information supports these distinctions:
+
+| Desired result | Supportable now? | Reason |
+| --- | --- | --- |
+| A hardware media session exists | Yes, using private IOKit classes | User-client presence |
+| The system recently performed some encode work | Partially | Strong energy, bandwidth, and floor signals on the tested M5 Max |
+| The system recently performed some decode work | Partially | Session presence and performance-floor changes on the tested M5 Max |
+| Device encoder or decoder utilization in percent | No | No capacity-normalized counter |
+| Per-process encoder or decoder utilization | No | Work is attributed to shared VideoToolbox helpers |
+
+Media activity should therefore not be written into `encoder_rate`,
+`decoder_rate`, `encode_usage`, or `decode_usage` without discovering a new
+source that already represents utilization as a percentage or provides both
+busy time and a meaningful capacity denominator.
+
+## Graphics-versus-compute detection
+
+### Public Metal APIs
+
+Metal distinguishes render and compute work inside the application submitting
+the commands. Applications create distinct render and compute command encoders,
+as described in Apple's
+[Metal command-structure documentation](https://developer.apple.com/documentation/metal/gpu_devices_and_work_submission/setting_up_a_command_structure).
+
+Metal capture, counter sampling, and Xcode's visual timeline can preserve that
+distinction while profiling an instrumented application. They are not passive,
+cross-process monitoring APIs. `MTLCaptureManager`, for example, captures Metal
+commands associated with the profiled application's command buffers.
+
+See
+[`MTLCaptureManager`](https://developer.apple.com/documentation/metal/MTLCaptureManager)
+and Apple's
+[Metal visual timeline documentation](https://developer.apple.com/documentation/xcode/analyzing-apple-gpu-performance-using-a-visual-timeline).
+
+### Per-process AGX information
+
+The Apple backend currently enumerates `AGXDeviceUserClient` objects and sums
+each client's `AppUsage[].accumulatedGPUTime`. This supplies a useful aggregate
+per-process GPU-time delta.
+
+The observed `AppUsage` entries also contained an `API` string, with these
+values:
+
+- `Metal`; and
+- `GL/CL`.
+
+Neither value permits graphics/compute classification:
+
+- a Metal command queue may contain render work, compute work, or both; and
+- `GL/CL` combines OpenGL and OpenCL, which collapses graphics and compute into
+  a single value.
+
+No render-pass, compute-pass, pipeline-type, or engine-time property was found
+in the per-client registry data. `lastSubmittedTime` describes submission
+timing, not command type.
+
+The parser for this information is in
+[`src/extract_gpuinfo_apple_utils.m`](src/extract_gpuinfo_apple_utils.m).
+
+### Device-level pipeline signals
+
+The AGX accelerator publishes global values named:
+
+- `Device Utilization %`;
+- `Renderer Utilization %`; and
+- `Tiler Utilization %`.
+
+Its IOReport legend also advertises global GPU pipeline and context-switch
+channels, including names such as `GTP`, `FRG`, `CDM`, `TA`, `3D`, and `CL`.
+These can potentially describe the shape of total GPU activity, but they do not
+carry a PID. Under concurrent workloads there is no reliable way to assign a
+change in one of these global counters to a particular process.
+
+Correlating a global renderer, tiler, or compute-like counter with the process
+that happened to accumulate the most GPU time during the same interval would
+be a heuristic. It would fail as soon as graphical and compute workloads run
+concurrently.
+
+Other macOS monitoring implementations reach the same practical boundary. For
+example, [`metop`](https://pypi.org/project/metop/) reports global device,
+renderer, and tiler values and derives per-process activity from
+`accumulatedGPUTime`; it does not derive a per-process graphics/compute split.
+
+### Current nvtop behavior
+
+The Apple backend currently assigns every discovered process
+`gpu_process_graphical_compute`, displayed as `Both G+C`. This occurs in
+[`src/extract_gpuinfo_apple_utils.m`](src/extract_gpuinfo_apple_utils.m).
+
+`Both G+C` is not a verified classification, but changing the backend to
+`gpu_process_unknown` would currently introduce a worse error in the
+interactive interface. The UI displays:
+
+- `Both G+C` for `gpu_process_graphical_compute`;
+- `Graphic` for `gpu_process_graphical`; and
+- `Compute` for every other value, including `gpu_process_unknown`.
+
+That behavior is in [`src/interface.c`](src/interface.c). Proper unknown-state
+rendering is required before the Apple backend can report uncertainty honestly.
+
+The API string should not be used to relabel processes. In particular, Metal
+must not be treated as evidence of both graphics and compute, and `GL/CL` must
+not be classified from the process name.
+
+### Graphics/compute conclusion
+
+| Desired result | Supportable now? | Reason |
+| --- | --- | --- |
+| Aggregate per-process GPU activity | Yes | Delta of `accumulatedGPUTime` |
+| Process used the Metal API | Yes, privately | `AppUsage[].API` |
+| Process performed graphics work | No | No per-client render-engine signal |
+| Process performed compute work | No | No per-client compute-engine signal |
+| System-wide renderer/tiler activity | Yes, privately | Global AGX performance properties |
+| Attribute renderer/tiler activity to a PID | No | Counters are global |
+
+## Implications for a future design
+
+The first design change should make the model represent what macOS actually
+provides instead of coercing activity into percentage fields.
+
+1. Add correct interactive display support for `gpu_process_unknown`.
+2. Report Apple process type as unknown, or retain `Both G+C` until the UI can
+   represent unknown without calling it compute.
+3. If media detection is added, represent session presence or recent activity
+   separately from encoder and decoder utilization percentages.
+4. Leave `ENC%` and `DEC%` unavailable unless a capacity-normalized source is
+   found.
+5. Consider exposing Renderer and Tiler as separate Apple device-level metrics,
+   not as evidence for per-process classification.
+6. Keep private-property and IOReport channel resolution isolated behind the
+   Apple backend because names and availability are not stable APIs.
+
+The Apple backend currently sets `encode_decode_shared = true` in
+[`src/extract_gpuinfo_apple.m`](src/extract_gpuinfo_apple.m). The observed Apple
+media engines are distinct, so this appears to be placeholder behavior rather
+than a description of the hardware. It should become false if separate media
+metrics or activity indicators are eventually exposed. Changing it alone would
+not add usable metrics.
+
+## Validation required before implementation
+
+Any future implementation based on private telemetry should be tested across:
+
+- M1, M2, M3, M4, and M5 families;
+- idle sessions versus continuously active sessions;
+- low-resolution and 4K workloads;
+- H.264, HEVC, AV1, and ProRes where supported;
+- simultaneous encode and decode;
+- simultaneous graphical and compute GPU workloads; and
+- ordinary-user operation without root, entitlements, or additional privacy
+  permissions.
+
+For each media workload, validation should distinguish session existence,
+recent activity, power, bandwidth, performance-state residency, busy time, and
+true capacity-normalized utilization. A source should only populate nvtop's
+percentage fields if the final category is actually available.
+
+## Summary
+
+- Private IOKit data can identify open Apple decoder, encoder, and ProRes
+  sessions, subject to chip and OS differences.
+- M5 Max IOReport channels can reveal some system-wide media activity, but the
+  signals are energy, bandwidth, or performance-state residency rather than
+  utilization percentages.
+- VideoToolbox XPC helpers prevent reliable attribution of media sessions to
+  originating applications.
+- Per-process AGX data provides aggregate GPU time and an ambiguous API label,
+  not render-versus-compute engine time.
+- Global Renderer and Tiler values may be useful device metrics but cannot
+  classify processes.
+- The honest near-term behavior is to leave media percentages unavailable and
+  represent process type as unknown once the interface supports it correctly.