GPU monitoring (INTEL)

[nicolargo]

Description

As implemented in the issue #170 for the NVidia GPUs, the goal is to enhance the GPU plugin for Intel GPUs.

First of all, we have to find a Python Lib to grab stats, then change the current plugin to be able to monitor Intel GPU.

[kdbanman]

As shared in #993, this and this were the starting points of my research

---

we have to find a Python Lib to grab stats

The intel-gpu-tools library is usually used through a command line tool called intel_gpu_top to monitor intel gpus.

The command line tool could be used by spawning a subprocess and parsing its text output from within python. Alternatively, the necessary parts of intel-gpu-tools API could be accessed in python through ctypes. The ctypes option will be less sensitive because it won't break from text output format changes in the command line tool.

But the API is complex to consume. Here is how intel_gpu_top consumes the API to extract simple utilization stats. The library API is used to directly read from certain registers in hardware with memory address constants.

Even with the simpler command line tool text parsing option, it will take some research to understand the relationship between the stats that are reported and the stats that Glances needs. See this StackOverflow answer to see what I mean.

Hopefully there is a simpler option that I don't yet know of!

[jamiew]

Is there still any interest in this, or willingness to accept a (rough) PR? I am new to the glances codebase but would be happy to take a crack

[Swiftnesses]

Just checking in to see if there was progress here?

[nicolargo]

Intel GPU top on my laptop.

sudo apt install intel-gpu-tools

then:

sudo intel_gpu_top

Result:

and

===
intel_gpu_top could also be ran from the command line to generate JSON format (for example every refresh = 2 seconds):

# sudo intel_gpu_top -J -s 2000
{
	"period": {
		"duration": 2000.468394,
		"unit": "ms"
	},
	"frequency": {
		"requested": 15.496371,
		"actual": 15.496371,
		"unit": "MHz"
	},
	"interrupts": {
		"count": 11.997190,
		"unit": "irq/s"
	},
	"rc6": {
		"value": 96.077379,
		"unit": "%"
	},
	"power": {
		"GPU": 0.058702,
		"Package": 1.576352,
		"unit": "W"
	},
	"imc-bandwidth": {
		"reads": 694.191091,
		"writes": 75.975951,
		"unit": "MiB/s"
	},
	"engines": {
		"Render/3D/0": {
			"busy": 1.244288,
			"sema": 0.000000,
			"wait": 0.000000,
			"unit": "%"
		},
		"Blitter/0": {
			"busy": 0.000000,
			"sema": 0.000000,
			"wait": 0.000000,
			"unit": "%"
		},
		"Video/0": {
			"busy": 0.000000,
			"sema": 0.000000,
			"wait": 0.000000,
			"unit": "%"
		},
		"VideoEnhance/0": {
			"busy": 0.000000,
			"sema": 0.000000,
			"wait": 0.000000,
			"unit": "%"
		}
	}
}

A Glances subprocess could be used to run this command line.

PRO: the output is easy to parse (but we had to compute the global %, is it render/xxx/busy or rc6/value ? more info here https://stackoverflow.com/questions/28876242/interpretation-of-intel-gpu-top-output/38087477#38087477)
CON: the command line should be ran as root (so the command should be added in the sudoers file, not very user friendly and need root access to the server)

[RazCrimson]

We'll go ahead and extend the gpu plugin similar to the docker -> containers?

CON: the command line should be ran as root (so the command should be added in the sudoers file, not very user friendly and need root access to the server)

Does making it a executable that runs with root perms work fine?

[nicolargo]

@RazCrimson just open a "meta" issue for #2409

[nicolargo]

NVtop support the following GPUs:

• AMD
• Intel
• NVIDIA
• Adreno
• Apple
• Ascend (only tested on 910B)

Note: Intel introduced the fdinfo interface in kernel 5.19 (browse kernel source). Hence, you will need a kernel with a version greater or equal to 5.19 to see the processes using Intel GPUs. Intel is working on exposing more hardware information through an HWMON interface. The patches are still a work in progress: see patch series. The fdinfo interface does not expose the memory allocated by the process. The field in the process list is therefore empty.

[tomerh2001]

Any updates?

[nicolargo]

@tomerh2001 Unfortunately and for the moment GPU stats for my laptop where Glances is developed are not available. See following screenshot of the NVTop software:

My Linux kernel is:

Linux nicolargo-xps15 6.8.0-40-generic #40-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 10:34:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux