Energy Benchmarks

Energy Benchmarks

The energy is currently measured in-programm with zeus. Due to how zeus and our implementation of energy measurements work, root privileges are required for CPU measurements. GPU measurements can be done without any additional privileges.

The measurements are performed by defining a starting window before and an ending window after the generation process using a given model. Since the measurement window only covers the generation phase.

The data used is generated due to some limitations of the code structure. However a dataset and tokenizer is still required so certain required values with qonnx models are initialized.

Energy Command

Run Examples

Example without any special configurations:

qtransform run=energy model=energy_bench dataset=tsV2 tokenizer=tsV2 run.data_points=1 run.preheat.data_points=1 wandb.enabled=False

Example using run configurations for idle time and the maximum number of iterations:

qtransform run=energy model=energy_bench dataset=tsV2 tokenizer=tsV2 run.data_points=10 run.idle_time=60 wandb.enabled=False

Complete example:

qtransform run=energy model=energy_bench dataset=tsV2 tokenizer=tsV2 model.cstr=MGPT-s256-t2048-l1-h2-e256-AReLU-NBatchNormTranspose-Plearned run.data_points=1000 run.preheat.data_points=5 run.idle_time=60 run.temperature=0.7 run.top_k=200 run.batch_size=1 run.max_new_tokens=256 run.out.path=/home/student/energy/gpt2-s256-t2048-l1-h2-e256 wandb.enabled=False

Note that model.cstr overwrites model.

Run Options

idle_time: 0

preheat:
  data_points: 0

# generation configuration
max_new_tokens: 512
temperature: 0.7
top_k: 200

data_points: ???

batch_size: 1

out:
  path: null # path where the results should be saved to

Most run options are already explained on the qtransform page. Check there for more info.

idle_time

Set by default to 0. When idle_time is greater than 0, energy while the program is idling is measured. The energy while idling is measured before any generation is done.

The energy measured during idle time is intended to be used to see the spike from idling to preheating/generation. This is useful for visualization purposes.

preheat

Generation with the given model will be done before the actual measuring of energy during generation. This serves to load up the caches and reduce possible initialization costs of the inference device.

data_points

Set by default to 0. Preheating won't occur unless the value is greater than 0.

batch_size

The batch size, with which generation should run.

out

path

The path where the results should be saved to. If not specified the results will be printed to console instead.

Structure of results

Example structure of how the results are stored with run.out.path="/home/user/qtransform/energy_measurements" and multiple runs with batch sizes 1 and 2, a max new token count of 256, a temperature of 0.7 and top k of 200:

qtransform/
├── energy_measurements
│   ├── batch-size1_token256_temp0.7_top200
│   │   ├── energy_verbose.csv
│   │   └── run_cfg.txt
│   ├── batch-size2_token256_temp0.7_top200
│   │   ├── energy_verbose.csv
│   │   └── run_cfg.txt
│   └── generation_energy_summary.csv
...

energy_verbose

Saves for each batch the measured energy in Joule and the time in seconds, start and end timestamp, alongside with the type ("idle", "preheat", "generation"). The type is for visualization and filtering purposes. The .csv file storing these values has the following header:

time(s)	cpu_energy(J)	gpu_energy(J)	start	end	type

generation_energy_summary.csv

Saves for each run the per batch averages and total sum of the measured energy in Joule, the time in seconds and the number of tokens. These values are only from the "generation" type. The .csv file storing these values has the following header:

run	total_time(s)	avg_time(s)	total_cpu_energy(J)	avg_cpu_energy(J)	total_gpu_energy(J)	avg_gpu_energy(J)	total_tokens	avg_tokens	batch_size

run_cfg

Stores the parameters, with which the benchmark was executed.