Skip to content

Commit f016725

Browse files
GregBowyerGregBowyer
committed
Release of eval scripts
1 parent a50e394 commit f016725

File tree

6 files changed

+354
-0
lines changed

6 files changed

+354
-0
lines changed

Diff for: ‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2023 Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

Diff for: ‎README.md

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Eval Scripts
2+
3+
## TL;DR
4+
5+
These are the _very basic_ scripts are for running submissions of the competition. Scripts herein are very dumb, intentionally so. PRs welcome for better scripts!
6+
7+
## Setup
8+
9+
The following tools need to be setup on an evaluation machine.
10+
11+
* `docker` and `nvidia-docker` for running the helm suite and the submissions
12+
* `git` for cloning repos
13+
* `curl` for running the healthchecks to ensure containers are running
14+
15+
With these installed run the `setup.sh` script with the number of GPUs in the evaluation machine.
16+
17+
```sh
18+
./setup.sh $NUM_GPUS
19+
```
20+
21+
... for example to use 8 gpus in an eval machine
22+
23+
```sh
24+
./setup.sh 8
25+
```
26+
27+
### What this does
28+
This script will ensure a few tools are present and functional, as well as setting up a degree of isolation on the machine.
29+
This takes the form of docker networks and port exposures for each individual GPU.
30+
31+
We also checkout the helm evaluation version used in the competition, into the `./private-helm` folder.
32+
33+
## Running a submission
34+
Make sure the submissions are in `./submissions` folder relative to this script.
35+
36+
To run a single submission do:
37+
38+
```sh
39+
./eval-repo.sh \
40+
'$gpu_device' \
41+
'$isolation' \
42+
'$hardware_track' \
43+
'$helm_config' \
44+
'$submission'
45+
```
46+
47+
What this does:
48+
49+
* `'$gpu_device'` Specifies the GPU string to pass to docker for the GPU(s) to run the submission on
50+
* `'$isolation'` An isolation factor used to divide submissions between multiple GPUs on a single server. We recommend to keep this number the same as the `$gpu_device` string.
51+
* `'$helm_config'` The config used for helm, must be in a path visibile to the helm container. Private-helm contains configs within the container for the 111 competition.
52+
* `'$hardware_track'` The hardware track to build for in the submissions folder. Essentially a top level folder in `./submissions` that differentiates between different hardware tracks/
53+
* `'$submission'` The submission folder to build
54+
55+
The `$submission` is the folder that contains the submission to evaluate. A bare `Dockerfile` is expected at this location to build the submissions container.
56+
57+
### Layout of the submissions folder
58+
Submissions are laid out in `./submissions` folder in the following fashion:
59+
60+
```
61+
./submissions
62+
├── 4090
63+
│   └── $user
64+
│      └── $repo
65+
│      ├── README.md
66+
│      ├── submission_1
67+
│      │   ├── Dockerfile
68+
│      │   └── ...
69+
│      └── submission_2
70+
│         ├── Dockerfile
71+
│         └── ...
72+
└── A100
73+
   └── $user
74+
      └── $repo
75+
      ├── README.md
76+
      ├── submission_1
77+
      │   ├── Dockerfile
78+
      │   └── ...
79+
      └── submission_2
80+
         ├── Dockerfile
81+
         └── ...
82+
```
83+
84+
### Putting this together for a simple run
85+
86+
Using the above layout to run a submission for A100 `$user` `$repo/submission_2` on the second GPU you would do the following:
87+
88+
```sh
89+
./eval-repo.sh \
90+
'device=1' \
91+
'1' \
92+
'A100' \
93+
'/helm/config/some_helm_config.conf' \
94+
'$user/$repo/submission_2'
95+
```
96+
97+
## Changing locations
98+
Should you want to change the locations of the `./private-helm`, `./submissions` or `./results` folders you will need to edit `utils.sh`
99+
100+
* For `./submissions` change `$BASE_SUB_DIR` in [`utils.sh`](utils.sh)
101+
* For `./results` change `$OUT_DIR` in [`utils.sh`](utils.sh)
102+
* For `./private-helm` change [`build-eval-container.sh`](build-eval-container.sh)

Diff for: ‎build-eval-container.sh

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/usr/bin/env bash
2+
3+
cd private-helm || exit
4+
docker build -t llm-eval .

Diff for: ‎eval-repo.sh

+152
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
#!/usr/bin/env bash
2+
3+
source ./utils.sh
4+
5+
cleanup() {
6+
local llm_eval_container
7+
llm_eval_container="$(cat "$PID_DIR/llm_docker.name")"
8+
docker stop "$llm_eval_container"
9+
10+
local sub_name
11+
sub_name="$(cat "$PID_DIR/submission_docker.name")"
12+
docker stop "$sub_name"
13+
14+
kill -15 "$(cat "$PID_DIR/submission_log.pid")"
15+
sleep 10
16+
17+
docker rmi "$sub_name"
18+
}
19+
20+
trap cleanup EXIT
21+
trap cleanup SIGINT
22+
23+
submission_name() {
24+
echo "${1//\//_}" | tr '-' '_' | tr '[:upper:]' '[:lower:]'
25+
}
26+
27+
gaurentee_dirs() {
28+
mkdir -p "$PID_DIR"
29+
mkdir -p "$BASE_SUB_DIR"
30+
mkdir -p "$OUT_DIR"
31+
}
32+
33+
build_submission() {
34+
local hardware_track="$1"
35+
local submission="$2"
36+
37+
local sub_name
38+
sub_name=$(submission_name "$submission")
39+
40+
enter "$BASE_SUB_DIR/$hardware_track/$submission"
41+
42+
docker build -t "$sub_name" . 2>&1 \
43+
| tee "$OUT_DIR/$sub_name-build.log" \
44+
|| die "Could not build $sub_name"
45+
46+
leave
47+
}
48+
49+
healthcheck() {
50+
local port="$1"
51+
52+
local max_retries=10 # Maximum number of retries
53+
local retry_delay=120 # Delay between retries in seconds
54+
55+
local url="http://localhost:$port/process"
56+
local data='{"prompt": "The capital of France is "}'
57+
local accept='Content-Type: application/json'
58+
59+
for ((i = 0; i < max_retries; i++)); do
60+
sleep $retry_delay
61+
if curl -q -X POST -H "$accept" -d "$data" "$url" ; then
62+
return 0
63+
else
64+
echo "Retrying healthcheck (Attempt $i)..."
65+
fi
66+
done
67+
68+
die "Could not healthcheck after $max_retries retries"
69+
}
70+
71+
run_submission() {
72+
local submission="$1"
73+
local isolation="$2"
74+
local gpus="$3"
75+
76+
local network
77+
local sub_name
78+
79+
network="llm_eval_$isolation"
80+
sub_name=$(submission_name "$submission")
81+
82+
local port="808$isolation"
83+
84+
docker run \
85+
-d \
86+
--rm \
87+
--name "$sub_name" \
88+
--network "$network" \
89+
--runtime=nvidia \
90+
--gpus "$gpus" \
91+
-p "$port:80" \
92+
"$sub_name" || die "Could not run $sub_name"
93+
94+
echo "$sub_name" > "$PID_DIR/submission_docker.name"
95+
96+
( docker logs -f "$sub_name" > "$OUT_DIR/$sub_name-run.log" 2>&1 ) > /dev/null &
97+
echo "$!" > "$PID_DIR/submission_log.pid"
98+
99+
healthcheck "$port"
100+
}
101+
102+
get_ip() {
103+
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$1"
104+
}
105+
106+
run_helm() {
107+
local submission="$1"
108+
local isolation="$2"
109+
local config="$3"
110+
111+
local sub_name
112+
local ip
113+
local llm_eval_name="llm_eval_${isolation}"
114+
115+
sub_name=$(submission_name "$submission")
116+
ip="$(get_ip "$sub_name")"
117+
118+
echo "$llm_eval_name" > "$PID_DIR/llm_docker.name"
119+
120+
docker run \
121+
--rm \
122+
--name "$llm_eval_name" \
123+
--env HELM_HTTP_MODEL_BASE_URL="http://$ip" \
124+
--network "$llm_eval_name" \
125+
-v "$OUT_DIR:/results" \
126+
llm-eval \
127+
/helm/do-run.sh "$config" "$sub_name" || die "Could not run helm"
128+
}
129+
130+
main() {
131+
local gpus="$1"
132+
local isolation="$2"
133+
local hardware_track="$3"
134+
local config="$4"
135+
local submission="$5"
136+
137+
# Isolate for specific runs on multi-gpus
138+
export PID_DIR="$EVAL_ROOT/state/$isolation"
139+
140+
if [[ $# != 5 ]]; then
141+
echo "Usage $0: gpu-spec isolation hardware-track config repo submission"
142+
exit 1
143+
fi
144+
145+
gaurentee_dirs
146+
147+
build_submission "$hardware_track" "$submission"
148+
run_submission "$submission" "$isolation" "$gpus"
149+
run_helm "$submission" "$isolation" "$config"
150+
}
151+
152+
main "$@"

Diff for: ‎setup.sh

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
3+
source ./utils.sh
4+
5+
git_clone() {
6+
git clone "$1"
7+
}
8+
9+
setup_docker_network() {
10+
local num_gpus="$1"
11+
for isolation in $(seq 1 "$num_gpus"); do
12+
if ! docker network inspect "llm_eval_$isolation" > /dev/null 2>&1 ; then
13+
docker network create "llm_eval_$isolation" || die "Unable to create llm-eval docker network"
14+
fi
15+
done
16+
}
17+
18+
main() {
19+
if [[ $# -ne 1 ]]; then
20+
die "Usage $0: [number-gpus]"
21+
fi
22+
23+
check_cmd curl
24+
check_cmd docker
25+
check_cmd git
26+
27+
git_clone "[email protected]:llm-efficiency-challenge/private-helm.git"
28+
29+
enter private-helm
30+
git checkout neurips_eval
31+
leave
32+
33+
./build-eval-container.sh || die "Cannot build the eval container"
34+
setup_docker_network "$2"
35+
36+
echo "Make sure submissions are present in submissons"
37+
}
38+
39+
main "$@"

Diff for: ‎utils.sh

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/usr/bin/env bash
2+
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
3+
4+
export EVAL_ROOT="$SCRIPT_DIR"
5+
export BASE_SUB_DIR="$EVAL_ROOT/submissions"
6+
export OUT_DIR="$EVAL_ROOT/benchmark-results"
7+
8+
die() {
9+
local sub_name
10+
sub_name="$(cat "$PID_DIR/submission_docker.name" 2> /dev/null || echo "" )"
11+
12+
if [[ "$sub_name" ]]; then
13+
echo "$sub_name" >> "$EVAL_ROOT/failures.txt"
14+
else
15+
echo "4sub_name" >> "$EVAL_ROOT/successes.txt"
16+
fi
17+
18+
echo "$1"
19+
20+
exit 1
21+
}
22+
23+
enter() {
24+
pushd "$1" > /dev/null || die "Could not enter $1"
25+
}
26+
27+
leave() {
28+
popd > /dev/null || die "Could not exit"
29+
}
30+
31+
check_cmd() {
32+
if ! command -v "$1" &> /dev/null; then
33+
echo "$1 could not be found install it"
34+
exit 1
35+
fi
36+
}

0 commit comments

Comments
 (0)