Skip to content

Commit 575387a

Browse files
BBC-EsqBBC-Esq
authored
v2.0
1 parent f2074af commit 575387a

15 files changed

Lines changed: 1074 additions & 0 deletions

‎User_Manual/float.png‎

472 KB
Loading

‎User_Manual/number_format.html‎

Lines changed: 339 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,339 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Floating-Point Formats</title>
7+
<style>
8+
body {
9+
font-family: Arial, sans-serif;
10+
line-height: 1.6;
11+
margin: 0;
12+
padding: 0;
13+
background-color: #333; /* Dark grey background */
14+
color: #f0f0f0; /* Off-white text color */
15+
}
16+
17+
header {
18+
text-align: center;
19+
background-color: #3498db;
20+
color: #fff;
21+
padding: 20px;
22+
position: sticky;
23+
top: 0;
24+
z-index: 999;
25+
}
26+
27+
main {
28+
max-width: 800px;
29+
margin: 0 auto;
30+
padding: 20px;
31+
}
32+
33+
img {
34+
display: block;
35+
margin: 0 auto;
36+
max-width: 100%;
37+
height: auto;
38+
}
39+
40+
h1, h2 {
41+
color: #333;
42+
}
43+
44+
p {
45+
text-indent: 35px; /* Indent paragraphs */
46+
}
47+
48+
table {
49+
border-collapse: collapse;
50+
width: 100%;
51+
}
52+
53+
th, td {
54+
text-align: left;
55+
padding: 8px;
56+
border-bottom: 1px solid #ddd;
57+
}
58+
59+
th {
60+
background-color: #f2f2f2;
61+
color: #000;
62+
}
63+
64+
footer {
65+
text-align: center;
66+
background-color: #333;
67+
color: #fff;
68+
padding: 10px;
69+
}
70+
</style>
71+
72+
</head>
73+
74+
<body>
75+
<header>
76+
<h1>Floating-Point Formats</h1>
77+
</header>
78+
79+
<main>
80+
<section>
81+
<img src="./float.png" alt="Floating Point">
82+
</section>
83+
84+
<section>
85+
<h2 style="color: #f0f0f0;">Introduction to Floating-Point Formats</h2>
86+
<p>Running an embedding model or a large language model requires a lot of math calculations and computers don't understand decimal
87+
numbers (1,2,3) like you and me. Rather, they use a series of ones and zeros to represent a number, which are called "bits." The
88+
more bits that are used, the more VRAM/RAM and computational horsepower required. However, more bits also "generally" means a
89+
higher quality result.</p>
90+
91+
<p>I say "generally" becuase even if the same number of bits are used the quality also depends on how many of those bits are
92+
"exponent" versus "fraction" bits. The term "floating point format" is defined by both the total number of "bits" used
93+
as well as how many of the bits are "exponent" versus "fraction" bits. The three most common floating point formats above
94+
illustrate these concepts. For example, both float16 and bfloat16 have the same total bits (16) but a different number of
95+
"exponent" versus "fraction" bits.</p>
96+
97+
<p>"Exponent" bits basically determine the "range" of numbers that a neural network can utilize when doing math. For example,
98+
since Float32 has 8 "exponent" bits lets pretend that this allows the neural network to use any integer
99+
between one and one-hundred for the math calculations. It's "range," therefore, is 1-100. Bfloat16 would have
100+
the same "range" because it also has 8 "exponent" bits. However, float16 only has 5 "exponent" bits so "range" might be 1-50.</p>
101+
102+
<p></p>
103+
104+
<p>In contrast, "fraction" bits basically determine the number of values that can be used within that "range." The number
105+
of "fraction" bits is informally referred to a neural network's "precision." To give another hypothetical, since float32 has 23
106+
"fraction" bits let's assume it can use every whole number between 1-100 when doing math calculations Therefore, it has 100
107+
"values" it can use. In contrast, because bfloat16 only has 7 "fraction" bits could only 25 integers within the "range" of 1-100.</p>
108+
109+
<p>These are hypotheticals, however, the actual ranges and precision are summarized in this table:</p>
110+
111+
<table border="1">
112+
<tr>
113+
<th>Floating Point Format</th>
114+
<th>Range (Based on Exponent)</th>
115+
<th>Discrete Values (Based on Fraction)</th>
116+
</tr>
117+
<tr>
118+
<td>float32</td>
119+
<td>~3.4×10<sup>38</sup></td>
120+
<td>8,388,608</td>
121+
</tr>
122+
<tr>
123+
<td>float16</td>
124+
<td>±65,504</td>
125+
<td>1,024</td>
126+
</tr>
127+
<tr>
128+
<td>bfloat16</td>
129+
<td>~3.4×10<sup>38</sup></td>
130+
<td>128</td>
131+
</tr>
132+
</table>
133+
134+
</section>
135+
136+
<p>Overall, both "range" and "precision" determine the "quality" of an output but in different ways. The specifics are complex.
137+
However, in general the particular use case determines the best floating point format to use. For example, Google created
138+
bfloat16 and found that it was overall better for neural networks but that float16 was better for scientific calculations.</p>
139+
140+
<p>You can see the floating point format used to create the various embedding models in this program by looking at the
141+
"config.json" file for each model.</p>
142+
143+
<section>
144+
<h2 style="color: #f0f0f0;">What is Quantization?</h2>
145+
146+
<p>"Quantization" refers to converting the original floating point format to one with a smaller "range"
147+
and/or "precision" - usually both. Projects like LLAMA.CPP and AutoGPTQ do this with slightly different algorithms but
148+
the same general concept applies. The overall goal is to reduce the memory and computational power needed while only
149+
experiencing a "reasonable" loss in quality. Specific "quantizations" like "Q8_0" or "8-bit" refer to the "floating point
150+
format" of "int8," for example. Technically, "int8" is no longer "floating" but you don't need to delve into the nuances
151+
of this to understand the basic concepts I'm trying to communicate.</p>
152+
153+
<p>You can obviously see the huge change of "range" and "precision" when using "int8" compared to the above floating
154+
point formats:</p>
155+
156+
<table border="1">
157+
<tr>
158+
<th>Floating Point Format</th>
159+
<th>Range (Based on Exponent)</th>
160+
<th>Discrete Values (Based on Fraction)</th>
161+
</tr>
162+
<tr>
163+
<td>int8</td>
164+
<td>-128 to 127</td>
165+
<td>±127 (within integer range)</td>
166+
</tr>
167+
</table>
168+
</section>
169+
170+
<section>
171+
<h2 style="color: #f0f0f0;">What is Ctranslate2?</h2>
172+
173+
<p>Ctranslate2 is a C++ & Python library that both quantizes and runs large langauge models, but better than ggml,
174+
gguf, and gptq. Moreover, it supports more floating point formats:</p>
175+
176+
<table border="1">
177+
<tr>
178+
<th>Floating Point Format</th>
179+
<th>Quantized Model Size</th>
180+
<th>Summary</th>
181+
</tr>
182+
<tr>
183+
<td>float32</td>
184+
<td>100%</td>
185+
<td>Original</td>
186+
</tr>
187+
<tr>
188+
<td>int16</td>
189+
<td>51.37%</td>
190+
<td>Not used for neural networks.</td>
191+
</tr>
192+
<tr>
193+
<td>float16</td>
194+
<td>50.00%</td>
195+
<td>"Old school," not suited for neural networks.</td>
196+
</tr>
197+
<tr>
198+
<td>bfloat16</td>
199+
<td>50.00%</td>
200+
<td>Best for neural networks except for float32.</td>
201+
</tr>
202+
<tr>
203+
<td>int8_float32</td>
204+
<td>27.47%</td>
205+
<td>Good for neural networks despite low quantization.</td>
206+
</tr>
207+
<tr>
208+
<td>int8_bfloat16</td>
209+
<td>26.10%</td>
210+
<td>Good for neural networks despite low quantization, but not as good as int8_float32.</td>
211+
</tr>
212+
<tr>
213+
<td>int8_float16</td>
214+
<td>26.10%</td>
215+
<td>Slightly better than int8.</td>
216+
</tr>
217+
<tr>
218+
<td>int8</td>
219+
<td>25%</td>
220+
<td>Mediocre</td>
221+
</tr>
222+
</table>
223+
224+
<p>In other words, a model converted to ctranslate2 format and run using "int8" quantization will run
225+
faster, produce a higher quality result, and require less vram/ram and computational power than the same
226+
model quantized to int8 with the ggml, gguf or gptq algorithms. Moreover, ctranslate2 supports the floating point
227+
formats that are more suitable for neural networks. All of this makes it more powerful.</p>
228+
229+
<p>For example, here's a simple comparison an identical 7 billion parameter Llama2-based model:</p>
230+
<table border="1">
231+
<tr>
232+
<th>Floating Point Format</th>
233+
<th>Backend Tech</th>
234+
<th>VRAM/RAM Needed</th>
235+
</tr>
236+
<tr>
237+
<td>float16</td>
238+
<td>ctranslate2</td>
239+
<td>15.5 GB</td>
240+
</tr>
241+
<tr>
242+
<td>bfloat16</td>
243+
<td>ctranslate2</td>
244+
<td>15.4 GB</td>
245+
</tr>
246+
<tr>
247+
<td>int8 ("Q8_0")</td>
248+
<td>ggml/gguf</td>
249+
<td>12.4 GB</td>
250+
</tr>
251+
<tr>
252+
<td>"Q6_0"</td>
253+
<td>ggml/gguf</td>
254+
<td>11.6 GB</td>
255+
</tr>
256+
<tr>
257+
<td>"Q5_k_m"</td>
258+
<td>ggml/gguf</td>
259+
<td>11.4 GB</td>
260+
</tr>
261+
<tr>
262+
<td>"Q4_k_m"</td>
263+
<td>ggml/gguf</td>
264+
<td>11.3 GB</td>
265+
</tr>
266+
<tr>
267+
<td>"Q3_k_l"</td>
268+
<td>ggml/gguf</td>
269+
<td>10.5 GB</td>
270+
</tr>
271+
<tr>
272+
<td>"Q3_k_m"</td>
273+
<td>ggml/gguf</td>
274+
<td>10.3 GB</td>
275+
</tr>
276+
<tr>
277+
<td>"Q3_k_s"</td>
278+
<td>ggml/gguf</td>
279+
<td>10 GB</td>
280+
</tr>
281+
<tr>
282+
<td>int8_float32</td>
283+
<td>ctranslate2</td>
284+
<td>9.4 GB</td>
285+
</tr>
286+
<tr>
287+
<td>int8_float16</td>
288+
<td>ctranslate2</td>
289+
<td>9.0 GB</td>
290+
</tr>
291+
<tr>
292+
<td>int8_bfloat16</td>
293+
<td>ctranslate2</td>
294+
<td>9.0 GB</td>
295+
</tr>
296+
<tr>
297+
<td>int8</td>
298+
<td>ctranslate2</td>
299+
<td>9.0 GB</td>
300+
</tr>
301+
</table>
302+
303+
<p>For example, let's say you only have 12 GB of VRAM. This allows you to run an "int8_float32" quantization of
304+
a model converted with ctranslate versus only a "Q6_0" version converted using ggml/gguf. This is huge.
305+
You can easily see from the above table that the model quantized to int8_float32 using Ctranslate2 uses even
306+
LESS MEMORY than the much lower quality Q3_k_s ggml/gguf converstion of the same model!</p>
307+
308+
<ul>
309+
<li>Ctranslate2 has numerous other benefits as well including but not limited to:</li>
310+
<ul>
311+
<li>Automatically choosing the next best quantization level if what you choose isn't supported by
312+
your CPU/GPU</li>
313+
<li>Having built-in CPU acceleration in the form of MKL (Intel's Math Kernel Library)</li>
314+
<li>Allowing a user to download a single model and then switching between whatever quantizations
315+
you want at runtime. GGML/GGUF/GPTQ all require you to download a separate model for each quantization.</li>
316+
</ul>
317+
</ul>
318+
</section>
319+
320+
<section>
321+
<h2 style="color: #f0f0f0;">But is the Quality Loss Noticeable?</h2>
322+
323+
<p>Yes. Anyone who's played with LLMs or embedding models knows that there's a significant loss in quality
324+
between, say, a Q8_0 and Q3_k_m model. One way to measure this is by analyzing the "perplexity" of a model.</p>
325+
</section>
326+
327+
<img src="perplexity_loss.png" alt="Perplexity Loss">
328+
<section>
329+
<h2 style="color: #f0f0f0;">So Why isn't Everybody using Ctranslate2?</h2>
330+
331+
<p>This is only my opinion, but the primary reason is because the documentation for Ctranslate2 is written
332+
"by programmers for programmers" and can be difficult to understand and there aren't many examples out there.</p>
333+
</main>
334+
335+
<footer>
336+
<nav><a href="http://www.chintellalaw.com" target="_blank">www.chintellalaw.com</a></nav>
337+
</footer>
338+
</body>
339+
</html>

‎User_Manual/perplexity_loss.png‎

48.1 KB
Loading

‎check_gpu.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
import sys
2+
from PySide6.QtWidgets import QApplication, QMessageBox
3+
import torch
4+
5+
def display_info():
6+
app = QApplication(sys.argv)
7+
info_message = ""
8+
9+
if torch.cuda.is_available():
10+
info_message += "CUDA is available!\n"
11+
info_message += "CUDA version: {}\n\n".format(torch.version.cuda)
12+
else:
13+
info_message += "CUDA is not available.\n\n"
14+
15+
if torch.backends.mps.is_available():
16+
info_message += "Metal/MPS is available!\n\n"
17+
else:
18+
info_message += "Metal/MPS is not available.\n\n"
19+
20+
info_message += "If you want to check the version of Metal and MPS on your macOS device, you can go to \"About This Mac\" -> \"System Report\" -> \"Graphics/Displays\" and look for information related to Metal and MPS.\n\n"
21+
22+
if torch.version.hip is not None:
23+
info_message += "ROCm is available!\n"
24+
info_message += "ROCm version: {}\n".format(torch.version.hip)
25+
else:
26+
info_message += "ROCm is not available.\n"
27+
28+
msg_box = QMessageBox(QMessageBox.Information, "GPU Acceleration Available?", info_message)
29+
msg_box.exec()
30+
31+
if __name__ == "__main__":
32+
display_info()

‎choose_documents.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import os
2+
import shutil
3+
from PySide6.QtWidgets import QApplication, QFileDialog
4+
5+
def choose_documents_directory():
6+
current_dir = os.path.dirname(os.path.realpath(__file__))
7+
docs_folder = os.path.join(current_dir, "Docs_for_DB")
8+
file_dialog = QFileDialog()
9+
file_dialog.setFileMode(QFileDialog.ExistingFiles)
10+
file_paths, _ = file_dialog.getOpenFileNames(None, "Choose Documents for Database", current_dir)
11+
12+
if file_paths:
13+
if not os.path.exists(docs_folder):
14+
os.mkdir(docs_folder)
15+
16+
for file_path in file_paths:
17+
shutil.copy(file_path, docs_folder)
18+
19+
if __name__ == '__main__':
20+
app = QApplication([])
21+
choose_documents_directory()
22+
app.exec()

0 commit comments

Comments
 (0)