External benchmarks for older solc versions #15561
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Benchmark resultsValues averaged over 2 runs. Compilation time
Peak memory use
|
Plots of all dataHere are plots showing all the info from the tables. First compilation time. On two separate plots, because the slower projects dwarf everything else: I did not bother with that for memory, because there is nothing interesting happening there. Looks like the inclusion of unnecessary JSON artifacts in 0.8.21 made it explode for a few versions, but other than that memory use is not a significant factor. Plots of relevant dataAnd here's a subset that's actually worth showing:
Generating the plotsIn case there's a need to adjust those, here's the Python script I used to create them (requires matplotlib). Script used to generate the plotsimport matplotlib.pyplot as plt
import numpy as np
time = [
["version", "OpenZeppelin 5.0.2", "OpenZeppelin 4.9.0", "OpenZeppelin 4.8.0", "OpenZeppelin 4.7.0", "Uniswap 2022-06-16", "Liquity 2024-10-30", "Uniswap 2024-06-06", "Eigenlayer 0.3.0"],
[ "0.8.10", None, 100, 139, 118, None, None, None, None],
[ "0.8.11", None, 92, 138, 117, None, None, None, None],
[ "0.8.12", None, 93, 136, 122, 64, None, None, None],
[ "0.8.13", None, 102, 144, 123, 63, None, None, None],
[ "0.8.14", None, 101, 140, 122, 60, None, None, None],
[ "0.8.15", None, None, 154, 142, 73, None, None, None],
[ "0.8.16", None, None, 138, 126, 68, None, None, None],
[ "0.8.17", None, None, 139, 123, 68, None, None, None],
[ "0.8.18", None, None, 137, 120, 64, None, None, 2958],
[ "0.8.19", None, None, 138, 122, 63, None, None, 2896],
[ "0.8.20", None, None, 142, 120, 64, None, None, 2738],
[ "0.8.21", None, None, 114, 100, 51, None, None, 2282],
[ "0.8.22", None, None, 118, 103, 52, None, None, 2253],
[ "0.8.23", None, None, 112, 102, 53, None, None, 2254],
[ "0.8.24", None, None, 112, 96, 50, 124, 424, 2194],
[ "0.8.25", None, None, 113, 96, 51, 124, 433, 2270],
[ "0.8.26", 38, 46, 70, 63, 28, 69, 236, 1227],
[ "0.8.27", 37, 46, 79, 67, 23, 52, 158, 684],
[ "0.8.28", 33, 44, 69, 63, 20, 50, 134, 560],
]
memory = [
["version", "OpenZeppelin 5.0.2", "OpenZeppelin 4.9.0", "OpenZeppelin 4.8.0", "OpenZeppelin 4.7.0", "Uniswap 2022-06-16", "Liquity 2024-10-30", "Uniswap 2024-06-06", "Eigenlayer 0.3.0"],
[ "0.8.10", None, 558, 660, 512, None, None, None, None],
[ "0.8.11", None, 557, 660, 515, None, None, None, None],
[ "0.8.12", None, 556, 663, 524, 274, None, None, None],
[ "0.8.13", None, 566, 668, 527, 277, None, None, None],
[ "0.8.14", None, 582, 684, 526, 278, None, None, None],
[ "0.8.15", None, None, 716, 578, 307, None, None, None],
[ "0.8.16", None, None, 716, 576, 305, None, None, None],
[ "0.8.17", None, None, 717, 578, 306, None, None, None],
[ "0.8.18", None, None, 706, 576, 304, None, None, 4892],
[ "0.8.19", None, None, 707, 576, 304, None, None, 4758],
[ "0.8.20", None, None, 726, 593, 314, None, None, 4882],
[ "0.8.21", None, None, 2049, 1722, 808, None, None, 22200],
[ "0.8.22", None, None, 2053, 1726, 811, None, None, 22270],
[ "0.8.23", None, None, 2053, 1725, 811, None, None, 22269],
[ "0.8.24", None, None, 2053, 1726, 812, 1766, 5244, 22272],
[ "0.8.25", None, None, 2048, 1730, 817, 1747, 5202, 22122],
[ "0.8.26", 1074, 1240, 1774, 1490, 712, 1524, 4478, 19011],
[ "0.8.27", 1218, 1420, 2058, 1743, 789, 1686, 4806, 20368],
[ "0.8.28", 505, 580, 736, 584, 304, 675, 1498, 4464],
]
def column(array, selected_column_index):
return [
item
for row_index, row in enumerate(array)
for column_index, item in enumerate(row)
if column_index == selected_column_index and row_index > 0
]
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Compilation time of benchmarked projects across compiler versions (smaller projects)")
axes.set_xlabel("solc version")
axes.set_ylabel("Compilation time (seconds)")
for y_column in range(1, len(time[0]) - 2):
plot = axes.plot(column(time, 0), column(time, y_column), label=time[0][y_column])
axes.set_ylim(bottom=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-smaller-projects.svg')
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Compilation time of benchmarked projects across compiler versions (bigger projects)")
axes.set_xlabel("solc version")
axes.set_ylabel("Compilation time (seconds)")
for y_column in range(len(time[0]) - 2, len(time[0])):
plot = axes.plot(column(time, 0), column(time, y_column), label=time[0][y_column])
axes.set_ylim(bottom=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-bigger-projects.svg')
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Peak memory use during compilation of benchmarked projects across compiler versions")
axes.set_xlabel("solc version")
axes.set_ylabel("Peak memory (MiB)")
for y_column in range(1, len(memory[0])):
plot = axes.plot(column(memory, 0), column(memory, y_column), label=memory[0][y_column])
axes.set_ylim(bottom=0)
axes.legend()
axes.grid()
plt.savefig('peak-memory-use.svg')
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Compilation time of benchmarked projects across compiler versions (part 1)")
axes.set_xlabel("solc version")
axes.set_ylabel("Compilation time (seconds)")
plot = axes.plot(column(time, 0), column(time, 3), label="OpenZeppelin")
plot = axes.plot(column(time, 0), column(time, 5), label="Uniswap")
axes.set_ylim(bottom=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-oz-uniswap.svg')
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Compilation time of benchmarked projects across compiler versions (part 2)")
axes.set_xlabel("solc version")
axes.set_ylabel("Compilation time (seconds)")
plot = axes.plot(column(time, 0), column(time, 8), label="Eigenlayer")
axes.set_ylim(bottom=0)
axes.set_xlim(left=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-eigenlayer.svg')
plt.show() |
SummaryOverall impactOverall we can summarize that compared to 0.8.18 compilation time went down by:
Or, compared to 0.8.15, which seems to have been the slowest:
This is of course for older versions of projects, which should be noted:
But this is in line with what we're seeing in incomplete data for newer versions. If anything, the newer versions seem to benefit more from these changes. Plots vs changelogThe plots seem to mostly align with the entries from the changelog that I expected to have the biggest impact:
There were a lot of smaller optimizations in optimizer steps before 0.8.21, but I'm not sure they are all listed in the changelog. Surprisingly, their impact seems to have been either very limited or overshadowed by other changes, because, at least based on the data we have here, the compilation time only started going significantly down with 0.8.21. |
cameel
force-pushed
the
external-benchmarks-for-older-solc
branch
3 times, most recently
from
e2dc74c
to
43f97b1
Compare
- Also use explicit versions for all projects
cameel
force-pushed
the
external-benchmarks-for-older-solc
branch
from
43f97b1
to
249c335
Compare
|
We could have a plot with the speedup with respect to baseline (oldest version), so |
clonker
approved these changes
Nov 1, 2024
|
Here's an attempt at the relative plots. Turned out surprisingly well despite the fact that we don't have a common baseline. I guess that's because the plots do mostly line up in the middle part.
Script used to generate the plots# Uses imports, data and `column()` from the previous script.
def make_relative(values, reference_value):
return [
round(value / reference_value * 100) if value is not None else None
for value
in values
]
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Change in compilation time of benchmarked projects across compiler versions, compared to solc 0.8.18")
axes.set_xlabel("solc version")
axes.set_ylabel("Compilation time (% of 0.8.18 time)")
for y_column in [4, 5, 8]:
x_values = column(time, 0)
y_values = column(time, y_column)
reference_value = y_values[x_values.index("0.8.18")]
plot = axes.plot(x_values, make_relative(y_values, reference_value), label=time[0][y_column])
axes.set_ylim(bottom=0)
axes.set_xlim(left=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-change-vs-0.8.18.svg')
(figure, axes) = plt.subplots(figsize=(20, 10))
axes.set_title("Change in compilation time of benchmarked projects across compiler versions (normalized)")
axes.set_ylabel("Compilation time (% of worst time)")
for y_column in [4, 5, 8]:
x_values = column(time, 0)
absolute_y_values = column(time, y_column)
reference_value = max(x for x in absolute_y_values if x is not None)
relative_y_values = make_relative(absolute_y_values, reference_value)
plot = axes.plot(x_values, relative_y_values, label=time[0][y_column])
first_non_empty_index = next(i for i, v in enumerate(absolute_y_values) if v is not None)
axes.annotate(
f"{absolute_y_values[first_non_empty_index]} s",
(x_values[first_non_empty_index], relative_y_values[first_non_empty_index])
)
axes.set_ylim(bottom=0)
axes.set_xlim(left=0)
axes.legend()
axes.grid()
plt.savefig('compilation-time-change-normalized.svg')
plt.show() |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds benchmarks for older OpenZeppelin and Uniswap versions, which can be compiled by older solc versions (down to 0.8.10 in some cases).
I also tried to quickly find some new projects that would compile with a larger range of solc versions. Two seemed promising: liquity and farcaster. Unfortunately they did not prove useful for this. They may be useful in general though so I'm still adding them.
Finally, the PR refactors the setup script to make it less repetitive, especially when using multiple versions of the same project.