chore: limit symdb uploaders under spawn #15140
Conversation
|
|
P403n1x87
force-pushed
the
refactor/symdb-cap-uploads-via-ipc
branch
2 times, most recently
from
77ab2c8 to
3263680
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 245 ± 5 ms. The average import time from base is: 250 ± 10 ms. The import time difference between this PR and base is: -9.3 ± 0.5 ms. Import time breakdownThe following import paths have appeared:
|
Performance SLOsComparing candidate refactor/symdb-cap-uploads-via-ipc (2030061) with baseline main (81acf6d) 🟡 Near SLO Breach (5 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 20.457ms (SLO: <22.300ms -8.3%) vs baseline: +0.1% Memory: ✅ 66.032MB (SLO: <67.000MB 🟡 -1.4%) vs baseline: +4.9% ✅ exception-replay-enabledTime: ✅ 1.341ms (SLO: <1.450ms -7.5%) vs baseline: +0.2% Memory: ✅ 64.295MB (SLO: <67.000MB -4.0%) vs baseline: +5.0% ✅ iastTime: ✅ 20.453ms (SLO: <22.250ms -8.1%) vs baseline: -0.1% Memory: ✅ 65.998MB (SLO: <67.000MB 🟡 -1.5%) vs baseline: +4.8% ✅ profilerTime: ✅ 15.556ms (SLO: <16.550ms -6.0%) vs baseline: +0.2% Memory: ✅ 54.009MB (SLO: <54.500MB 🟡 -0.9%) vs baseline: +4.7% ✅ resource-renamingTime: ✅ 20.538ms (SLO: <21.750ms -5.6%) vs baseline: -0.1% Memory: ✅ 66.034MB (SLO: <67.000MB 🟡 -1.4%) vs baseline: +4.8% ✅ span-code-originTime: ✅ 25.378ms (SLO: <28.200ms 📉 -10.0%) vs baseline: -0.3% Memory: ✅ 67.583MB (SLO: <69.500MB -2.8%) vs baseline: +5.1% ✅ tracerTime: ✅ 20.492ms (SLO: <21.750ms -5.8%) vs baseline: ~same Memory: ✅ 65.953MB (SLO: <67.000MB 🟡 -1.6%) vs baseline: +4.7% ✅ tracer-and-profilerTime: ✅ 22.684ms (SLO: <23.500ms -3.5%) vs baseline: -0.3% Memory: ✅ 67.850MB (SLO: <68.000MB 🟡 -0.2%) vs baseline: +4.6% ✅ tracer-dont-create-db-spansTime: ✅ 19.287ms (SLO: <21.500ms 📉 -10.3%) vs baseline: -0.7% Memory: ✅ 66.081MB (SLO: <67.000MB 🟡 -1.4%) vs baseline: +4.8% ✅ tracer-minimalTime: ✅ 16.668ms (SLO: <17.500ms -4.8%) vs baseline: +0.4% Memory: ✅ 66.031MB (SLO: <67.000MB 🟡 -1.4%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 20.497ms (SLO: <21.750ms -5.8%) vs baseline: +0.4% Memory: ✅ 67.987MB (SLO: <72.500MB -6.2%) vs baseline: +4.8% ✅ tracer-no-cachesTime: ✅ 18.440ms (SLO: <19.650ms -6.2%) vs baseline: -0.5% Memory: ✅ 66.176MB (SLO: <67.000MB 🟡 -1.2%) vs baseline: +5.1% ✅ tracer-no-databasesTime: ✅ 18.756ms (SLO: <20.100ms -6.7%) vs baseline: -0.3% Memory: ✅ 65.814MB (SLO: <67.000MB 🟡 -1.8%) vs baseline: +4.7% ✅ tracer-no-middlewareTime: ✅ 20.139ms (SLO: <21.500ms -6.3%) vs baseline: +0.1% Memory: ✅ 66.124MB (SLO: <67.000MB 🟡 -1.3%) vs baseline: +4.8% ✅ tracer-no-templatesTime: ✅ 20.281ms (SLO: <22.000ms -7.8%) vs baseline: ~same Memory: ✅ 66.163MB (SLO: <67.000MB 🟡 -1.2%) vs baseline: +4.9% 🟡 errortrackingdjangosimple - 6/6✅ errortracking-enabled-allTime: ✅ 18.062ms (SLO: <19.850ms -9.0%) vs baseline: -0.2% Memory: ✅ 66.071MB (SLO: <66.500MB 🟡 -0.6%) vs baseline: +4.9% ✅ errortracking-enabled-userTime: ✅ 18.097ms (SLO: <19.400ms -6.7%) vs baseline: +0.1% Memory: ✅ 66.037MB (SLO: <66.500MB 🟡 -0.7%) vs baseline: +4.9% ✅ tracer-enabledTime: ✅ 18.092ms (SLO: <19.450ms -7.0%) vs baseline: +0.1% Memory: ✅ 66.023MB (SLO: <66.500MB 🟡 -0.7%) vs baseline: +5.0% 🟡 errortrackingflasksqli - 6/6✅ errortracking-enabled-allTime: ✅ 2.068ms (SLO: <2.300ms 📉 -10.1%) vs baseline: +0.4% Memory: ✅ 52.534MB (SLO: <53.500MB 🟡 -1.8%) vs baseline: +4.8% ✅ errortracking-enabled-userTime: ✅ 2.067ms (SLO: <2.250ms -8.2%) vs baseline: -0.5% Memory: ✅ 52.534MB (SLO: <53.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ tracer-enabledTime: ✅ 2.063ms (SLO: <2.300ms 📉 -10.3%) vs baseline: -0.3% Memory: ✅ 52.671MB (SLO: <53.500MB 🟡 -1.5%) vs baseline: +4.9% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 4.599ms (SLO: <4.750ms -3.2%) vs baseline: +0.2% Memory: ✅ 62.345MB (SLO: <65.000MB -4.1%) vs baseline: +4.8% ✅ appsec-postTime: ✅ 6.640ms (SLO: <6.750ms 🟡 -1.6%) vs baseline: ~same Memory: ✅ 62.322MB (SLO: <65.000MB -4.1%) vs baseline: +4.9% ✅ appsec-telemetryTime: ✅ 4.594ms (SLO: <4.750ms -3.3%) vs baseline: ~same Memory: ✅ 62.347MB (SLO: <65.000MB -4.1%) vs baseline: +4.8% ✅ debuggerTime: ✅ 1.862ms (SLO: <2.000ms -6.9%) vs baseline: -0.2% Memory: ✅ 45.349MB (SLO: <47.000MB -3.5%) vs baseline: +5.2% ✅ iast-getTime: ✅ 1.858ms (SLO: <2.000ms -7.1%) vs baseline: ~same Memory: ✅ 42.064MB (SLO: <49.000MB 📉 -14.2%) vs baseline: +4.6% ✅ profilerTime: ✅ 1.915ms (SLO: <2.100ms -8.8%) vs baseline: +0.3% Memory: ✅ 46.714MB (SLO: <47.000MB 🟡 -0.6%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 3.362ms (SLO: <3.650ms -7.9%) vs baseline: -0.2% Memory: ✅ 52.635MB (SLO: <53.500MB 🟡 -1.6%) vs baseline: +4.9% ✅ tracerTime: ✅ 3.353ms (SLO: <3.650ms -8.1%) vs baseline: ~same Memory: ✅ 52.567MB (SLO: <53.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ tracer-nativeTime: ✅ 3.353ms (SLO: <3.650ms -8.1%) vs baseline: +0.1% Memory: ✅ 54.360MB (SLO: <60.000MB -9.4%) vs baseline: +4.9% 🟡 flasksqli - 6/6✅ appsec-enabledTime: ✅ 3.961ms (SLO: <4.200ms -5.7%) vs baseline: -0.3% Memory: ✅ 62.344MB (SLO: <66.000MB -5.5%) vs baseline: +4.9% ✅ iast-enabledTime: ✅ 2.449ms (SLO: <2.800ms 📉 -12.5%) vs baseline: ~same Memory: ✅ 58.825MB (SLO: <60.000MB 🟡 -2.0%) vs baseline: +4.9% ✅ tracer-enabledTime: ✅ 2.059ms (SLO: <2.250ms -8.5%) vs baseline: -0.7% Memory: ✅ 52.632MB (SLO: <54.500MB -3.4%) vs baseline: +4.9%
|
P403n1x87
force-pushed
the
refactor/symdb-cap-uploads-via-ipc
branch
from
3263680 to
ab746d4
Compare
P403n1x87
added
the
changelog/no-changelog
P403n1x87
force-pushed
the
refactor/symdb-cap-uploads-via-ipc
branch
3 times, most recently
from
ce33357 to
c79a0d2
Compare
We use file-based IPC to ensure that Symbol DB has as most 2 active uploader processes under more general circumstances than fork, such as spawn.
P403n1x87
force-pushed
the
refactor/symdb-cap-uploads-via-ipc
branch
4 times, most recently
from
97af775 to
89b66f2
Compare
P403n1x87
force-pushed
the
refactor/symdb-cap-uploads-via-ipc
branch
from
89b66f2 to
432f73d
Compare
|
|
||
| def __init__(self) -> None: | ||
| self.filename: typing.Optional[str] = str(TMPDIR / secrets.token_hex(8)) if TMPDIR is not None else None | ||
| def __init__(self, name: typing.Optional[str] = None) -> None: |
There was a problem hiding this comment.
I'm worried about how this will behave with an unwritable temp directory, for example some folks use this setting in K8s:
securityContext:
readOnlyRootFilesystem: true
There was a problem hiding this comment.
Have we considered an anonymous mmap? It might fail on Windows in certain forking scenarios, but if we always fallback to 0 in those cases it's probably safer?
m = mmap.mmap(-1, SIZE, tagname=f"Global\\{name}")
There was a problem hiding this comment.
Yes this will definitely fail in some configurations, but it is the solution that other parts of the library are currently using the same IPC solution. I think the long-term solution is a more robust cross-platform IPC feature which we currently don't have 🙁
There was a problem hiding this comment.
Makes sense, it's not a big deal since the fallback just means we're performing the same behavior as we are today. You might want to make a comment that it could fail in restricted environments (it's somewhat common because it can be considered a K8s best-practice) so no mission-critical pieces uses it in the interim.
Description
We use file-based IPC to ensure that Symbol DB has as most 2 active uploader processes under more general circumstances than fork, such as spawn.