You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since #112 the access log records per request latency, and since #113 the requests that exhaust the solver are kept. Both are deployed. This is what they show.
About 2.6 percent of POST /optimize/charge-schedule calls run into OPTIMIZER_TIME_LIMIT and return whatever CBC has at 20 seconds. The p50 is around 0.5 s, so the p99 is not a slow median, it is a cliff: 495 of 19006 calls over a two hour window, and the p99 sits at 20.5 s in every five minute bucket.
What the slow requests look like
293 captured requests, compared against the 19 golden cases as the only fast reference available:
feature
slow median
golden median
distinct import prices, share of steps
0.007
0.200
longest run of identical prices
141
5
steps
192
35
headroom to s_max, share
0.20
0.76
first dt
504 s
1944 s
uniform dt
0 %
26 %
192 of the 293 have exactly one distinct import price across the whole horizon, 216 have two or fewer, and the export price is single valued in 256. The typical shape is a fixed tariff, a 33 to 35 hour horizon at 900 s with a partial first interval, and two batteries that are already close to full.
Battery symmetry is not involved: only 10 percent have two or more identical batteries, and that subset is not slower.
Why they are slow
The tie breakers, not the physics and not the horizon. Same captured request, same 7794 rows and 1732 integer variables, only charging_strategy swapped:
strategy
solve
none
0.39 s
attenuate_demand_peaks
1.55 s
attenuate_feedin_peaks
1.96 s
attenuate_grid_peaks
7.90 s
charge_before_export
28.60 s
A 433 step request, 4.5 days of horizon, solves in 0.39 s with no strategy and takes 73 times as long with a cost neutral one. On a second request, 192 steps and 4 batteries, none is 4.90 s and attenuate_grid_peaks is 8.62 s.
The two findings combine into one mechanism. With a constant price every schedule that moves the same energy costs the same, so the optimum is a plateau rather than a point. The only thing that distinguishes vertices of that plateau is the strategy terms, whose coefficients are around min_import_price * 1e-6, below CBC's optimality and integrality tolerances. The solver cannot prune with bounds and walks the plateau until the time limit. The golden cases have genuinely varying prices, which is why the suite has never caught this.
Model building is not a factor, 0.05 s for the largest captured model.
What does not fix it
A relative gap.gapRel 1e-6 takes one request from 29.7 s to 1.5 s for an objective difference of 0.000037 percent, but the schedule moves by up to 2760 Wh on 53 to 97 individual steps. That is precisely the tie breaking the strategies exist to encode, so the speed is bought by discarding the feature.
Breaking the price ties numerically. A deterministic tilt on p_N responds erratically: one request goes 28.8 s to 5.8 s at 1e-6, another goes 8.4 s to 9.7 s at 1e-4 and only improves at 1e-2. Not a dependable lever, and it changes the economics.
Solve in two stages instead of encoding the tie breaking below solver tolerance:
economics only, no strategy terms, which is the 0.39 s solve
pin the economic optimum as a constraint and optimise the strategy terms over that face
That keeps the strategies exactly cost neutral, which is their contract, while giving CBC an objective it can actually prune on in both stages. The cheaper alternative is to raise the strategy coefficients above CBC's tolerance and accept a small bounded economic loss, but that gives up cost neutrality.
Reproducing
make slow collects the dumps from every replica. Each line is one request that hit the limit, shaped like test_cases/*.json so it replays directly. Note the file is per replica and ephemeral, and az containerapp exec leaks a NUL every few KiB into the stream, which make slow strips.
Worth capturing shape counters on every request, not only the slow ones: there is currently no baseline distribution, so the share of all traffic that is a flat tariff on a long horizon is unknown.
Since #112 the access log records per request latency, and since #113 the requests that exhaust the solver are kept. Both are deployed. This is what they show.
About 2.6 percent of
POST /optimize/charge-schedulecalls run intoOPTIMIZER_TIME_LIMITand return whatever CBC has at 20 seconds. The p50 is around 0.5 s, so the p99 is not a slow median, it is a cliff: 495 of 19006 calls over a two hour window, and the p99 sits at 20.5 s in every five minute bucket.What the slow requests look like
293 captured requests, compared against the 19 golden cases as the only fast reference available:
s_max, sharedtdt192 of the 293 have exactly one distinct import price across the whole horizon, 216 have two or fewer, and the export price is single valued in 256. The typical shape is a fixed tariff, a 33 to 35 hour horizon at 900 s with a partial first interval, and two batteries that are already close to full.
Battery symmetry is not involved: only 10 percent have two or more identical batteries, and that subset is not slower.
Why they are slow
The tie breakers, not the physics and not the horizon. Same captured request, same 7794 rows and 1732 integer variables, only
charging_strategyswapped:noneattenuate_demand_peaksattenuate_feedin_peaksattenuate_grid_peakscharge_before_exportA 433 step request, 4.5 days of horizon, solves in 0.39 s with no strategy and takes 73 times as long with a cost neutral one. On a second request, 192 steps and 4 batteries,
noneis 4.90 s andattenuate_grid_peaksis 8.62 s.The two findings combine into one mechanism. With a constant price every schedule that moves the same energy costs the same, so the optimum is a plateau rather than a point. The only thing that distinguishes vertices of that plateau is the strategy terms, whose coefficients are around
min_import_price * 1e-6, below CBC's optimality and integrality tolerances. The solver cannot prune with bounds and walks the plateau until the time limit. The golden cases have genuinely varying prices, which is why the suite has never caught this.Model building is not a factor, 0.05 s for the largest captured model.
What does not fix it
gapRel1e-6 takes one request from 29.7 s to 1.5 s for an objective difference of 0.000037 percent, but the schedule moves by up to 2760 Wh on 53 to 97 individual steps. That is precisely the tie breaking the strategies exist to encode, so the speed is bought by discarding the feature.p_Nresponds erratically: one request goes 28.8 s to 5.8 s at 1e-6, another goes 8.4 s to 9.7 s at 1e-4 and only improves at 1e-2. Not a dependable lever, and it changes the economics.Suggested direction
Solve in two stages instead of encoding the tie breaking below solver tolerance:
That keeps the strategies exactly cost neutral, which is their contract, while giving CBC an objective it can actually prune on in both stages. The cheaper alternative is to raise the strategy coefficients above CBC's tolerance and accept a small bounded economic loss, but that gives up cost neutrality.
Reproducing
make slowcollects the dumps from every replica. Each line is one request that hit the limit, shaped liketest_cases/*.jsonso it replays directly. Note the file is per replica and ephemeral, andaz containerapp execleaks a NUL every few KiB into the stream, whichmake slowstrips.Worth capturing shape counters on every request, not only the slow ones: there is currently no baseline distribution, so the share of all traffic that is a flat tariff on a long horizon is unknown.