Skip to content

Commit 03fe256

Browse files
committed
bug #1669 Fix the E2E tests workflow (javiereguiluz)
This PR was merged into the main branch. Discussion ---------- Fix the E2E tests workflow Fixes #1662 The daily E2E workflow has been failing every day since June 26. This PR fixes the root cause and makes two improvements to the workflow. **Why the tests were failing** Only the Test Composer Create Project job failed: the install worked, bin/console about worked, the server started, but the first request to the homepage returned 502 Bad Gateway. The root cause is a PHP-FPM segfault caused by the opcache JIT: ``` php-fpm8.5[5018]: segfault at 7f273e598ffc ip 00005581c66e35a6 ... in php-fpm8.5 ``` The conditions that had to combine: * This job is the only one using php-version: latest (currently PHP 8.5.7); the other jobs pin PHP 8.4 and kept passing. * setup-php enables the opcache JIT by default (opcache.jit=1235, 256M buffer). Stock PHP builds (official Docker images, sury packages, Homebrew) ship with the JIT disabled, which is why this was not reproducible outside GitHub runners * PHP 8.5.7's JIT crashes on a hot code path in Symfony 8.1. Failures started exactly when demo v3.1.0 (the Symfony 8.1 upgrade) was released on June 25: the last green run installed v3.0.2/Symfony 8.0 on the same PHP 8.5.7. PHP 8.4's JIT is not affected. The crashed FPM worker made the Symfony CLI proxy return 502; an immediate retry succeeded (fresh worker), which is how the crash was narrowed down. **Changes** * Disable the opcache JIT in the composer job (ini-values: opcache.jit=disable) * Add a "Show diagnostics on failure" step to each test job: PHP version and key ini values, a retry of the request (to tell first-request-only failures from persistent ones), the application logs and the kernel messages (dmesg, which is what exposed the segfault). Until now the workflow captured nothing on failure, making issues like this one invisible. * Create at most one E2E failure issue at a time: previously the workflow created one issue per day (see #1663#1668, all closed manually). Now, if any issue whose title starts with E2E Test Failure is still open, the workflow doesn't create a new issue. Commits ------- 7e7f61b Fix the E2E tests workflow
2 parents 8f84bf8 + 7e7f61b commit 03fe256

1 file changed

Lines changed: 64 additions & 10 deletions

File tree

.github/workflows/end_to_end_tests.yaml

Lines changed: 64 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,23 @@ jobs:
4242
curl --fail --max-time 5 --no-progress-meter --output home.html -v http://127.0.0.1:8000/
4343
symfony server:stop
4444
45+
- name: Show diagnostics on failure
46+
if: ${{ failure() }}
47+
working-directory: ${{ github.job }}
48+
run: |
49+
php -v
50+
php -i | grep -E "memory_limit|opcache.enable|opcache.jit" || true
51+
# retry without --fail to see if the failure is first-request-only
52+
curl -sS --max-time 15 -o /tmp/retry.html -w 'retry: %{http_code} in %{time_total}s\n' http://127.0.0.1:8000/ || true
53+
head -c 300 /tmp/retry.html || true
54+
echo '===== app logs (filtered) ====='
55+
grep -hvE "security.DEBUG|http_client.INFO|cache.INFO" var/log/*.log | tail -n 60 || true
56+
echo '===== symfony CLI / PHP-FPM logs ====='
57+
find "$HOME" -maxdepth 6 -type d -name "*symfony5*" 2>/dev/null || true
58+
for f in $(find "$HOME/.symfony5" "${XDG_DATA_HOME:-$HOME/.local/share}/symfony5" "${XDG_CONFIG_HOME:-$HOME/.config}/symfony5" -type f -name "*.log" 2>/dev/null); do echo "--- $f"; cat "$f"; done
59+
echo '===== kernel messages (crashes / OOM) ====='
60+
sudo dmesg 2>/dev/null | tail -n 30 || true
61+
4562
test-composer-create-project:
4663
name: Test Composer Create Project
4764
runs-on: ubuntu-latest
@@ -52,6 +69,12 @@ jobs:
5269
with:
5370
coverage: none
5471
extensions: none, ctype, dom, iconv, intl, mbstring, pdo_sqlite, simplexml, tokenizer, xml, xmlwriter
72+
# setup-php enables the opcache JIT by default, but stock PHP disables it.
73+
# The tracing JIT crashes PHP-FPM while serving the first request of a
74+
# Symfony 8.1 app (reproduced with PHP 8.4 and 8.5, on x86_64 and arm64),
75+
# which made this job fail with a 502 error. Disable the JIT to test the
76+
# default PHP configuration that users actually run.
77+
ini-values: opcache.jit=disable
5578
php-version: latest
5679
tools: symfony-cli
5780

@@ -66,6 +89,23 @@ jobs:
6689
curl --fail --max-time 5 --no-progress-meter --output home.html -v http://127.0.0.1:8000/
6790
symfony server:stop
6891
92+
- name: Show diagnostics on failure
93+
if: ${{ failure() }}
94+
working-directory: ${{ github.job }}
95+
run: |
96+
php -v
97+
php -i | grep -E "memory_limit|opcache.enable|opcache.jit" || true
98+
# retry without --fail to see if the failure is first-request-only
99+
curl -sS --max-time 15 -o /tmp/retry.html -w 'retry: %{http_code} in %{time_total}s\n' http://127.0.0.1:8000/ || true
100+
head -c 300 /tmp/retry.html || true
101+
echo '===== app logs (filtered) ====='
102+
grep -hvE "security.DEBUG|http_client.INFO|cache.INFO" var/log/*.log | tail -n 60 || true
103+
echo '===== symfony CLI / PHP-FPM logs ====='
104+
find "$HOME" -maxdepth 6 -type d -name "*symfony5*" 2>/dev/null || true
105+
for f in $(find "$HOME/.symfony5" "${XDG_DATA_HOME:-$HOME/.local/share}/symfony5" "${XDG_CONFIG_HOME:-$HOME/.config}/symfony5" -type f -name "*.log" 2>/dev/null); do echo "--- $f"; cat "$f"; done
106+
echo '===== kernel messages (crashes / OOM) ====='
107+
sudo dmesg 2>/dev/null | tail -n 30 || true
108+
69109
test-git-clone-installation:
70110
name: Test Git Clone Installation
71111
runs-on: ubuntu-latest
@@ -93,6 +133,23 @@ jobs:
93133
curl --fail --max-time 5 --no-progress-meter --output home.html -v http://127.0.0.1:8000/
94134
symfony server:stop
95135
136+
- name: Show diagnostics on failure
137+
if: ${{ failure() }}
138+
working-directory: ${{ github.job }}
139+
run: |
140+
php -v
141+
php -i | grep -E "memory_limit|opcache.enable|opcache.jit" || true
142+
# retry without --fail to see if the failure is first-request-only
143+
curl -sS --max-time 15 -o /tmp/retry.html -w 'retry: %{http_code} in %{time_total}s\n' http://127.0.0.1:8000/ || true
144+
head -c 300 /tmp/retry.html || true
145+
echo '===== app logs (filtered) ====='
146+
grep -hvE "security.DEBUG|http_client.INFO|cache.INFO" var/log/*.log | tail -n 60 || true
147+
echo '===== symfony CLI / PHP-FPM logs ====='
148+
find "$HOME" -maxdepth 6 -type d -name "*symfony5*" 2>/dev/null || true
149+
for f in $(find "$HOME/.symfony5" "${XDG_DATA_HOME:-$HOME/.local/share}/symfony5" "${XDG_CONFIG_HOME:-$HOME/.config}/symfony5" -type f -name "*.log" 2>/dev/null); do echo "--- $f"; cat "$f"; done
150+
echo '===== kernel messages (crashes / OOM) ====='
151+
sudo dmesg 2>/dev/null | tail -n 30 || true
152+
96153
notify-on-failure:
97154
if: ${{ always() && contains(needs.*.result, 'failure') }}
98155
name: Notify on Failure
@@ -151,17 +208,14 @@ jobs:
151208
});
152209
}
153210
154-
// Reuse an open issue for today if it already exists to avoid duplicates
211+
// Skip if any E2E failure issue is already open, to avoid daily noise
155212
const { data: issues } = await github.rest.issues.listForRepo({
156-
owner, repo, state: 'open', labels: labelName, per_page: 100
213+
owner, repo, state: 'open', per_page: 100
157214
});
158-
const existing = issues.find(i => i.title === title);
159-
215+
const existing = issues.find(i => i.title.startsWith('E2E Test Failure'));
160216
if (existing) {
161-
await github.rest.issues.createComment({
162-
owner, repo, issue_number: existing.number,
163-
body: `Another failing run detected.\n\n${body}`
164-
});
165-
} else {
166-
await github.rest.issues.create({ owner, repo, title, body, labels: [labelName] });
217+
core.info(`Issue #${existing.number} about E2E failures is still open; not creating another one.`);
218+
return;
167219
}
220+
221+
await github.rest.issues.create({ owner, repo, title, body, labels: [labelName] });

0 commit comments

Comments
 (0)