Playwright #18927
Playwright #18927
Conversation
AdrienClairembault
force-pushed
the
playwright
branch
8 times, most recently
from
5c5bc1a to
48aafcd
Compare
|
Don't worry about sonar cloud failure, it report a new OS command execution but it should be safe given its just a call to glpi's console:
|
Iv'e marked it as safe |
AdrienClairembault
force-pushed
the
playwright
branch
from
479b84b to
3a81949
Compare
|
Not a fan of using TypeScript. That's both a personal thing and also a worry that it makes writing E2E tests less desirable for the developers that already don't work with JavaScript a lot. Also, working with TypeScript is incredibly frustrating in PHPStorm. There have been too many times I've Ctrl + Click a JS function related to a library to try and see the code for it, only to be directed to the useless .d.ts file for it. Same issue started happening when TS files were being added to the Cypress tests. I don't see the benefit of it at all, and only downsides. I voiced my opinion about it before, but that was when we only had a .d.ts file added for Cypress commands all to fix VSCode behavior (which I also don't like but I know others are using it and it was an OK solution to fix autocomplete). |
It is the recommended way to setup playwright. Here is an extract form the documentation:
|
You can add a |
I'll restore the tests but unsure about the reporters. |
Whatever the reporter used, we need to be able to compare metrics. The |
|
Regarding general performances feedback, here is a quick overview on a test that I've migrated this morning. I've made care to migrate it as close as possible to its cypress version (even if it could be optimized a bit more) to make sure it is a good comparison. Here are the two tests:
Running the test on cypress take 5s, with a total 12s execution time (cypress is very slow to initialize - but note that this mater less when running multiple tests as this initialization is only done once).
Comparing with playwright, we get a 3.5s execution time with a 11s total execution time. Note that the 11s total is "penalized" by the mandatory This extra setup takes a good 5s by itself (but we are still a bit faster overall for the total execution time, showcasing that playwright initialization is incredibly fast compared to cypress).
So over one single test, on my laptop, playwright seems faster (3.5s vs 5s). Note that an individual test might appear slower when running things in parallel, as the GLPI's server can get a bit overwhelmed.
This is explained because this test is executed at the very start of the process. This might seems a potential issue but it isn't because the parallelization gain are more important that this initial "speed bump".
Sadly, these parallelization gains are less effective on the CI because we only have 2 threads available. |
|
On a side note, there is also the option of running only a subset of the tests on the CI ("smoke tests") and run the whole suite only on the nightly action and before releases. This seems to be a very common practice, but this require the tests to not be flaky at all (which is not the state of the cypress suite, and too early to say for the playwright suite). |
I don't run tests locally unless I've already pushed to a PR and the CI reports issues, or I am working on specific tests. E2E tests take half an hour, so I push to let the hosted CI run them and switch to another task instead of sitting and waiting for a complete suite to run. |
|
More feedback on the CI execution time, it seems the 15 migrated tests take 1m05 on cypress for 58s on playwright. |
|
I updated the local docker scripts and ran both the Playwright and Cypress tests and then compared the times for all common tests. Cypress took 59.66 seconds while Playwright took 92.73 seconds. That made Playwright 55.4% slower than Cypress just on individual test execution. Rerunning Playwright tests a few times did not noticeably improve test execution time so it isn't likely to be a one-time performance issue. It also didn't seem to solve the reliability issues. Both Cypress and Playwright had tests outright fail all retries at least once while I was comparing and they had to be run a few times to get a full run without failures. None of the 2x slower tests on Playwright were tests that had to be retried. There is quite the difference between what I see locally and in CI for both performance and reliability so maybe I missed something when configuring things to run in the local docker environment? |
|
Do you have any screenshots of the results (how many threads were used ?) and maybe the changes to the tests scripts so I can try it on my side ? For the flakiness, note that if you run the full suite (cypress then playwright), some flakyness from cypress can "leak" into the playwrights tests. For example, if the display preference test from cypress fails, it will create a failure on playwright due to an unexpected "Pending reason" column being displayed:
|
This is something I suspect as well, I feel like it should be much quicker on the CI than the actual result (compared to what I get locally). |
|
Running with 2 thread on a local docker seems reliable on my side and comparable to the execution time of the github actions CI given than I am a laptop CPU (still a lot slower than running locally without docker tho...).
|
All I have is the ctf-report and an HTML report which don't show the threads used. I didn't touch the configuration, but instead just moved the commands run in the CI workflow to scripts like we did with the other test suites. I'm not even sure where the thread count is configured. Even with a single thread, I expected Playwright to be the same or faster than Cypress, at least once the initial setup was done.
To be fair to both frameworks and rule out me missing something with the test runner to handle interactive mode properly, I had the containers recreated every time but also I ran the initial Playwright tests first. I split the old E2E tests and Playwright tests into their own "suites" so running For the CLI output, all I see is: I published my changes to a branch on my fork for you to see what I did. |
|
You can't see the total time and thread infos because I've re-added them only in the Add dot reporter to CI commit which is not yet pulled on your branch. I've hardcoded the workers for the
But as I've said this is still disappointing compared to running it without docker, even with only 2 workers it is much faster:
Not sure what we can do about it. |
|
Oh, ok. How long does it take Playwright to run with a single thread in CI? Seems only fair to compare Cypress and Playwright with the same number of threads. Beyond the ease of running tests in parallel, I thought it was said that Playwright was fundamentally more performant than Cypress. I had it in mind that even on a single thread, Playwright would be the same or faster than Cypress. If it isn't, I wonder if there is something with the test code itself rather than the framework that is causing the slowdown. Or I misunderstood the previous discussions on this subject. |
|
On single thread I get the following results. Cypress (docker): Cypress (without docker): Playwright (docker, 1 worker): Playwright (without docker, 1 worker): So playwright does seem faster in single thread too (by a lesser margin with docker tho). Disclaimer that all tests have not been migrated 100% identically (e.g. I've merged login + logout in playwright to gain time and sometimes tweaked a bit some assertions). |
|
Here are the execution times on my machine, using docker.
|
|
Sound good, probably just a WSL issue on my side then. |
|
Summary for @orthagh as requested in our meeting. Note that I try to stay objective but for some points I have to pick an opinion as there are no objective universal answers. Is playwright faster ? On single thread -> Not much, it is only about 10% faster. On 8 threads -> Yes, it is at least 3x faster than cypress in this case (I suspect this number will even be better when we migrate everything because starting 8 thread has a big "initial cost" so with our small test samples of only 15 tests it artificially slow down the results compared to a real situation where we would run 100+ tests). On the github CI -> Not much, even if we have access to 2 threads the machine does not make a good use of it. On our potential own runner dedicated to E2E tests -> probably great for a reasonable cost (to be verified). Is playwright less flaky ? I didn't encounter flakyness at this time BUT there are not a lot of tests that are migrated so it is too soon to have a good opinion on this. We must also take into account that multi-threading will increase the potential flakyness risk if our tests are not well designed. Are playwright tests easier to develop/debug ? The tests feels much better to write due to a structure that reuse known OOP principles instead of the "magic" cypress command pattern that can lead to confusions and complexity in many cases. Debugging seems better too with the trace system (especially for CI failures). The "UI mode" seems a bit more complicated to understand that the one used by cypress (but I don't use it much personally). There is some dedicated VSCode integration but I didn't test it too much too, I prefer working from the command line. Is playwright more future proof ? It seems so:
Should we use it ? Yes but I still don't think it is an urgent matter. Cypress is "good enough" for now and we need to focus on more important things before doing this full migration to playwright. |
|
As discussed with @orthagh, we put on on hold till next monday meeting. |
Description
Setup playwright and migrate two tests cypress tests files.
It does seems faster locally, the 2 migrated tests take ~15 seconds with 6 threads on my machine (vs 28 seconds for their cypress equivalent on the CI).
On the CI, the improvement seems less impressive (unsure why, it even seems slower).
Maybe there isn't enough tests to properly compare.
Anyway, I've saved a trace of its execution to investigate it next week (the work can still be reviewed).
Regarding the tests themselves, they feel nicer to write.
Cypress was great at the start because it is easy to get started, however as you go on you face many issues when you try to do more complicated things due to its design choices (thenable everywhere that is not a real promise, command pattern, ...).
For that, playwright feels much better IMO.
It requires a bit more time to understand it at the start but after that it get quite smooth due to its design which is way more predictable.