The axe element will be the same each time, but you can do that without any AI shenanigans - just run it in a normal Playwright test.
My question was really about the page interactions and the assertions being driven by AI: if they are going to be generating different code every time the test runs, how can you have any confidence in the test not having false positives and false negatives at least some of the time, unless you read the generated script each time?
That sounds like a lot more work than just writing the test once in the traditional way (codegen or manually) and tweaking it only when there's a breaking change to the page.
If people are genuinely using this approach then there must be something I'm missing.
We are going to be adding the ability to trigger more actions (beyond the normal clicks/keyboard) without AI by using the in-browser control panel. We wanted to add it for this ShowHN, but we ran out of time on our self-imposed deadline. :(
Regarding variability of flows, you can cement a given flow by pressing the `rerun` button. That takes AI out of the driver's seat and the flow will rerun the set of actions decided on in the original flow as if it's on rails.
Regarding creating a test manually, that will be a best fit for pages that have consistent selector logic for elements, though we found that as soon as a page starts randomizing element IDs, this approach starts to struggle. We get around this by creating a prioritized list of selectors for every action that touches the DOM, so that if `document.querySelector("#shenanigansId")` fails, the run can still continue by choosing the next-best selector, and so on. Thankfully this logic requires no AI at all, though it is heuristic at the end of the day.
The Playwright dev team would probably say that you should avoid using IDs as selectors and instead favour the use of selectors based on user-visible aspects, e.g. "a link including the text 'cat'" or "a button with the label 'register now'". That way your tests are immune to under-the-hood changes the user is oblivious to. The range of selector options (locators in their world) are a real strength of Playwright.
My question was really about the page interactions and the assertions being driven by AI: if they are going to be generating different code every time the test runs, how can you have any confidence in the test not having false positives and false negatives at least some of the time, unless you read the generated script each time?
That sounds like a lot more work than just writing the test once in the traditional way (codegen or manually) and tweaking it only when there's a breaking change to the page.
If people are genuinely using this approach then there must be something I'm missing.