Skip to content

Implement soft assertions#2750

Open
mbroton wants to merge 2 commits intomicrosoft:mainfrom
mbroton:soft-assert
Open

Implement soft assertions#2750
mbroton wants to merge 2 commits intomicrosoft:mainfrom
mbroton:soft-assert

Conversation

@mbroton
Copy link

@mbroton mbroton commented Feb 18, 2025

Related to #1272

This PR adds support for soft assertions, allowing multiple assertion failures to be collected before raising them together. I implemented this using a context manager pattern since it feels more natural and Pythonic:

with expect.soft() as soft_expect:
    await soft_expect(page).to_have_title("wrong title")
    await soft_expect(page.locator("#div1")).to_have_text("wrong")
    response = await page.request.get("...")
    await soft_expect(response).to_be_ok()

Example error report:

>       with expect.soft() as soft_expect:
E       AssertionError: Soft assertion failures
E       1. Page title expected to be 'wrong title'
E       Actual value: Page Title
E       Call log:
E         - PageAssertions.to_have_title with timeout 5000ms
E         -   - waiting for locator(":root")
E         -     9 × locator resolved to <html>…</html>
E         -       - unexpected value "Page Title"
E
E       2. Locator expected to have text 'wrong'
E       Actual value: Text1
E       Call log:
E         - LocatorAssertions.to_have_text with timeout 5000ms
E         -   - waiting for locator("#div1")
E         -     9 × locator resolved to <div id="div1">Text1</div>
E         -       - unexpected value "Text1"
E
E       3. Response status expected to be within [200..299] range, was '404'
E       Call log:
E       → GET http://localhost:2137/non-existent
E         -   user-agent: (...)
E         -   accept: */*
E         -   accept-encoding: gzip,deflate,br
E         - ← 404 Not Found
E         -   content-type: text/plain
E         -   transfer-encoding: chunked

Implementation Details

  • The implementation maintains backward compatibility with all existing assertion methods
  • Both sync and async APIs are fully supported with identical behavior
  • Error messages preserve the order of failures and include clear numbering (1., 2., etc.)
  • Custom error messages are properly handled and included in the final error report
  • The implementation handles nested errors correctly (when other exceptions occur within the soft assertion block)
  • Timeout settings are respected both at the global and individual assertion level

Initially, I used contextlib.contextmanager for expect.soft() as it was a simpler implementation. However, this approach included some unrelevant information in the traceback when assertions failed. To address this, I created a custom SoftAssertionContextManager class that gives us more control over the error handling and traceback presentation.

This is just one way to add soft assertions in Python Playwright. I think the context manager pattern works well with Python, but I’m totally open to other ideas if they fit the project better.

@mbroton
Copy link
Author

mbroton commented Feb 18, 2025

@microsoft-github-policy-service agree

@mbroton
Copy link
Author

mbroton commented Mar 5, 2025

I reviewed the failing checks and also tried to reproduce some of them locally without success. I couldn't find any indication that these failures are connected to my changes, so it seems they might be flaky or not related? Please let me know if you think further investigation is needed on my end.

@mbroton
Copy link
Author

mbroton commented Jul 7, 2025

Just wanted to follow up on this PR - happy to address any feedback or make adjustments if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mbroton