Skip to content

Latest commit

 

History

History
290 lines (209 loc) · 10.5 KB

TESTING.md

File metadata and controls

290 lines (209 loc) · 10.5 KB

DevContainer Template Test Suite

This test suite validates the DevContainer template configuration. These tests ensure that when the template is copied to a new project, all features work correctly.

Location: This test suite lives in the devcontainer_tests/ directory of the template repository.

Purpose: Validate the template itself - these tests are not meant to be copied to projects using the template.

Test Components

Test Fixtures (Sample Code)

  • test_fixtures/cli_app.py - Simple CLI program with argument parsing
  • test_fixtures/webapp/static/index.html - Minimal test webpage with interactive elements

Test Files

  • tests/test_file_visibility.py (5 tests) - Validates workspace mount and file visibility
  • tests/test_cli_program.py (6 tests) - Validates Python CLI execution
  • tests/test_webapp_playwright.py (9 tests) - Validates Playwright with all 3 browsers

Running Tests

Option 1: Inside the Container (Standard Tests)

Run these commands from the devcontainer_tests/ directory when you're inside the DevContainer:

# Change to test directory
cd devcontainer_tests/

# First-time setup
./setup_tests.sh

# Run all tests
./run_tests.sh

# Run specific test files
uv run pytest tests/test_cli_program.py -v
uv run pytest tests/test_webapp_playwright.py -k chromium -v

# Run with markers
uv run pytest -m browser  # Only browser tests
uv run pytest -m "not slow"  # Skip slow tests

# Or use pytest directly if not using uv
pytest tests/test_cli_program.py -v

What this validates:

  • ✅ Files created inside container are readable
  • ✅ Python programs execute correctly
  • ✅ All 3 Playwright browsers work (Chromium, Firefox, WebKit)
  • ✅ Workspace mount exists and is writable

Note on relative paths: All test scripts expect to be run from within the devcontainer_tests/ directory. Test fixtures are located in test_fixtures/ relative to this directory.

Option 2: From the Host (Comprehensive Integration Test)

Run this from the devcontainer_tests/ directory OUTSIDE the container (on your host machine) to validate the entire DevContainer:

# From template root, change to test directory
cd devcontainer_tests/

# Run the integration test
./test_from_host.sh

What this validates:

  • ✅ Files created on HOST are visible INSIDE container
  • ✅ Files created INSIDE container are visible on HOST
  • ✅ Complete bidirectional file synchronization works
  • ✅ All 20 tests run (file visibility, CLI, Playwright browsers)
  • ✅ End-to-end container environment validation

This script:

  1. Creates a marker file on the host with a timestamp
  2. Automatically finds your running DevContainer (or uses the image)
  3. Executes ./run_tests.sh inside the container via docker exec
  4. Passes marker file path and content as command-line arguments (--marker-file, --marker-content)
  5. Runs the complete test suite (all 20 tests)
  6. Verifies the marker file is visible and readable inside the container
  7. Cleans up the marker file after tests complete

Test Coverage

Total: 20 tests (5 file visibility + 6 CLI + 9 Playwright)

  • When run inside container: 19 tests (skips test_host_file_visible_in_container)
  • When run via test_from_host.sh: All 20 tests execute

File Visibility Tests (5 tests)

  1. test_file_created_in_workspace_is_readable - Creates file, verifies it's in workspace mount
  2. test_file_persists_in_mount - Verifies files persist and have correct permissions
  3. test_nested_directory_in_workspace - Tests nested directory structures work
  4. test_workspace_root_is_accessible - Validates workspace mount point exists
  5. test_host_file_visible_in_container - Verifies host files are visible inside (requires --marker-file parameter from test_from_host.sh)

CLI Program Tests (6 tests)

  1. test_cli_basic_execution - No arguments, default behavior
  2. test_cli_with_name_argument - Custom --name argument
  3. test_cli_with_uppercase_flag - --upper flag functionality
  4. test_cli_combined_arguments - Multiple arguments together
  5. test_cli_help_flag - Help documentation works
  6. test_cli_as_module - Can run as Python module

Playwright Browser Tests (9 tests)

  1. test_webapp_loads_in_all_browsers[chromium] - Chromium browser validation
  2. test_webapp_loads_in_all_browsers[firefox] - Firefox browser validation
  3. test_webapp_loads_in_all_browsers[webkit] - WebKit browser validation
  4. test_webapp_content_in_chromium - Detailed content validation
  5. test_webapp_interactivity - JavaScript execution and DOM updates
  6. test_webapp_responsive_in_all_browsers[chromium] - Chromium HTTP response
  7. test_webapp_responsive_in_all_browsers[firefox] - Firefox HTTP response
  8. test_webapp_responsive_in_all_browsers[webkit] - WebKit HTTP response
  9. test_playwright_browsers_are_installed - Verifies all 3 browsers available

Troubleshooting

Port 8000 Already in Use

The webapp server runs on port 8000. If you see "Address already in use":

# Kill existing server
lsof -ti:8000 | xargs kill -9

The run_tests.sh script automatically cleans up port 8000 before starting.

pytest Not Found

Run the setup script first:

./setup_tests.sh

The setup script installs dependencies using uv pip install and verifies pytest is accessible via uv run python3 -c "import pytest". The test runner automatically uses uv run pytest when uv is detected.

Playwright Browsers Not Found

Browsers are pre-installed at /ms-playwright during container build. If missing:

playwright install chromium firefox webkit

test_from_host.sh Can't Find Container

Make sure you've opened the project in VS Code with the Dev Containers extension at least once to build the container image.

Alternatively, manually build:

docker build -t devcontainer-python3-devcontainer .devcontainer

Test Output Examples

Successful Test Run

=========================================
DevContainer Test Suite
=========================================

🧹 Cleaning up any existing servers on port 8000...

🧪 Running tests...
=========================================

tests/test_file_visibility.py::test_file_created_in_workspace_is_readable PASSED
tests/test_file_visibility.py::test_file_persists_in_mount PASSED
tests/test_cli_program.py::test_cli_basic_execution PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[chromium] PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[firefox] PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[webkit] PASSED

=========================================
✓ All tests passed!
=========================================

Host Test Run (Complete Suite)

=========================================
DevContainer Test Suite (Host Runner)
=========================================

This script runs FROM THE HOST and validates
that files are visible across container boundaries.

📝 Step 1: Creating marker file on host...
   Created: .host_marker_1234567890.txt
   Content: Created from host at 2026-01-08T03:33:31Z

🔍 Step 2: Finding DevContainer...
   Found running container: abc123def456
   Container name: my-devcontainer

🧪 Step 3: Running tests inside container...
=========================================

Running ALL tests in existing container...

=========================================
DevContainer Test Suite
=========================================

🧹 Cleaning up any existing servers on port 8000...

Integration mode: marker-file=.host_marker_1234567890.txt
Integration mode: marker-content=Created from host at 2026-01-08T03:33:31Z

🧪 Running tests...
=========================================

tests/test_file_visibility.py::test_file_created_in_workspace_is_readable PASSED
tests/test_file_visibility.py::test_file_persists_in_mount PASSED
tests/test_file_visibility.py::test_nested_directory_in_workspace PASSED
tests/test_file_visibility.py::test_workspace_root_is_accessible PASSED
tests/test_file_visibility.py::test_host_file_visible_in_container PASSED
tests/test_cli_program.py::test_cli_basic_execution PASSED
tests/test_cli_program.py::test_cli_with_name_argument PASSED
tests/test_cli_program.py::test_cli_with_uppercase_flag PASSED
tests/test_cli_program.py::test_cli_combined_arguments PASSED
tests/test_cli_program.py::test_cli_help_flag PASSED
tests/test_cli_program.py::test_cli_as_module PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[chromium] PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[firefox] PASSED
tests/test_webapp_playwright.py::test_webapp_loads_in_all_browsers[webkit] PASSED
tests/test_webapp_playwright.py::test_webapp_content_in_chromium PASSED
tests/test_webapp_playwright.py::test_webapp_interactivity PASSED
tests/test_webapp_playwright.py::test_webapp_responsive_in_all_browsers[chromium] PASSED
tests/test_webapp_playwright.py::test_webapp_responsive_in_all_browsers[firefox] PASSED
tests/test_webapp_playwright.py::test_webapp_responsive_in_all_browsers[webkit] PASSED
tests/test_webapp_playwright.py::test_playwright_browsers_are_installed PASSED

========== 20 passed in 12.34s ==========

=========================================
✓ All tests passed!
=========================================

🧹 Cleaning up...
   Removed marker file: .host_marker_1234567890.txt

=========================================
✓ All tests passed!

File visibility verified:
  ✓ Host → Container: Marker file visible inside container
  ✓ Container → Host: Test artifacts visible on host
=========================================

CI/CD Integration

Both test scripts return proper exit codes:

  • Exit 0: All tests passed
  • Exit non-zero: Tests failed

Example GitHub Actions workflow:

- name: Run DevContainer Tests
  run: |
    docker build -t test-container .devcontainer
    ./test_from_host.sh

Dependencies

  • pytest ≥8.0.0 - Test framework
  • pytest-playwright ≥0.4.4 - Playwright pytest plugin
  • playwright ≥1.48.0 - Browser automation (Python 3.13 compatible)

All dependencies are compatible with Python 3.13.

Package Management

Dependencies are installed using uv (fast Python package manager) via:

uv pip install -r requirements.txt

Tests are executed using uv run pytest to ensure packages installed by uv are accessible. The scripts automatically detect and use uv run when available, falling back to direct pytest execution if uv is not installed.