Maintenance Push

[jirhiker]

maintenance push

[Copilot]

Pull request overview

Maintenance-oriented update that expands test coverage/fixtures, refactors app/test bootstrapping, and aligns schema/services/admin tooling for legacy data workflows.

Changes:

• Added/updated BDD feature tests + step implementations (CLI CSV imports, admin views, legacy notes).
• Refactored app/test initialization paths (create_app, test env defaults) and enhanced CSV parsing/summary logic.
• Introduced multiple DB/schema/admin changes (new legacy fields, constraints, data migrations, docker/CI updates).

Reviewed changes

Copilot reviewed 195 out of 283 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tests/integration/init.py	Adds integration test package metadata/docs
tests/features/well-inventory-real-user-csv.feature	Adds BDD scenario for real-world well inventory CSV
tests/features/water-level-csv.feature	Clarifies water level CSV step wording
tests/features/steps/well-notes.py	Renames Behave step functions (notes assertions)
tests/features/steps/well-location.py	Renames/reformats Behave steps for location retrieval
tests/features/steps/well-inventory-real-user-csv.py	Adds step assertions for well inventory import summary/errors
tests/features/steps/transducer.py	Renames Behave step functions; minor import reorder
tests/features/steps/thing-path.py	Renames Behave step functions (thing path filters)
tests/features/steps/sensor-notes.py	Renames Behave step functions (sensor notes)
tests/features/steps/post_migration_legacy_data.py	Import cleanup + assertion message tweaks
tests/features/steps/location-notes.py	Renames Behave step functions (location notes)
tests/features/steps/geojson-response.py	Renames Behave step functions (GeoJSON endpoints)
tests/features/steps/cli_common.py	Adds shared CLI/auth setup steps for feature tests
tests/features/steps/api_common.py	Refactors API test bootstrap + adds status code assertions
tests/features/data/well-inventory-valid.csv	Adds well inventory CSV fixture (valid)
tests/features/data/well-inventory-valid-reordered.csv	Adds reordered-header CSV fixture
tests/features/data/well-inventory-valid-extra-columns.csv	Adds extra-columns CSV fixture
tests/features/data/well-inventory-valid-comma-in-quotes.csv	Adds quoted-comma CSV fixture
tests/features/data/well-inventory-no-data.csv	Adds header-only CSV fixture
tests/features/data/well-inventory-no-data-headers.csv	Adds alternate header-only CSV fixture
tests/features/data/well-inventory-missing-wl-fields.csv	Adds missing-fields CSV fixture
tests/features/data/well-inventory-missing-required.csv	Adds missing-required CSV fixture
tests/features/data/well-inventory-missing-phone-type.csv	Adds missing phone-type fixture
tests/features/data/well-inventory-missing-email-type.csv	Adds missing email-type fixture
tests/features/data/well-inventory-missing-contact-type.csv	Adds missing contact-type fixture
tests/features/data/well-inventory-missing-contact-role.csv	Adds missing contact-role fixture
tests/features/data/well-inventory-missing-address-type.csv	Adds missing address-type fixture
tests/features/data/well-inventory-invalid.csv	Adds invalid CSV fixture
tests/features/data/well-inventory-invalid-utm.csv	Adds invalid-UTM fixture
tests/features/data/well-inventory-invalid-postal-code.csv	Adds invalid postal-code fixture
tests/features/data/well-inventory-invalid-phone-number.csv	Adds invalid phone-number fixture
tests/features/data/well-inventory-invalid-partial.csv	Adds partially-invalid fixture
tests/features/data/well-inventory-invalid-numeric.csv	Adds invalid numeric fixture
tests/features/data/well-inventory-invalid-lexicon.csv	Adds invalid lexicon fixture
tests/features/data/well-inventory-invalid-email.csv	Adds invalid email fixture
tests/features/data/well-inventory-invalid-date.csv	Adds invalid date fixture
tests/features/data/well-inventory-invalid-date-format.csv	Adds invalid date-format fixture
tests/features/data/well-inventory-invalid-contact-type.csv	Adds invalid contact-type fixture
tests/features/data/well-inventory-invalid-boolean-value-maybe.csv	Adds invalid boolean fixture
tests/features/data/well-inventory-duplicate.csv	Adds duplicate-id fixture
tests/features/data/well-inventory-duplicate-columns.csv	Adds duplicate-columns fixture
tests/features/admin-minor-trace-chemistry.feature	Adds admin-view BDD feature for minor/trace chemistry
tests/conftest.py	Adjusts test env defaults + seeds contact notes
tests/init.py	Refactors test bootstrap to import main app + env defaults
tests/README.md	Adds documentation for running tests
services/water_level_csv.py	Adds header aliasing + row-issue counting + parsing tweaks
services/util.py	Adds naive->aware timezone attachment helper
services/query_helper.py	Extends paginated getter to accept a prebuilt SQL query
services/ngwmn_helper.py	Fixes legacy table name for pressure daily SQL
services/gcs_helper.py	Switches to ADC-based storage client creation
services/contact_helper.py	Enhances contact creation (optional commit + notes + association guard)
scripts/check_waterlevels_measured_by.py	Adds utility script for mapper completeness checks
schemas/validators.py	Minor formatting/import spacing
schemas/sample.py	Makes participant/contact optional in sample schemas
schemas/permission_history.py	Import reorder
schemas/observation.py	Adds nma_data_quality to observation response
schemas/notes.py	Tightens note_type to enum + makes BaseNote a BaseModel
schemas/location.py	Updates UTM zone type + surfaces legacy note/reliability fields
schemas/geologic_formation.py	Import reorder
schemas/deployment.py	Makes installation_date optional in response
schemas/contact.py	Adds notes to CreateContact + relaxes address requirements + adds note lists to response
schemas/aquifer_system.py	Import/format cleanup
schemas/init.py	Expands PastOrToday validator to support datetime + adds PastOrTodayDatetime
main.py	Introduces create_app() and conditional Sentry init
features/admin/well_data_relationships.feature	Adds data-integrity admin feature specification
features/admin/minor_trace_chemistry_admin.feature	Adds detailed admin UI feature specification
docker/db/Dockerfile	Builds PostGIS image with pg_cron package
docker-compose.yml	Switches db service to custom build + enables pg_cron + port mapping change
db/thing_aquifer_association.py	Minor formatting/import cleanup
db/permission_history.py	Import reorder
db/observation.py	Adds nma_data_quality column to Observation model
db/measuring_point_history.py	Makes measuring_point_height nullable
db/location.py	Renames legacy notes field + adds nma_data_reliability
db/initialization.py	Makes app_read role parsing robust + adds pg_cron extension + trigger sync guard
db/group.py	Fixes mapped_column import + changes uniqueness to (name, group_type)
db/engine.py	Loads dotenv without overriding externally-set env vars
db/deployment.py	Makes installation_date nullable + adds legacy WI_* fields
db/data_provenance.py	Import organization cleanup
db/contact.py	Adds NotesMixin + nullable address fields + note accessors
db/init.py	Reorders/expands exports and search-related imports
db/README.md	Adds DB module documentation
data_migrations/registry.py	Adds discovery/lookup for data migrations
data_migrations/migrations/_template.py	Adds data migration template
data_migrations/migrations/init.py	Initializes migrations package
data_migrations/migrations/20260205_0001_move_nma_location_notes.py	Adds migration to backfill polymorphic notes from legacy location notes
data_migrations/base.py	Introduces DataMigration definition
data_migrations/init.py	Initializes data_migrations package
core/pygeoapi-config.yml	Adds pygeoapi configuration template
core/enums.py	Adds enums for data_reliability, note_type; renames origin enum
core/constants.py	Adds SRID + state code constants
core/app.py	Removes route registration from lifespan (centralizes init)
cli/service_adapter.py	Adds well-inventory CSV import wrapper + changes water-level return
cli/README.md	Adds CLI documentation
api/ogc/schemas.py	Removes in-app OGC schemas (pygeoapi now used)
api/ogc/router.py	Removes in-app OGC router (pygeoapi now used)
api/ogc/conformance.py	Removes in-app conformance constants
api/ogc/collections.py	Removes in-app collections registry
api/ogc/init.py	Removes OGC package marker
api/lexicon.py	Adds name-filter option for lexicon categories
api/README.md	Adds API module documentation
alembic/versions/h1b2c3d4e5f6_update_group_unique_constraint_to_name_type.py	Adds migration for group (name, group_type) uniqueness
alembic/versions/g4a5b6c7d8e9_change_minor_trace_volume_to_int.py	Changes minor/trace volume type to integer
alembic/versions/f1a2b3c4d5e6_add_nma_formation_zone_to_thing.py	Updates revision chain for thing formation zone
alembic/versions/f0c9d8e7b6a5_align_pressure_daily_uuid_columns.py	Aligns UUID types in legacy pressure table
alembic/versions/e8a7c6b5d4f3_add_thing_id_to_nma_waterlevelscontinuous_pressure_daily.py	Adds FK thing_id to pressure daily table
alembic/versions/e71807682f57_add_sample_point_fields_to_minor_trace.py	Adds SamplePointID field to minor/trace chemistry
alembic/versions/e123456789ab_add_observation_data_quality.py	Adds observation.nma_data_quality columns
alembic/versions/d9f1e2c3b4a5_drop_thing_id_from_nma_radionuclides.py	Drops thing_id from radionuclides table
alembic/versions/c7f8a9b0c1d2_add_thing_id_to_nma_surface_water_data.py	Adds/enforces thing_id FK in surface water data
alembic/versions/c4d5e6f7a8b9_make_address_city_state_nullable.py	Makes address city/state nullable
alembic/versions/c1d2e3f4a5b6_create_nma_field_parameters.py	Creates legacy NMA_FieldParameters table
alembic/versions/b3c4d5e6f7a8_make_wellscreen_depths_nullable.py	Makes well screen depths nullable
alembic/versions/b12e3919077e_add_missing_legacy_fields.py	Adds legacy fields + renames location notes column + adds reliability
alembic/versions/a1b2c3d4e5f7_make_deployment_installation_date_nullable.py	Makes deployment installation_date nullable
alembic/versions/9a0b1c2d3e4f_make_address_postal_code_nullable.py	Makes address postal_code nullable
alembic/versions/8c9d0e1f2a3b_make_measuring_point_height_nullable.py	Makes measuring_point_height nullable
alembic/versions/7c02d9f8f412_create_nmawaterlevelscontinuouspressuredaily.py	Clarifies legacy table naming in migration header
alembic/versions/7b8c9d0e1f2a_delete_is_suitable_for_datalogger.py	Drops is_suitable_for_datalogger from thing tables
alembic/versions/76e3ae8b99cb_enforce_thing_fk_for_nma_legacy_models.py	Enforces thing FK integrity for legacy NMA tables
alembic/versions/71a4c6b3d2e8_add_nma_wclab_id_to_minor_trace.py	Adds WCLab ID legacy column
alembic/versions/6e1c90f6135a_add_unique_constraint_to_.py	Migration header cleanup (minor/trace naming)
alembic/versions/5336a52336df_drop_minor_trace_chemistry_unique_.py	Drops unique constraint from minor/trace chemistry
alembic/versions/50d1c2a3b4c5_add_unique_index_ngwmn_wellconstruction.py	Adds unique index for NGWMN view
alembic/versions/3a9c1f5b7d2e_align_nma_minor_trace_columns.py	Keeps revision chain with no-op migration
alembic/versions/263109252fb1_add_legacy_equipment_fields.py	Adds legacy deployment equipment fields
alembic/versions/1d2c3b4a5e67_create_nma_stratigraphy_table.py	Adjusts stratigraphy schema types/constraints
alembic/env.py	Makes Alembic logging config opt-in + fixes include_object(None)
admin/views/weather_photos.py	Adds read-only admin view for legacy weather photos
admin/views/weather_data.py	Adds read-only admin view for legacy weather data
admin/views/waterlevelscontinuous_pressure_daily.py	Adds read-only admin view for legacy pressure daily table
admin/views/thing.py	Adds identity + removes deleted field from view
admin/views/surface_water_photos.py	Adds read-only admin view for surface water photos
admin/views/surface_water.py	Renames surface water view + enforces read-only
admin/views/stratigraphy.py	Adds read-only admin view for stratigraphy
admin/views/soil_rock_results.py	Adds read-only admin view for soil/rock results
admin/views/minor_trace_chemistry.py	Adds read-only admin view for minor/trace chemistry
admin/views/location.py	Updates legacy location fields exposed in admin
admin/views/hydraulicsdata.py	Updates hydraulics admin view for integer PK + legacy fields
admin/views/field_parameters.py	Adds read-only admin view for field parameters
admin/views/deployment.py	Adds legacy deployment fields to admin views
admin/views/associated_data.py	Adds read-only admin view for associated data
admin/views/init.py	Registers new/updated admin views
AGENTS.MD	Adds playbook for high-volume transfers + testing notes
.github/workflows/release.yml	Updates action versions for release automation
.github/workflows/format_code.yml	Updates action versions for formatting workflow
.github/workflows/dependabot_automerge.yml	Adds workflow to auto-merge Dependabot PRs
.github/workflows/CD_staging.yml	Refactors staging CD to render app.yaml via template/envsubst
.github/workflows/CD_production.yml	Refactors production CD to render app.yaml via template/envsubst
.github/dependabot.yml	Expands Dependabot config + grouping + github-actions updates
.github/app.template.yaml	Adds App Engine config template
.env.example	Updates default db port + adds transfer/test flags

Comments suppressed due to low confidence (1)

services/water_level_csv.py:1

• HEADER_ALIASES maps measuring_person to sampler, but the required column is field_staff. With this mapping, CSVs using measuring_person won’t satisfy required-field validation and will fail unexpectedly. If measuring_person is intended as a legacy synonym, map it to field_staff (or update validation/required fields to accept sampler).

# ==============================================================================

[Copilot]

Setting POSTGRES_PORT to 5432 contradicts the safety comment and can accidentally point tests at a developer/default Postgres instance (often running on 5432). Prefer using os.environ.setdefault("POSTGRES_PORT", "54321") (or another dedicated test port) and keep the test-only overrides in one place (either here or pytest_configure), to avoid unsafe/fragile environment behavior.

[Copilot]

The step receives well_name but looks up context.well_name, which is not set in this step and will return the wrong record (or None). Use the well_name parameter for the lookup (and/or assign it to context explicitly if later steps depend on it).

[Copilot]

The step receives well_name but looks up context.well_name, which is not set in this step and will return the wrong record (or None). Use the well_name parameter for the lookup (and/or assign it to context explicitly if later steps depend on it).

Suggested change

	context.result = context.database.get(context.well_name)
	context.well_name = well_name
	context.result = context.database.get(well_name)

[Copilot]

This assertion is logically inconsistent: when v == "", v is None can never be true, so the step will fail if any empty string is present. To validate “no placeholder strings,” assert that values are not empty strings (e.g., assert v != "") and separately handle actual None values if needed.

Suggested change

	if v == "":
	assert v is None, f"Value for key {k} is an empty string but should be null"
	assert v != "", f"Value for key {k} is an empty string but should be null"

[Copilot]

Several step function names were changed to non-descriptive placeholders (step_step_step*), which makes failures harder to debug and reduces maintainability. Rename these functions to meaningful, unique names that reflect their step text (similar to the other renamed steps in this PR).

[Copilot]

Several step function names were changed to non-descriptive placeholders (step_step_step*), which makes failures harder to debug and reduces maintainability. Rename these functions to meaningful, unique names that reflect their step text (similar to the other renamed steps in this PR).

[Copilot]

Several step function names were changed to non-descriptive placeholders (step_step_step*), which makes failures harder to debug and reduces maintainability. Rename these functions to meaningful, unique names that reflect their step text (similar to the other renamed steps in this PR).

[Copilot]

HEADER_ALIASES maps measuring_person to sampler, but the required column is field_staff. With this mapping, CSVs using measuring_person won’t satisfy required-field validation and will fail unexpectedly. If measuring_person is intended as a legacy synonym, map it to field_staff (or update validation/required fields to accept sampler).

Suggested change

	"measuring_person": "sampler",
	"measuring_person": "field_staff",

[Copilot]

water_levels_csv previously returned an exit code (int) but now returns the full BulkUploadResult. If any Typer command or caller expects an int for process exit handling, this change will break that control flow. Consider keeping backward compatibility (return result.exit_code) and/or introducing a new function that returns the full result object.

[Copilot]

water_levels_csv previously returned an exit code (int) but now returns the full BulkUploadResult. If any Typer command or caller expects an int for process exit handling, this change will break that control flow. Consider keeping backward compatibility (return result.exit_code) and/or introducing a new function that returns the full result object.

Suggested change

	return result
	return result.exit_code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31facf709b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fallback to existing DB credentials for pygeoapi config

register_routes() now mounts pygeoapi during app startup, but _pygeoapi_db_settings() hard-fails if PYGEOAPI_POSTGRES_USER is not explicitly set. Existing setups in this commit still rely on POSTGRES_* variables (for example the app service environment in docker-compose.yml), so starting the app in those environments now raises a RuntimeError before the API boots. Please add a fallback to the existing database env vars (or make pygeoapi mounting conditional) to avoid breaking current deployments and local development flows.

Useful? React with 👍 / 👎.