Skip to content

Feature collection Views #581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jirhiker merged 13 commits into from
Mar 3, 2026
Merged

Feature collection Views #581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
98cb0ad
feat: remove pg_cron dependency and related scheduling logic for impr…
jirhiker Mar 2, 2026
0ba9e54
Formatting changes
jirhiker Mar 2, 2026
a18c1d8
Merge pull request #579 from DataIntegrationGroup/jir-remove-pg-cron
jirhiker Mar 2, 2026
924652c
feat: add materialized views for latest TDS, depth to water trend, an…
jirhiker Mar 3, 2026
b33700d
Formatting changes
jirhiker Mar 3, 2026
c21c254
feat: remove pg_cron extension from test database setup for improved …
jirhiker Mar 3, 2026
cbd7449
feat: update latest TDS materialized view to use observation datetime…
jirhiker Mar 3, 2026
73f26bf
feat: normalize database password environment variable and enhance co…
jirhiker Mar 3, 2026
9d24983
feat: simplify password retrieval logic for pygeoapi configuration
jirhiker Mar 3, 2026
72c6b00
feat: streamline password validation for pygeoapi configuration
jirhiker Mar 3, 2026
6510d43
Update alembic/versions/i2b3c4d5e6f7_add_latest_tds_pygeoapi_material…
jirhiker Mar 3, 2026
945d140
feat: add handling for null slope in water trend materialized view
jirhiker Mar 3, 2026
0d51464
Merge pull request #580 from DataIntegrationGroup/jir-feature-collect…
jirhiker Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- name: Check out source repository
uses: actions/checkout@v6.0.2

- name: Start database (PostGIS + pg_cron)
- name: Start database (PostGIS)
run: |
docker compose build db
docker compose up -d db
Expand Down Expand Up @@ -81,7 +81,6 @@ jobs:
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'ocotilloapi_test'" | grep -q 1 || \
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -c "CREATE DATABASE ocotilloapi_test"
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d ocotilloapi_test -c "CREATE EXTENSION IF NOT EXISTS postgis"
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d ocotilloapi_test -c "CREATE EXTENSION IF NOT EXISTS pg_cron"

- name: Run tests
run: uv run pytest -vv --durations=20 --cov --cov-report=xml --junitxml=junit.xml --ignore=tests/transfers
Expand Down Expand Up @@ -121,7 +120,7 @@ jobs:
- name: Check out source repository
uses: actions/checkout@v6.0.2

- name: Start database (PostGIS + pg_cron)
- name: Start database (PostGIS)
run: |
docker compose build db
docker compose up -d db
Expand Down Expand Up @@ -167,7 +166,6 @@ jobs:
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -tc "SELECT 1 FROM pg_database WHERE datname = 'ocotilloapi_test'" | grep -q 1 || \
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -c "CREATE DATABASE ocotilloapi_test"
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d ocotilloapi_test -c "CREATE EXTENSION IF NOT EXISTS postgis"
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d ocotilloapi_test -c "CREATE EXTENSION IF NOT EXISTS pg_cron"

- name: Run BDD tests
run: uv run behave tests/features --tags="@backend and @production and not @skip" --no-capture
Expand Down
95 changes: 4 additions & 91 deletions alembic/versions/d5e6f7a8b9c0_create_pygeoapi_supporting_views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
REFRESH_FUNCTION_NAME = "refresh_pygeoapi_materialized_views"
REFRESH_JOB_NAME = "refresh_pygeoapi_matviews_nightly"
REFRESH_SCHEDULE = "0 3 * * *"

THING_COLLECTIONS = [
("water_wells", "water well"),
Expand Down Expand Up @@ -73,9 +71,7 @@ def _create_thing_view(view_id: str, thing_type: str) -> str:
SELECT
t.id,
t.name,
t.thing_type,
t.first_visit_date,
t.spring_type,
t.nma_pk_welldata,
t.well_depth,
t.hole_depth,
Expand All @@ -89,6 +85,7 @@ def _create_thing_view(view_id: str, thing_type: str) -> str:
t.formation_completion_code,
t.nma_formation_zone,
t.release_status,
l.elevation,
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move view-definition changes into a new Alembic revision

This commit changes the SQL projection in revision d5e6f7a8b9c0 (for example adding l.elevation here and removing prior columns), but Alembic will never re-run a revision that is already applied. Environments that had already migrated to d5e6f7a8b9c0 keep the old ogc_* thing-view schemas, while fresh installs get the new schema, and later revisions in this commit only recreate TDS/trend/summary views. That creates divergent schemas at the same head revision and inconsistent OGC collection payloads across deployments.

Useful? React with 👍 / 👎.

l.point
FROM thing AS t
JOIN latest_location AS ll ON ll.thing_id = t.id
Expand Down Expand Up @@ -154,7 +151,7 @@ def _create_avg_tds_view() -> str:
SELECT
csi.thing_id,
mc.id AS major_chemistry_id,
mc."AnalysisDate" AS analysis_date,
COALESCE(mc."AnalysisDate", csi."CollectionDate")::date AS observation_date,
mc."SampleValue" AS sample_value,
mc."Units" AS units
FROM "NMA_MajorChemistry" AS mc
Expand All @@ -178,8 +175,8 @@ def _create_avg_tds_view() -> str:
t.thing_type,
COUNT(to2.major_chemistry_id)::integer AS tds_observation_count,
AVG(to2.sample_value)::double precision AS avg_tds_value,
MIN(to2.analysis_date) AS first_tds_observation_datetime,
MAX(to2.analysis_date) AS latest_tds_observation_datetime,
MIN(to2.observation_date) AS first_tds_observation_date,
MAX(to2.observation_date) AS last_tds_observation_date,
l.point
FROM tds_obs AS to2
JOIN thing AS t ON t.id = to2.thing_id
Expand Down Expand Up @@ -231,68 +228,6 @@ def _create_refresh_function() -> str:
"""


def _schedule_refresh_job() -> str:
return f"""
DO $do$
BEGIN
BEGIN
-- Avoid direct SELECT on cron.job because managed Postgres
-- environments may deny access to the cron schema table.
PERFORM cron.unschedule('{REFRESH_JOB_NAME}');
EXCEPTION
WHEN undefined_function THEN
NULL;
WHEN invalid_parameter_value THEN
NULL;
WHEN internal_error THEN
-- Some pg_cron builds raise internal_error when the named
-- job does not exist. Treat this as already-unscheduled.
NULL;
WHEN insufficient_privilege THEN
RAISE NOTICE
'Skipping pg_cron unschedule for % due to insufficient privileges.',
'{REFRESH_JOB_NAME}';
RETURN;
END;

PERFORM cron.schedule(
'{REFRESH_JOB_NAME}',
'{REFRESH_SCHEDULE}',
$cmd$SELECT public.{REFRESH_FUNCTION_NAME}();$cmd$
);
EXCEPTION
WHEN insufficient_privilege THEN
RAISE NOTICE
'Skipping pg_cron schedule for % due to insufficient privileges.',
'{REFRESH_JOB_NAME}';
END
$do$;
"""


def _unschedule_refresh_job() -> str:
return f"""
DO $do$
BEGIN
BEGIN
PERFORM cron.unschedule('{REFRESH_JOB_NAME}');
EXCEPTION
WHEN undefined_function THEN
NULL;
WHEN invalid_parameter_value THEN
NULL;
WHEN internal_error THEN
NULL;
WHEN insufficient_privilege THEN
RAISE NOTICE
'Skipping pg_cron unschedule for % due to insufficient privileges.',
'{REFRESH_JOB_NAME}';
END;
END
$do$;
"""


def upgrade() -> None:
bind = op.get_bind()
inspector = inspect(bind)
Expand All @@ -307,16 +242,6 @@ def upgrade() -> None:
f"tables are missing: {missing_tables_str}"
)

pg_cron_available = bind.execute(
text(
"SELECT EXISTS ("
"SELECT 1 FROM pg_available_extensions WHERE name = 'pg_cron'"
")"
)
).scalar()
if pg_cron_available:
op.execute(text("CREATE EXTENSION IF NOT EXISTS pg_cron"))

for view_id, thing_type in THING_COLLECTIONS:
safe_view_id = _safe_view_id(view_id)
op.execute(text(f"DROP VIEW IF EXISTS ogc_{safe_view_id}"))
Expand Down Expand Up @@ -360,21 +285,9 @@ def upgrade() -> None:
_create_matview_indexes()

op.execute(text(_create_refresh_function()))
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore automatic refresh for pygeoapi materialized views

After this change, the migration still creates refresh_pygeoapi_materialized_views() but no longer schedules or otherwise invokes it automatically, so ogc_* materialized views stop updating after creation unless operators run the CLI command manually. In deployments that previously relied on built-in nightly refresh, OGC endpoints will serve stale depth/TDS/trend/summary data as source tables change.

Useful? React with 👍 / 👎.

if pg_cron_available:
op.execute(text(_schedule_refresh_job()))


def downgrade() -> None:
bind = op.get_bind()
pg_cron_available = bind.execute(
text(
"SELECT EXISTS ("
"SELECT 1 FROM pg_available_extensions WHERE name = 'pg_cron'"
")"
)
).scalar()
if pg_cron_available:
op.execute(text(_unschedule_refresh_job()))
op.execute(text(f"DROP FUNCTION IF EXISTS public.{REFRESH_FUNCTION_NAME}()"))
_drop_view_or_materialized_view("ogc_avg_tds_wells")
_drop_view_or_materialized_view("ogc_latest_depth_to_water_wells")
Expand Down
Loading