REQUIREMENTS

[nf-s]

First version of high-level requirements

Front-End Web Application

• Fullstack web app proof-of-concept #4
• Geospatial Visualization: Interactive map component for visualizing and querying standardized geospatial datasets (Zarr, GeoParquet, tiled formats)
  • Map Visualisation proof-of-concept #2
• Unstructured Data Presentation: User interface for discovering, viewing, and interacting with reports linked to geographic bounds
• Modular Dashboards: Modular visualization components for standardized ecosystem reporting (design and report structure to be provided)
  • Dashboard/Report generation proof-of-concept #3
• User-Driven Workflows (see back-end Data Pipeline Integration)
  • Support uploading geometries and triggering predefined workflows for summary statistics
  • Curated catalogue interface for method selection with implementation guidance
  • Support for structured data uploads (Zarr, GeoParquet) and simple file uploads

Back-End Services & Database

• Fullstack web app proof-of-concept #4
• Authz
  • Implement robust authentication and authorization.
  • Enforce attribute-based access control (ABAC) to manage tiered access for jurisdiction-specific data.
• REST API: Deployed to Vercel with containerized deployment support
• Database: PostgreSQL (NeonDB/AWS RDS) with JSON indexing strategy for unstructured reports
• Data Pipeline Integration: Integrate with the data pipeline API (e.g., Argo Workflows, Pachyderm) to:
  • Ingest data product outputs into database
  • Submit workflow runs for custom geometry processing
  • Monitor data pipeline status
  • Support method chaining through pipeline workflow definitions (not user-driven)
• PDF Certificate Generation: Standardized certificates (requirements to be provided)

[nf-s]

Notes from Project refresh + planning meeting - July 11

• Nick is going to focus on the stack for dashboard(s); Alex/Matt will focus on the data pipeline infrastructure
• Nick is going to take guidance from the current DB + dashboard implementation, but actually re-write/build it from scratch (which is not as big of a deal as it sounds now with the template of what's needed)
• We agreed that D3 is a nice option for vis and components (mixed with whatever other front-end stuff is required like react etc.), but that use of specific Observable code/framework/notebooks should be limited to either embeds or generating code (rather than reliance on generating the app/dashboard/static content itself)
• Further thought and trade-offs in the short term TBD in context of what exactly an “export” from the dashboard will entail – i.e. is it compatible with a simple render + print, what pre-baked outputs are possible etc. The motivation being that these are either very easy or very complicated, with potentially not much in between!
• We all landed on that we will (at least for now) switch to postGIS for DB, as 1) there are no use cases yet that require mongoDB and 2) it's simpler and faster for now for both development and user interaction. This does not preclude a switch to mongoDB in the future

[nf-s]

Next steps for me

• More thinking about architecture for Fullstack web app proof-of-concept #4 (what frameworks, tech, etc)
• Create skeleton app
• Deploy to Vercel and NeonDB
• Create first version of DB schema (and basic APIs)
  • Ingest dataset, geometries and products
  • Basic CRUD API for datasets, geometries and products
• Create very basic UI to browse datasets/geometries/products
• Move onto Map Visualisation proof-of-concept #2 or Dashboard/Report generation proof-of-concept #3

[nf-s]

Notes from meeting

Data pipeline

• existing pipeline
• moved to "meta" level - with "basic" UI
• ability to upload data, apply "pre-canned" method
  • data and method is "agnostic"
• Note: data types may not be varied - but methods may be varied

Data storage

• "NoSQL" paradigm...
  • We are starting with Postgres/PostGIS, we previously discussed
    • Geospatial goodness of PostGIS
    • End-to-end "type safety", that is the entire codebase understands what each table represents, so the data, data model, api, and front-end code will never be out of sync. Generally relational DBs are preferred for "system-level data model" (eg users, access control, ...) as they are strictly-defined/structured at the DB level.
  • Ben on Mongo: "Would prefer to take the complexity hit upfront" - "the metadata spec will change every 4 months"
  • Nick: postgres schema is very easy to change (through ORM - code that managed the database), we can also support JSON (and index JSON) - also we still need to define a schema
  • Conclusion: will stick with postgres for now, we can change to Mongo later if needed. Also look into a mix of Json payloads in postgres - how can we create complex queries linking multiple JSON payloads together. Also, in the future we can have a hybrid postgres/mongo approach
• "Appropriate tokenisation" - unchangeable record
• ability to track provenance of arbitrary data...
• Interoperability - API query from D3, or another static web app

Data query / visualisation

• Served/dynamic API + Visualisation
• Static generated (examples using Hardcoded JSON - rendered using D3/Observable)
  • Hooks up to zenodo API - if changes, then new DOI is minted
  • Ad hoc input files in a repo
    • For technical users
  • Won't need this to be "non-coding" environment - won't be "customising the configuration"
  • Doesn't need to be an automatic process initially

Qs

• How to query across jsonb's
• We may have many schemas now - but will converge
• Look at maldives example - in Atlas now... Nick to validate with the current data model

[nf-s]

Notes from meeting

Note - I only have notes related to the web-app

FJ potential first customer!

We talked a bit about generic “pre-canned” components and also specific “hifi” dashboards

• Think climate change one, diversity one, etc for different departments
• Also think higher-level one (eg for whole of gov)
• Important that it is usable by non-technical users
  • This includes the ability to create dashboards/reports using "pre-canned" components

FJ stats - very concerned about total lack of provenance for Data. So we talked about potential metadata standards for non-spatial data

• Think big spreadsheet (non-spatial) on someones laptop with zero metadata
• See Mitch’s metadata entry form
• Logic QA/QC on entry - think metadata side
• Then enforce schema
• Schema may be at the indicator level - with different levels of strictness

Talked about console vs user-facing app - do we have separate apps or control visibility

• I prefer to keep everything in the app and then control visibility - the app was designed to be modular, so there are lots of re-usable components that can be used to quickly create new screens
• Also possible to use API and API key for separate besoke apps (or prototyping - eg mitches data entry)

Also talked about possible future features around intersection of geometries

• For example how much is overlap between sediment compartments and eez

Workflow process steps - Ben asked about how we are going to model the workflow in the platform

• We haven't started thinking about that architecture yet
• The first step is to trigger workflows from the app (eg on Product creation)
• Then we think about what metadata needs to be defined at the product level to trigger an appropriate workflow
  • Eg dataset, geometries, other arguments (eg column names or something)
• Then we go a level deeper - and start to break up workflow steps and then incorporate them into the data model (TBD)
• ... Alex and Nick to discuss further

Some todos for Nick

• Rename Variables to Indicators
• Keyword search - for all pages (too many geometries to click through)
• Create more charts
• Maybe experiment with CSV import... allowing easy import of "weak provenance" data makes the platform more compelling to demo - also makes it easy to develop chart components, and flesh out dashboard/report system

[nf-s]

We also have quite a few features that still need to be developed

• Creating chart/map components to be used in reporting and dashboards (in progress)
• Examples using Observable notebooks (plus code snippets in the app)
• Dataset visualisation components (in progress - Will is working on this)
• Dashboard/report publishing (sharing with external users)
• PDF/certificate generation
• Triggering pipelines from the app (and allowing customisation of workflow procedures/parameters) No longer planned
  • This includes user uploaded datasets/geometries
• Search improvements
  • Search more fields in the console (eg geometry output names in product outputs search)
  • App wide search (find reports based on geographic regions, etc)
• User/role based access control
  • User owned reports/dashboards
  • Granting users access to specific datasets, etc
• MCP and chatbot-like UI investigation