From 93e0008d8fe305ef170b379a486122046ea173ec Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 10 Mar 2026 14:41:03 -0700 Subject: [PATCH 1/7] CDA-74 Created ADR for timeseries csv formatting --- .../decisions/timeseries-csv-format.rst | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/source/decisions/timeseries-csv-format.rst diff --git a/docs/source/decisions/timeseries-csv-format.rst b/docs/source/decisions/timeseries-csv-format.rst new file mode 100644 index 000000000..f44e22086 --- /dev/null +++ b/docs/source/decisions/timeseries-csv-format.rst @@ -0,0 +1,85 @@ +##### +CSV Format for TimeSeries +##### + + +Summary +======= + +This ADR defines a standardized CSV representation for TimeSeries. It specifies a row-per-record CSV format that preserves essential metadata and ensures consistent ingestion by analytics, automation, and warehousing systems. + + +Opinions +======== + +Opinion 1 +--------- + +@brysonspilman + +Summary: +Since the intended use of the CSV format is for retrieval only, a customized format that follows standardize csv practices is appropriate. + +Key points: + +- CSV will be serialized via the jackson api, with either a custom csv DTO, or csv annotations on the existing DTO on specific fields to include. +- Timeseries CSV will include the date-time, value, and units fields as well as optionally the time-series id, office, quality, and version-date. +- By default, optional fields will not be included. +- A flag to include optional fields as metadata comments at the top of the file may be added. This would be a commented line indicating the amount of metadata followed by commented metadata rows using key:value pairs. See example below. +- Column names use kebab-case for consistency with JSON and XML. +- Units will either be included in metadata comments or as a column. Recommend not including this in the value column-header, as column names should match DTO fields. A custom serializer could be created if units need to be included in the value-column-header. +- date-time values will be serialized as ISO-8601 strings. NOTE this differs from JSON and XML for date-time which are serialized as epoch-millis. +- Null values are empty fields. Missing values use quality-code = 5, for consistency with JSON and XML. +- UTF-8 encoding, comma delimiter, LF line endings, header always included. +- One row is produced per Record. +- Multi-retrieve never includes multiple time-series IDs. + +Example CSVs: + +1) All optionals turned off, and no metadata comments: +date-time, value, units +2021-06-21T00:00:00Z, 0.0, ft +2021-06-22T00:00:00Z, 1.0, ft +2021-06-23T00:00:00Z, 2.0, ft +2021-06-24T00:00:00Z, 3.0, ft + +2) All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: +date-time, value (ft) +2021-06-21T00:00:00Z, 0.0 +2021-06-22T00:00:00Z, 1.0 +2021-06-23T00:00:00Z, 2.0 +2021-06-24T00:00:00Z, 3.0 + +3) All optionals turned on, with metadata-as-comments turned on: +# metadata-count: 5 +# time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI +# office-id: SWT +# version-date: 2021-06-21T00:00:00Z +# quality-code: 1 +# units: ft +date-time, value +2021-06-21T00:00:00Z, 0.0 +2021-06-22T00:00:00Z, 1.0 +2021-06-23T00:00:00Z, 2.0 +2021-06-24T00:00:00Z, 3.0 + +4) All optionals turned on, with metadata-as-comments not turned on: +time-series-id, office-id, date-time, value, units, version-date, quality-code +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 +ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 + +(Note that if we go with option 2 for units, then the units column would not be included in this example in 4), instead it would be included in the value column header) + +Decision Status +=============== + +(Status: proposed) + + +References +========== + +Related Types: cwms.cda.data.dto.TimeSeries, TimeSeries.Record +Issue/Discussion: https://github.com/USACE/cwms-data-api/issues/1525 \ No newline at end of file From a39233f2b2e1de5683a79d9400f33d207f433e88 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Wed, 11 Mar 2026 09:15:57 -0700 Subject: [PATCH 2/7] CDA-74 Updated ADR for timeseries csv to include doc number and added to index.rst --- ...{timeseries-csv-format.rst => 0008-timeseries-csv-format.rst} | 0 docs/source/decisions/index.rst | 1 + 2 files changed, 1 insertion(+) rename docs/source/decisions/{timeseries-csv-format.rst => 0008-timeseries-csv-format.rst} (100%) diff --git a/docs/source/decisions/timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst similarity index 100% rename from docs/source/decisions/timeseries-csv-format.rst rename to docs/source/decisions/0008-timeseries-csv-format.rst diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst index cc4e5d92e..0dc16c28f 100644 --- a/docs/source/decisions/index.rst +++ b/docs/source/decisions/index.rst @@ -24,3 +24,4 @@ Some decisions may also be a proposal and marked appropriately. Authorization Middleware <./0005-data-authorization-middleware.md> CDA Authorization Filtering <./0006-cda-authorization-filtering.md> Access Management Clients <./0007-access-management-clients.md> + Timeseries CSV Format <./0008-timeseries-csv-format.rst> From 4823b58c0cca0c1ca6942d91170a069e713d96d1 Mon Sep 17 00:00:00 2001 From: Bryson Spilman <87150647+rma-bryson@users.noreply.github.com> Date: Thu, 12 Mar 2026 15:00:37 -0700 Subject: [PATCH 3/7] Update docs/source/decisions/0008-timeseries-csv-format.rst Co-authored-by: Mike Neilson --- docs/source/decisions/0008-timeseries-csv-format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index f44e22086..5464bb34a 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -18,7 +18,7 @@ Opinion 1 @brysonspilman Summary: -Since the intended use of the CSV format is for retrieval only, a customized format that follows standardize csv practices is appropriate. +Since the intended use of the CSV format is for retrieval only, a customized format that follows standardized csv practices is appropriate. Key points: From 24e8b63790c494ce56ef866083c9c6720bef5b6f Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:24:17 -0700 Subject: [PATCH 4/7] Updated numbering and sectioning of csv ADR --- .../decisions/0008-timeseries-csv-format.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 5464bb34a..253ee4bc4 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -17,10 +17,12 @@ Opinion 1 @brysonspilman -Summary: +Summary +~~~~~~~ Since the intended use of the CSV format is for retrieval only, a customized format that follows standardized csv practices is appropriate. -Key points: +Key points +~~~~~~~~~~ - CSV will be serialized via the jackson api, with either a custom csv DTO, or csv annotations on the existing DTO on specific fields to include. - Timeseries CSV will include the date-time, value, and units fields as well as optionally the time-series id, office, quality, and version-date. @@ -34,23 +36,24 @@ Key points: - One row is produced per Record. - Multi-retrieve never includes multiple time-series IDs. -Example CSVs: +Example CSVs +~~~~~~~~~~~~ -1) All optionals turned off, and no metadata comments: +1. All optionals turned off, and no metadata comments: date-time, value, units 2021-06-21T00:00:00Z, 0.0, ft 2021-06-22T00:00:00Z, 1.0, ft 2021-06-23T00:00:00Z, 2.0, ft 2021-06-24T00:00:00Z, 3.0, ft -2) All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: +2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: date-time, value (ft) 2021-06-21T00:00:00Z, 0.0 2021-06-22T00:00:00Z, 1.0 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -3) All optionals turned on, with metadata-as-comments turned on: +3. All optionals turned on, with metadata-as-comments turned on: # metadata-count: 5 # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI # office-id: SWT @@ -63,14 +66,14 @@ date-time, value 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -4) All optionals turned on, with metadata-as-comments not turned on: +4. All optionals turned on, with metadata-as-comments not turned on: time-series-id, office-id, date-time, value, units, version-date, quality-code ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 -(Note that if we go with option 2 for units, then the units column would not be included in this example in 4), instead it would be included in the value column header) +(Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) Decision Status =============== From 59611f945d0bcb88f4b382ccaf27c878dd8428d3 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:31:39 -0700 Subject: [PATCH 5/7] Updated csv to be in code block --- .../decisions/0008-timeseries-csv-format.rst | 64 +++++++++++-------- 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 253ee4bc4..c5b0d1929 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -40,38 +40,50 @@ Example CSVs ~~~~~~~~~~~~ 1. All optionals turned off, and no metadata comments: -date-time, value, units -2021-06-21T00:00:00Z, 0.0, ft -2021-06-22T00:00:00Z, 1.0, ft -2021-06-23T00:00:00Z, 2.0, ft -2021-06-24T00:00:00Z, 3.0, ft + + .. code-block:: text + + date-time, value, units + 2021-06-21T00:00:00Z, 0.0, ft + 2021-06-22T00:00:00Z, 1.0, ft + 2021-06-23T00:00:00Z, 2.0, ft + 2021-06-24T00:00:00Z, 3.0, ft 2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: -date-time, value (ft) -2021-06-21T00:00:00Z, 0.0 -2021-06-22T00:00:00Z, 1.0 -2021-06-23T00:00:00Z, 2.0 -2021-06-24T00:00:00Z, 3.0 + + .. code-block:: text + + date-time, value (ft) + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 3. All optionals turned on, with metadata-as-comments turned on: -# metadata-count: 5 -# time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI -# office-id: SWT -# version-date: 2021-06-21T00:00:00Z -# quality-code: 1 -# units: ft -date-time, value -2021-06-21T00:00:00Z, 0.0 -2021-06-22T00:00:00Z, 1.0 -2021-06-23T00:00:00Z, 2.0 -2021-06-24T00:00:00Z, 3.0 + + .. code-block:: text + + # metadata-count: 5 + # time-series-id: ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI + # office-id: SWT + # version-date: 2021-06-21T00:00:00Z + # quality-code: 1 + # units: ft + date-time, value + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 4. All optionals turned on, with metadata-as-comments not turned on: -time-series-id, office-id, date-time, value, units, version-date, quality-code -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 -ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 + + .. code-block:: text + + time-series-id, office-id, date-time, value, units, version-date, quality-code + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-21T00:00:00Z, 0.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-22T00:00:00Z, 1.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 + ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 (Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) From 2e47552726595551371334a9d6263b996e7c04f6 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:37:36 -0700 Subject: [PATCH 6/7] Removed option to include units in headers. While possible to achieve, it breaks the standard of column to field naming used by jackson. --- .../decisions/0008-timeseries-csv-format.rst | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index c5b0d1929..4ddc054bc 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -49,17 +49,7 @@ Example CSVs 2021-06-23T00:00:00Z, 2.0, ft 2021-06-24T00:00:00Z, 3.0, ft -2. All optionals turned on, and no metadata comments, with custom serializer used for value field to include units in the column-header: - - .. code-block:: text - - date-time, value (ft) - 2021-06-21T00:00:00Z, 0.0 - 2021-06-22T00:00:00Z, 1.0 - 2021-06-23T00:00:00Z, 2.0 - 2021-06-24T00:00:00Z, 3.0 - -3. All optionals turned on, with metadata-as-comments turned on: +2. All optionals turned on, with metadata-as-comments turned on: .. code-block:: text @@ -75,7 +65,7 @@ Example CSVs 2021-06-23T00:00:00Z, 2.0 2021-06-24T00:00:00Z, 3.0 -4. All optionals turned on, with metadata-as-comments not turned on: +3. All optionals turned on, with metadata-as-comments not turned on: .. code-block:: text @@ -85,8 +75,6 @@ Example CSVs ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-23T00:00:00Z, 2.0, ft, 2021-06-21T00:00:00Z, 1 ALAT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI, SWT, 2021-06-24T00:00:00Z, 3.0, ft, 2021-06-21T00:00:00Z, 1 -(Note that if we go with option 2 for units, then the units column would not be included in this example in 4., instead it would be included in the value column header) - Decision Status =============== From 458186635c5712605b27e8e0af258f1ac335f7e9 Mon Sep 17 00:00:00 2001 From: Bryson Spilman Date: Tue, 17 Mar 2026 14:40:18 -0700 Subject: [PATCH 7/7] Updated to not include units comment if optionals turned off --- docs/source/decisions/0008-timeseries-csv-format.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/decisions/0008-timeseries-csv-format.rst b/docs/source/decisions/0008-timeseries-csv-format.rst index 4ddc054bc..1aab6a4bf 100644 --- a/docs/source/decisions/0008-timeseries-csv-format.rst +++ b/docs/source/decisions/0008-timeseries-csv-format.rst @@ -43,11 +43,11 @@ Example CSVs .. code-block:: text - date-time, value, units - 2021-06-21T00:00:00Z, 0.0, ft - 2021-06-22T00:00:00Z, 1.0, ft - 2021-06-23T00:00:00Z, 2.0, ft - 2021-06-24T00:00:00Z, 3.0, ft + date-time, value + 2021-06-21T00:00:00Z, 0.0 + 2021-06-22T00:00:00Z, 1.0 + 2021-06-23T00:00:00Z, 2.0 + 2021-06-24T00:00:00Z, 3.0 2. All optionals turned on, with metadata-as-comments turned on: