Feature/scripts-849-nf

[guilhermeltm]

Summary

This pull request adds the following features to the MONAN workflow:

1) Personalized mesh generation

This has been added as an (optional) second step in the MONAN workflow (2.create_mesh.bash). The main idea here is that, if needed, one can readily create a global or regional mesh with or without refinement, with all mesh characteristics set by the user according to the requirements of the experiment of interest. If one instead prefers to use a publicly available mesh (e.g. created by NCAR), this is also possible. The control over which mesh is used is set by the parameter MESH in 0.run_all.bash, which now substitutes the previous RES parameter, which indicated the number of cells of the mesh, but was only applicable to the particular case of NCAR meshes (following their standard naming x1.${RES}.grid.nc).

As hinted at above, the personalized mesh is generated by executing 2.create_mesh.bash. The characteristics of the mesh are set in mesh_input_file.txt.

The source code for this mesh generation step uses jigsaw (https://github.com/dengwirda/jigsaw) and adaptations from vtx-mpas-meshes (https://github.com/marta-gil/vtx-mpas-meshes) and MPAS-Limited-Area (https://github.com/MPAS-Dev/MPAS-Limited-Area). Since these adaptations are not available in the original repositories, I added them directly to sources/.

2) Option for running global or regional simulation

The previous workflow allowed only for global simulations. Now it is also possible to run regional simulations. A regional mesh can be directly created (without the need for separately cutting the mesh) by the mesh functionality explained above. The regional simulation then follows by setting parameters REGIONAL (flag to choose between regional and global simulation) and LBCINT (variable indicating the interval in seconds separating each update of lateral boundary conditions in the regional run) in 0.run_all.bash. Once REGIONALis set to Y, the workflow automatically generates by 3.pre_processing.bashthe initial and lateral boundary conditions needed for the simulation.

3) Option for running simulations with GFS or ERA5 data

In the previous workflow it was only possible to run simulations with GFS data. Now one can also use ERA5 data. To choose which data to use, one just needs to set EXP=GFSor EXP=ERA5 in 0.run_all.bash. Both options work both for global and for regional simulation setups.

4) Option for running standard idealized test cases for dynamics evaluation

For testing purely the dynamical core of MONAN it may be useful to run idealized simulations where all physics parameterizations are switched off. We now have the possibility of running standard dynamics test cases following the MPAS user guide (se https://www2.mmm.ucar.edu/projects/mpas/mpas_atmosphere_users_guide_8.2.0.pdf; section 7.1). At the moment the only test case completely implemented is the Jablonowski and Williamson baroclinic wave, with initial perturbation (number 2 in the user guide), but with the current setup this can easily be extended to the remaining test cases. Running this test case is basically a matter of choosing EXP=IDEALIZED2 in 0.run_all.bash.

5) Postprocessing on native MPAS grid (grid plotting, scalar field plotting and animation)

Now it is possible to plot both the generated MPAS grid and scalar fields from simulations directly on the native grid. Plotting the grid is achieved by plot_mpas_grid.bash, while plotting scalar fields on the native grid can be done via the new (optional) step 5 5.run_post_on_mpas_grid.bash. The source code in Python for these plots can be found under sources/CGFD-USP-Post-Proc. Using the source code one can also make animations by running mpas_animate.py, but no bash script is at the moment available for such animations under scripts/.

6) Script for automatically downloading ERA5 data

The python script download_era5_data.py in scripts allows for automatically downloading raw ERA5 data, either global or regional. All the setting for the download is done directly in download_era5_data.py, and the download itself is achieved by running python download_era5_data.py after activating our vtx_envconda environment (see remark 1 below).

Tutorials for testing

For learning how to use and testing these new features, 5 tutorials were created (see docs/):

1. tutorial1-meshes.md: generating your first personalized meshes
2. tutorial2-ideal-case.md: running a global simulation with idealized initial conditions and no physics parametrizations
3. tutorial3-real-case-global.md: running a global simulation with ERA5 data
4. tutorial4-real-case-regional-ERA5.md: running a regional simulation with ERA5 data
5. tutorial5-real-case-regional-GFS.md: running a regional simulation with GFS data

Remarks

1. The conda environment needed to run many of the scripts above is installed in the shared directory prepared for the WMO course under /pesq; how to use it is explained in the tutorials.
2. The directory where ERA5 data are saved needs to be adjusted (I didn't have the permissions to open a directory similar to that prepared for GFS data in EGEON).
3. Although implementation was performed both for ERA5 and GFS data, tests were more thoroughly performed for ERA5 (tutorials 3 and 4). For GFS I only tested if the whole workflow runs properly following tutorial 5.
4. Regional tests with ERA5 (tutorial 4) and GFS (tutorial 5) data worked for all preprocessing steps of the model. The runs themselves failed for MONAN, but worked for MPAS (I succesfully ran tutorials 4 and 5 using the preprocessed data generated by MONAN but using the MPAS code from our USP repository to perform the simualtions), so the preprocessing must be correct, but there seems to be an issue with the MONAN core_atmosphereregional configuration. Please have a look at this.
5. I haven't thoroughly worked on sanity checks and error messages for the new features. Please feel free to adapt the existing code as needed.

Acknowledgements

The code implemented here benefited from the work of many people:
Pedro S. Peixoto, Danilo C. de Souza, Felipe A. V. B. Alves, Guilherme L. Torres Mendonça (USP) - CGFD-USP: https://github.com/CGFD-USP
Darren Engwirda (CSIRO) - jigsaw: https://github.com/dengwirda/jigsaw
Marta G. Bardají (Vortex) - vtx-mpas-meshes: https://github.com/marta-gil/vtx-mpas-meshes
MPAS-Limited-Area team: https://github.com/MPAS-Dev/MPAS-Limited-Area
MPAS-Tools team: https://mpas-dev.github.io/MPAS-Tools/master/authors.html

[Copilot]

Pull request overview

This PR extends the MONAN workflow to support user-defined mesh generation, global vs. regional runs (including LBC generation), ERA5 forcing alongside GFS, idealized dynamics test cases, and new post-processing utilities for plotting on the native MPAS grid.

Changes:

• Vendored mesh/limited-area tooling into sources/ (MPAS-Limited-Area + vtx-mpas-meshes derivatives) and added scripts to generate/plot meshes.
• Updated workflow scripts/namelists to use MESH (instead of NCAR-only RES), enable regional runs with LBCs, and add ERA5/idealized options.
• Added new plotting and ERA5-download utilities plus an initial tutorial for mesh generation.

Reviewed changes

Copilot reviewed 79 out of 104 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
sources/MPAS-Limited-Area/requirements.txt	Adds Python requirements for vendored limited-area tool
sources/MPAS-Limited-Area/limited_area/region_spec.py	Region specification/geometry generation utilities
sources/MPAS-Limited-Area/limited_area/points.py	Points/PTS parser for region definitions
sources/MPAS-Limited-Area/limited_area/init.py	Package init (empty)
sources/MPAS-Limited-Area/docs/points-examples/tropics.channel.pts	Example PTS file for channel region
sources/MPAS-Limited-Area/docs/points-examples/pnw.custom.pts	Example PTS file for custom polygon
sources/MPAS-Limited-Area/docs/points-examples/japan.ellipse.pts	Example PTS file for ellipse
sources/MPAS-Limited-Area/docs/points-examples/india.circle.pts	Example PTS file for circle
sources/MPAS-Limited-Area/docs/points-examples/conus.custom.pts	Example PTS file for custom polygon (CONUS)
sources/MPAS-Limited-Area/create_region_n_layers_as_input	CLI helper to create region with explicit layer count
sources/MPAS-Limited-Area/create_region	CLI helper to create/plot regional subset
sources/MPAS-Limited-Area/README.md	Vendored documentation for limited-area tool
sources/CGFD-USP-Post-Proc/mpas_plot_grid.py	Adds MPAS grid plotting script
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/vtxmpasmeshes/mpas_plots.py	Mesh plotting utilities for regional meshes
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/vtxmpasmeshes/jigsaw_generator.py	JIGSAW spherical mesh generation wrapper
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/vtxmpasmeshes/init.py	Package init (empty)
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/vtxmpasmeshes.egg-info/PKG-INFO	Packaged metadata artifact
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/setup.py	Python packaging for vtxmpasmeshes
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/plot_regional_cl.py	CLI wrapper to plot regional mesh
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/gtm_tests/test_onlyregional_gtm_largermesh20240201.py	Test script for generating a larger regional mesh
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/gtm_tests/test_onlyregional_gtm.py	Test script for generating a regional mesh
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/gtm_tests/test2_gtm.py	Sensitivity-style mesh generation/plot script
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/examples/view_mpas_grid.py	Example to view a grid file
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/examples/sensitivity_test.py	Example sensitivity workflow for mesh parameters
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/examples/personalize_variable_resolution.py	Example variable-resolution map generation
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/examples/generate_mesh.py	Example mesh generation driver
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/examples/compare_meshes.py	Example comparing multiple meshes
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/environment.yml	Conda environment definition for mesh tooling
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/data/.gitkeep	Keeps data/ tracked while ignoring contents
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/create_regional_mesh_inwards.py	Script to generate regional mesh and partitions (inwards variant)
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/create_regional_mesh.py	Script to generate regional mesh and partitions
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/build/lib/vtxmpasmeshes/plot_utilities.py	Built artifact copy of plotting utilities
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/build/lib/vtxmpasmeshes/mpas_plots.py	Built artifact copy of mesh plotting utilities
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/build/lib/vtxmpasmeshes/mesh_generator.py	Built artifact copy of mesh generator
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/build/lib/vtxmpasmeshes/jigsaw_generator.py	Built artifact copy of JIGSAW wrapper
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/build/lib/vtxmpasmeshes/init.py	Built artifact package init (empty)
sources/CGFD-USP-Create-Mesh/vtx-mpas-meshes/README.md	Vendored README for vtx mesh tooling
scripts/utils.bash	Adds date arithmetic + forecast-hour list helper functions
scripts/setenv.bash	Updates MONAN path and adds LBCS phase resources
scripts/plot_mpas_grid.bash	Adds MPAS grid plotting runner
scripts/namelists/streams.init_atmosphere.TEMPLATE_IDEALIZED	Streams template for idealized init_atmosphere
scripts/namelists/streams.init_atmosphere.TEMPLATE	Updates init streams to use #MESH#
scripts/namelists/streams.init_atmosphere.STATIC	Updates static streams to use #MESH#
scripts/namelists/streams.init_atmosphere.LBCS	Adds streams template for LBC generation
scripts/namelists/streams.atmosphere.TEMPLATE	Updates atmosphere streams for #MESH# and LBC input
scripts/namelists/namelist.wps.TEMPLATE	Updates end date + interval templating for LBC cadence
scripts/namelists/namelist.init_atmosphere.TEMPLATE_IDEALIZED	Adds idealized init namelist template
scripts/namelists/namelist.init_atmosphere.TEMPLATE	Updates init namelist to use #MESH#/#EXP# and regional terrain blend toggle
scripts/namelists/namelist.init_atmosphere.STATIC	Updates static namelist to use #MESH#
scripts/namelists/namelist.init_atmosphere.LBCS	Adds init_atmosphere namelist for generating LBCs
scripts/namelists/namelist.atmosphere.TEMPLATE_IDEALIZED	Adds atmosphere namelist template for idealized cases
scripts/namelists/namelist.atmosphere.TEMPLATE	Updates atmosphere namelist with #MESH#, #CONFIG_LEN_DISP#, and #APPLY_LBCS#
scripts/mesh_input_file.txt	Adds configuration file for personalized mesh generation
scripts/make_template.bash	Updates post templating to support MESH naming
scripts/make_static.bash	Updates static generation workflow to use MESH
scripts/make_lbcs.bash	Adds LBC generation step using init_atmosphere
scripts/make_initatmos_idealized.bash	Adds init-atmosphere runner for idealized testcases
scripts/make_initatmos.bash	Updates init-atmosphere runner for MESH + regional flag
scripts/make_degrib_GFS.bash	Adds regional-aware GFS degribbing (IC + LBC timesteps)
scripts/make_degrib_ERA5.bash	Adds ERA5 degribbing (global and regional modes)
scripts/download_era5_data.py	Adds ERA5 download utility via CDS API
scripts/5.run_post_on_mpas_grid.bash	Adds optional post step for native-grid plotting
scripts/5.run_post.bash	Updates post workflow to use MESH and explicit km-resolution argument
scripts/4.run_model.bash	Updates model runner for MESH, regional LBCs, ERA5, and idealized runs
scripts/3.pre_processing.bash	Updates preprocessing orchestration for MESH, regional, ERA5/idealized, and LBC gen
scripts/2.create_mesh.bash	Adds mesh generation step driven by mesh_input_file.txt
scripts/0.run_all.bash	Updates workflow entrypoint variables and step invocations
docs/tutorial1-meshes.md	Adds tutorial for personalized mesh generation
.gitignore	Refines ignores and keeps mesh data directory placeholder

Comments suppressed due to low confidence (1)

scripts/make_degrib_GFS.bash:110

• For regional runs, boundary file names are built as ...f0${hour}.... This works up to 99h, but for forecast hours >= 100 it will produce f0100, f0120, etc., which don't match standard GFS naming (f100, f120, ...). Format the forecast hour with 3 digits (e.g., %03d) and drop the hardcoded 0 prefix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In colorvalue(), the fallback vmin/vmax computation uses xr.DataArray.min()/.max() without an instance, which will raise at runtime if vmin or vmax are not provided. Use the actual data array (or require callers to always pass vmin/vmax) so this function is safe when called directly.

[Copilot]

eval(args.plot_rings) executes arbitrary Python from the command line (e.g., --plot_rings='__import__("os").system("...")'), which is a code-injection risk. Parse this as a proper boolean flag instead (e.g., action='store_true'/store_false, or a safe string-to-bool converter) and pass the resulting boolean to view_mpas_regional_mesh.

[Copilot]

After mv {regional_mesh} {regional_mesh_info} ... {OUTPUT_DIR}, the subsequent cp {regional_mesh} {regional_mesh_info} ../ uses the old absolute paths in DATA_DIR, which no longer exist post-move. This step will silently fail and the run directory won't get the mesh files. Copy from {OUTPUT_DIR} (or update the variables after moving) instead.

[Copilot]

The graph-info filename is inconsistent: regional_mesh_info is defined as ...{name}.region.graph.info, but gpmetis is invoked on {name}.region.grid.graph.info (note the extra .grid). This will fail because that file won't exist. Use the same filename pattern in both places (and in the earlier mv).

[Copilot]

4.run_model.bash requires 7 arguments (including LBCINT), but this call passes only 6. As written, LBCINT will be empty in the model run, and the LBC input stream templating will be wrong for regional runs. Pass ${LBCINT} as the 7th argument here.

Suggested change

	time ${SCRIPTS}/4.run_model.bash ${EXP} ${MESH} ${YYYYMMDDHHi} ${FCST} ${RES} ${REGIONAL}
	time ${SCRIPTS}/4.run_model.bash ${EXP} ${MESH} ${YYYYMMDDHHi} ${FCST} ${RES} ${REGIONAL} ${LBCINT}

[Copilot]

normalize_cords() claims to return lon in the MPAS range 0..2π, but it only converts degrees to radians and leaves negative longitudes negative. This can produce inconsistent lon conventions compared to other mesh utilities (e.g., xyz_to_latlon() returns 0..2π). Consider normalizing lon after conversion (e.g., modulo 2π) or update the docstring to match the actual behavior.

[Copilot]

The __main__ block calls parse_points(sys.argv[1]), but there is no parse_points function in this module (the parser function is named PointsParser). Running this file directly will crash with NameError. Either remove the __main__ block or update it to call the correct entrypoint.