re-do concat_ice_daily on fail

[aekiss]

payu runs concat_ice_daily.py after each run segment (via postscript.sh), but this occasionally fails with dask errors (in my experience 4-5 times out of 66 one-year run segments).

It would be nice if this would be automatically recovered from, but at present concat_ice_daily.py is only applied to the most recent run output, so previous failed concatentations never get fixed.

Should it do this instead?

• loop over all output directories
  • if output directory contains files of the form *cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].nc then we have a concatenation failure (because these files are deleted on successful concatenation), so
    • delete (or move to a DELETE subdirectory for safety?) any concatenated files of the form *cice.1day.*.[0-9][0-9][0-9][0-9]-[0-1][0-9].nc, as these will stop new attempts at concatenation and may be corrupt
    • do python3 $SCRIPTS_DIR/payu_config/postscript/concat_ice_daily.py --directory <output_directory_path> so that payu will clean up any previous failures in subsequent runs

IIRC this is the approach taken with collation and syncing.

This assumes all the required unconcatenated files*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].nc are still present.

[AndyHoggANU]

Yeah, I have to say that I'm really struggling to post process some of the cice concatenation directories that failed. It seems pretty flaky, TBH, and a pain to go back and concatenate the leftovers...

Maybe we should look into whether we can make the script itself a bit more robust? It's a pretty simple operation, so not sure why it has so many fails??

[anton-seaice]

My plan is to remove the post-processing and build this into the model.

There was a shell script version of this, which was slow but possibly more reliable. It may make sense in the interim to revert to that, although from memory it might have been very slow? 1 hour per model year to do the concatentation.