Deprecation warning when loading old H5 files with Numpy >= 2.4

[gdementen]

It seems like it is caused by unpickling some data inside the H5 file.

The issue is in pytables. I hope it is PyTables/PyTables#1290 and thus already fixed for pytables 3.11, but I am unsure.

I also do not know if the bad code is because of an old pytables version (and the code is embedded inside the pickle string in the datafile) or if that datafile triggers some code path in the currently installed pytables version. If that's the former, this is extremely annoying because we would need to tell our users to update their old data files before numpy stops working with this bad code.

If it is the later, we can just ignore the warning and wait for pytables to be updated (which might be the case already in 3.11 but even if not it will be so eventually). I will leave this issue open until I know which option it is.

[gdementen]

Oh well, it turns out the bad code is embedded in the datafile 😭 . See PyTables/PyTables#1310 and numpy/numpy#29301

[gdementen]

And I really need to handle it now because CI currently fails because of this. Sigh... Ideally, I should :

1. catch the warning and re-output it with a clearer error message (and better stacklevel).
2. change the test_session.h5 test file to not use it
3. create and add an MRE/small test file reproducing the problem. Edit: I created an MRE (without using larray) in the pytables issue.
4. add a test using that test file (which should use must_warn)

[gdementen]

Here is an MRE using larray (had to install larray in both environment mentioned in pytables issue and downgrade Pandas to "pandas=2" in the "new" environment to get it to work):

> conda activate pickle_dtype_old
> python -c "import larray;larray.Axis('a=a0..a9')['a1,a3'].to_hdf('test.h5', 'test')"
> conda activate pickle_dtype_new
> python -c "import larray;larray.read_hdf('test.h5', 'test')"
<env>\Lib\site-packages\pandas\compat\pickle_compat.py:80: VisibleDeprecationWarning: dtype(): align should be passed as Python or NumPy boolean but got `align=0`. Did you mean to pass a tuple to create a subarray type? (Deprecated NumPy 2.4)
  stack[-1] = func(*args)

I am hesitant to include it in our test suite because that test file in 1Mb ! 😞