int64 is not JSON serializable

[jeiros]

Hi,

I'm running into an error that I've not seen before. I'm using already-featurized trajectories stored as text numpy arrays, to compare the performance of different featurizers.

Here's the osprey config file:

estimator:
    eval: |
        Pipeline([
                ('tica', tICA()),
                ('cluster', MiniBatchKMeans()),
                ('msm', MarkovStateModel(lag_time=100, n_timescales=5, verbose=True)),
        ])
    eval_scope: msmbuilder


strategy:
    name: random

search_space:
  cluster__n_clusters:
    min: 10
    max: 10000
    type: int
    

  tica__lag_time:
    min: 1
    max: 1000
    type: jump
    warp: log
    var_type: int
    num: 10

  tica__commute_mapping:
    choices:
      - True
      - False
    type: enum

  tica__n_components:
    min: 1
    max: 20
    type: int

cv:
  name: shufflesplit
  params:
    n_iter: 5
    test_size: 0.25

dataset_loader:
  name: numpy
  params:
    filenames: ftrajs/*
trials:

  uri: sqlite:///osprey-trials.db

And here's the output: https://pastebin.com/gtZxxwEt

I've previously done this kind of workflow, so I don't really know what might've gone wrong.

Best regards,

Juan

[cxhernandez]

Hi @jeiros!

Thanks for the report! Could you also share your conda environment? Relevant package versions would be msmbuilder, numpy, sklearn, cython, etc.

[jeiros]

Sure! Forgot to do that.

Heres the output of conda list:

# packages in environment at /home/juan/miniconda3:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.10           py36h306e16b_0
ambertools                18.0                          0    http://ambermd.org/downloads/ambertools/conda
asn1crypto                0.24.0                   py36_0
babel                     2.5.3                    py36_0
bleach                    2.1.3                      py_0    conda-forge
bzip2                     1.0.6                h9a117a8_4
ca-certificates           2018.03.07                    0
certifi                   2018.4.16                py36_0
cffi                      1.11.4           py36h9745a5d_0
chardet                   3.0.4            py36h0f667ec_1
conda                     4.5.9                    py36_0
conda-env                 2.6.0                h36134e3_1
corner                    2.0.0                    py36_0    omnia
cryptography              2.1.4            py36hd09be54_0
cycler                    0.10.0           py36h93f1223_0
dbus                      1.13.2               h714fa37_1
decorator                 4.2.1                    py36_0    conda-forge
docutils                  0.14             py36hb0f60f5_0
entrypoints               0.2.3                    py36_1    conda-forge
expat                     2.2.5                he0dffb1_0
fastcluster               1.1.20              np112py36_0    omnia
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_0
glib                      2.56.1               h000015b_0
gmp                       6.1.2                         0    conda-forge
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
hdf5                      1.10.1               h9caa474_1
html5lib                  1.0.1                      py_0    conda-forge
icu                       58.2                 h9c2bf20_1
idna                      2.6              py36h82fb2a8_1
imagesize                 1.0.0                    py36_0
intel-openmp              2018.0.0                      8
ipykernel                 4.8.2                    py36_0    conda-forge
ipython                   6.2.1                    py36_1    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
jedi                      0.11.1                   py36_0    conda-forge
jinja2                    2.10                     py36_0    conda-forge
jpeg                      9b                   h024ee3a_2
jsonschema                2.6.0                    py36_1    conda-forge
jupyter_client            5.2.3                    py36_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                0.31.12                  py36_1    conda-forge
jupyterlab_launcher       0.10.5                   py36_0    conda-forge
kiwisolver                1.0.1            py36h764f252_0
libedit                   3.1                  heed3624_0
libffi                    3.2.1                hd88cf55_4
libgcc                    7.2.0                h69d50b8_2
libgcc-ng                 7.2.0                h7cc24e2_2
libgfortran-ng            7.2.0                hdf63c60_3
libpng                    1.6.34               hb9fc6fc_0
libsodium                 1.0.15                        1    conda-forge
libstdcxx-ng              7.2.0                h7a57d05_2
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.8                hf84eae3_0
lzo                       2.10                 h49e0be7_2
markupsafe                1.0                      py36_0    conda-forge
matplotlib                2.2.2            py36hb69df0a_2
mdtraj                    1.9.1                    py36_0    omnia
mistune                   0.8.3                      py_0    conda-forge
mkl                       2018.0.2                      1
msmadapter                0.1rc0                    <pip>
msmbuilder                3.8.0               np112py36_0    omnia
msmexplorer               1.1.0                    py36_0    omnia
natsort                   5.3.0                     <pip>
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                    py36_0    conda-forge
ncurses                   6.0                  h9df7e31_2
networkx                  2.1                      py36_0
notebook                  5.4.1                    py36_0    conda-forge
numexpr                   2.6.4            py36hc4a3f9a_0
numpy                     1.12.1           py36he24570b_1
numpydoc                  0.7.0            py36h18f165f_0
openssl                   1.0.2o               h14c3975_1
osprey                    1.1.0                    py36_0    omnia
packaging                 17.1                     py36_0
pandas                    0.22.0           py36hf484d3e_0
pandoc                    2.1.3                         0    conda-forge
pandocfilters             1.4.1                    py36_0    conda-forge
parso                     0.1.1                      py_0    conda-forge
patsy                     0.5.0                    py36_0
pcre                      8.42                 h439df22_0
pexpect                   4.4.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pip                       9.0.1            py36h6c6f9ce_4
prompt_toolkit            1.0.15                   py36_0    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
pycosat                   0.6.3            py36h0a5515d_0
pycparser                 2.18             py36hf9f622e_1
pygments                  2.2.0                    py36_0    conda-forge
pyhmc                     0.1.2               np112py36_1    omnia
pyopenssl                 17.5.0           py36h20ba746_0
pyparsing                 2.2.0            py36hee85983_1
pyqt                      5.9.2            py36h22d08a2_0
pysocks                   1.6.7            py36hd97a5b1_1
pytables                  3.4.2            py36h3b5282a_2
python                    3.6.4                hc3d631a_1
python-dateutil           2.7.2                      py_0    conda-forge
pytz                      2018.3                   py36_0
pyyaml                    3.12             py36hafb9ca4_1
pyzmq                     17.0.0                   py36_3    conda-forge
qt                        5.9.6                h52aff34_0
readline                  7.0                  ha6073c6_4
requests                  2.18.4           py36he2e5f8d_1
ruamel_yaml               0.15.35          py36h14c3975_1
scikit-learn              0.19.1           py36h7aa7ec6_0
scipy                     1.0.0            py36hbf646e7_0
seaborn                   0.9.0                    py36_0
send2trash                1.5.0                      py_0    conda-forge
setuptools                38.4.0                   py36_0
simplegeneric             0.8.1                    py36_0    conda-forge
sip                       4.19.8           py36hf484d3e_0
six                       1.11.0           py36h372c433_1
snowballstemmer           1.2.1            py36h6febd40_0
sphinx                    1.7.1                    py36_0
sphinxcontrib             1.0              py36h6d0f590_1
sphinxcontrib-websupport  1.0.1            py36hb5cb234_1
sqlalchemy                1.2.10           py36h14c3975_0
sqlite                    3.23.1               he433501_0
statsmodels               0.8.0            py36h8533d0b_0
terminado                 0.8.1                    py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
tk                        8.6.7                hc745277_3
tornado                   5.0.1                    py36_1    conda-forge
tqdm                      4.24.0                   py36_0
traitlets                 4.3.2                    py36_0    conda-forge
typing                    3.6.4                    py36_0
uncertainties             3.0.2                     <pip>
urllib3                   1.22             py36hbe7ace6_0
wcwidth                   0.1.7                    py36_0    conda-forge
webencodings              0.5                      py36_0    conda-forge
wheel                     0.30.0           py36hfd4bba0_1
xz                        5.2.3                h55aa19d_2
yaml                      0.1.7                had09818_2
zeromq                    4.2.3                         2    conda-forge
zlib                      1.2.11               ha838bed_2

Python version is 3.6.

This is running on the Ubuntu shell that comes now with Windows 10, however, I've been able to replicate the problem in a different machine using OSX.

[jeiros]

Here's a small subset of the data with which the error can be reproduced

dihedrals.zip

[jeiros]

Did you manage to replicate the error?

[jeiros]

So I think the problem is in specifying a list of integers in a log warped space:

  tica__lag_time:
    min: 1
    max: 1000
    type: jump
    warp: log
    var_type: int
    num: 10

I changed it to a type: int and now it works.
Perhaps the int64 is not JSON serializable error comes from within the Osprey code that deals with the creation of the integers in the log space? Is it maybe using numpy, which stores the integers as the int64 dtype and that confuses the json package? No idea, since I'm not familiar with osprey's code, it's just a guess.

Let me know what you think

Edit: I'm pretty certain that's the problem, apparently only floats accept a log space. Maybe it'd be a useful addition to accept integers?

Edit2: I saw this PR which seems to deal with the same issue but for gp optimization.

[cxhernandez]

Sorry! I've been busy with moving, and haven't had much time to check this out.

Perhaps the int64 is not JSON serializable error comes from within the Osprey code that deals with the creation of the integers in the log space? Is it maybe using numpy, which stores the integers as the int64 dtype and that confuses the json package? No idea, since I'm not familiar with osprey's code, it's just a guess.

This is quite plausible. Does the #236 solve the issue for you, or does this require an additional fix? That PR needs some additional work to make it py27 compatible (although, I've very open to discontinuing py27 support).