linear speedup on new H5readConcurrent

[JohnLCaron]

experimental concurrent reading all data one variable from

netcdf4 /home/all/testdata/cdmUnitTest/formats/netcdf4/hiig_forec_20140208.nc 1918.164279 Mbytes
variable salt

netcdf4 with Btree1 layout.

[JohnLCaron]

log-log plot show linear scaling up to 20 threads on my computer. (24 CPUs)

[JohnLCaron]

Ive modified the chunked data reading to use readChunksConcurrent() transparent to the user.

readArrayData returns the array as ArrayTyped<T>
chunkIterator returns the data chunks as Iterator<ArraySection>
concurrentChunks returns each data chunk in a multhreaded callback(chunk: <ArraySection>)

where

ArraySection<T>(val array : ArrayTyped<T>, val chunkSection : Section)

netcdf4 /home/all/testdata/cdmUnitTest/formats/netcdf4/hiig_forec_20140208.nc 1918.164279 Mbytes
salt nelems = 118,041,300 float

[JohnLCaron]

netcdf4 ../core/src/commonTest/data/netcdf4/tiling.nc4 18.895259999999997 Mbytes
Turbulence_SIGMET_AIRMET nelems = 66,859,300 float (deflate)

[JohnLCaron]

@lesserwhirls can you run

timeH5compareReading() in com/sunya/netchdf/hdf5/H5readConcurrentTest.kt

sometime?

[lesserwhirls]

Looks pretty similar:

netcdf4 /Users/sarms/dev/local/data/thredds-test-data/cdmUnitTest/formats/netcdf4/hiig_forec_20140208.nc 1918.164279 Mbytes
  float salt[57, 36, 195, 295] nelems = 118041300
readArrayData
nthreads,   time in secs
1, 2.479725625
2, 1.260284458
4, 0.680148959
8, 0.454596125
10, 0.378277542
16, 0.420353792
20, 0.33219283400000005
24, 0.33358395900000004
32, 0.348690958
40, 0.33962416700000003
48, 0.34222612500000005

chunkIterator
nthreads,   time in secs
1, 2.235157875
2, 1.1451482910000002
4, 0.601439625
8, 0.327130333
10, 0.298860834
16, 0.28041345900000003
20, 0.285436917
24, 0.278139541
32, 0.293609375
40, 0.294536625
48, 0.28689220800000004

chunksConcurrent
nthreads,   time in secs
1, 2.246344209
2, 1.1365295420000001
4, 0.5887669170000001
8, 0.30914508300000004
10, 0.292193333
16, 0.273588125
20, 0.280758583
24, 0.278522166
32, 0.27272729100000004
40, 0.275331041
48, 0.269412625

Read chunked data
nthreads,  time in secs
      , readArrayData,  chunkIterator,  chunksConcurrent, 
    1, 2.479725625,2.235157875,2.246344209,
    2, 1.260284458,1.1451482910000002,1.1365295420000001,
    4, 0.680148959,0.601439625,0.5887669170000001,
    8, 0.454596125,0.327130333,0.30914508300000004,
    10, 0.378277542,0.298860834,0.292193333,
    16, 0.420353792,0.28041345900000003,0.273588125,
    20, 0.33219283400000005,0.285436917,0.280758583,
    24, 0.33358395900000004,0.278139541,0.278522166,
    32, 0.348690958,0.293609375,0.27272729100000004,
    40, 0.33962416700000003,0.294536625,0.275331041,
    48, 0.34222612500000005,0.28689220800000004,0.269412625,

Chip: Apple M2 Pro
Total Number of Cores: 12 (8 performance and 4 efficiency)
Memory: 32 GB
Storage: APPLE SSD AP2048Z
  Sequential Reads: 6720 MiB/S
  Random Reads: 2863 MiB/S

[lesserwhirls]

I can try to run this reading data from a spinning disk to see if we get the same behavior.

[JohnLCaron]

Mines a SSD but slower, something like 2050 MB/s read.
Ive updated the plot above yours with the hardware description.

[JohnLCaron]

Previous tests have shown that java deflate is a 2x slowdown compared to native deflate. Might be worth playing with native library for that. Might be tricky to figure that out transparently.