Skip to content

Implement Webworkers in Datafetching #601

@TheJeran

Description

@TheJeran

The process of grabbing chunks and converting to Float16 can be a bit slow for very large or poorly chunked data.
Since both Zarr and Netcdf4 support parallel I/O, it should be possible to distribute this process into webworkers.

Right now the main thread calls copyChunk2Array() which writes the chunk values to the main data-array.
I think I can create the main data-array with a SharedArrayBuffer
If I understand this article correctly
https://dev.to/rigalpatel001/high-performance-javascript-simplified-web-workers-sharedarraybuffer-and-atomics-3ig1
I can then call copyChunk2Array() in a webworker while passing the shared buffer to it and the worker will mutate the main array.

This should dramatically speed up datafetching.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions