Feat/hypercore11 migration

[RangerMauve]

Closes #1042

[awana-lockfile-bot]

package-lock.json changes

Summary

Status	Count
	27
	16
	47

Click to toggle table visibility

Name	Status	Previous	Current
@bufbuild/buf-darwin-arm64		1.26.1	-
@bufbuild/buf-darwin-x64		1.26.1	-
@bufbuild/buf-linux-aarch64		1.26.1	-
@bufbuild/buf-win32-arm64		1.26.1	-
@bufbuild/buf-win32-x64		1.26.1	-
@emnapi/core		1.3.1	-
@emnapi/runtime		1.3.1	-
@emnapi/wasi-threads		1.0.1	-
@esbuild/aix-ppc64		0.25.10	-
@esbuild/android-arm		0.25.10	-
@esbuild/android-arm64		0.25.10	-
@esbuild/android-x64		0.25.10	-
@esbuild/darwin-arm64		0.25.10	-
@esbuild/darwin-x64		0.25.10	-
@esbuild/freebsd-arm64		0.25.10	-
@esbuild/freebsd-x64		0.25.10	-
@esbuild/linux-arm		0.25.10	-
@esbuild/linux-arm64		0.25.10	-
@esbuild/linux-ia32		0.25.10	-
@esbuild/linux-loong64		0.25.10	-
@esbuild/linux-mips64el		0.25.10	-
@esbuild/linux-ppc64		0.25.10	-
@esbuild/linux-riscv64		0.25.10	-
@esbuild/linux-s390x		0.25.10	-
@esbuild/netbsd-arm64		0.25.10	-
@esbuild/netbsd-x64		0.25.10	-
@esbuild/openbsd-arm64		0.25.10	-
@esbuild/openbsd-x64		0.25.10	-
@esbuild/openharmony-arm64		0.25.10	-
@esbuild/sunos-x64		0.25.10	-
@esbuild/win32-arm64		0.25.10	-
@esbuild/win32-ia32		0.25.10	-
@esbuild/win32-x64		0.25.10	-
@napi-rs/wasm-runtime		0.2.6	-
@node-rs/crc32-android-arm-eabi		1.10.6	-
@node-rs/crc32-android-arm64		1.10.6	-
@node-rs/crc32-darwin-arm64		1.10.6	-
@node-rs/crc32-darwin-x64		1.10.6	-
@node-rs/crc32-freebsd-x64		1.10.6	-
@node-rs/crc32-linux-arm-gnueabihf		1.10.6	-
@node-rs/crc32-linux-arm64-gnu		1.10.6	-
@node-rs/crc32-linux-arm64-musl		1.10.6	-
@node-rs/crc32-wasm32-wasi		1.10.6	-
@node-rs/crc32-win32-arm64-msvc		1.10.6	-
@node-rs/crc32-win32-ia32-msvc		1.10.6	-
@node-rs/crc32-win32-x64-msvc		1.10.6	-
@tybys/wasm-util		0.9.0	-
b4a		1.6.6	1.8.0
bare-addon-resolve		-	1.10.0
bare-ansi-escapes		-	2.2.3
bare-assert		-	1.2.0
bare-events		2.4.2	2.8.2
bare-inspect		-	3.1.4
bare-module-resolve		-	1.12.1
bare-semver		-	1.0.2
bare-stream		2.1.3	2.8.1
bare-type		-	1.1.0
bare-url		-	2.3.2
compact-encoding		2.15.0	2.19.0
corestore		6.8.4	7.9.1
device-file		-	2.3.1
events-universal		-	1.0.1
fd-lock		-	2.1.1
flat-tree		1.9.0	1.13.0
fs-native-extensions		1.3.2	1.4.5
generate-object-property		-	2.0.0
generate-string		-	1.0.1
hyperblobs		2.3.0	2.9.0
hypercore-crypto		3.4.2	3.6.1
hypercore-errors		1.0.0	1.5.0
hypercore-id-encoding		-	1.3.0
hypercore-storage		-	2.7.1
hypercore		10.19.0	11.27.13
hyperdrive		11.5.3	13.3.0
hyperschema		-	1.20.1
index-encoder		-	3.5.0
is-property		-	1.0.2
ready-resource		1.0.0	1.2.0
refcounter		-	1.0.0
require-addon		-	1.2.0
resolve-reject-promise		-	1.1.0
resource-on-exit		-	1.0.0
rocksdb-native		-	3.13.2
scope-lock		-	1.2.4
signal-promise		-	1.0.3
sodium-native		4.3.1	5.1.0
sodium-universal		4.0.1	5.0.1
speedometer		-	1.1.0
streamx		2.19.0	2.23.0
teex		-	1.0.1

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	ready-resource@​1.0.0 ⏵ 1.2.0			+3	+4
	hyperblobs@​2.9.0
	sodium-universal@​5.0.1
	hypercore-crypto@​3.6.1
	corestore@​7.9.1
	hyperdrive@​13.3.0
	b4a@​1.6.6 ⏵ 1.8.0				-2
	compact-encoding@​2.15.0 ⏵ 2.19.0	+3			+6
	streamx@​2.19.0 ⏵ 2.23.0	+1			+9
	hypercore@​11.27.13

View full report

[RangerMauve]

Issue right now: Hypercore 11 introduced a "remote contiguous range" feature to track which blocks a remote peer has in a row. This has been combined with not sending ranges to peers if they already have data that is available locally. This breaks our code's assumptions because ranges the have value no longer gets those values in the bitfield.

We either need to use something else to calculate the have, or figure out another approach for detecting that a peer still "wants" some blocks they do not yet have.

e.g. broadcastRange early returns now since the local core has the data already and the remote peer knows that.

holepunchto/hypercore@v10.19.0...v11.27.14#diff-3344a84994d8e172768f33ec4bf514ba2bc28eb7eeffff10fcbbd294efe56edcR619

[RangerMauve]

We could also monkey patch the replication logic to get rid of the early returns just for comapeo peers.

[RangerMauve]

Adding proper support for the remoteContiguousLength fixed the want calculation. still need to figure out how to bypass the changes to broadcastRange and the such

[RangerMauve]

Another patched hole is the onhave event. Whenever we got a new bit of data we'd broadcast that info to our peers. This has since been patched to not notify peers we have the data if we already know they have it and don't need it.

I'm going to try to monkey patch the replicator to override its onhave. Or maybe we could monkeypatch the broascastRange event?