feat: Add Push Mode

[james-mcnulty]

Linear

STREAM-820

Description

Currently, taskworkers pull tasks from taskbrokers via RPC. This approach works, but has some drawbacks. Therefore, we want taskbrokers to push tasks to taskworkers instead. Read this page on Notion for more information.

This PR allows users to run the taskbroker in push mode. It introduces seven new configuration parameters...

Parameter	Type	Default	Description
TASKBROKER_PUSH_MODE	bool	false	Enables push mode.
TASKBROKER_FETCH_THREADS	usize	1	Sets the number of fetch threads to run.
TASKBROKER_PUSH_THREADS	usize	1	Sets the number of push threads to run.
TASKBROKER_PUSH_QUEUE_SIZE	usize	1	Sets the capacity of the channel sitting in front of the push thread pool.
TASKBROKER_WORKER_ENDPOINT	String	http://127.0.0.1:50052	Sets the worker service endpoint.
TASKBROKER_CALLBACK_ADDR	String	0.0.0.0	Sets the host used in the callback URL.
TASKBROKER_CALLBACK_PORT	usize	50051	Sets the port used in the callback URL.

Push Threads

On startup, the taskbroker now creates a "push pool," which is a pool of push threads. All of them wait to receive activations from the same MPMC channel provided by the flume crate. When a push thread receives an activation, it sends it to the worker service. Note that each push thread has its own connection to the worker service.

Push threads are grouped together by the PushPool data structure, which exposes a start method to actually spawn the threads and a submit method to receive activations.

Fetch Threads

On startup, the taskbroker also creates a "fetch pool," which is a pool of fetch threads. Each one retrieves a pending activation from the store, passes it to the push pool (waiting until it accepts), and repeats.

Notes on Naming

Fetch threads and push threads are actually asynchronous tasks provided by the Tokio crate. They are not real threads.

Details

Dependencies

• Add flume 0.12.0 as a dependency (I didn't want to add any dependencies, but Tokio does not provide an asynchronous MPMC queue - only MPSC)
• Upgrade sentry-protos from 0.4.11 to 0.8.5 (to use the new worker service schema)
• Upgrade tonic, tonic-health, prost, and prost-types to 0.14 (to match the version used by sentry-protos)

Additions

• Add FetchPool abstraction in src/fetch.rs
• Add PushPool abstraction in src/push.rs
• Use push pool and fetch pool abstractions in src/main.rs
• Add configuration parameters for push mode

Modifications

• Return "permission denied" error with explanatory message for get_task when operating in push mode

Future Changes

• Fetch and send tasks in batches
• Update tasks in batches
• Combine upkeep row count queries into a single query
• Delete completed tasks immediately