ALCF Reconstruction and Segmentation for bl832 by davramov · Pull Request #114 · als-computing/splash_flows

davramov · 2026-01-23T18:32:04Z

This PR adds segmentation logic to the HPCTomographyController, a Prefect task for segmentation, and updated Globus endpoint configuration.

… up)

…t for reconstruction on ALCF

…t for segmentation on ALCF. Still needs to be configured for GPU and the environment with dependencies

…e for the TomographyController. Turning off TIFF to ZARR on ALCF for the demo

…, and segmented data

…conversion

…e deployment

…parate option

…ant to collapse these into the final version, but for testing purposes I'm leaving both codes

…olaris that scales well to multiple gpu nodes

… its own subfolder

…ctions for each segmentation step

…sly separated from dino segmentation

…n, multiresolution flows

… nature comms paper

xiaoyachong

Thanks for your PR. I’ve left a few comments. I didn’t repeat similar issues in nersc.py, but they follow the same pattern. We could address the generalization aspect in a future PR to make ALCF support different segmentation routines.

xiaoyachong · 2026-03-18T18:49:58Z

orchestration/flows/bl832/alcf.py

+        return result
+
+    @staticmethod
+    def _segmentation_sam3_wrapper(


Could we also define a _segmentation_sam3_wrapper() alongside the segmentation_sam3 function in nersc.py? This separation would make the code clearer and improve readability.

xiaoyachong · 2026-03-18T18:52:06Z

orchestration/flows/bl832/alcf.py

+            master_addr = "localhost"
+            print("No PBS_NODEFILE, single node mode")
+
+        venv_path = "/eagle/SYNAPS-I/segmentation/env"


Shall we add venv_path to the function’s argument list?

xiaoyachong · 2026-03-18T18:58:47Z

orchestration/flows/bl832/alcf.py


    @staticmethod
    def _reconstruct_wrapper(
-        rundir: str = "/eagle/IRIProd/ALS/data/raw",


It looks like reconstruct calls _reconstruct_wrapper_multinode, so _reconstruct_wrapper may no longer be used. Should we remove it, or merge the two into a single reconstruct wrapper?

xiaoyachong · 2026-03-18T19:03:50Z

orchestration/flows/bl832/alcf.py

+
+    if dino_success and sam3_success:
+        logger.info("Running segmentation combination (SAM3+DINO).")
+        combine_success = tomography_controller.combine_segmentations(


alcf_forge_recon_multisegment_flow calls combine_segmentations directly on the controller, rather than through alcf_combine_segmentations_task. That seems inconsistent with nersc.py. Should we use the same pattern here?

xiaoyachong · 2026-03-18T19:08:39Z

orchestration/flows/bl832/alcf.py


-            # STEP 2B: Run the Tiff to Zarr Globus Flow
-            logger.info(f"Starting ALCF tiff to zarr flow for {file_path=}")
-            alcf_multi_res_success = tomography_controller.build_multi_resolution(


It seems the new alcf_forge_recon_segment_flow has removed the Zarr conversion. Is that intentional?

xiaoyachong · 2026-03-18T19:12:19Z

orchestration/flows/bl832/prefect.yaml

+  entrypoint: orchestration/flows/bl832/alcf.py:alcf_forge_recon_multisegment_flow
+  work_pool:
+    name: alcf_recon_flow_pool
+    work_queue_name: alcf_forge_recon_segment_flow_queue


It seems that alcf_forge_recon_multisegment_flow and alcf_forge_recon_segment_flow share the same queue name.

xiaoyachong · 2026-03-18T19:15:29Z

scripts/create_latex_for_segmentation_timing.py

This file seems to be intended for the manuscript table, so it includes some hardcoded logic.

xiaoyachong · 2026-03-18T19:20:18Z

scripts/polaris/globus_compute_recon_config.yaml

+    worker_init: "module use /soft/modulefiles; module load conda; conda activate /eagle/SYNAPS-I/reconstruction/env/tomopy; export PATH=$PATH:/eagle/SYNAPSE-I/; cd $HOME/.globus_compute/globus_compute_reconstruction"
+
+    walltime: 00:60:00 # Jobs will end after 60 minutes
+    nodes_per_block: 2 # All jobs will have 1 node


The comment does not match nodes_per_block and max_blocks.

xiaoyachong · 2026-03-18T19:21:29Z

scripts/polaris/globus_compute_recon_config_multinode.yaml

+    scheduler_options: "#PBS -l filesystems=home:eagle"
+
+    # Node setup: activate necessary conda environment and such
+    worker_init: "module use /soft/modulefiles; module load conda; conda activate /eagle/SYNAPS-I/reconstruction/env/tomopy; export PATH=$PATH:/eagle/SYNAPSE-I/; cd $HOME/.globus_compute/globus_compute_reconstruction"


There’s a typo in worker_init: “SYNAPSE-I” should be “SYNAPS-I.”

xiaoyachong · 2026-03-18T19:24:05Z

orchestration/flows/bl832/alcf.py

+        )
+
+    # Prune TIFFs from ALCF scratch/segmentation
+    if alcf_segmentation_success:


Shall we check both alcf_segmentation_success and segment_transfer_success

davramov added 18 commits January 28, 2026 11:50

Updating ALCF endpoints to include the synaps-i allocation (to be set…

31e4c9a

… up)

Updating bl832 config.py to distinguish IRI and SYNAPS-I ALCF endpoints

bf712a4

Adding the config.yaml file for setting up the globus compute endpoin…

77d6bc8

…t for reconstruction on ALCF

Adding the config.yaml file for setting up the globus compute endpoin…

f4f9efa

…t for segmentation on ALCF. Still needs to be configured for GPU and the environment with dependencies

Adding segmentation Prefect task, and segmentation globus compute cod…

0f5d5c9

…e for the TomographyController. Turning off TIFF to ZARR on ALCF for the demo

ensuring self.config for scicat and ghcr images

d3ad219

linting

a96c4a8

Making separate ALCF SYNAPS-I endpoint configs for raw, reconstructed…

5873d94

…, and segmented data

Refactoring ALCF reconstruction flow to use the prune_controller class

49e6e7f

Removing old commented out prune code

922b715

linting and docstrings

c903411

Docstrings, linting, and type hints

5a5cff4

Updating globus compute config for segmentation

b5e0ba9

turning ALCF recon+segmentation into a separate flow from recon+zarr …

54dab5d

…conversion

updating pytest for alcf reconstruction

6361a33

Adjusting endpoint names for synaps

f96c5cd

adding the alcf_forge_recon_segment flow to prefect.yaml as a separat…

8324b7d

…e deployment

updating bl832 dispatcher to include alcf_forge_recon_segment as a se…

7599f2e

…parate option

davramov force-pushed the alcf_recon_segmentation branch from 3b2c506 to 7599f2e Compare January 28, 2026 19:50

davramov added 3 commits February 2, 2026 10:56

adding transfer client uuid for ALCF SYNAPS-I

d78c98d

this configuration worked for launching segmentation on 1 GPU

3c0f25e

Updating segmentation compute endpoint config

e6ebd1f

davramov mentioned this pull request Feb 2, 2026

ALCF Integration #120

Open

davramov added 7 commits February 3, 2026 13:05

Adding a separate wrapper for segmentation v2. At some point we may w…

3293d68

…ant to collapse these into the final version, but for testing purposes I'm leaving both codes

adding globus compute configs for segmentation (single and multinode)

78e9c79

removing old segment config

0778cde

Segmentation wrapper v2 calls the segmentation inference version on p…

93da03a

…olaris that scales well to multiple gpu nodes

removing stale comments

b2ee3ce

removing stale comments

e38815b

Cleaning, adding helpful comments

738ada2

docstring

c73b91a

davramov mentioned this pull request Feb 9, 2026

[bl832] Forge dispatcher for segmentation flows #122

Open

davramov added 24 commits February 10, 2026 10:48

fixing typo

c58e3cd

Segmentation with inference_v4 working

dab25f7

Adding checkpoint v2

477b0de

adding tasks/flows for segmentation with dino and cellpose on ALCF

3c4adb5

Adding combine segmentation results function to the ALCF controller

ea20271

Updating segmentation directory paths to put each models outputs into…

0808a7f

… its own subfolder

Adding combine_segmentations to the multiseg flow

1694b61

updating ALCF conda env for dino/cellpose

ee9ffd0

Updating the combine step to use the dino/cellpose env

0747683

Adding globus compute configs for dino, cellpose, and combine steps

12b02a1

linting

9dd7b70

Adding globus compute endpoint uuids

82ab4f0

updating globus compute endpoint uuid and adding integration test fun…

2a97e75

…ctions for each segmentation step

linting

1fd15aa

adding new flow to prefect.yaml for deployment

2eee18e

updaing dispatcher flow to handle the multisegment option

3a7937a

Updating to match NERSC deployment

f239784

removing commented code

c368653

removing cellpose flows/tasks

9d88b3f

renaming segmentation flow/tasks to segmentation_sam3 so it is obviou…

df663ef

…sly separated from dino segmentation

Docstrings

ede6ff9

Adding _tests/bl832/test_alcf.py to test: reconstruction, segmentatio…

4d2980e

…n, multiresolution flows

Adding script used to generate latex segmentation timing table in the…

5795b11

… nature comms paper

linting

6a30a6c

davramov marked this pull request as ready for review March 13, 2026 21:35

davramov requested review from xiaoyachong March 13, 2026 21:36

xiaoyachong reviewed Mar 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALCF Reconstruction and Segmentation for bl832#114

ALCF Reconstruction and Segmentation for bl832#114
davramov wants to merge 56 commits intoals-computing:mainfrom
davramov:alcf_recon_segmentation

davramov commented Jan 23, 2026

Uh oh!

xiaoyachong left a comment

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

xiaoyachong Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

davramov commented Jan 23, 2026

Uh oh!

xiaoyachong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants