Solution for the problem of the10x Chromium 3' scRNA-seq data stuck in the "salmon_bulk_quantification" step

[jinjinzh1190-cyber]

Hi there! The design of your team's Scalpel tool has left a deep impression on me, and I truly appreciate your open-source contribution! However, I’ve encountered an issue while using it and would greatly appreciate your guidance.
I’m analyzing 14 single-cell samples, which are 10x Genomics 3' sequencing data. The pipeline has been stuck at the annotation_preprocessing:**salmon_bulk_quantification** step. Currently, the progress is "11 of 14, 78%", and it has remained unchanged for nearly 24 hours with no error messages reported.
I wonder if you could help clarify:

1. Is this kind of stagnation at this step common?
2. Could it be caused by data format, parameter settings, or other factors?
3. Are there any suggestions for troubleshooting or resolving this issue?
   I’m happy to provide additional information (such as full logs or configuration files) if needed. I apologize for taking up your time and look forward to your reply!

[mschilli87]

@jinjinzh1190-cyber: Could you tell us a bit about the system you are running this one? How many cores and how much memory do you have? How large are the scRNA-seq samples? Is the system responsive? Are you working on a local disc or a network mount? How much free space do you have on the file system you are running SCALPEL on? Can you open another session on the same system, run htop, make sure you are in Tree view (rather than List view, toggle with F5), filter for processes of your user (using u, selecting your user, and selecting with RETURN), search for SCALPEL's nextflow job (using /, typing nextflow) and check / show us the status (column S) of it's child processes, especially the salmon one?

@Franzx7: Are the salmon logs written on the fly? Do you think we could check them on the fly without aborting the current nextflow run or could they require a flush to disk first?

[jinjinzh1190-cyber]

Hi, here are the details about my system and the dataset:
System configuration:
I’m running SCALPEL on a high-performance computing node with 128 CPU cores and 1 TB of RAM.
The system is generally responsive during the runs.
Storage setup:
The workflow is running under /home on a network-mounted file system (/eaas), with about 100 TB free space out of 189 TB total (48% usage).
Sample size:
Each paired-end scRNA-seq FASTQ file is around 7–14 GB (R1) and 23–45 GB (R2) in size — roughly 30–50 GB per sample.
There are multiple samples (SRR29763195–SRR29763208).
Process monitoring:
When I run htop in Tree view (F5) and filter for my user processes (u → zhoujin → RETURN), the nextflow parent process and its child processes (including salmon) show status “S” (sleeping) most of the time.
The system remains responsive, so they don’t appear to be stuck due to load.
Log behavior:
The salmon .command.err files seem to be updated continuously (I can see progress lines like “processed XXX fragments”), so I think the logs are written on the fly, not buffered.

[mschilli87]

In this case my best guess is that it just takes this long. The XXX in your progress log should give you an ETA.

If I had to guess the bottleneck I'd say it's likely the network-mointed FS. Nextflow is rather I/O heavy and the fact that your salmon jobs are running (as per their progress logs) but spend a lot of time in sleeping state argues they are not CPU bound.
Maybe you could use something along the lines of

pidstat -dul 10 -p $(ps -a | awk '$NF=="salmon"{printf $1;exit}')

to check the actual (CPU utilization and) read and write speeds.
If you have a large enough local drive, I'd consider (temporarily) copying the input data there, running SCALPEL from a working directory on that storage and copying the final results files to your network mount for permanent storage as a (potential) means to speed up the overall (wall clock) time.

---

@Franzx7: Could you comment on expected runtimes for the pseudobulk salmon step given the sample sizes reported above?

[Franzx7]

Hi @jinjinzh1190-cyber ,

very sorry for my delayed response as I was entangled into adminstrative issues and side works.

The execution time you describe is indeed unusually long for samples of that size, which suggests a memory or resource allocation issue.

By default, SCALPEL uses 16 CPUs and 40 GB of memory, as defined in the nextflow.config file in SCALPEL folder:

params.cpus = 16
params.memory = 40.GB

These values can be increased according to your system capacity. Since you’re running on a high-performance node with 1 TB RAM and 128 cores, I recommend raising both CPU and memory limits as you provided 14 samples.

Please give this a try and let me know if the runtime improves.

Best,
Franz

[jinjinzh1190-cyber]

Hello! Sorry to bother you again. I'm encountering some minor issues with my analysis and would like to ask for your help.

As shown in the first image, my process had already run 793 steps, but it stopped due to a memory storage error. However, when I increased the memory and restarted, it began again from the bam_splitting step, as shown in the second image. I already used the -resume option in my code—why didn’t it continue from where it left off?

Looking forward to your reply. Thank you very much!

[mschilli87]

@jinjinzh1190-cyber: Is this still relevant?
@Franzx7: In case it is, would you mind having a look at the screenshots @jinjinzh1190-cyber provided and try to help them (or at least let them know what additional information you'd need to be able to do so)?