Adding other file formats option

[jonbrenas]

All the data that the API produces and uses during its analyses is in the Zarr format (https://zarr.readthedocs.io/en/stable/). However, many common external tools (e.g., ADMIXTURE, bcftools, ...) use other file format. Additionally, many genomic research groups use R (https://www.r-project.org) for many of their analyses and Zarr doesn't have modules for use in R.

In the interest of expanding the options for our partners, it would be helpful if some of our data could be exported into other commonly used formats (e.g., PLINK, VCF, ...). Some limited support for Plink-conversions already exists but it could certainly be extended to other types of data.

[Tanisha127]

"Hi, I'm interested in working on this. Could you clarify the scope? For example, should I start with VCF export or PLINK format? Are there existing methods I should build on?"

[joshitha1808]

Hi @jonbrenas ,

Since the data is stored in Zarr format, adding export options to formats like VCF or PLINK would make it much easier to use with tools such as ADMIXTURE, bcftools, or even workflows in R.

It might also be useful to handle the conversion in chunks (for example by genomic windows) so that large genotype datasets don’t have to be loaded fully into memory. I noticed there is already some support for PLINK export , so maybe this could be extended into a more general export utility for multiple formats.

I’d be interested in learning more about which formats would be the most useful to prioritize.

[shauryam2807]

Hello @jonbrenas
I would love to pick this up and expand our export options for external tools like ADMIXTURE and R.

I was just looking through the architecture and noticed we currently have some setup in

to_plink.py
specifically for biallelic_snps_to_plink(). To build this out, would you prefer I start by expanding the broader PLINK functionality (e.g., support for multiallelic variants), or should I focus on establishing a robust VCF export methodology first?

Happy to start engineering an export pipeline for whichever format you think is the highest priority for our partners right now!

[jonbrenas]

Thanks @Tanisha127, @joshitha1808 and @shauryam2807.

The scope is as large or as narrow as you may want. If you think that something can be useful (and you can argue why), then it is probably worth adding to the API.

[adilraza99]

Hi @jonbrenas,

I spent some time looking through the codebase to understand how export functionality currently fits into the architecture.

From what I can see, the repository already has an established pattern for exports through cooperative mixin classes. For example, the PLINK export is implemented via PlinkConverter in to_plink.py, which extends AnophelesSnpData and is mixed into AnophelesDataResource. This suggests that new export capabilities are expected to follow the same pattern rather than being implemented as standalone utilities.

Based on that structure, it seems a natural extension would be to add a similar export class (e.g. VcfExporter) under malariagen_data/anoph/, exposing something like a to_vcf() method that reuses the existing snp_calls() or biallelic_snp_calls() APIs.

One thing that stood out while reviewing the data pipeline is that genotype data is accessed lazily through Dask arrays backed by Zarr. Because of the potential dataset size, a VCF export would likely need to process data chunk-by-chunk rather than materializing the full genotype matrix in memory (unlike the current PLINK export which computes the array before writing).

If this direction aligns with the intended architecture, I’d be happy to work on implementing a VCF export following the existing PlinkConverter pattern and keeping the export region-scoped to match the current API design.

Curious to hear your thoughts.

[Tanisha127]

Thanks @jonbrenas! After looking at the existing PlinkConverter pattern in to_plink.py, I'd like to implement a to_vcf() method following the same mixin architecture. I'll start with biallelic SNP export handling data in chunks given the Dask/Zarr backend. I'll open a draft PR soon.

[shauryam2807]

Thanks @jonbrenas
Since bcftools and similar downstream analysis tools are primarily built around standard VCF formats, I can argue that a VCF exporter will have the highest immediate impact across the largest swath of external tools.

I’m going to start researching and architecting a .to_vcf() method within the API (similar to the current .to_plink() setup) to export SNP/Indel data natively from Zarr back into Variant Call Format. I'll open a Draft PR soon!

[jonbrenas]

I want to reiterate that you are free (and actually encouraged) to come up with your own ideas and plans. I want to point out that, and it might be something that @adilraza99 alluded to, VCF files are very heavy and it is something that you may want to factor in.

[Tanisha127]

Thank you for the feedback @jonbrenas!
I've opened a draft PR with the initial implementation. I'm now working on chunked writing to handle the VCF file size concern you mentioned, so the full genotype matrix won't need to be loaded into memory at once. Will update the PR once that's addressed along with unit tests.

[joshitha1808]

Thanks for the clarification @jonbrenas.

In that case, I’m thinking it might be useful to focus on improving interoperability with common population genetics tools. A practical first step could be supporting a workflow like:

1. Fetch SNP calls from the API
2. Apply filtering / LD pruning
3. Export to PLINK format for tools like ADMIXTURE

Since the datasets can be quite large, implementing this with chunked processing could help keep memory usage manageable.

I’ll explore how this could integrate with the existing biallelic_snps_to_plink functionality and open a draft PR soon.

[Tanisha127]

PR #1060 is now ready for review. Used allel.write_vcf() for memory-efficient writing as suggested @jonbrenas. Happy to add unit tests or make any changes based on your feedback.