double free or corruption (out)

[Flu09]

sce <- zellkonverter::readH5AD(h5ad_path, verbose = TRUE)

Registered S3 methods overwritten by 'zellkonverter':
method from
py_to_r.numpy.ndarray reticulate
py_to_r.pandas.core.arrays.categorical.Categorical reticulate
Loading required package: SeuratObject
Loading required package: sp
‘SeuratObject’ was built with package ‘Matrix’ 1.6.3 but the current
version is 1.6.5; it is recomended that you reinstall ‘SeuratObject’ as
the ABI for ‘Matrix’ may have changed

Attaching package: ‘SeuratObject’

The following object is masked from ‘package:base’:

intersect

ℹ Using the Python reader
ℹ Using anndata version 0.10.2
⠙ Reading '/.../.../.../.../a5463d8f-07df-4870-8cae-bc504de762c8.h5ad '
✔ Read '/.../.../.../.../a5463d8f-07df-4870-8cae-bc504de762c8.h5ad' [18m 37.6s]

ℹ Converting AnnData to SingleCellExperiment
ℹ Converting uns to metadata
ℹ Converting uns
ℹ Converting uns$batch_condition
✔ uns$batch_condition converted [11ms]

ℹ Converting uns$citation
✔ uns$citation converted [12ms]

ℹ Converting uns$schema_reference
✔ uns$schema_reference converted [12ms]

ℹ Converting uns$schema_version
✔ uns$schema_version converted [12ms]

ℹ Converting uns$title
✔ uns$title converted [12ms]

✔ uns converted [181ms]

✔ Converting uns to metadata ... done

ℹ Converting X matrix to assayCellExperiment
double free or corruption (out)
/var/spool/slurm/job32682951/slurm_script: line 26: 2767444 Aborted

I am trying to convert an h5ad object into sce and I have encountered this issue. I will if you can help. Thank you.

[lazappi]

Hi. From a quick internet search this looks this a C error. We don't directly use any C code in {zellkonverter} so it could be coming from another R package, a Python package or maybe something to do with the SLURM job you seem to be using. Finding out exactly where would be difficult but it might help to do things interactively so that you can see the full output. It could have something to do with your R/Python installation so I would check that (the warnings when you load packages also suggest this). We don't use {SeuratObject} in {zellkonverter} so I'm not sure why that is loaded unless you are doing other things in this script.

[Flu09]

i tried only with library(zellkonverter) and the issue persisted. Yes, I am using slurm and conda environment.

Registered S3 methods overwritten by 'zellkonverter':
method from
py_to_r.numpy.ndarray reticulate
py_to_r.pandas.core.arrays.categorical.Categorical reticulate
ℹ Using the Python reader
ℹ Using anndata version 0.10.2
⠙ Reading '/.../.../.../.../a5463d8f-07df-4870-8cae-bc504de762c8.h5ad '
✔ Read '/.../.../.../.../a5463d8f-07df-4870-8cae-bc504de762c8.h5ad' [12m 17.3s]

ℹ Converting AnnData to SingleCellExperiment
ℹ Converting uns to metadata
ℹ Converting uns
ℹ Converting uns$batch_condition
✔ uns$batch_condition converted [12ms]

ℹ Converting uns$citation
✔ uns$citation converted [11ms]

ℹ Converting uns$schema_reference
✔ uns$schema_reference converted [11ms]

ℹ Converting uns$schema_version
✔ uns$schema_version converted [11ms]

ℹ Converting uns$title
✔ uns$title converted [11ms]

✔ uns converted [133ms]

✔ Converting uns to metadata ... done

ℹ Converting X matrix to assayCellExperiment
double free or corruption (out)
/var/spool/slurm/job32692094/slurm_script: line 26: 1922865 Aborted

sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Rocky Linux 9.1 (Blue Onyx)

Matrix products: default
BLAS/LAPACK: /path/conda-environments/R_tools/lib/libopenblasp-r0.3.26.so; LAPACK version 3.12.0

locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Asia/Riyadh
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] zellkonverter_1.12.1

loaded via a namespace (and not attached):
[1] crayon_1.5.2 cli_3.6.2
[3] rlang_1.1.3 png_0.1-8
[5] jsonlite_1.8.8 DelayedArray_0.28.0
[7] dir.expiry_1.10.0 SummarizedExperiment_1.32.0
[9] S4Vectors_0.40.2 RCurl_1.98-1.14
[11] stats4_4.3.2 MatrixGenerics_1.14.0
[13] Biobase_2.62.0 grid_4.3.2
[15] filelock_1.0.3 abind_1.4-5
[17] bitops_1.0-7 SingleCellExperiment_1.24.0
[19] IRanges_2.36.0 basilisk_1.14.3
[21] GenomeInfoDb_1.38.6 compiler_4.3.2
[23] Rcpp_1.0.12 XVector_0.42.0
[25] lattice_0.22-5 reticulate_1.35.0
[27] SparseArray_1.2.4 parallel_4.3.2
[29] GenomeInfoDbData_1.2.11 GenomicRanges_1.54.1
[31] Matrix_1.6-5 tools_4.3.2
[33] matrixStats_1.2.0 zlibbioc_1.48.0
[35] S4Arrays_1.2.0 basilisk.utils_1.14.1
[37] BiocGenerics_0.48.1

[lazappi]

Running readH5AD() inside a conda environment can sometimes be problematic. In that case using AnnData2SCE() can work better but you need to be somewhat familiar with {reticulate}/Python to try that.

Would it be possible for you to share the file? That would help work out if it is something to do with the file or your system. If that's not possible maybe you have another machine you could try running it on?

[Flu09]

Thank you. I will give the AnnData2SCE() a try. It is the neurons object at https://cellxgene.cziscience.com/collections/283d65eb-dd53-496d-adb7-7570c7caa443 (wget https://datasets.cellxgene.cziscience.com/a5463d8f-07df-4870-8cae-bc504de762c8.h5ad)

[Flu09]

I tried AnnData2SCE() with and without conda and still got the same issue : double free or corruption (out).

With module load R

Java Development Kit version 19.0.1 is now loaded
GNU 12.2.0 is now loaded
Loading module for R

R 4.3.2, JDK 19.0.1 and GNU 12.2.0 are now loaded

Loading R/4.3.2/gnu-12.2.0
Loading requirement: java/19.0.1 gcc/12.2.0

Registered S3 methods overwritten by 'zellkonverter':
method from
py_to_r.numpy.ndarray reticulate
py_to_r.pandas.core.arrays.categorical.Categorical reticulate

sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Rocky Linux 9.1 (Blue Onyx)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblas-r0.3.15.so; LAPACK version 3.9.0

locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] zellkonverter_1.12.1

loaded via a namespace (and not attached):
[1] crayon_1.5.2 cli_3.6.2
[3] rlang_1.1.3 png_0.1-8
[5] jsonlite_1.8.8 DelayedArray_0.28.0
[7] dir.expiry_1.10.0 SummarizedExperiment_1.32.0
[9] S4Vectors_0.40.2 stats4_4.3.2
[11] MatrixGenerics_1.14.0 Biobase_2.62.0
[13] grid_4.3.2 abind_1.4-5
[15] filelock_1.0.3 SingleCellExperiment_1.24.0
[17] IRanges_2.36.0 basilisk_1.14.3
[19] GenomeInfoDb_1.38.7 compiler_4.3.2
[21] Rcpp_1.0.12 XVector_0.42.0
[23] lattice_0.21-9 reticulate_1.35.0
[25] SparseArray_1.2.4 parallel_4.3.2
[27] GenomeInfoDbData_1.2.11 GenomicRanges_1.54.1
[29] Matrix_1.6-5 tools_4.3.2
[31] matrixStats_1.2.0 zlibbioc_1.48.0
[33] S4Arrays_1.2.1 basilisk.utils_1.14.1
[35] BiocGenerics_0.48.1

double free or corruption (out)
/var/spool/slurm/job32708472/slurm_script: line 27: 589082 Aborted

With a new conda environment:

sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Rocky Linux 9.1 (Blue Onyx)

Matrix products: default
BLAS/LAPACK: /path/conda-environments/Rtools/lib/libopenblasp-r0.3.26.so; LAPACK version 3.12.0

locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] zellkonverter_1.12.1

loaded via a namespace (and not attached):
[1] crayon_1.5.2 cli_3.6.2
[3] rlang_1.1.3 png_0.1-8
[5] jsonlite_1.8.8 DelayedArray_0.28.0
[7] dir.expiry_1.10.0 SummarizedExperiment_1.32.0
[9] S4Vectors_0.40.2 RCurl_1.98-1.14
[11] stats4_4.3.2 MatrixGenerics_1.14.0
[13] Biobase_2.62.0 grid_4.3.2
[15] filelock_1.0.3 abind_1.4-5
[17] bitops_1.0-7 SingleCellExperiment_1.24.0
[19] IRanges_2.36.0 basilisk_1.14.1
[21] GenomeInfoDb_1.38.1 compiler_4.3.2
[23] Rcpp_1.0.12 XVector_0.42.0
[25] lattice_0.22-5 reticulate_1.35.0
[27] SparseArray_1.2.2 parallel_4.3.2
[29] GenomeInfoDbData_1.2.11 GenomicRanges_1.54.1
[31] Matrix_1.6-3 tools_4.3.2
[33] matrixStats_1.2.0 zlibbioc_1.48.0
[35] S4Arrays_1.2.0 basilisk.utils_1.14.1
[37] BiocGenerics_0.48.1

[lazappi]

I was able to load the file in Python but I ran out of memory trying to read it with {zellkonverter}. Maybe we are hitting the limits of what R can handle in a normal array.

Can you please try:

• Reading the file with Python (to double check your version of the file is ok)
• Saving a subset of the data to a new file
• Reading the new file with R

At the moment, I'm not sure if it's a problem with file, R limitations, a bug in {zellkonverter}, something to do with your setup or something else.

[lazappi]

@Flu09 Were you able to try any of these things?

[lazappi]

Closing this issue. Please comment if you want to follow up.