Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note that CoMorMent containers are organized using several GitHub repositories:

All of the above repositories are covered by this CHANGELOG. They will have the same version tags on github. In addition, we have repositories containing specific tools, e.g. https://github.com/comorment/HDL, which will be covered by their own CHANGELOG.md file.

To identify the version of a .sif file, run md5sum <container>.sif command and find the MD5 checksum in the list below. If MD5 sum is not listed for a certain release then it means that the container hasn’t been changed from the previous release.

[Unreleased]

Added

  • Added options --extract, --extract-step1, --extract-step2, --exclude, --exclude-step1, and --exclude-step2 to the gwas.py script to enable inclusion and exclusion of SNPs

  • Added Rstudio-server and R packages info to r.sif container documentation

Updated

  • Rebuilt gwas.sif container with md5sum checksum:

    4e295149f3a5e25588cc4a1f1d39876c  singularity/gwas.sif

  • Compile regenie with HAS_BOOST_IOSTREAM=1 and HTSLIB_PATH options

  • Change LDpred2 usage example to use the OpenSNP based datasets

  • Bundle of sphinx documentation build updates/restructures

  • Refer to the project as “COSGAP-containers”

  • Minor changes to documentation + suggestion of TOC

  • migrate online documentation to cosgap.readthedocs.io

  • updated documentation to reflect the new project name

  • added references/urls to software tables in the documentation for singularity containers

  • update citation info

Fixed

  • Fixed brittle tests if TMPDIR is not /tmp

Removed

  • Removed Saige support and Saige-related files

Misc

  • Miscellaneous goes here

[1.8.1] - 2024-03-05

Fixed

  • Fixed parsing of IID field in pheno.dict

  • Fixed issue with files with different suffixes produced by plink2 for binary phenotypes in gwas.py

[1.8.0] - 2024-02-22

Added

  • Added scripts to analyze and filter bigSNPR LD matrixes (scripts/pgs/LDpred2/analyzeLD.R, scripts/pgs/LDpred2/splitLD.R).

[1.7.2] - 2024-02-14

Updated

  • Rebuilt r.sif container with md5sum checksum:

    3d69fc2168ef98d1eda3da05391cd6e4  singularity/r.sif

Added

  • added CC-GWAS R package to r.sif container

[1.7.1] - 2024-02-06

Fixed

  • Fixed parsing of --genomic-build hg18/hg38 in ldpred2.R

[1.7.0] - 2024-02-02

Added

  • Added samtools 1.19.2, bedtools 2.31.1, liftOver (latest) to gwas.sif container

  • Added corresponding unit tests

Updated

  • Updated the following binaries (not listing apt package updates) in gwas.sif built

    • bcftools to 1.19

    • bolt to 2.4.1

    • gcta to 1.94.1

    • gctb to 2.04.3

    • htslib to 1.19.1

    • king to 2.3.2

    • minimac4 to 4.1.6

    • plink to v1.90b7.2 64-bit (11 Dec 2023)

    • plink2 to v2.00a5.10LM 64-bit Intel (5 Jan 2024)

    • plink2_avx2 to v2.00a5.10LM AVX2 Intel (5 Jan 2024)

    • PRSice_linux to 2.3.5

    • regenie to 3.4.1

    • vcftools to git SHA: d511f469e87c2ac9779bcdc3670b2b51667935fe (0.1.17dev)

  • Rebuilt gwas.sif w. md5sum checksum:

    a775f4216b15b731471821d0c2a0da43  singularity/gwas.sif

  • updated installer scripts

Fixed

  • Broken docker/scripts/build_docker.sh script

[1.6.0] - 2023-12-12

Added

  • Added gdb debugger, ldak and snptest binaries to gwas.sif container

  • Added tests for ldak and snptest binaries in gwas.sif container

Updated

  • updated metal to version 2020-05-05 in gwas.sif

  • updated qctool to v2.2.2 and added related binaries inthinnerator, hptest, ldbird and selfmap to gwas.sif

  • rebuilt gwas.sif (md5 checksum b6104b58d21f862f9d61a86d9d4802a6)

[1.5.1] - 2023-10-20

Fixed

  • Fixed broken ReadTheDocs documentation build

[1.5.0] - 2023-10-17

Added

  • Added <containers>/scripts/pgs/pgs_toolkit, a Python toolkit for computing PGS using LDpred2, PRSice2 or PLINK

  • Added <containers>docker/scripts/build_docker.sh script replacing corresponding build statement in Makefile

  • Added test for gcta

Updated

  • Updated r.sif build with many additional R packages, with corresponding updates to build recipes and tests

  • Use https://packagemanager.posit.co/cran/__linux__/focal/2023-02-16 as main R package repo

  • r.sif md5 checksum:

    1280ba24d99664d450b2e4c4a9c00587  singularity/r.sif

  • Updated GitHub workflow versions to current versions

Removed

  • removed logging of docker build ... in docker/Makefile (issues with piping to tee in case of build errors)

[1.4.0] - 2023-10-17

Added

  • Added phasing/imputation tools beagle, duohmm, eagle, shapeit5, switchError, to gwas.sif container + updated tests

Fixed

  • Fix issue that shell script wouldn’t capture failing statements

Updated

  • Updated gwas.sif Dockerfile and installed shell scripts (misc. dependencies updates, installing gcta version 1.93.3beta2)

  • Rebuilt gwas.sif using Docker --no-cache option to fix missing minimac4 binary, w. md5 checksum:

    a1dd235221902741bf5773945a584e47  singularity/gwas.sif

Removed

  • Removed unused install_miniconda.sh script from src/scripts folder

[1.3.9] - 2023-10-17

Added

  • User-set directory option for temporary files during LDpred2 runs, by default base::tempdir()

[1.3.8] - 2023-10-17

Fixed

  • Added --genomic-build hg18/hg19/hg38 option to ldpred2.R to use correct LD reference meta file pos column name

[1.3.7] - 2023-10-17

Added

  • Added a feature to read and convert BGEN (.bgen) files to scripts/pgs/LDpred2/createBackingFile.R

[1.3.7] - 2023-10-12

  • User-set directory for temporary files during LDpred2 runs, by default base::tempdir()

[1.3.6] - 2023-08-17

Fixed

  • Ignore LDpred2 --col-bp <column> arg in case --merge-by-rsid is used

[1.3.5] - 2023-08-17

Updated

  • Updated LDpred2 README file

[1.3.4] - 2023-06-22

Updated

  • Update regenie to v3.2.8

Fixed

  • #187 - Regression in gwas.py in handling of info, maf, hwe and geno filters

[1.3.3] - 2023-06-14

Updated

  • Removed time consuming genotype missingness check from ldpred2.R.

[1.3.2] - 2023-06-12

Fixed

  • Fixed misc. issues with cross references in online documentation

[1.3.1] - 2023-06-07

Added

  • Added unittest for uppercase chromosome column name in sumstats files, that may also contain chromosomes encoded as character(s)

Fixed

  • Fixed issue with character encoding in sumstats files, in case chromosome column name is uppercase.

[1.3.0] - 2023-05-19

Added

  • Added to ldpred2.R: Multi-threading of snp_ldsc, arguments for parameters to snp_ldpred2_auto, and alternative effective sample-size calculation through --n-cases and n-controls.

Fixed

  • Solved error due to case-sensitive handling of --col-chr in ldpred2.R and naming of diagnostic plot when using --name-score.

[1.2] - 2023-05-11

Added

  • Added RELEASES.md file explaining steps needed to make releases.

  • Added PRSice_linux to r.sif

  • Added tests for gwas.py

  • Added package GWASTools to r.sif.

  • Added confidence intervals to qq plots created by gwas.py using GWASTools R package.

  • Added status badges and citation.cff file

Updated

  • Updated file and folder layout, fixing minor documentation issues. Moving from m2r2 to Myst-parser for Sphinx-generated online docs.

  • Rebuilt the R container

  • 5ecbfc50f96bc6b25f61858927283e2d  singularity/r.sif

  • Rebuilt the R container

    23d195a10b84603b15d0e8c42df40fbd  singularity/r.sif

Fixed

  • Set version file info to 1.2.dev (was 0.1.1dev)

  • Fixed bad parsing of arbitrary length list of args in usecases/LDpred2/complementSumstats.R

  • Made usecases/LDpred2/complementSumstats.R write output file by default, not stdout.

  • Fixed print statement in usecases/LDpred2/complementSumstats.R causing crash w. --file-output arg.

  • Fixed ldpred2.R script in case --file-pheno/--col-pheno/--col-pheno-from-fam args were used, by removing these options altogether.

  • Use packagemanager.rstudio.com/cran/linux/focal/2023-02-16 as main R package repo

  • gwas.py --variance-standardize option now throws an error when applied to columns with no variance

Removed

  • Removed redundant usecases/LDpred2_tutorial files

Misc

  • Python code max line length of 120 chars, ignore number of newlines between functions

Misc

  • Python code max line length of 120 chars, ignore number of newlines between functions

[1.1] - 2022-12-01

Maintenance/feature release with the following main software incorporated into each container:

container

OS/tool

version

license

hello.sif

ubuntu

20.04

Creative Commons CC-BY-SA version 3.0 UK licence

hello.sif

plink

v1.90b6.18 64-bit (16 Jun 2020)

GPLv3

gwas.sif

ubuntu

20.04

Creative Commons CC-BY-SA version 3.0 UK licence

gwas.sif

plink

v1.90b6.18 64-bit (16 Jun 2020)

GPLv3

gwas.sif

plink2

v2.00a3.6LM 64-bit Intel (14 Aug 2022)

GPLv3

gwas.sif

plink2_avx2

v2.00a3.6LM AVX2 Intel (24 Jan 2020)

GPLv3

gwas.sif

PRSice_linux

2.3.3 (2020-08-05)

GPLv3

gwas.sif

simu_linux

v0.9.4

GPLv3

gwas.sif

bolt

v2.4 July 22, 2022

GPLv3

gwas.sif

gcta64

version 1.93.2 beta Linux

GPLv3

gwas.sif

gctb

2.02

MIT

gwas.sif

qctool

2.0.6, revision 18b8f17

Boost

gwas.sif

king

2.2.9 - ©

permissive

gwas.sif

metal

version released on 2011-03-25

-

gwas.sif

vcftools

0.1.17

GPLv3

gwas.sif

bcftools

1.12 (using htslib 1.12)

MIT/Expat/GPLv3

gwas.sif

flashpca_x86-64

2.0

GPLv3

gwas.sif

regenie

v2.0.2.gz

MIT/Boost

gwas.sif

GWAMA

2.2.2

BSD-3-Clause

gwas.sif

minimac4

v4.1.0

GPLv3

gwas.sif

bgenix

1.1.7

Boost

gwas.sif

cat-bgen

same version as bgenix

Boost

gwas.sif

edit-bgen

same version as bgenix

Boost

gwas.sif

HTSlib

1.12

MIT/Expat/Modified-BSD

gwas.sif

shapeit4.2

v4.2.2

MIT

python3.sif

ubuntu

20.04 (LTS)

Creative Commons CC-BY-SA version 3.0 UK licence

python3.sif

python3

python 3.10.6 + numpy, pandas, etc.

PSF

python3.sif

LDpred

1.0.11

MIT

python3.sif

python_convert

github commit bcde562

GPLv3

python3.sif

plink

v1.90b6.18 64-bit (16 Jun 2020)

GPLv3

r.sif

ubuntu

20.04

Creative Commons CC-BY-SA version 3.0 UK licence

r.sif

R

4.0.5 (2021-03-31) + data.table, ggplot, etc.

misc

r.sif

gcta64

version 1.93.2 beta Linux

GPLv3

r.sif

PRSice_linux

2.3.3 (2020-08-05)

GPLv3

r.sif

rareGWAMA

dajiangliu/rareGWAMA@72e962d

-

r.sif

GenomicSEM

GenomicSEM/GenomicSEM@bcbbaff

GPLv3

r.sif

TwoSampleMR

MRCIEU/TwoSampleMR@c174107

unknown/MIT

r.sif

GSMR

v1.0.9

GPL>=v2

r.sif

snpStats

v1.40.0

GPLv3

saige.sif

ubuntu

16.04

Creative Commons CC-BY-SA version 3.0 UK licence

saige.sif

SAIGE

version 0.43

GPLv3

Main changes since release version 1.0.0:

Added

  • add option to append usecases/LDpred2/ldpred.R score output to an existing file

  • add script usecases/LDpred2/complementSumstats.R to append chromosome and position to summary statistics

  • add polygenic score output tests for usecases/LDpred2/ldpred.R

  • add usecases/LDpred2/imputeGenotypes.R for imputing genotypes using R-package bigSNPR

  • add usecases/LDpred2/calculateLD.R for calculation LD using R-package bigSNPR.

  • add autobuilt online documentation from repository sources at https://comorment-containers.readthedocs.io/en/latest/

  • add R libraries for LDpred2 analysis to r.sif + corresponding example.

  • add tests for metal and qctool in gwas.sif build

  • add basic GitHub actions from https://github.com/precimed/container_template.git

  • add FaST-LMM (version 0.6.3) to future python3.sif, and corresponding test

  • add shapeit4.2 binary (shapeit4 v.4.2.2) and HTSlib (1.11) to future gwas.sif builds, and corresponding test

  • added additional tests for software in gwas.sif, python3.sif builds

  • add versions identifiers for all explicitly installed software across hello.sif, gwas.sif, python3.sif, r.sif, listed in docker/README.md

  • replaced Ubuntu 18.04 with 20.04 (LTS) as base image for hello.sif, gwas.sif, python3.sif

  • replaced src/scripts/install_miniconda3.sh by scr/scripts/install_mambaforge.sh which is now used in future python3.sif builds

  • add tests for bgenix and Minimac4 software in gwas.sif, removing build-time dependencies for these from container

  • add basic test that KING software runs in gwas.sif

  • add Dockerfiles and install scripts for gwas.sif, hello.sif, python3.sif, r.sif, saige.sif from gwas.

  • add CHANGELOG.md (this file)

  • add gwas.py --analysis saige option, allowing to run SAIGE analysis

  • add gwas.py --analysis figures option, using R qqman for QQ and manhattan plots

  • add gwas.py --pheno-sep and --dict-sep options to specify delimiter for the phenotype file and phenotype dictionary file

  • add package qqman to r.sif

  • add package yaml to python3.sif

  • add gctb_2.0_tutorial.zip reference files under reference/examples/gctb_2.0_tutorial

  • add config.yaml file with configuration options, which can be specified via gwas.py --config option

  • add --chunk-size-bp and --bim option, allowing to run SAIGE analysis in smaller chunks

  • add --keep and --remove options to gwas.py, allowing to keep and remove subsets of individuals from analysis; the functions work similarly to plink2 as described here.

Updated

  • rebuilt the following containers following version pinning in Dockerfiles, install scripts, etc. (see above additions):

    bb7a8e0b977e29e03067d75d19803913  singularity/gwas.sif
11ac9e8fe69df07d650bd5e1e7cdeee5  singularity/hello.sif
c78d57397471ee802d37837ca5f8b797  singularity/python3.sif
e8f26b23a8b44f15f3dfff2b02623780  singularity/r.sif
a3f1d8411e1e3cf8670551b7f334a58d  singularity/saige.sif

Fixed

  • usecases/LDpred2/ldpred2.R error when sumstats contain characters in chromosome column.

  • use afterok spec instead of afterany in SLURM dependencies so that next steps of the pipeline don’t run if a previous step has failed (fix #26)

  • use SLURM’s cpus_per_task=1 for SAIGE step2, because it doesn’t support –nThreads (see https://github.com/saigegit/SAIGE/issues/9)

Removed

  • removed --geno-impute from usecases/LDpred2/ldpred2.R. Functionality replaced by --geno-impute-zero and usecases/LDpred2/imputeGenotypes.R

  • removed misc. source/data files in /tools/* from container builds

  • removed unused libquadmath0 library from builds (affecting future gwas.sif, hello.sif, and python3.sif builds)

  • the following command-line options are removed; instead, they can be specified via config.yaml file: --slurm-job-name, --slurm-account, --slurm-time, --slurm-cpus-per-task, --slurm-mem-per-cpu, --module-load, --comorment-folder, --singularity-bind. Note that config.yaml file is now required.

  • gwas.py --analysis loci manh qq options as removed (fix #22)

  • --bed-fit, --bed-test, --bgen-fit, --bgen-test options of gwas.py are removed; use new options --geno-fit-file and --geno-file instead

  • remove regenie.sif and regenie3.sif, because regenie software is also included in gwas.sif

  • remove MiXeR package from python3.sif container, because MiXeR is now available as a separate container (https://github.com/comorment/mixer). This is also where you will find MiXeR’s use-cases.

  • MAGMA, LAVA and ldblock software is moved to https://github.com/comorment/magma. MAGMA reference files are also moved to this repository.

  • enigma-cnv.sif and enigma-cnv.sif is moved to https://github.com/comorment/iPsychCNV enigma-cnv.sif is also available here: in https://github.com/ENIGMA-git/ENIGMA-CNV/tree/main/CNVCalling/containers

  • tryggve_query.sif is moved to https://github.com/comorment/Tryggve_psych

  • matlabruntime.sif container is moved to https://github.com/comorment/matlabruntime. pleioFDR reference files are also moved to this repository.

[1.0.0] - 2020-10-20

Added

  • initial release of the following containers:

    70502c11d662218181ac79a846a0937a  enigma-cnv.sif
1ddd2831fcab99371a0ff61a8b2b0970  gwas.sif
b02fe60c087ea83aaf1b5f8c14e71bdf  hello.sif
1ab5d82cf9d03ee770b4539bda44a5ba  ipsychcnv.sif
6d024aed591d8612e1cc628f97d889cc  ldsc.sif
2e638d1acb584b42c6bab569676a92f8  matlabruntime.sif
331688fb4fb386aadaee90f443b50f8c  python3.sif
cdbfbddc9e5827ad9ef2ad8d346e6b82  r.sif
b8c1727227dc07e3006c0c8070f4e22e  regenie.sif
97f75a45a39f0a2b3d728f0b8e85a401  regenie3.sif
20e01618bfb4b0825ef8246c5a63aec5  saige.sif
5de579f750fb5633753bfda549822a32  tryggve_query.sif

    Here is the list of tools available in prebuilt containers:

    container

    tool

    version

    hello.sif

    demo example

    gwas.sif

    plink

    v1.90b6.18 64-bit (16 Jun 2020)

    gwas.sif

    plink2

    v2.00a2.3LM 64-bit Intel (24 Jan 2020)

    gwas.sif

    plink2_avx2

    v2.00a2.3LM AVX2 Intel (24 Jan 2020)

    gwas.sif

    PRSice_linux

    2.3.3 (2020-08-05)

    gwas.sif

    simu_linux

    Version v0.9.4

    gwas.sif

    bolt

    v2.3.5 March 20, 2021

    gwas.sif

    gcta64

    version 1.93.2 beta Linux

    gwas.sif

    gctb

    GCTB 2.02

    gwas.sif

    qctool

    version: 2.0.6, revision 18b8f17

    gwas.sif

    king

    KING 2.2.6 - ©

    gwas.sif

    metal

    version released on 2011-03-25

    gwas.sif

    vcftools

    VCFtools (0.1.17)

    gwas.sif

    bcftools

    Version: 1.12 (using htslib 1.12)

    gwas.sif

    flashpca_x86-64

    flashpca 2.0

    gwas.sif

    regenie

    REGENIE v2.0.2.gz

    gwas.sif

    GWAMA

    GWAMA_v2.2.2.zip

    gwas.sif

    magma

    magma_v1.09a_static.zip

    gwas.sif

    shapeit2

    Version : v2.r904

    gwas.sif

    impute4

    impute4.1.2_r300.3

    gwas.sif

    minimac4

    Version: 1.0.2; Built: Fri Sep 3 13:25:51

    gwas.sif

    bgenix

    version: 1.1.7, revision

    gwas.sif

    cat-bgen

    same version as bgenix

    gwas.sif

    edit-bgen

    same version as bgenix

    python3.sif

    python3

    python 3.10 + standard packages (numpy, pandas, etc)

    python3.sif

    ldpred

    ?

    python3.sif

    mixer

    mixer v1.3

    python3.sif

    python_convert

    github commit bcde562f0286f3ff271dbb54d486d4ca1d40ae36

    r.sif

    R

    version 4.0.3 + standard packages (data.table, ggplot, etc)

    r.sif

    seqminer

    ?

    r.sif

    rareGWAMA

    ?

    r.sif

    GenomicSEM

    ?

    r.sif

    TwoSampleMR

    ?

    r.sif

    GSMR

    v1.0.9

    r.sif

    LAVA

    ?

    r.sif

    LAVA partitioning

    ?

    saige.sif

    SAIGE

    version 0.43

    enigma-cnv.sif

    PennCNV

    version 1.0.5

    ldsc.sif

    LDSC

    version 1.0.1

    ipsychcnv.sif

    ???

    missing Dockerfile

    matlabruntime.sif

    ???

    work in progress

    regenie.sif

    ???

    ?

    regenie3.sif

    ???

    ?