# Introduction ## What is MiGA? MiGA is a data management and processing system for microbial genomes and metagenomes. Its main goal is to provide a uniform system for genome-based taxonomic classification and diversity studies, but its base can be used for other purposes. * [How can MiGA help me?](/part1/pitch) * [Who's using MiGA?](/part1/use-cases) * [Definitions](/part1/glossary) ## Getting started with MiGA MiGA itself doesn't require much but processing large data collections may require large infrastructures. With that in mind, MiGA is designed to support both single machines and cluster infrastructures. * [Requirements](/part2/requirements) * [Installation](/part2/installation) * [MiGA types](/part2/types) * [Input data](/part2/input) * [Distances](/part2/distances) * [Clustering](/part2/clustering) ## MiGA Interfaces You can interact with MiGA through different interfaces. These interfaces have different purposes, but they also have some degree of overlap, because different users with different aims sometimes want to do the same thing. The API and the CLI are the most comprehensive and flexible interfaces, but the other two are friendlier to humans. MiGA is mostly written in Ruby, with an object-oriented approach, and all the interfaces are just thin layers atop the Ruby core. That means that you can write your own interfaces (or pieces) if you know how to talk to these Ruby objects. Sometimes I even use `irb`, which is an interactive shell for Ruby, but that's mostly for debugging. * [MiGA API](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/api.md) * [MiGA CLI](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/cli.md) * [MiGA Web](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/web.md) * See also: [Help for MiGA Online](https://help.microbial-genomes.org/) ## Deploying examples Once you have installed MiGA, you might want to follow one (or several) of these tutorials to familiarize yourself with the MiGA environment. * [RefSeq in MiGA](/part4/deploy-refseq) * [Build a clade collection](/part4/deploy-clade) * [Launching daemons](/part4/daemons) * [Setting up MiGA in a cluster](/part4/cluster) ## MiGA in detail Ready for more? Here are some technical details for advanced users. * [Advanced configuration](/part5/advanced-configuration) * [MiGA workflow](/part5/workflow) * [Metadata](/part5/metadata) * [External Software](/part5/external) ## Workflows MiGA comes with ready-to-use pre-configured workflows for different analyses: * [Quality](/part6/quality_wf) * [Dereplicate](/part6/derep_wf) * [Classify](/part6/classify_wf) * [Preprocess](/part6/preproc_wf) * [Index](/part6/index_wf) * [Summaries](/part6/summaries) # Part I: What is MiGA? MiGA is a [data management](/part1#data-management) and [processing](/part1#processing) system for [microbial genomes and metagenomes](/part1#data-types). Its main goal is to provide a [uniform system](/part1#standards) for [genome-based taxonomic classification](/part1#taxonomy) and [diversity](/part1#diversity) studies, and its base can be used for [other purposes](/part1#more). ## Data management MiGA organizes your data in a consistent, well-organized fashion independent of centralized databases. This makes MiGA projects the ideal system to store data even if you don't use MiGA for anything else. MiGA is completely based on filesystem structures, so it can easily be transferred, backed-up, and stored long-term. Moreover, MiGA projects can be easily browsed, with descriptive folder names and a simple structure that is easy to understand. **MiGA is not** designed to support versioning or database storage, other than individual file-based databases, in order to keep the overhead on any of the tasks above (and the system requirements) at a minimum. ## Processing MiGA performs general-purpose analyses to pre-process genomic and metagenomic data. The main purpose of MiGA is [genome-based taxonomy](/part1#taxonomy), but some pre-processing steps are necessary regardless, so they can be used for many other purposes. For example, the initial data in most genomic and metagenomic projects is sequencing data. For almost any project, this means that trimming, clipping, and read-quality assessment are necessary steps for any downstream analyses. In most cases, assembly and gene prediction are also necessary, and other analyses like rRNA and essential genes detection is very useful. All of this is automatically done by MiGA! **MiGA is not** a workflow manager system. MiGA only supports short-read data (and it's optimized for Illumina data) or already assembled sequences. MiGA's goal is to keep analyses as simple and standardized as possible, so only critical customization is supported. ## Data types MiGA is designed to process genomes and can handle metagenomes (with some restrictions). MiGA is optimized for short-read datasets or assembled datasets. MiGA is optimized to process prokaryotic data (Archaeal and Bacterial), but it has some readily available customizations for viral metagenomes (or viromes). For more details, see the [types](/part2/types) of datasets and projects and the [input data](/part2/input) supported. **MiGA does not** have custom settings for eukaryotic or viral genomes, nor for transcriptomic data. The data management design (and perhaps some of the processing steps) can be used for these and other purposes, but thread carefully. ## Standards MiGA has a general-purpose design with some presets designed for the different [data types](/part2/types) supported. All internal configuration and metadata are stored as individual JSON files. Sequence quality is stored as FastQ, and sequences are stored as FastA; these two cover most of the data in the system. There are also some graphic reports in PDF and HTML, raw-text reports and logs, and general statistics in JSON. Finally, all of the [pair-wise comparisons](/part2/distances) are stored in SQLite3 files [described here](/part2/distances#sqlite3-schema). ### Filesystem structure * **daemon/**: Daemons lair. * **daemon.json**: Daemon settings. * ...: Several daemon log files. * **data/**: All the data is stored here. * **01.raw\_reads/**: Raw reads in FastQ format ([raw\_reads](/part5/workflow#raw-reads)). * **02.trimmed\_reads/**: Trimmed/clipped reads in FastQ format ([trimmed\_reads](/part5/workflow#trimmed-reads)). * **03.read\_quality/**: Read quality reports in HTML and PDF formats ([read\_quality](/part5/workflow#read-quality)). * **04.trimmed\_fasta/**: Trimmed/clipped and interposed reads in FastA format ([trimmed\_fasta](/part5/workflow#trimmed-fasta)). * **05.assembly/**: Assemblies in FastA format ([assembly](/part5/workflow#assembly)). * **06.cds/**: Gene predictions in FastA (genes and proteins) and GFF formats ([cds](/part5/workflow#cds)). * **07.annotation/**: Data annotations. * **01.function/**: Functional annotations. * **01.essential/**: Essential prokaryotic gene detections ([essential\_genes](/part5/workflow#essential-genes)). * **02.ssu/**: Ribosomal RNA (small subunit) sequence annotations ([ssu](/part5/workflow#ssu)). * **02.taxonomy/**: Taxonomic annotations. * **01.mytaxa/**: MyTaxa fragment annotations ([mytaxa](/part5/workflow#mytaxa)). * **03.qa/**: Quality assessments. * **01.checkm/**: (Currently not in use). * **02.mytaxa\_scan/**: Gene-window assessment of taxonomic distributions ([mytaxa\_scan](/part5/workflow#mytaxa-scan)). * **08.mapping/**: (Currently not in use). * **09.distances/**: Pair-wise comparisons ([distances](/part5/workflow#distances)). * **01.haai/**: Heuristic Average Amino Acid Identity (essential proteins; [distances](/part5/workflow#distances) and [haai\_distances](/part5/workflow#haai-distances)). * **02.aai/**: Average Amino Acid Identity (all proteins; [distances](/part5/workflow#distances) and [aai\_distances](/part5/workflow#aai-distances)). * **03.ani/**: Average Nucleotide Identity (genomic fragments; [distances](/part5/workflow#distances) and [ani\_distances](/part5/workflow#ani-distances)). * **04.ssu/**: (Currently not in use). * **05.taxonomy/**: Taxonomy based on reference projects ([taxonomy](/part5/workflow#taxonomy)). * **10.clades/**: Dataset clustering at various resolution levels. * **01.find/**: Identification of naturally-forming AAI clades at species level and above ([clade\_finding](/part5/workflow#clade-finding)). * **02.ani/**: Identification of naturally-forming ANI clades at species level and below ([subclades](/part5/workflow#subclades)). * **03.ogs/**: Extraction of orthologous groups of proteins and pan-genome statistics ([ogs](/part5/workflow#ogs)). * **04.phylogeny/**: (Currently not in use). * **05.metadata/**: (Currently not in use). * **90.stats/**: Results metadata for dataset stats ([stats](/part5/workflow#stats) and project-wide indexing and statistics ([project\_stats](/part5/workflow#project-stats)). * **metadata/**: Collection of JSON files with datasets metadata. * **miga.project.json**: JSON file with project metadata. ## Taxonomy MiGA's ultimate goal is to provide a standardized set of tools for consistent genome-wide taxonomic analyses. For this reason, MiGA **does not** provide nor favor any one taxonomic database. This *authority-agnostic* approach allows us to focus on the underlying analyses, supporting as many schemas as possible. With that being said, MiGA does support automated taxonomy annotation for some databases in EBI and NCBI linked to NCBI Taxonomy, and it does support some automated adjustments for the JGI schema (in particular for metagenomes). Instead of forcing groups by external taxonomies that may have varying degrees of accuracy and completeness, MiGA follows a data-driven clustering based on [naturally-forming groups](/part2/clustering) based on AAI and ANI analyses. Hence, MiGA projects can be used to classify novel genomes using any reference taxonomy (or none!). **MiGA does not** provide or endorse any particular taxonomic authority. ## Diversity MiGA can catalogue datasets, even in the absence of a reference taxonomy. This allows many advanced analyses, including (but not restricted to): * Phylogenomic reconstructions using [orthologous groups of proteins](/part5/workflow#ogs) * Multi-Locus sequence analysis using [essential genes](/part5/workflow#essential-genes) * Characterization of collections of [single-cell](/part2/types#single-cell-genome) or [population](/part2/types#population-genome) genomes * Characterization of [intra-population diversity](/part2/clustering#ani-clades) * [Metagenome](/part2/types#metagenome) or [virome](/part2/types#virome) analyses ## More The [intermediate analyses](/part5/workflow) performed by MiGA can be used for many other purposes. For example, we use MiGA's initial pre-processing (such as [read trimming](/part5/workflow#trimmed-reads) / [quality check](/part5/workflow#read-quality), [assembly](/part5/workflow#assembly), and [gene prediction](/part5/workflow#cds)) in most of our genomic and metagenomic projects. # How can MiGA help me? MiGA's analyses can accurately classify genomes providing a statistical classification support, identify taxonomic novelty, and annotate and evaluate sequencing reads and assembly quality, among others. MiGA can take in a variety of inputs such as raw, unassembled reads, assembled isolate genomes, metagenome-assembled genomes (MAGs), and single-cell amplified genomes (SAGs). MiGA uses a combination of the genome-aggregate average nucleotide identity concept or ANI and the average amino-acid identity, AAI, to taxonomically classify a query genomic sequence against the genome sequences in its reference database. Part of MiGA’s strength lies in the 10,000+ reference genomes that make up its database and an efficient heuristic algorithm to search the query genome against all this database. The reference database is regularly updated and improved with minimal downtime, ensuring consistently improved classification accuracy. # Who is using MiGA? MiGA provides a wide range of utility to suit a variety of users. Exploration of existing datasets can be beneficial for newcomers and experienced scientists alike. For inexperienced users, MiGA can help especially with the initial pre-processing step. By using a set of tools from the [Enveomics collection](http://enveomics.ce.gatech.edu/enveomics), MiGA simplifies the first steps of genomic analysis which can be overwhelming until users are familiar with the variety of tools utilized by the field. MiGA's analytical ability is also robust enough that even experienced users can find utility within. MiGA can also work as a powerful educational tool by providing easy analysis and visualization of results for use in and out of the classroom. # Who is behind MiGA? ## Conception and Design * [Luis M Rodriguez-R](https://rodriguez-r.com/) * [Konstantinos T Konstantinidis](http://enve-omics.gatech.edu/) * James R Cole * James M Tiedje * Ramon Rosselló-Mora ## Development * [Luis M Rodriguez-R](https://rodriguez-r.com/) * Fang Yuan * Santosh Gunturu * [Zilong Huang](mailto:zhuang390@gatech.edu) ## Documentation * [Luis M Rodriguez-R](https://rodriguez-r.com/) * [MaryFrances Johnson](mailto:mjohnson394@gatech.edu) * [John Quensen](https://john-quensen.com/) * William T Harvey * Tanya Kiryutina ## Testing and Evaluation * [Luis M Rodriguez-R](https://rodriguez-r.com/) * Brittany J Suttner * Fang Yuan * Santosh Gunturu * Jianshu Zhao * Roth Conrad # Definitions ## MiGA API Ruby Application Program Interface to MiGA. ## MiGA CLI Command Line Interface to MiGA. ## MiGA Web Web-based interface to MiGA. ## MiGA Names MiGA names are non-empty strings composed exclusively of alphanumerics and underscores. All the dataset names in MiGA must conform to this restriction. ## MiGA Dates The official format in which MiGA represents date/times is the default of Ruby's `Time.now.to_s`, as generated by the CLI action `miga date`. In the \*nix `date` utility this corresponds to the format: `+%Y-%m-%d %H:%M:%S %z`. # Part II: Getting started To get started with MiGA, first make sure you have all the [system requirements](/part2/requirements). Next, [install and configure MiGA](/part2/installation) in order to use it for processing. You can also skip the configuration if you simply want to access projects previously processed. Finally, you may want to get familiar with the [MiGA types](/part2/types), the [input data for MiGA](/part2/input), and the methods for [distances](/part2/distances) and [clustering](/part2/clustering). # Requirements MiGA requires a system (single computer, cluster, or cloud-computing infrastructure) with Linux or MacOS. If you simply want to access projects previously processed, all you need is `ruby` 3.1+ and the `miga-base` gem. If you want to use MiGA to process your data, please follow these instructions. MiGA has a relatively long list of requirements, but most of them are easy to install. Find your system below and follow the instructions using: * [Homebrew](/part2/requirements/brew) (preferred), * [`apt-get`](/part2/requirements/apt-get), * [Anaconda](/part2/requirements/conda), * [Installing from source](/part2/requirements/source). Additionally you will need [MyTaxa utils](/part2/requirements/mytaxa) in order to activate the [MyTaxa](/part5/workflow#mytaxa) and [MyTaxa Scan](/part5/workflow#mytaxa-scan) steps (optional). # Using Homebrew ## Live notebook If you prefer to see code in action, the full installation process with homebrew is available as a [Notebook in Google Colab](https://colab.research.google.com/gist/lmrodriguezr/4af3b345067e862dc7903e4c78ad40ac). ## Installing requirements using Homebrew You can use [Homebrew](https://brew.sh/) to install most of the software required by MiGA. If you don't have Homebrew, execute (and follow the instructions): ```bash /bin/bash -c "$(curl -fsSL \ https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` ### A recent Ruby It's possible you already have a recent version of ruby (2.3+). To check which version you have, simply run: `ruby -v`. If your version of Ruby is older than 2.3, you can install a recent version using: ```bash brew install libyaml rbenv rbenv install 3.3.0 rbenv global 3.3.0 ``` ### Packages Now that you have Homebrew, execute: ```bash brew tap brewsci/bio # In Linux, replace 'temurin' with 'openjdk' brew install \ r sqlite3 python numpy pigz temurin \ blast hmmer bedtools prodigal gmp spades mcl krona \ barrnap diamond fastani faqcs brewsci/bio/falco seqtk fastp trnascan ``` Additionally, recent changes in `pip` might break some installations, and as such we recommend pre-installing python packages in a brew environment: ```bash pip install --break-system-packages numpy pyhmmer pyrodigal ``` We also recommend installing the `sqlite3` gem beforehand using the brew libraries, to avoid headaches down the road (but this is optional): ```bash gem install sqlite3 -- --with-sqlite3-dir="$(brew --prefix sqlite3)" ``` ### MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/blob/main/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/blob/main/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/part2/requirements/mytaxa). # Using apt-get ## Installing requirements using `apt-get` ### Live notebook If you prefer to see code in action, the full installation process with apt-get is available as a [Notebook in Google Colab](https://colab.research.google.com/gist/lmrodriguezr/78f2f48eadce96bc2dd526fd194fb00a). ### Packages Run: ```bash sudo apt-get update sudo apt-get install \ ruby ruby-sqlite3 r-base sqlite3 libcurl4-openssl-dev zlib1g zlib1g-dev pigz \ ncbi-blast+ hmmer bedtools spades prodigal mcl barrnap diamond-aligner \ fastp fastani trnascan-se seqtk ``` ### Additional Software Some of the software required by MiGA is not available in aptitude, but you can install it in your 64bit Linux machine using: ```bash ## FaQCs wget -O FaQCs \ "https://github.com/LANL-Bioinformatics/FaQCs/releases/download/2.10/FaQCs_linux_x86_64" sudo install FaQCs /usr/bin/ && rm FaQCs ## Falco wget -O falco.tar.gz \ "https://github.com/smithlabcode/falco/releases/download/v1.2.5/falco-1.2.5.tar.gz" tar zxf falco.tar.gz ( cd falco-1.2.5 \ && ./configure CXXFLAGS="-O3 -Wall" \ && make && sudo make install ) rm -rf falco-1.2.5 falco.tar.gz ``` ## JAVA VM If you want support for RDP classifications, you'll need any working Java VM. For example, you could install Temurin as follows: ```bash wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public \ | sudo apt-key add - echo "deb https://packages.adoptium.net/artifactory/deb \ $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" \ | sudo tee /etc/apt/sources.list.d/adoptium.list sudo apt-get update sudo apt-get install temurin-21-jdk ``` ### MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/blob/main/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/blob/main/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/part2/requirements/mytaxa). # Using Conda You can use [conda](https://conda.io/) to install most of the Software required by MiGA. If you don't have Conda, you can follow the [Installation instructions](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). ## Live notebook If you prefer to see code in action, the full installation process with conda is available as a [Notebook in Google Colab](https://colab.research.google.com/gist/lmrodriguezr/3fe4db8df4e5038ae603fde18214b148). ## Note for macOS users > In general, it is recommended to use [Homebrew](/part2/requirements/brew) to install MiGA if possible. The MiGA installation in macOS using conda is known to be fragile and extremely time-consuming (it could take upwards of 20 minutes just solving the environment). ## Packages Now, install all the required packages using conda: ```bash # Install prerequisites conda env create \ -f "https://raw.githubusercontent.com/bio-miga/miga/main/conda.yml" # Tell MiGA to activate the proper conda environment echo 'eval "$(conda shell.bash hook)" && conda activate miga' > ~/.miga_modules # Activate the environment . ~/.miga_modules ``` # Installing from source ## Essentials If you don't have apt-get, Homebrew, or conda, here's the list of requirements and URLs with installation instructions: ## MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/part2/requirements/mytaxa). # MyTaxa Utils If you followed the instructions for [conda](/part2/requirements/conda), you already have all the necessary MyTaxa utils. **The MyTaxa utilities are optional**, but without them the [MyTaxa](/part5/workflow#mytaxa) and [MyTaxa scan](/part5/workflow#mytaxa-scan) analyses are disabled. Note that MyTaxa requires about **15Gb** of disk available to store the database. We will install the necessary software in `$HOME/apps`. You can change this directory if you prefer: ```bash [[ -d $HOME/apps/bin ]] || mkdir -p $HOME/apps/bin cd $HOME/apps echo 'export PATH=$HOME/apps/bin:$PATH' >> ~/.miga_modules ``` ## MyTaxa ```bash wget -O - \ https://github.com/bio-miga/MyTaxa/archive/master.tar.gz | tar zx cd MyTaxa-master make python3 utils/download_db.py echo 'export PATH="'$PWD':$PATH"' >> ~/.miga_modules cd .. ``` ## Krona If you followed the instructions for [conda](/part2/requirements/conda) or [Homebrew](/part2/requirements/brew), you already have Krona. If you still need to install Krona, simply execute: ```bash wget -O - \ https://github.com/marbl/Krona/archive/master.tar.gz | tar zx ( cd Krona-master/KronaTools && ./install.pl --prefix ../.. ) ``` # Installation ## Prerequisites If you simply want to access projects previously processed, you can skip this step. To install the prerequisites follow [system requirements](/part2/requirements). ## Getting MiGA To install MiGA itself, you'll just need: ```bash gem install miga-base ``` If necessary, you can use `sudo gem install miga-base` or `gem install --user miga-base` instead. ### Getting MiGA source If you want to get MiGA working from source instead of using the gem, you can use: ```bash # Get the source. Make sure you use --recursive, to clone submodules: git clone --recursive https://github.com/bio-miga/miga.git cd miga # You can use bundle to make sure you have the required gems, # or simply install them manually: bundle # And finally make MiGA available in the PATH. # This is not mandatory but it saves time and effort: echo "export PATH=\"$(pwd)/bin:\$PATH\"" >> ~/.bashrc source ~/.bashrc ``` ## Initializing MiGA To initialize MiGA for data processing, simply execute and follow the instructions: ```bash miga init ``` For additional initialization parameters, use `miga init -h`. ## Pre-loading Software If some of the prerequisites are not already available in your `$PATH`, you'll have to first tell MiGA how to find them. For that, create (or modify) a `bash` configuration file at `~/.miga_modules` loading any necessary packages. For example, if you're in a cluster architecture using `modules`, this is where you should load them. Note that some contents may already be present in this file from previous steps. Do not remove or modify those lines if you're not sure what they do. Here's an example of a configuration file for a single computer: ```bash #!/bin/bash # File: ~/.miga_modules # Enable software installed in this user export PATH="$HOME/apps/bin:$PATH" # Enable MyTaxa export PATH="$HOME/apps/MyTaxa:$PATH" ``` Here is an example of a configuration file for a cluster: ```bash #!/bin/bash # File: ~/.miga_modules shopt -s expand_aliases module purge module load gcc/4.9.0 module load ruby/2.1.5 module load R/3.3.2 module load prodigal/2.6.1 module load bedtools/2.21.0 module load scythe/0.993 module load fastqc/0.11.2 module load spades/4.0 module load hmmer/3.1b1 # Enable MyTaxa export PATH="$HOME/shared3/apps/MyTaxa:$PATH" # Workaround for broken FastQC in the cluster alias fastqc="perl $(which --skip-alias --skip-functions fastqc)" # Workaround for broken KronaTools in the cluster alias ktImportText="perl -I$HOME/shared3/apps/KronaTools-2.5/lib/ \ $HOME/shared3/bin/ktImportText" ``` Once you complete this file, rerun: ```bash miga init ``` # MiGA types ## MiGA types MiGA has predefined settings and analyses. In order to maximize simplicity and reproducibility while ensuring a wide array of use cases, MiGA uses presets called "types". There are different [types of projects](/part2/types#project-types) and [types of datasets](/part2/types#dataset-types): ### Project types When you create a project, the type of project defines which project-wide analyses are going to be executed (and how). The different types are: #### Mixed **Symbol**: `mixed`. A mixed collection of genomes, metagenomes, and viromes. This is the most basic type of project, with no support for any project-wide analyses. It is intended for projects that are only concerned with datasets preprocessing, *e.g.*, read trimming, assembly, etc. #### Genomes **Symbol**: `genomes`. A collection of genomes. This is the most typical type of project, storing a set of genomes from different taxonomic groups. It can be useful for anything from indexing a reference database, to create a collection of metagenomic bins, and anything in between. #### Clade **Symbol**: `clade`. A collection of closely-related genomes (ANI >= 90%). This is a project for a collection of genomes in the same species (or closely-related species) that require higher resolution but don't require support for a large distance range. #### Metagenomes **Symbol**: `metagenomes`. A collection of metagenomes and/or viromes. This is an experimental type, currently identical to [Mixed](/part2/types#mixed). ### Dataset types Once you have a project, the type of the datasets define which analyses are going to be executed for that particular entry (and how). The different types are: #### Genome **Symbol**: `genome`. The genome from an isolate. This is the most typical case, in which you have a genome (complete or draft) from a pure culture (excluding SAGs). #### Single-cell genome **Symbol**: `scgenome`. A Single-cell Amplified Genome (SAG). This is the particular case in which you are dealing with an amplified genome from a single cell. These datasets typically have very uneven coverage (resulting in very incomplete assemblies) and sometimes have contamination from external DNA. #### Population genome **Symbol**: `popgenome`. A population genome (including metagenomic bins). This is the type of dataset that includes sequences from different strains of the same species, such as metagenomic bins or metagenomes of highly enriched (but not pure) cultures. #### Metagenome **Symbol**: `metagenome`. A metagenome (excluding viromes). #### Virome **Symbol**: `virome`. A viral metagenome. ## Query vs reference datasets In addition to the dataset types, some analyses may differ depending on the status of a dataset as query or reference. Reference datasets are those that integrate the database of the project; *i.e.*, those that can be queried by analyses with other datasets like [distances](/part5/workflow#distances). In contrast, query datasets are more isolated: they can use data from other datasets (or the project), but don't get to form part of the project database. Defining query datasets is useful when, for example, you have a reference framework for taxonomy (formed by reference datasets) and want to find the best classification for a genome without affecting the project itself. By default, datasets are created as reference datasets. # Input data MiGA datasets can be created from three different points: [Raw reads](/part2/input#raw-reads), [Trimmed reads](/part2/input#trimmed-reads), and [Assemblies](/part2/input#assemblies). The input files can be added through the [CLI](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/cli.md) using `miga add` or any of the available [workflows](/part6). Files can also be added through the [Web](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/web.md) interface. ## Raw reads Raw (unprocessed) sequencing reads. MiGA can handle different sequencing technologies, but it has been optimized for short reads. * **Format**: FastQ, optionally gzipped (with .gz extension) * **Workflow step**: [Raw reads](/part5/workflow#raw-reads) ## Trimmed reads Sequencing reads already processed to remove low quality or other artifacts. MiGA can handle different sequencing technologies, but has been optimized for short reads. * **Format**: FastA, optionally gzipped (with .gz extension) * **Workflow step**: [Trimmed FastA](/part5/workflow#trimmed-fasta) ## Assemblies Assembled contigs/scaffolds. Ideally, but not necessarily, sequences longer than 1 Kbp. * **Format**: FastA, optionally gzipped (with .gz extension) * **Workflow step**: [Assembly](/part5/workflow#assembly) # Distances MiGA estimates distances (or similarities) between datasets using different techniques. Only genome-to-genome comparisons have been implemented, including the genomes of isolates, metagenome-assembled genomes, or single-cell amplified genomes. No metagenome-to-metagenome or genome-to-metagenome distances are currently available in MiGA. ## Hierarchical approach to distances For any given pair of genomes, MiGA attempts a hierarchical approach to identify the most appopriate metric of similarity: **1**. First, the genomes are compared using [hAAI](/part2/distances#haai). If this method is skipped, if it fails, or if the value is greater than 90%, MiGA continues to step 2. Otherwise, this value is used to estimate the AAI, both values are recorded, and the comparison ends. **2**. Next, MiGA compares genomes using [AAI](/part2/distances#aai). Whenever the AAI is 85% or higher, MiGA continues to step 3. Otherwise the comparison ends. **3**. Finally, MiGA estimates [ANI](/part2/distances#ani). ## hAAI **Heuristic Average Amino Acid Identity**. The hAAI is the average amino acid identity between the highly conserved proteins of two genomes, as identified by [essential genes](/part5/workflow#essential-genes). It is used to estimate AAI for distant pairs, but it loses resolution between close relatives. This metric is completely bypassed in projects of [type clade](/part2/types#clade) as well as projects with the [metadata field](/part5/metadata#projects) `haai_p=no`. This field also controls the Software used: `blast+` (default), `blast`, `blat`, or `diamond`. ## AAI **Average Amino Acid Identity**. The AAI is the average amino acid identity between all proteins of two genomes, as identified by [cds](/part5/workflow#cds). When running this analysis, the intermediate reciprocal best matches (RBMs) are also stored in projects of [type clade](/part2/types#clade). This feature can be turned off to save storage space or forced to be on in any project type using the [metadata field](/part5/metadata#projects) `aai_save_rbm=false` or `aai_save_rbm=true`, respectively. The Software used as a search engine can be controlled using the [metadata field](/part5/metadata#projects) `haai_p`: `blast+` (default), `blast`, `blat`, or `diamond`. [Workflows](/part6) use `blast+` by default, or `diamond` if the flag `--fast` is passed (whenever available). ## ANI **Average Nucleotide Identity**. The ANI is the average nucleotide identity between fragments of two genomes. The Software used as a search engine can be controlled using the [metadata field](/part5/metadata#projects) `haai_p`: `blast+` (default), `blast`, `blat`, or `fastani`. [Workflows](/part6) use `blast+` by default, or `fastani` if the flag `--fast` is passed (whenever available). ## SQLite3 schema The information on the different similarity metrics above is stored in SQLite3 database files. The general schema for [hAAI](/part2/distances#haai) and [AAI](/part2/distances#aai) is: ```sql CREATE TABLE aai( seq1 varchar(256), seq2 varchar(256), aai float, sd float, n int, omega int ); CREATE TABLE rbm( seq1 varchar(256), seq2 varchar(256), id1 varchar(256), id2 varchar(256), id float, evalue float, bitscore float ); ``` The `aai` table holds the metric values, including: the name of the genomes compared `seq1` and `seq2`, the AAI or hAAI as percentage `aai`, the standard deviation across proteins when available `sd`, the total number of RBMs `n`, and the smaller number of proteins from the two genomes `omega`. The `rbm` table holds the RBMs for [AAI](/part2/distances#aai) (whenever available if stored) and is always empty for [hAAI](/part2/distances#haai). The general schema for [ANI](/part2/distances#ani) is: ```sql CREATE TABLE ani( seq1 varchar(256), seq2 varchar(256), ani float, sd float, n int, omega int ); CREATE TABLE rbm( seq1 varchar(256), seq2 varchar(256), id1 int, id2 int, id float, evalue float, bitscore float ); CREATE TABLE regions( seq varchar(256), id int, source varchar(256), `start` int, `end` int ); ``` The `ani` table holds the metric values, including: the name of the genomes compared `seq1` and `seq2`, the ANI as percentage `ani`, the standard deviation across fragments when available `sd`, the total number of RBM fragments `n`, and the smaller number of fragments from the two genomes `omega`. The tables `rbm` and `regions` are always empty. # Clustering MiGA generates a clustering-based indexing of databases using ANI distances (for clade projects) or AAI distances (for all other projects). This indexing enables quick searching of databases with query genomes. ## General algorithm The AAI or ANI values are transformed to distances (1 - identity), and the all-vs-all distance matrix is used to generate a *k*-medoids partition (PAM: Partition Around Medoids). *k* is selected to simultaneously optimize for maximum Silhouette average width and minimum Silhouette negative area, between 2 and 100 (or the number of genomes minus 1, whichever is smaller). Once the partitions are defined, the same algorithm is applied recursively to each partition with 8 or more genomes. The resulting clustering-based indexing is used to speed-up query searches. In some cases, it can also be used as *de novo* typing scheme, in particular for ANI distances (clade projects). ## Genomospecies proposals In addition to the above clustering-based indexing, MiGA clusters genomes by Markov Clustering (MCL) using all ANI values above 95% as edges. The result is a collection of discrete genomospecies. The list of genomes per genomospecies is sorted by medoid-ranking, in which the first genome has the minimum average distance to all other genomes in the genomospecies. # Part III: Interfaces The core of MiGA is not as useful if we cannot interact with it, and there is a variety of interfaces for that. At the lowest level, the documented [API](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/api.md) is available for developers to communicate directly with MiGA, either through scripts or through the console. A convenient wrapper including most common tasks (and some not-so-common) is available as the MiGA [CLI](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/cli.md). Finally, a graphical interface allowing result browsing and querying of projects is available as [MiGA Web](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/web.md). # MiGA API # MiGA CLI # MiGA Web # Part IV: Deploying examples Now that your system is ready with MiGA, you may want to deploy some examples for a test run. Here you can find how to get the entire [RefSeq in MiGA](/part4/deploy-refseq) and how [build a clade collection](/part4/deploy-clade). For more details on running MiGA, see how to [launch daemons](/part4/daemons) and how to [setup MiGA in a cluster](/part4/cluster). # RefSeq in MiGA In this tutorial, we will create a genomes project including all the representative genomes available in RefSeq using MiGA alone. If you want to explore a more manual approach using `bash`, see the [RefSeq in MiGA using BASH example](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part4/deploy-refseq-bash.md). ## 0. Initialize the project ```bash miga new -P RefSeq -t genomes cd RefSeq ``` ## 1. Download publicly available genomes **Re-running and updating**: If the following code fails at any point, for example due to a network interruption, you can simply re-run it, and it will take it from where it failed. ```bash miga ncbi_get -P . --reference -v -T Archaea ``` Of course, you can repeat for `-T Bacteria` to get all prokaryotic genomes. It is strongly recommended to use an [NCBI API Key](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/) to increase the number of allowed requests. Once you obtain one, you can pass it as an argument: ```bash miga ncbi_get -P . --reference --api-key ABCD123 -v -T Archaea ``` Or you can set it globally as an environmental variable before running `miga`: ```bash export NCBI_API_KEY=ABCD123 ``` ## 2. Launch the daemon Now that your data is ready, you can fire up the daemon to start processing the data. For additional details, see [launching daemons](/part4/daemons): ```bash miga daemon start -P . ``` # Build a clade collection In this tutorial, we will create a clade project including all the genomes available for a species in RefSeq as well as any additional genomes you may have using MiGA alone. If you want to explore a more manual approach using `bash`, see the [Build a clade collection using BASH example](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part4/deploy-clade-bash.md). We will use *Escherichia coli* as the target species, but you can use any species (or any taxon) you want. ## 0. Initialize the project ```bash miga new -P E_coli -t clade cd E_coli ``` ## 1. Download publicly available genomes There are different stages of completeness defined in the NCBI Genome database, and you may want to include only some cases depending on your analysis. The stages (from higher to lower quality) are: 1. *Complete*: Genomes including all replicons in the organism(s) sequenced. 2. *Chromosome*: Genomes with complete chromosome (but missing other replicons). 3. *Scaffold*: Draft genomes with scaffold status. 4. *Contig*: Draft genomes with contig status. In this example, we'll skip the draft genomes. However, if you want all of them, simply use the `--all` option of `miga ncbi_get`. **Re-running and updating**: If the following code fails at any point, for example due to a network interruption, you can simply re-run it, and it will take it from where it failed. ```bash miga ncbi_get -T "Escherichia coli" -P . --complete --chromosome -v ``` Note that you can change the value of `-T` from `"Escherichia coli"` to any other species name, or even taxa of any rank such as genus or family. It is strongly recommended to use an [NCBI API Key](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/) to increase the number of allowed requests. Once you obtain one, you can pass it as an argument: ```bash miga ncbi_get -T "Escherichia coli" -P . --complete --chromosome --api-key ABCD123 -v ``` Or you can set it globally as an environmental variable before running `miga`: ```bash export NCBI_API_KEY=ABCD123 ``` ## 2. Add your own genomes If you have any unreleased genomes, you can simply add them to the same project to be processed together with those publicly available. You can initialize datasets at different points, see [input data](/part2/input). For the purposes of this tutorial, we'll assume that you have raw coupled reads from two sequencing lanes (1 and 2) in Gzipped FastQ files: ```bash # Unzip and/or concatenate input files # this is not necessary if your files are ready to use: gzip -d -c ~/some/file/d1_ACTG_L[12]_R1.fastq.gz > /tmp/sister.1.fastq gzip -d -c ~/some/file/d1_ACTG_L[12]_R2.fastq.gz > /tmp/sister.2.fastq # Register the dataset # change the dataset name MyDataset to whichever name you want: miga add -P . -D MyDataset -t genome -i trimmed_reads_paired /tmp/sister.[12].fastq ``` ## 3. Launch the daemon Now that your data is ready, you can fire up the daemon to start processing the data. For additional details, see [launching daemons](/part4/daemons): ```bash miga daemon start -P . ``` # Launching daemons MiGA Daemons are centralized processes in MiGA that define which jobs need to be executed, when, where, and how. All of this can be controlled by customizing the daemons: ## Customizing daemons ### Daemon JSON file When you initialize MiGA (`miga init`), the final step is generating a daemon JSON file stored in `~/.miga_daemon.json`. This file will control the type of daemon (see below) and the default configuration. In addition to this user-wide file, each MiGA project has a `daemon/daemon.json` file. Any variables defined here overwrite the user-wide file. ### `miga daemon` When a daemon is launched using `miga daemon`, several configuration variables can be overwritten using the different command-line flags. If undefined, the variables will be defined by the `daemon/daemon.json` file in the project, or by the daemon JSON file defined by `--json` if passed. Finally, and if a variable is still missing from there, it will default to the values in `~/.miga_daemon.json`. ### Workflows Daemons are also launched by workflows. This process is not manually controlled by the user, but it can also be controlled in a number of ways. First, the flags `--jobs` and `--threads` control the maximum number of jobs (`maxjobs`) and the number of CPUs per job (`ppn`), respectively. Next, a daemon JSON file can be defined using the flag `--daemon`. Finally, variables that are not defined by either method will default to the user-wide configuration in `~/.miga_daemon.json`. ## Daemon types MiGA currently supports three daemon modes, and they each have specific details: ### Local daemons Local daemons, or `bash` daemons, are the simplest mode. You can use this type of daemon if you are launching MiGA in a single computer. The most important consideration here is the **total number of available CPUs**. When you launch a MiGA daemon, the maximum number of jobs spawned will be determined by `--max-jobs` (or `maxjobs` in daemon JSON files, or `--jobs` in workflows), whereas the number of CPUs each job can use is determined by `--ppn` (or `ppn` in daemon JSON files, or `--threads` in workflows). The number of jobs times the number of CPUs per job should never exceed the number of CPUs available. For example, if you have 12 cores in your computer, the default configuration of 6 jobs and 2 CPUs per jobs could use up to 100% of the available CPUs. ### Remote daemons Remote daemons, or `ssh` daemons, are daemons that can communicate with other machines to launch tasks remotely using a login shell through SSH. The first consideration here is: the absolute path to the MiGA project and to the MiGA system **must** be the same in all nodes (launcher and running nodes). If this is not the case, remote executions will fail. Next, in this type of daemon the trickiest part is to configure the remote nodes correctly. This is, to correctly **define the node list file**. The node list file is a raw text file that contains the list of all remote nodes available (hostnames), one per line. It can be set as an environmental variable (*e.g.*, `$MIGA_NODELIST`) that will be read at execution or explicitly as a path to the file. Setting the node list file as a variable is useful when launching MiGA from a scheduler-controlled system. For example, when using Torque, this can be set to `$PBS_NODEFILE`. On the other hand, a fixed file can be a useful alternative if you have a defined set of nodes that are freely available to you (*e.g.*, a dedicated infrastructure). There are several ways to define the node list. First, it can be set using the command-line flag `--node-list` in `miga daemon` (no equivalent is available for workflows). If this is not defined, daemons will look for the value of `nodelist` in the daemon JSON files: first in `daemon/daemon.json` or in the file set by `--daemon` (for workflows) or `--json` (for `miga daemon`), and finally in `~/.miga_daemon.json`. If the node list is set to a variable (starting with `$`), it must be defined as an environmental varible. For example, you could execute `export MIGA_NODELIST=/path/to/file.txt` before running MiGA if the node list is set to `$MIGA_NODELIST`. Finally, **remote daemons ignore `maxjobs`** (or `--max-jobs` or `--jobs`). Instead, the maximum number of jobs is determined by the number of lines in the node list. This allows specifying how many jobs can be launched to each node. For example, if a given node is present three times in the node list, MiGA will run up to three jobs at the same time in that node. Be careful when directly using node files defined by schedulers. Some schedulers will list nodes on the basis of dedicated cores, which may be a problem if the number of threads is more than 1 (`--ppn`, `ppn`, or `--threads`). For example, consider the following PBS (Torque) script which you can use as a template (assuming that `nodelist` is `$MIGA_NODELIST` and `type` is `ssh` in `~/.miga_daemon.list`): ```bash #PBS -q my_nice_queue #PBS -l nodes=12:ppn=3 #PBS -l walltime=12:00:00 # Create the node list file # Change the '3' by the number of CPUs you are using (the value of ppn): awk 'NR % 3 == 0' < "$PBS_NODEFILE" > hosts.txt # Define the nodelist variable: export MIGA_NODELIST=hosts.txt # Run MiGA (change the '3' by the value of ppn): miga quality_wf -T 'Macondimonas' -o Macondimonas -v --threads 3 ``` ### Scheduler daemons Finally, daemons can also communicate directly with schedulers. Currently, MiGA supports: `qsub` (Torque), `msub` (Moab), and `slurm` (SLURM). If your HPCC runs a different scheduler and you'd like to see it added to this list, [please contact us](http://support.microbial-genomes.org). This is the preferred method when the flow of tasks is variable, because it can easily adapt to different task loads. For example, when maintaining a website that automatically processes incoming jobs (like [MiGA Online](http://microbial-genomes.org)), the daemons are low-resource tasks that launch as many or as few jobs as necessary. Importantly, this method tends to require large numbers of jobs submitted to the schedulers, and often wastes some requested resources since the resource use configuration does not adapt to each task, both of which may be against your HPCC policies. Therefore, when processing a defined task in HPCC, it is preferred to use either of the two modes above instead. For example, if you want to evaluate the quality of four genomes, a good option would be to launch a [local daemon](/part4/daemons#local-daemons) with a maximum of 4 jobs (`--max-jobs 4`). On the other hand, if you have hundreds of genomes to process, you could launch instead a [remote daemon](/part4/daemons#remote-daemons). # Setting up MiGA in a cluster MiGA is developed to run on clusters with a TORQUE scheduler. The administrator should install the [dependencies](/part2/requirements) and [MiGA-base](/part2/installation) on a networked file system. When you first initialize MiGA, the program will prompt the user for the type of daemon. For a cluster with TORQUE, the user can choose a qsub daemon. During the initialization, MiGA will also ask for the number of jobs to submit and the number of CPUs per job. After you have setup a MiGA project, you can create a job script to start daemon. The daemon requires to be alive until all the jobs for the have finished, so you should add a long-enough walltime. If you didn't give enough walltime for your job, the MiGA daemon will get killed by the system. In this case, you can restart the daemon again to finish processing the project. Example job script: ```bash #PBS -N MiGADaemon #PBS -l mem=2gb #PBS -l nodes=1:ppn2 #PBS -l walltime=48:00:00 cd $HOME/Path-to-project/ miga daemon start -P . ``` # Part V: Additional details # Advanced configuration MiGA allows flexible execution using a number of techniques, mainly setting [project metadata flags](/part5/metadata#project-flags), [dataset metadata flags](/part5/metadata#dataset-flags), and through the [daemon configuration](/part4/daemons). These are some examples of advanced behaviors that can be configured in MiGA ## Controlling many daemons at once The [MiGA CLI](https://github.com/bio-miga/miga/tree/7d5b5d8ca7b592d2dd8e085f33813de3ea42030a/manual/part3/miga-cli.md) includes support for mass-controlled daemons when several MiGA projects are in the same folder using the `miga lair` command. The MiGA Lair will find all the MiGA projects in a folder (or subfolders) and control all the corresponding daemons together: ```bash # Check the current status of all the daemons miga lair list -p /path/to/folder # Terminate all the daemons at once miga lair terminate -p /path/to/folder # Launch and daemonize a process to keep all daemons running miga lair start -p /path/to/folder # See more options miga lair -h ``` ## Sending an email when the project is ready If your project runs for a long time and you want to be notified by email when it's ready, you can use [project hooks](/part5/metadata#project-hooks): ```bash # First cd to the project folder cd /path/to/project # And open the MiGA Console miga c ``` In the MiGA Console: ```ruby # Set the hook "on_preprocessing_ready" to # execute a shell command ("run_cmd") consisting of # sending an email with the project path ("{{project}}") MiGA::Project.load('.').tap do |p| p.metadata[:on_preprocessing_ready] = [ ['run_cmd', 'echo \'Project ready: {{project}}\' | sendmail me@example.com'] ] end.save quit ``` Make sure to change `/path/to/project` to the project path and `me@example.com` to the email where you want to receive the notification. Also, note that depending on your `sendmail` configuration you might get that email in the spam folder (or flat-out rejected), so make sure to test `sendmail` first. # MiGA workflow ## MiGA Workflow This is the general overview of the MiGA workflow: ![MiGA Workflow](/files/-LBP6DvubxUjSQkvMTSs) For each step, performed analyses may include the use of [external Software](/part5/external), and produce one or more result files (indexed in a hash). In most steps, different utilities from the [Enveomics Collection](/part5/external#enveomics-collection) are used in addition to the Software detailed below. Some files are mandatory to continue with the analysis (marked with *req*), some can be gzipped during or after the analysis (marked with *gz*), and some are directories (marked with *dir*). ### List of individual steps * [Dataset Results](#dataset-results) * [Raw Reads](#raw-reads) * [Trimmed Reads](#trimmed-reads) * [Read Quality](#read-quality) * [Trimmed FastA](#trimmed-fasta) * [Assembly](#assembly) * [CDS](#cds) * [Essential Genes](#essential-genes) * [MyTaxa](#mytaxa) * [MyTaxa Scan](#mytaxa-scan) * [Distances](#distances) * [Taxonomy](#taxonomy) * [SSU](#ssu) * [Stats](#stats) * [Project Results](#project-results) * [hAAI Distances](#haai-distances) * [AAI Distances](#aai-distances) * [ANI Distances](#ani-distances) * [Clade Finding](#clade-finding) * [Subclades](#subclades) * [OGS](#ogs) * [Project Stats](#project-stats) ## Dataset Results ### Raw Reads This step is never actually performed by MiGA, instead it serves as the entry point for raw reads input. Supported file keys: * **For single reads only** * `single` (*req*, *gz*): FastQ file containing the raw reads * **For paired-end reads only** * `pair1` (*req*, *gz*): FastQ file containing the raw forward reads * `pair2` (*req*, *gz*): FastQ file containing the raw reverse reads Statistics: * **For single reads only** * `reads`: Total number of reads * `length_average`: Average read length (in bp) * `length_standard_deviation`: Standard deviation of read length (in bp) * `g_c_content`: G+C content of all reads (in %) * `x_content`: Undetermined bases content of all reads (in %) * `a_t_skew`: A-T sequence skew across all reads (in %) * `g_c_skew`: G-C sequence skew across all reads (in %) * **For paired-end reads only** * `read_pairs`: Total number of read pairs * `forward_length_average`: Average forward read length (in bp) * `forward_length_standard_deviation`: Standard deviation of forward read length (in bp) * `forward_g_c_content`: G+C content of forward reads (in %) * `forward_x_content`: Undetermined bases content of forward reads (in %) * `forward_a_t_skew`: A-T sequence skew across forward reads (in %) * `forward_g_c_skew`: G-C sequence skew across forward reads (in %) * `reverse_length_average`, `reverse_length_standard_deviation`, * `reverse_g_c_content`: Same as above, for reverse reads * `reverse_x_content`: Undetermined bases content of reverse reads (in %) * `reverse_a_t_skew`: A-T sequence skew across reverse reads (in %) * `reverse_g_c_skew`: G-C sequence skew across reverse reads (in %) MiGA symbol: `raw_reads`. ### Trimmed Reads This is part of *Trimming & read quality* in the above diagram. In this step, MiGA trims reads and clips potential adapters with an adaptive strategy using a combination of [FaQCs](/part5/external#faqcs), [Seqtk](/part5/external#seqtk), and [fastp](/part5/external#fastp). The full pipeline is implemented as a stand-alone submodule called [multitrim](/part5/external#multitrim). Supported file keys: * **For single reads only** * `single` (*req*, *gz*): FastQ file containing trimmed/clipped reads * **For paired-end reads only** * `pair1` (*req*, *gz*): FastQ file containing trimmed/clipped forward reads * `pair2` (*req*, *gz*): FastQ file containing trimmed/clipped reverse reads * `single` (*req*, *gz*): FastQ file containing trimmed/clipped reads with only one sister passing quality control * **Deprecated** (for backwards-compatibility) * `trimming_summary`: Raw text file containing a summary of the trimmed sequences MiGA symbol: `trimmed_reads`. ### Read Quality This is a quality-control step included as part of *Trimming & read quality* in the diagram above. In this step, MiGA generates quality reports of the trimmed/clipped reads using [Falco](/part5/external#falco) via [multitrim](/part5/external#multitrim). Supported file keys: * **For single and paired reads** * `pre_qc_1`: HTML file with QC-report of the forward reads before trimming * `post_qc_1` (*req*): HTML file with QC-report of the forward reads after trimming * `adapter_detection`: List of adapters identified in the first pass * **For paired reads only** * `pre_qc_2`: HTML file with QC-report of the reverse reads before trimming * `post_qc_2`: HTML file with QC-report of the reverse reads after trimming * **Deprecated** (for backwards-compatibility) * `solexaqa` (*dir*): Folder containing the SolexaQA++ quality-control summaries * `fastqc` (*dir*): Folder containing the FastQC quality-control analyses MiGA symbol: `read_quality`. ### Trimmed FastA This is the final step included in *Trimming & read quality* in the diagram above, in which MiGA generates FastA files with the trimmed/clipped reads. Supported file keys: * `coupled` (*req* for coupled reads, unless `pair1` and `pair2` exist): Interposed FastA file containing quality-checked paired reads. If this file doesn't exist, it is automatically generated from `pair1` and `pair2` * `single` (*req* for single reads, *gz* for coupled reads): FastA file with quality-checked single-end reads * `pair1` (*gz*): FastA file containing forward sisters of quality-checked paired-end reads * `pair2` (*gz*): FastA file containing reverse sisters of quality-checked paired-end reads Statistics: * `reads`: Total number of reads * `length_average`: Average read length (in bp) * `length_standard_deviation`: Standard deviation of read length (in bp) * `g_c_content`: G+C content of all reads (in %) * `x_content`: Undetermined bases content of all reads (in %) * `a_t_skew`: A-T sequence skew across all reads (in %) * `g_c_skew`: G-C sequence skew across all reads (in %) MiGA symbol: `trimmed_fasta`. ### Assembly In this step MiGA assembles trimmed FastA reads using [SPADES](/part5/external#spades). Supported file keys: * `largecontigs` (*req*): FastA file containing large contigs or scaffolds (>1Kbp) * `allcontigs`: FastA file containing all contigs or scaffolds (including large) * `assembly_data` (*dir*): Folder containing some intermediate files generated during the assembly Statistics: * `contigs`: Total number of (large) contigs * `n50`: N50 of (large) contigs (in bp) * `total_length`: Total length of large contigs (in bp) * `longest_sequence`: Length of the longest contig (in bp) * `n_content`: Undetermined bases content of large contigs (in %) * `g_c_content`: G+C content of large contigs (in %) * `x_content`: Undetermined bases content of large contigs (in %) * `a_t_skew`: A-T sequence skew across large contigs (in %) * `g_c_skew`: G-C sequence skew across large contigs (in %) MiGA symbol: `assembly`. ### CDS This step corresponds to *Gene prediction* in the diagram above. MiGA predicts coding sequences (putative genes and proteins) using [Prodigal](/part5/external#prodigal), and automatically calculates the most likely codon table between 11 and 4. Supported file keys: * `proteins` (*req*): FastA file containing translated protein sequences * `genes`: FastA file containing putative gene sequences * `gff3` (*gz*): GFF v3 file containing the coordinates of coding sequences. This file is not required, but [MyTaxa](#mytaxa) depends on it (or `gff2` or `tab`, whichever is available) * `gff2` (*gz*): GFF v2 file containing the coordinates of coding sequences. This file is not produced by MiGA, but it's supported for backwards compatibility with earlier versions using MetaGeneMark * `tab` (*gz*): Tabular-delimited file containing the columns: gene ID, gene length, and contig ID. This file is not produced by MiGA, but it's supported to allow [MyTaxa](#mytaxa) to run when more detailed information about the gene prediction is missing Statistics: * `predicted_proteins`: Total number of predicted proteins * `average_length`: Average length of predicted proteins (in aa) * `coding_density`: Coding density of the genome (in %) * `codon_table`: Optimal coding table (4 or 11) MiGA symbol: `cds`. ### Essential Genes In this step, MiGA uses `HMM.essential.rb` from the [Enveomics Collection](/part5/external#enveomics-collection) to identify a set of genes typically present in single-copy in Bacterial and Archaeal genomes. In this step, protein translations of those *essential* genes are extracted for other analyses in MiGA (*e.g.*, hAAI in [distances](#distances)) or outside (*e.g.*, phylogeny or MLSA for [diversity analyses](/part1#diversity)). In addition, this step generates a report that can be used for quality control including estimations of completeness and contamination (for genomes) and median number of copies of single-copy genes (for metagenomes and viromes). Supported file keys: * `ess_genes` (*req*): FastA file containing all extracted protein translations from *essential* genes (.faa) or archived collection (proteins.tar.gz) * `collection` (*req*): Folder containing individual FastA files with protein translations from *essential* genes * `report` (*req*): Raw text report including derived statistics, as well as *essential* genes missing or detected in multiple copies (for genomes) or copy counts (for metagenomes and viromes) * `alignments`: Generated for all genomes (non-multi types). It contains the best matching protein for each detected model aligned to the model * `fastaai_index`: A FastAAI index now deprecated (for backwards-compatibility) * `fastaai_index_2`: A FastAAI index with the second format version (SQLite) * `bac_report`: If present, this is the original report, and it indicates that a corrected report has been generated to accomodate particular features of the dataset Statistics: * **For metagenomes and viromes** * `mean_copies`: Average copy number across essential genes * `median_copies`: Median copy number across essential genes * **For genomes** * `completeness`: Estimated completeness of the genome, based on presence of essential genes (in %) * `contamination`: Estimated contamination of the genome, based on copy number of essential genes (in %) * `quality`: Completeness - 5 x Contamination MiGA symbol: `essential_genes`. ### MyTaxa This step is only supported for metagenomes and viromes, and it requires the (optional) MyTaxa [requirements installed](/part2/requirements). In this step, the most likely taxonomic classification of each contig is identified using [MyTaxa](/part5/external#mytaxa), and a report is generated using [Krona](/part5/external#krona). Supported file keys: * `mytaxa` (*req*): Output generated by MyTaxa * `blast` (*gz*): BLAST against the reference genomes database * `mytaxain` (*gz*): Re-formatted BLAST used as input for MyTaxa * `nomytaxa`: If it exists, MiGA assumes no support for MyTaxa modules, and none of the above files are required * `species`: Profile of species composition (in permil) as raw tab-delimited text * `genus`: Profile of genus composition (in permil) as raw tab-delimited text * `phylum`: Profile of phylum composition (in permil) as raw tab-delimited text * `innominate`: List of innominate taxa (groups without a name but containing lower-rank classifications) as raw text * `kronain`: Raw-text list of taxa used as input for Krona * `krona`: HTML output produced by Krona MiGA symbol: `mytaxa`. ### MyTaxa Scan This step is only supported for genomes (dataset types genome, popgenome, and scgenome), and it requires the (optional) MyTaxa [requirements installed](/part2/requirements). In this step, the genomes are scanned in windows of ten genes. For each window, the taxonomic distribution is determined using [MyTaxa](/part5/external#mytaxa) and compared against the distribution for the entire genome. This is a quality-control step for manual curation. Supported file keys: * `mytaxa` (*req*): MyTaxa output * `report` (*req*): PDF file containing the graphic report * `regions_archive` (*gz*): Archived folder containing FastA files with the sequences of the genes in regions identified as abnormal * `nomytaxa`: If it exists, MiGA assumes no support for MyTaxa modules, and none of the above files are required Deprecated file keys (for backwards-compatibility): * `wintax`: Taxonomic distribution of each window * `blast` (*gz*): BLAST against the reference genomes database * `mytaxain` (*gz*): Re-formatted BLAST used as input for MyTaxa * `regions` (*dir*): Folder containing FastA files with the sequences of the genes in regions identified as abnormal * `gene_ids`: List of genes per window * `region_ids`: List of regions identified as abnormal MiGA symbol: `mytaxa_scan`. ### Distances This step is only supported for genomes ([dataset types](/part2/types#dataset-types) genome, popgenome, and scgenome). In this step, each dataset is compared against all other datasets in the project. If the dataset is a [reference dataset](/part2/types#query-vs-reference-datasets), it is compared against all other reference datasets in the project. If it's a query dataset, it is compared iteratively against medoids. For more details on the strategy used in this step, see the manual [section on distances](/part2/distances). Supported file keys: * **For reference datasets** * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * **For query datasets** * `aai_medoids` (*req* except for clades projects): Best hits among medoids at different hierarchical levels in the AAI indexing * `ani_medoids` (*req* for clades projects): Best hits among medoids at different hierarchical levels in the ANI indexing * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * `ref_tree`: Newick file with the Bio-NJ tree including queried medoids and the query dataset * `ref_tree_pdf`: PDF rendering of `ref_tree` * `intax`: Raw text result of the taxonomy test against the reference genome MiGA symbol: `distances`. ### Taxonomy This step is only supported for genomes ([dataset types](/part2/types#dataset-types) genome, popgenome, and scgenome) that are [reference datasets](/part2/types#query-vs-reference-datasets), in projects with a set reference project (`:ref_project` in metadata). In this step, MiGA compares the genome against a reference project using the query search method, and imports the resulting taxonomy with p-value below 0.1 (or whichever value is set as `:tax_pvalue` in metadata). Supported file keys: * `intax`: Raw text result of the taxonomy test against the reference genome * `aai_medoids` (*req* except for reference clades projects): Best hits among medoids at different hierarchical levels in the AAI indexing * `ani_medoids` (*req* for reference clades projects): Best hits among medoids at different hierarchical levels in the ANI indexing * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * `ref_tree`: Newick file with the Bio-NJ tree including queried medoids and the query dataset * `ref_tree_pdf`: PDF rendering of `ref_tree` Statistics: * `closest_relative`: Name of the reference dataset with highest AAI * `aai`: AAI to the closest relative * `domain_pvalue`, `phylum_pvalue`, `class_pvalue`, `order_pvalue`, `family_pvalue`, `genus_pvalue`, `species_pvalue`, `subspecies_pvalue`: Empirical p-values for classification at each rank with respect to the closest relative, based on the observed AAI MiGA symbol: `taxonomy` ### SSU In this step, MiGA detects rRNA genes (16S and 23S) using [Barrnap](/part5/external#barrnap), extracts the sequences of the small subunit genes (16S) using [Bedtools](/part5/external#bedtools), and identifies tRNA elements using [tRNAscan-SE](/part5/external#trnascan-se). If configured, it will also classify all the 16S rRNA genes detected using the [RDP Naïve Bayes Classifier](/part5/external#rdp-classifier). Supported file keys: * `longest_ssu_gene` (*req*): FastA file containing the longest detected SSU gene * `gff` (*gz*): GFF v3 file containing the location of detected SSU genes * `all_ssu_genes` (*gz*): FastA file containing all the detected SSU genes * `classification`: Taxonomic classification with RDP taxonomy * `trna_list` (*gz*): Raw-text table with tRNA predictions, generated for all genome types (non-multi) Deprecated file keys (for backwards-compatibility): * `ssu_gff` (*gz*): GFF3 file containing pre-filtered SSU rRNA predictions Statistics: * `ssu`: Total number of detected SSU fragments * `complete_ssu`: Number of complete SSU loci * `ssu_fragment`: Maximum percentage covered for any detected SSU fragments * `lsu`: Total number of detected LSU fragments * `complete_lsu`: Number of complete LSU loci * `lsu_fragment`: Maximum percentage covered for any detected LSU fragments * `max_length`: Length of the longest detected SSU fragment * `trna_count`: Total number of tRNA elements detected (including pseudogenes) * `trna_aa`: Number of distinct amino acids for which tRNA elements were detected (excluding pseudogenes) MiGA symbol: `ssu`. ### Stats In this step, MiGA traces back all the results of the dataset and estimates summary statistics. In addition, it cleans any stored values in the distances database including datasets no longer registered in the project. Supported file keys: * `trna_list`: List of tRNA elements detected. This file is only produced for genome datasets with defined taxonomy within the Archaea, Bacteria, or Eukaryota domains MiGA symbol: `stats`. ## Project Results Once all datasets have been pre-processed (*i.e.*, once all the results above are available for all reference datasets), MiGA executes the following project-wide steps: ### hAAI Distances Consolidation of hAAI distances. Supported file keys: * `rds` (*req*): Pairwise values in a `data.frame` for `R` * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of hAAI values as raw tab-delimited file MiGA symbol: `haai_distances`. ### AAI Distances Consolidation of AAI distances. Supported file keys: * `rda` (*req*): Pairwise values for `R` in three vectors * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of AAI values as raw tab-delimited file * `rds` (*deprecated*): Pairwise values in a `data.frame` for `R` MiGA symbol: `aai_distances`. ### ANI Distances Consolidation of ANI distances. Supported file keys: * `rda` (*req*): Pairwise values for `R` in three vectors * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of ANI values as raw tab-delimited file * `rds` (*deprecated*): Pairwise values in a `data.frame` for `R` MiGA symbol: `ani_distances`. ### Clade Finding This step is only supported for project types [genomes](/part2/types#genomes) and [clade](/part2/types#clade). In this step, MiGA attempts to identify clades at species level or above using a combination of ANI and AAI values. MiGA generates [AAI clades](/part2/clustering#aai-clades) in this step for [genomes projects](/part2/types#genomes). Clades proposed at AAI > 90% and ANI > 95% are formed using the Markov Clustering algorithm implemented in [MCL](/part5/external#mcl). Most distance manipulation and tree estimation and manipulation utilities use the R packages [Ape](/part5/external#ape) and [Vegan](/part5/external#vegan). Supported file keys: * `report` (*req* for `genomes`): PDF file including a graphic report for the clustering * `class_table` (*req* for `genomes`): Tab-delimited file containing the classification of all datasets in AAI clusters * `class_tree` (*req* for `genomes`): Newick file containing the classification of all datasets in AAI clusters as a dendrogram * `classif` (*req* for `genomes`): Tab-delimited file containing the highest-level classification of each dataset, the medoid of the cluster, and the AAI against the corresponding medoid * `medoids` (*req* for `genomes`): List of medoids per cluster * `aai_tree`: Bio-NJ tree based on AAI distances in Newick format * `aai_dist_rds`: AAI-based distances in R data serialized format * `proposal` (*req*): Proposed species-level clades in the project, based on `clades_ani95`. One line per proposed clade, with tab-delimited dataset names. Only clades with 5 or more members are included * `clades_aai90`: Clades formed at AAI > 90%. One clade per line, with comma-delimited dataset names * `clades_ani95`: Clades formed at ANI > 95%. One clade per line, with comma-delimited dataset names * `medoids_ani95`: List of `clades_ani95` datasets with the smallest ANI distance to all members of its own ANI95 clade. The list is in the same order MiGA symbol: `clade_finding`. ### Subclades This step is only supported for project type [clade](/part2/types#clade). In this step, MiGA attempts to identify clades below species level using ANI values. MiGA generates [ANI clades](/part2/clustering#ani-clades) in this step. Most distance manipulation and tree estimation and manipulation utilities use the R packages [Ape](/part5/external#ape) and [Vegan](/part5/external#vegan). Supported file keys: * `report` (*req*): PDF file including a graphic report for the clustering * `class_table` (*req*): Tab-delimited file containing the classification of all datasets in ANI clusters * `class_tree` (*req*): Newick file containing the classification of all datasets in ANI clusters as a dendrogram * `classif` (*req*): Tab-delimited file containing the highest-level classification of each dataset, the medoid of the cluster, and the ANI against the corresponding medoid * `medoids` (*req*): List of medoids per cluster * `ani_tree`: Bio-NJ tree based on AAI distances in Newick format * `ani_dist_rds`: ANI-based distances in R data serialized format MiGA symbol: `subclades`. ### OGS This step is only supported for project type [clade](/part2/types#clade). In this step, MiGA generates groups of orthology using reciprocal best matches between all pairs of datasets in the project. Groups are generated using [MCL](/part5/external#mcl) with pairs weighted by bit score. Once computed, MiGA uses the matrix of OGS to estimate summary and rarefied statistics. Supported file keys: * `ogs` (*req*): Matrix of orthology groups, as tab-delimited raw file * `stats` (*req*): Summary statistics in JSON format * `abc` (*gz*): When available, it includes all the individual RBM files in ABC format. This file is typically produced as intermediate result and removed before finishing, but can be maintained using `miga option -P . --key clean_ogs --value false` in the project folder using the [CLI](https://github.com/bio-miga/miga/blob/main/manual/part3/cli.md) * `core_pan`: Summary statistics of rarefied core-genome/pangenome sizes in tab-delimited format * `core_pan_plot`: Plot of rarefied core-genome/pangenome sizes in PDF MiGA symbol: `ogs`. ### Project Stats In this step, MiGA traces back all the results of the project and estimates summary statistics. Supported file keys: * `taxonomy_index` (*req*): Index of datasets per taxonomy in JSON format * `metadata_index` (*req*): Searchable index of datasets metadata as SQLite3 database MiGA symbol: `project_stats`. # Metadata ## All objects All metadata objects support the following fields: | Field | Supported values | Description | | --------: | :--------------------------------: | ------------------- | | created\* | [Date](/part1/glossary#miga-dates) | Date of creation | | updated\* | [Date](/part1/glossary#miga-dates) | Date of last update | > **\*** Mandatory ## Projects The following metadata fields are recognized by different interfaces for **Projects**: ### Project Features Metadata with additional information and features about the project: | Field | Supported values | Description | | ----------: | :--------------------------------: | --------------------- | | comments | String | Free-form comments | | description | String | Free-form description | | name\* | [Name](/part1/glossary#miga-names) | Name‡ | > **\*** Mandatory ### Project System Metadata Metadata entries automatically set by MiGA: | Field | Supported values | Description | | ---------: | :--------------: | ---------------------------------- | | datasets\* | Array of String | List of datasets in the project | | type\* | String | [Type](/part2/types#project-types) | > **\*** Mandatory > > **‡** By default the base name of the project path ### Project Flags Metadata entries that trigger specific behaviors in MiGA: | Field | Supported values | Description | | --------------------: | :--------------: | ---------------------------------------------- | | ref\_project | Path | Project with reference taxonomy {1} | | db\_proj\_dir | Path | Directory containing database projects {1} {2} | | tax\_pvalue | Float \[0,1] | Max p-value to transfer taxonomy (def: 0.1) | | haai\_p | String | hAAI engine {3} (def: fastaai) | | aai\_p | String | AAI engine {3} (def: diamond) | | ani\_p | String | ANI engine {3} (def: fastani) | | max\_try | Integer | Max number of task attempts (def: 10) | | aai\_save\_rbm | Boolean | Should RBMs be saved for OGS analysis? | | ogs\_identity | Float \[0,100] | Min RBM identity for OGS (def: 80) | | clean\_ogs | Boolean | If false, keeps ABC (clades only) | | run\_clades | Boolean | Should clades be estimated from distances? | | run\_ogs | Boolean | Should orthologous groups be estimated? | | gsp\_ani | Float \[0,100] | ANI limit to propose gsp clades (def: 95) | | gsp\_aai | Float \[0,100] | AAI limit to propose gsp clades (def: 90) | | gsp\_metric | String | Metric to propose clades: `ani` (def), `aai` | | ess\_coll | String | Collection of essential genes to use {4} | | min\_qual | Float (or 'no') | Min. genome quality (or no filter; def: 25) | | distances\_checkpoint | Integer | Comparisons before storing data (def: 10) | > **{1}** This path can either be absolute or relative to the project's path > > **{2}** This is the location of the databases used by [db\_project](#dataset-flags). If not set, it is assumed to be the parent folder of the current project > > **{3}** Supported values: `blast`, `blat`, `diamond` (only for hAAI and AAI), `fastani` (only for ANI), `no` (only for hAAI and AAI), and `fastaai` (only for hAAI) > > **{4}** One of: `dupont_2012` (default), or `lee_2019` ### Project Hooks Additionally, hooks can be defined for projects as arrays of arrays containing the action name and the arguments (if any). For example, one can define: ``` on_processing_ready: [ ['run_cmd', 'date > {{project}}/ALL_DONE.txt'], ['run_cmd', 'sendmail ...'] ] ``` or ``` on_add_dataset: [ ['run_cmd', 'echo {{object}} > {{project}}/LATEST_DATASET.txt'] ] ``` Supported events: * `on_create()`: When created * `on_load()`: When loaded * `on_save()`: When saved * `on_add_dataset(object)`: When a dataset is added, with name `object` * `on_unlink_dataset(object)`: When dataset with name `object` is unlinked * `on_result_ready(object)`: When any result is ready, with key `object` * `on_result_ready_{result}()`: When `result` is ready * `on_processing_ready()`: When processing is complete Supported hooks: * `run_lambda(lambda, args...)` * `run_cmd(cmd)` ## Datasets The following metadata fields are recognized by different interfaces for **Datasets**: ### Dataset Features Metadata with additional information and features about the dataset: | Field | Supported values | Description | | ----------------: | :--------------: | ---------------------------------------- | | tax | MiGA::Taxonomy | Taxonomy of the dataset | | quality | String | Description of genome quality | | trna\_count | Integer | Number of tRNA elements detected | | trna\_aa | Integer | Number of distinct AA with tRNA elements | | dprotologue | String | Taxonumber in the Digital Protologue DB | | ncbi\_tax\_id | String | Linking ID(s) {1} for NCBI Taxonomy | | ncbi\_nuccore | String | Linking ID(s) {1} for NCBI Nucleotide | | ncbi\_asm | String | Linking ID(s) {1} for NCBI Assembly | | ebi\_embl | String | Linking ID(s) {1} for EBI EMBL | | ebi\_ena | String | Linking ID(s) {1} for EBI ENA | | web\_assembly | String | URL to download assembly | | web\_assembly\_gz | String | URL to download gzipped assembly | | see\_also | String | Link(s) {1} in the format text:url | | is\_type | Boolean | If it is type material | | is\_ref\_type | Boolean | If it is reference material {2} | | type\_rel | String | Relationship to type material | | suspect | Array(String) | Flags indicating a suspect dataset | > **{1}** Multiple values can be provided separated by commas or colons > > **{2}** This is not a valid type, but it represents the closest available dataset to material that is unavailable and unlikely to ever become available. See also [Federhen, 2015, NAR](https://doi.org/10.1093/nar/gku1127) ### Dataset System Metadata Metadata entries automatically set by MiGA: | Field | Supported values | Description | | -------------: | :--------------: | -------------------------------------------- | | type\* | String | [Type](/part2/types#dataset-types) | | ref | Boolean | [Reference](/part2/types#reference) | | inactive | Boolean | If auto-processing should stop | | metadata\_only | Boolean | Dataset with metadata but without input data | | status | String | Proc. status: complete, incomplete, inactive | | \_step | String | For internal control of processing | | \_try\_`step` | Integer | For internal control of processing | | ~~user~~ | String | Deprecated | > **\*** Mandatory ### Dataset Flags Metadata entries that trigger specific behaviors in MiGA: | Field | Supported values | Description | | ---------------------: | :--------------: | ---------------------------------------- | | run\_`step` | Boolean | Forces running or not `step` | | db\_project | Path | Project to use as database {1} | | dist\_req | Array of String | Run distances against these datasets {2} | | keep\_assembly\_graphs | Boolean | Do not clean assembly graphs {3} | > **{1}** By default, it uses its own project as database. The path can be absolute or relative to the parent folder of the project > > **{2}** When searching best-matching datasets, include these datasets even if they are not visited using the medoid tree > > **{3}** By default: false, meaning that assembly graphs are removed Any of these dataset flags can also be set as project metadata, which applies to all datasets in the project. If both a dataset and a project metadata flag are set, dataset flags take precedence. If neither is set, the default values are used. ### Dataset Hooks Additionally, hooks can be defined for datasets as arrays of arrays containing the action name and the arguments. See above ([project hooks](#project-hooks)) for examples. Supported events: * `on_load()`: When loaded * `on_save()`: When saved * `on_remove()`: When removed * `on_inactivate()`: When inactivated * `on_activate()`: When activated * `on_result_ready(object)`: When any result is ready, with key `object` * `on_result_ready_{result}()`: When `result` is ready * `on_preprocessing_ready()`: When preprocessing is complete Supported hooks: * `run_lambda(lambda, args...)` * `clear_run_counts()` * `run_cmd(cmd)` # External Software ## Enveomics Collection Rodriguez-R LM, and Konstantinidis KT (2016). The enveomics collection: a toolbox for specialized analyses of microbial genomes and metagenomes. [PeerJ Preprints 4:e1900v1](https://doi.org/10.7287/peerj.preprints.1900v1). ## NCBI BLAST+ Camacho C, Coulouris G, Avagyan V, Ma N, Papadopoulos J, Bealer K, and Madden TL (2009). BLAST+: architecture and applications. [BMC Bioinformatics 10:421](https://doi.org/10.1186/1471-2105-10-421). ## HMMer 3 Mistry J, Finn RD, Eddy SR, Bateman A, and Punta M (2013). Challenges in Homology Search: HMMER3 and Convergent Evolution of Coiled-Coil Regions. [Nucleic Acids Res 41:e121](https://doi.org/10.1093/nar/gkt263). ## FaQCs Chienchi L and Chain PSG (2014). Rapid evaluation and Quality Control of Next Generation Sequencing Data with FaQCs. [BMC Bioinformatics 15:366](https://doi.org/10.1186/s12859-014-0366-2). ## Seqtk [GitHub: lh3/seqtk](https://github.com/lh3/seqtk) by Heng Li et al. ## fastp Chen S, Zhou Y, Chen Y, Gu J (2018). fastp: an ultra-fast all-in-one FASTQ preprocessor. [Bioinformatics 34(17):i884–i890](https://doi.org/10.1093/bioinformatics/bty560) ## Falco de Sena BG and Smith AD (2021). Falco: high-speed FastQC emulation for quality control of sequencing data. [F1000Research 8:1874](https://doi.org/10.12688/f1000research.21142.2) ## multitrim [GitHub: KGerhardt/multitrim](https://github.com/KGerhardt/multitrim), by Kenji Gerhardt et al. ## SPADES Prjibelski A, Antipov D, Meleshko D, Lapidus A,and Korobeynikov A (2020). Using SPAdes De Novo Assembler. [Current Protocols in Bioinformatics](https://doi.org/10.1002/cpbi.102). Nurk S, Meleshko D, Korobeynikov A, and Pevzner PA (2017). metaSPAdes: a new versatile metagenomic assembler. [Genome Research, 27:824-834](https://doi.org/10.1101/gr.213959.116). ## Prodigal Hyatt D, Chen GL, Locascio PF, Land ML, Larimer FW, Hauser LJ (2010). Prodigal: prokaryotic gene recognition and translation initiation site identification. [BMC Bioinformatics, 11(1):119](https://doi.org/10.1186/1471-2105-11-119). ## Barrnap [GitHub: tseemann/barrnap](https://github.com/tseemann/barrnap), by Torsten Seemann. ## Bedtools [Bedtools site](http://bedtools.readthedocs.io/en/latest), by the Quinlan Laboratory. ## RDP Classifier Wang Q, Garrity GM, Tiedje JM, and Cole JR (2007). Naïve Bayesian Classifier for Rapid Assignment of rRNA Sequences into the New Bacterial Taxonomy. [Appl Environ Microbiol 73(16):5261-7](https://doi.org/10.1128/aem.00062-07). ## MyTaxa Luo C, Rodriguez-R LM, and Konstantinidis KT (2014). MyTaxa: an advanced taxonomic classifier for genomic and metagenomic sequences. [Nucleic Acids Res, 42(8):e73](https://doi.org/10.1093/nar/gku169). ## Krona Ondov BD, Bergman NH, and Phillippy AM (2011). Interactive metagenomic visualization in a Web browser. [BMC Bioinformatics 12:385](https://doi.org/10.1186/1471-2105-12-385). ## MCL Enright AJ, Van Dongen S, and Ouzounis CA (2002). An efficient algorithm for large-scale detection of protein families. [Nucleic Acids Res 30(7):1575-1584](https://doi.org/10.1093/nar/30.7.1575). ## Ape Paradis E, Claude J, and Strimmer K (2004). APE: Analyses of Phylogenetics and Evolution in R language. [Bioinformatics 20(2):289-290](https://doi.org/10.1093/bioinformatics/btg412). ## Vegan Oksanen J, Blanchet FG, Friendly M, Kindt R, Legendre P, McGlinn D, Minchin PR, O'Hara RB, Simpson GL, Solymos P, Stevens MHM, Szoecs E, and Wagner H (2017). Vegan: Community Ecology Package. [R package](https://CRAN.R-project.org/package=vegan). ## Diamond Buchfink B, XieC, and Huson DH (2015). Fast and Sensitive Protein Alignment using DIAMOND. [Nature Methods 12:59–60](https://doi.org/10.1093/10.1038/nmeth.3176). ## FastANI Jain C, Rodriguez-R LM, Phillippy AM, Konstantinidis KT, and Aluru S (2018). High throughput ANI analysis of 90K prokaryotic genomes reveals clear species boundaries. [Nature Comm 9:5114](https://doi.org/10.1038/s41467-018-07641-9). ## FastAAI [GitHub: cruizperez/FastAAI](https://github.com/cruizperez/FastAAI), by Carlos Ruiz-Perez et al. ## tRNAscan-SE Chan PP, Lin BY, Mak AJ, and Lowe TM (2021). tRNAscan-SE 2.0: improved detection and functional classification of transfer RNA genes. [Nucleic Acids Res 49:9077–9096](https://doi.org/10.1093/nar/gkab688). # Part VI: Workflows CLI Workflows automate common tasks in MiGA. Currently implemented workflows include: 1. [Quality](/part6/quality_wf) 2. [Dereplicate](/part6/derep_wf) 3. [Classify](/part6/classify_wf) 4. [Preprocess](/part6/preproc_wf) 5. [Index](/part6/index_wf) ## Using multiple workflows It is possible to concatenate workflows in the same project. First, run the first workflow as described in the documentation. For example: ```bash miga quality_wf -o my_project /path/to/genomes/*.fna ``` Next, execute any additional steps *without* specifying the input files, and using the same output directory. For example: ```bash miga classify_wf -o my_project miga rerep_wf -o my_project ``` In the examples above, input genomes will be processed to evaluate quality, next they'll be classified, and finally they'll be dereplicated. # Quality This workflow automates the evaluation of quality statistics on your genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs). To execute the workflow, run: ```bash miga quality_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga quality_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. In addition, if you pass the option `--mytaxa-scan`, you can expect the subdirectory `mytaxa_scan` including PDF reports for each input genome. # Dereplicate This workflow automates the dereplication of your collection of genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs). To execute the workflow, run: ```bash miga derep_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga derep_wf -h ``` Importantly, the dereplication can be performed on ANI (default) or AAI (passing the `--aai` flag) at a given threshold (by default 95%) that can be modified with the flag `--threshold`. Finally, the representative genomes can be selected to reflect the highest genome quality (default) or to be the most "central" genome in the clade in ANI or AAI space (passing the `--medoids` flag). ## Expected output Once your run is complete, you may expect the [standard summaries](/part6/summaries) for `cds`, `assembly`, and `essential_genes`, as well as a table (`genomospecies.tsv`) with three columns: (1) a clade name, (2) the name of the representative genome, and (3) the names of all the members in the clade separated by commas. Additionally you can expect the subdirectory `representatives` including assemblies (FastA files, nucleotides) of all representative genomes. This is the dereplicated set of genomes. # Classify This workflow automates the comparison of your genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs) against a reference project for taxonomy. A reference project has to be a separate MiGA project with a database of your choice, taxonomically annotated, and fully indexed. If you want to use the pre-processed database provided by MiGA Online, run: ```bash miga download ``` To execute the workflow, run: ```bash miga classify_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga classify_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `taxonomy`, as well as a summary classification table (`classification.tsv`) with two columns: (1) the genome name, and (2) the space-delimited classification of the genome with taxon names prefixed by the rank code and a colon. Something like: ``` NZ_CP010409_1 d:Bacteria p:Proteobacteria c:Gammaproteobacteria o:Xanthomonadales f:Xanthomonadaceae g:Xanthomonas s:Xanthomonas_sacchari NZ_CP016878_1 d:Bacteria p:Proteobacteria c:Gammaproteobacteria o:Xanthomonadales f:Xanthomonadaceae g:Xanthomonas s:Xanthomonas_hortoru [...] ``` # Preprocess This workflow automates the preprocessing of your genomic or metagenomic datasets. To execute the workflow, run: ```bash miga preproc_wf -o my_project -i raw_reads_single path/to/reads/*.fastq ``` Supported inputs include: * *raw\_reads\_single:* Single raw reads in a single FastQ file * *raw\_reads\_paired:* Paired raw reads in two FastQ files * *trimmed\_reads\_single:* Single trimmed reads in a single FastA file * *trimmed\_reads\_paired:* Paired trimmed reads in two FastA files * *trimmed\_reads\_interleaved:* Paired trimmed reads in a single FastA file * *assembly:* Assembled contigs or scaffolds in FastA format For additional options, run: ```bash miga preproc_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. Additionally all the intermediate files are preserved, including assemblies, predicted genes, and detected essential and ribosomal genes. # Index This workflow automates the indexing of your genomic collection. To execute the workflow, run: ```bash miga index_wf -o my_project path/to/genomes/*.fasta ``` For additional options, run: ```bash miga index_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. Additionally all the intermediate files are preserved, including assemblies, predicted genes, and detected essential and ribosomal genes. Importantly, all-vs-all comparisons are generated using Average Amino Acid Identity (AAI) and Average Nucleotide Identity (ANI), making this a queriable project. ## Indexing publicly available genomes It is also possible to use this workflow on genomes publicly available in NCBI. This enables taxonomic analysis, such as using this indexed project as a reference database for the [classification workflow](/part6/classify_wf). For example, to download and index all the genomes from the species **Xanthomonas vesicatoria** using Diamond for AAI and FastANI for ANI estimation (`--fast` flag), run: ```bash miga index_wf -o X_vesicatoria \ -T 'Xanthomonas vesicatoria' --project-type clade --fast -v ``` # Summaries All workflows generate summary tables, which are raw text tab-delimited files with information about the input datasets. The standard summaries include stats for: * `assembly.tsv`: [Assembly](/part5/workflow#assembly) * `cds.tsv`: [CDS](/part5/workflow#cds) * `essential_genes.tsv`: [Essential Genes](/part5/workflow#essential_genes) * `ssu.tsv`: [SSU](/part5/workflow#ssu) * `taxonomy.tsv`: [Taxonomy](/part5/workflow#taxonomy) # Introduction ## What is MiGA? MiGA is a data management and processing system for microbial genomes and metagenomes. Its main goal is to provide a uniform system for genome-based taxonomic classification and diversity studies, but its base can be used for other purposes. * [How can MiGA help me?](/master/part1/pitch) * [Who's using MiGA?](/master/part1/use-cases) * [Definitions](/master/part1/glossary) ## Getting started with MiGA MiGA itself doesn't require much but processing large data collections may require large infrastructures. With that in mind, MiGA is designed to support both single machines and cluster infrastructures. * [Requirements](/master/part2/requirements) * [Installation](/master/part2/installation) * [MiGA types](/master/part2/types) * [Input data](/master/part2/input) * [Distances](/master/part2/distances) * [Clustering](/master/part2/clustering) ## MiGA Interfaces You can interact with MiGA through different interfaces. These interfaces have different purposes, but they also have some degree of overlap, because different users with different aims sometimes want to do the same thing. The API and the CLI are the most comprehensive and flexible interfaces, but the other two are friendlier to humans. MiGA is mostly written in Ruby, with an object-oriented approach, and all the interfaces are just thin layers atop the Ruby core. That means that you can write your own interfaces (or pieces) if you know how to talk to these Ruby objects. Sometimes I even use `irb`, which is an interactive shell for Ruby, but that's mostly for debugging. * [MiGA API](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part3/api.md) * [MiGA CLI](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part3/cli.md) * [MiGA Web](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part3/web.md) * See also: [Help for MiGA Online](https://help.microbial-genomes.org/) ## Deploying examples Once you have installed MiGA, you might want to follow one (or several) of these tutorials to familiarize yourself with the MiGA environment. * [RefSeq in MiGA](/master/part4/deploy-refseq) * [Build a clade collection](/master/part4/deploy-clade) * [Launching daemons](/master/part4/daemons) * [Setting up MiGA in a cluster](/master/part4/cluster) ## MiGA in detail Ready for more? Here are some technical details for advanced users. * [Advanced configuration](/master/part5/advanced-configuration) * [MiGA workflow](/master/part5/workflow) * [Metadata](/master/part5/metadata) * [External Software](/master/part5/external) ## Workflows MiGA comes with ready-to-use pre-configured workflows for different analyses: * [Quality](/master/part6/quality_wf) * [Dereplicate](/master/part6/derep_wf) * [Classify](/master/part6/classify_wf) * [Preprocess](/master/part6/preproc_wf) * [Index](/master/part6/index_wf) * [Summaries](/master/part6/summaries) # Part I: What is MiGA? MiGA is a [data management](/master/part1#data-management) and [processing](/master/part1#processing) system for [microbial genomes and metagenomes](/master/part1#data-types). Its main goal is to provide a [uniform system](/master/part1#standards) for [genome-based taxonomic classification](/master/part1#taxonomy) and [diversity](/master/part1#diversity) studies, and its base can be used for [other purposes](/master/part1#more). ## Data management MiGA organizes your data in a consistent, well-organized fashion independent of centralized databases. This makes MiGA projects the ideal system to store data even if you don't use MiGA for anything else. MiGA is completely based on filesystem structures, so it can easily be transferred, backed-up, and stored long-term. Moreover, MiGA projects can be easily browsed, with descriptive folder names and a simple structure that is easy to understand. **MiGA is not** designed to support versioning or database storage, other than individual file-based databases, in order to keep the overhead on any of the tasks above (and the system requirements) at a minimum. ## Processing MiGA performs general-purpose analyses to pre-process genomic and metagenomic data. The main purpose of MiGA is [genome-based taxonomy](/master/part1#taxonomy), but some pre-processing steps are necessary regardless, so they can be used for many other purposes. For example, the initial data in most genomic and metagenomic projects is sequencing data. For almost any project, this means that trimming, clipping, and read-quality assessment are necessary steps for any downstream analyses. In most cases, assembly and gene prediction are also necessary, and other analyses like rRNA and essential genes detection is very useful. All of this is automatically done by MiGA! **MiGA is not** a workflow manager system. MiGA only supports short-read data (and it's optimized for Illumina data) or already assembled sequences. MiGA's goal is to keep analyses as simple and standardized as possible, so only critical customization is supported. ## Data types MiGA is designed to process genomes and can handle metagenomes (with some restrictions). MiGA is optimized for short-read datasets or assembled datasets. MiGA is optimized to process prokaryotic data (Archaeal and Bacterial), but it has some readily available customizations for viral metagenomes (or viromes). For more details, see the [types](/master/part2/types) of datasets and projects and the [input data](/master/part2/input) supported. **MiGA does not** have custom settings for eukaryotic or viral genomes, nor for transcriptomic data. The data management design (and perhaps some of the processing steps) can be used for these and other purposes, but thread carefully. ## Standards MiGA has a general-purpose design with some presets designed for the different [data types](/master/part2/types) supported. All internal configuration and metadata are stored as individual JSON files. Sequence quality is stored as FastQ, and sequences are stored as FastA; these two cover most of the data in the system. There are also some graphic reports in PDF and HTML, raw-text reports and logs, and general statistics in JSON. Finally, all of the [pair-wise comparisons](/master/part2/distances) are stored in SQLite3 files [described here](/master/part2/distances#sqlite3-schema). ### Filesystem structure * **daemon/**: Daemons lair. * **daemon.json**: Daemon settings. * ...: Several daemon log files. * **data/**: All the data is stored here. * **01.raw\_reads/**: Raw reads in FastQ format ([raw\_reads](/master/part5/workflow#raw-reads)). * **02.trimmed\_reads/**: Trimmed/clipped reads in FastQ format ([trimmed\_reads](/master/part5/workflow#trimmed-reads)). * **03.read\_quality/**: Read quality reports in HTML and PDF formats ([read\_quality](/master/part5/workflow#read-quality)). * **04.trimmed\_fasta/**: Trimmed/clipped and interposed reads in FastA format ([trimmed\_fasta](/master/part5/workflow#trimmed-fasta)). * **05.assembly/**: Assemblies in FastA format ([assembly](/master/part5/workflow#assembly)). * **06.cds/**: Gene predictions in FastA (genes and proteins) and GFF formats ([cds](/master/part5/workflow#cds)). * **07.annotation/**: Data annotations. * **01.function/**: Functional annotations. * **01.essential/**: Essential prokaryotic gene detections ([essential\_genes](/master/part5/workflow#essential-genes)). * **02.ssu/**: Ribosomal RNA (small subunit) sequence annotations ([ssu](/master/part5/workflow#ssu)). * **02.taxonomy/**: Taxonomic annotations. * **01.mytaxa/**: MyTaxa fragment annotations ([mytaxa](/master/part5/workflow#mytaxa)). * **03.qa/**: Quality assessments. * **01.checkm/**: (Currently not in use). * **02.mytaxa\_scan/**: Gene-window assessment of taxonomic distributions ([mytaxa\_scan](/master/part5/workflow#mytaxa-scan)). * **08.mapping/**: (Currently not in use). * **09.distances/**: Pair-wise comparisons ([distances](/master/part5/workflow#distances)). * **01.haai/**: Heuristic Average Amino Acid Identity (essential proteins; [distances](/master/part5/workflow#distances) and [haai\_distances](/master/part5/workflow#haai-distances)). * **02.aai/**: Average Amino Acid Identity (all proteins; [distances](/master/part5/workflow#distances) and [aai\_distances](/master/part5/workflow#aai-distances)). * **03.ani/**: Average Nucleotide Identity (genomic fragments; [distances](/master/part5/workflow#distances) and [ani\_distances](/master/part5/workflow#ani-distances)). * **04.ssu/**: (Currently not in use). * **05.taxonomy/**: Taxonomy based on reference projects ([taxonomy](/master/part5/workflow#taxonomy)). * **10.clades/**: Dataset clustering at various resolution levels. * **01.find/**: Identification of naturally-forming AAI clades at species level and above ([clade\_finding](/master/part5/workflow#clade-finding)). * **02.ani/**: Identification of naturally-forming ANI clades at species level and below ([subclades](/master/part5/workflow#subclades)). * **03.ogs/**: Extraction of orthologous groups of proteins and pan-genome statistics ([ogs](/master/part5/workflow#ogs)). * **04.phylogeny/**: (Currently not in use). * **05.metadata/**: (Currently not in use). * **90.stats/**: Results metadata for dataset stats ([stats](/master/part5/workflow#stats) and project-wide indexing and statistics ([project\_stats](/master/part5/workflow#project-stats)). * **metadata/**: Collection of JSON files with datasets metadata. * **miga.project.json**: JSON file with project metadata. ## Taxonomy MiGA's ultimate goal is to provide a standardized set of tools for consistent genome-wide taxonomic analyses. For this reason, MiGA **does not** provide nor favor any one taxonomic database. This *authority-agnostic* approach allows us to focus on the underlying analyses, supporting as many schemas as possible. With that being said, MiGA does support automated taxonomy annotation for some databases in EBI and NCBI linked to NCBI Taxonomy, and it does support some automated adjustments for the JGI schema (in particular for metagenomes). Instead of forcing groups by external taxonomies that may have varying degrees of accuracy and completeness, MiGA follows a data-driven clustering based on [naturally-forming groups](/master/part2/clustering) based on AAI and ANI analyses. Hence, MiGA projects can be used to classify novel genomes using any reference taxonomy (or none!). **MiGA does not** provide or endorse any particular taxonomic authority. ## Diversity MiGA can catalogue datasets, even in the absence of a reference taxonomy. This allows many advanced analyses, including (but not restricted to): * Phylogenomic reconstructions using [orthologous groups of proteins](/master/part5/workflow#ogs) * Multi-Locus sequence analysis using [essential genes](/master/part5/workflow#essential-genes) * Characterization of collections of [single-cell](/master/part2/types#single-cell-genome) or [population](/master/part2/types#population-genome) genomes * Characterization of [intra-population diversity](/master/part2/clustering#ani-clades) * [Metagenome](/master/part2/types#metagenome) or [virome](/master/part2/types#virome) analyses ## More The [intermediate analyses](/master/part5/workflow) performed by MiGA can be used for many other purposes. For example, we use MiGA's initial pre-processing (such as [read trimming](/master/part5/workflow#trimmed-reads) / [quality check](/master/part5/workflow#read-quality), [assembly](/master/part5/workflow#assembly), and [gene prediction](/master/part5/workflow#cds)) in most of our genomic and metagenomic projects. # How can MiGA help me? MiGA's analyses can accurately classify genomes providing a statistical classification support, identify taxonomic novelty, and annotate and evaluate sequencing reads and assembly quality, among others. MiGA can take in a variety of inputs such as raw, unassembled reads, assembled isolate genomes, metagenome-assembled genomes (MAGs), and single-cell amplified genomes (SAGs). MiGA uses a combination of the genome-aggregate average nucleotide identity concept or ANI and the average amino-acid identity, AAI, to taxonomically classify a query genomic sequence against the genome sequences in its reference database. Part of MiGA’s strength lies in the 10,000+ reference genomes that make up its database and an efficient heuristic algorithm to search the query genome against all this database. The reference database is regularly updated and improved with minimal downtime, ensuring consistently improved classification accuracy. # Who is using MiGA? MiGA provides a wide range of utility to suit a variety of users. Exploration of existing datasets can be beneficial for newcomers and experienced scientists alike. For inexperienced users, MiGA can help especially with the initial pre-processing step. By using a set of tools from the [Enveomics collection](http://enveomics.ce.gatech.edu/enveomics), MiGA simplifies the first steps of genomic analysis which can be overwhelming until users are familiar with the variety of tools utilized by the field. MiGA's analytical ability is also robust enough that even experienced users can find utility within. MiGA can also work as a powerful educational tool by providing easy analysis and visualization of results for use in and out of the classroom. # Who is behind MiGA? ## Conception and Design * [Luis M Rodriguez-R](https://rodriguez-r.com/) * [Konstantinos T Konstantinidis](http://enve-omics.gatech.edu/) * James R Cole * James M Tiedje * Ramon Rosselló-Mora ## Development * [Luis M Rodriguez-R](https://rodriguez-r.com/) * Fang Yuan * Santosh Gunturu * [Zilong Huang](mailto:zhuang390@gatech.edu) ## Documentation * [Luis M Rodriguez-R](https://rodriguez-r.com/) * [MaryFrances Johnson](mailto:mjohnson394@gatech.edu) * [John Quensen](https://john-quensen.com/) * William T Harvey * Tanya Kiryutina ## Testing and Evaluation * [Luis M Rodriguez-R](https://rodriguez-r.com/) * Brittany J Suttner * Fang Yuan * Santosh Gunturu * Jianshu Zhao * Roth Conrad # Definitions ## MiGA API Ruby Application Program Interface to MiGA. ## MiGA CLI Command Line Interface to MiGA. ## MiGA Web Web-based interface to MiGA. ## MiGA Names MiGA names are non-empty strings composed exclusively of alphanumerics and underscores. All the dataset names in MiGA must conform to this restriction. ## MiGA Dates The official format in which MiGA represents date/times is the default of Ruby's `Time.now.to_s`, as generated by the CLI action `miga date`. In the \*nix `date` utility this corresponds to the format: `+%Y-%m-%d %H:%M:%S %z`. # Part II: Getting started To get started with MiGA, first make sure you have all the [system requirements](/master/part2/requirements). Next, [install and configure MiGA](/master/part2/installation) in order to use it for processing. You can also skip the configuration if you simply want to access projects previously processed. Finally, you may want to get familiar with the [MiGA types](/master/part2/types), the [input data for MiGA](/master/part2/input), and the methods for [distances](/master/part2/distances) and [clustering](/master/part2/clustering). # Requirements MiGA requires a system (single computer, cluster, or cloud-computing infrastructure) with Linux or MacOS. If you simply want to access projects previously processed, all you need is `ruby` 2.3+ and the `miga-base` gem. If you want to use MiGA to process your data, please follow these instructions. MiGA has a relatively long list of requirements, but most of them are easy to install. Find your system below and follow the instructions using: * [Homebrew](/master/part2/requirements/brew) (preferred), * [`apt-get`](/master/part2/requirements/apt-get), * [Anaconda](/master/part2/requirements/conda), * [Installing from source](/master/part2/requirements/source). Additionally you will need [MyTaxa utils](/master/part2/requirements/mytaxa) in order to activate the [MyTaxa](/master/part5/workflow#mytaxa) and [MyTaxa Scan](/master/part5/workflow#mytaxa-scan) steps (optional). # Using Homebrew ## Live notebook If you prefer to see code in action, the full installation process with homebrew is available as a [Notebook in Google Collab](https://colab.research.google.com/drive/1Wv4uZwLGuzc5RiAT8NkgJ6B_IKAeM0KU). ## Installing requirements using Homebrew You can use [Homebrew](https://brew.sh/) to install most of the software required by MiGA. If you don't have Homebrew, execute (and follow the instructions): ```bash /bin/bash -c "$(curl -fsSL \ https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" ``` Next, make sure you have the brewsci/bio tap: ```bash brew tap brewsci/bio ``` ### Essentials Now that you have Homebrew and the science tap, execute: ```bash # General-purpose software brew install r sqlite3 python adoptopenjdk # Bioinformatics software brew install blast hmmer bedtools \ prodigal idba mcl barrnap fastqc solexaqa \ diamond fastani brew install jonchang/biology/scythe # Pending: Scythe in science (contacting authors) # See: https://github.com/brewsci/homebrew-bio/issues/23 # See also: https://github.com/vsbuffalo/scythe/pull/20 ``` ### A recent Ruby It's possible you already have a recent version of ruby (2.3+). To check which version you have, simply run: `ruby -v`. If your version of Ruby is older than 2.3, you can install a recent version using: ```bash brew install libyaml rbenv rbenv install 2.7.1 rbenv global 2.7.1 ``` We also recommend installing the `sqlite3` gem beforehand using the brew libraries, to avoid headaches down the road: ```bash gem install sqlite3 -- --with-sqlite3-dir="$(brew --prefix sqlite3)" ``` ### R packages The full list of R packages is automatically installed by MiGA. However, we will install one package here to make sure everything is properly initialized. ```bash Rscript -e "install.packages('ape', repos = 'http://cran.rstudio.com/')" ``` ### MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/master/part2/requirements/mytaxa). # Using apt-get ## Live notebook If you prefer to see code in action, the full installation process with apt-get is available as a [Notebook in Google Collab](https://colab.research.google.com/drive/1xP5l0p1osRwrzw39IyAplRhbVeCdgmsJ). ## Essentials If you have `apt-get`, execute: ```bash # General-purpose software sudo apt-get update sudo apt-get install ruby ruby-sqlite3 r-base sqlite3 python \ libcurl4-openssl-dev openjdk-8-jdk # Bioinformatics software sudo apt-get install ncbi-blast+ hmmer bedtools idba \ prodigal mcl barrnap scythe fastqc diamond-aligner sudo ln -s $(which diamond-aligner) /usr/bin/diamond ``` ## IDBA The index of some distros may still have an old version of IDBA that didn't build the IDBA-UD flavor (you'll need at least v1.1.3-2). Check first if you have it: ```bash idba_ud ``` If you don't, you can get it by adding the Ubuntu Universe source to APT: ```bash echo 'deb http://us.archive.ubuntu.com/ubuntu/ cosmic universe' \ | sudo tee /etc/apt/sources.list.d/ubuntu-universe.list sudo apt-get update sudo apt-get install idba ``` Or install it directly from the DEB package: ```bash wget http://ftp.br.debian.org/debian/pool/main/i/idba/idba_1.1.3-3_amd64.deb sudo apt-get install ./idba_1.1.3-3_amd64.deb ``` ## SolexaQA++ Next, you'll need to install SolexaQA++. If you have a 64-bits Linux: ```bash curl -L -o SolexaQA++_v3.1.7.1.zip \ "https://downloads.sourceforge.net/project/solexaqa/src/SolexaQA%2B%2B_v3.1.7.1.zip" unzip -p SolexaQA++_v3.1.7.1.zip Linux_x64/SolexaQA++ > SolexaQA++ sudo install SolexaQA++ /usr/bin/ ``` If you have 32-bits Linux, you can build SolexaQA++ from source: ```bash sudo apt-get install libboost-dev libboost-filesystem-dev \ libboost-regex-dev libboost-iostreams-dev curl -L -o SolexaQA++_v3.1.7.1.zip \ "https://downloads.sourceforge.net/project/solexaqa/src/SolexaQA%2B%2B_v3.1.7.1.zip" unzip SolexaQA++_v3.1.7.1.zip 'source/*' cd source && make sudo install source/SolexaQA++ /usr/bin/ ``` ## FastANI FastANI is optional, but it may be required to search certain databases. It can be used instead of BLAST ANI to speed up indexing. If you have a 64-bits Linux: ```bash curl -L -o fastani-Linux64-v1.3.zip \ "https://github.com/ParBLiSS/FastANI/releases/download/v1.3/fastani-Linux64-v1.3.zip" unzip fastani-Linux64-v1.3.zip fastANI sudo install fastANI /usr/bin/ ``` If you have a 32-bits Linux, you can build it from source following the [FastANI installation](https://github.com/ParBLiSS/FastANI/blob/master/INSTALL.txt). ## R packages The full list of R packages is automatically installed by MiGA. However, we will install one package here to make sure everything is properly initialized. ```bash Rscript -e "install.packages('ape', repos = 'http://cran.rstudio.com/')" ``` ## MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/master/part2/requirements/mytaxa). # Using Conda You can use [conda](https://conda.io/) to install most of the Software required by MiGA. If you don't have Conda, you can follow the [Installation instructions](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). ## Live notebook If you prefer to see code in action, the full installation process with conda is available as a [Notebook in Google Collab](https://colab.research.google.com/drive/1ybCCPbbZtJ41HC_1yNYed8Yf-q9CDY5a). ## Note for MacOS users > The bioconda fastani recipe currently depends on packages that force downgrading other important packages (including R). This can cause some issues, and we're currently recommending the use of [Homebrew](/master/part2/requirements/brew) whenever possible. If this is not an option for you, a good alternative might be to install FastANI from source instead of using conda. Finally, you could use the instructions below, if the risk of employing old libraries outweights the burden of installation. ## Essentials Now that you have Conda, activate it. If you want to create a separate clean environment (optional) you can simply execute: ```bash conda create -y -n miga python=3.7 conda activate miga ``` It's strongly recommended to activate conda in the `~/.miga_modules`: ```bash # Tell MiGA to activate the proper conda environment: echo 'eval "$(conda shell.bash hook)" && conda activate miga' > ~/.miga_modules ``` Next, install the requirements: ```bash conda install -y -c conda-forge r-base r conda install -y --strict-channel-priority -c conda-forge ruby conda install -y sqlite openjdk conda install -y -c conda-forge -c bioconda -c faircloth-lab \ scythe blast hmmer bedtools prodigal idba mcl barrnap \ fastqc diamond krona fastani ``` ## SolexaQA There is a version of SolexaQA in conda, but the recipe forced downgrading R and is only available for Linux. Therefore, a safer option is to obtain the precompiled binaries directly from the developers [here](https://downloads.sourceforge.net/project/solexaqa/src/SolexaQA++_v3.1.7.1.zip). Unzip that file, and locate the appropriate binary in a folder listed in your `$PATH`. ## R packages The full list of R packages is automatically installed by MiGA. However, we will install one package here to make sure everything is properly initialized. ```bash Rscript -e "install.packages('ape', repos = 'http://cran.rstudio.com/')" ``` ## MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/tree/2987d0a747581e7e0515fac7bf98f7935fe4b24b/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/master/part2/requirements/mytaxa). # Installing from source ## Essentials If you don't have apt-get, Homebrew, or conda, here's the list of requirements and URLs with installation instructions: * **Ruby**: . Required: v2.1+, recommended: v2.3+. * **Python**: . * **R**: . * **SQLite3**: . * **NCBI BLAST+**: ftp\://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST. * **HMMer**: . Required: v3.0+. * **Bedtools**: . * **Prodigal**: . * **IDBA**: . Required flavor: IDBA-UD. * **MCL**: . * **Barrnap**: . * **Scythe**: . Required: v0.991+. * **FastQC**: . * **SolexaQA++**: . Required: v3.1.3+. * **FastANI** (optional): . Required: v1.1+. * **Diamond** (optional): . Required: v0.9.20+. Diamond is optional but strongly recommended. Indexing can be performed much faster with Diamond, searching of some databases depend on it, and it's required by the MyTaxa utils. FastANI is also recommended, since searching some databases depend on it. ## R packages The full list of R packages is automatically installed by MiGA. However, we will install one package here to make sure everything is properly initialized. ```bash Rscript -e "install.packages('ape', repos = 'http://cran.rstudio.com/')" ``` ## MyTaxa utils If you want to activate the [MyTaxa](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part2/part5/workflow.md#mytaxa) and [MyTaxa Scan](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part2/part5/workflow.md#mytaxa-scan) steps, follow the instructions to install the [MyTaxa Utils](/master/part2/requirements/mytaxa). # MyTaxa Utils **The MyTaxa utilities are optional**, but without them the [MyTaxa](/master/part5/workflow#mytaxa) and [MyTaxa scan](/master/part5/workflow#mytaxa-scan) analyses are disabled. Note that MyTaxa requires about **15Gb** of disk available to store the database. We will install the necessary software in `$HOME/apps`. You can change this directory if you prefer: ```bash [[ -d $HOME/apps/bin ]] || mkdir -p $HOME/apps/bin cd $HOME/apps echo 'export PATH=$HOME/apps/bin:$PATH' >> ~/.miga_modules ``` ## MyTaxa ```bash curl -L \ https://github.com/luo-chengwei/MyTaxa/archive/master.tar.gz | tar zx cd MyTaxa-master make python2 utils/download_db.py curl -O \ http://enve-omics.ce.gatech.edu/data/public_mytaxa/AllGenomes.faa.dmnd echo 'export PATH='$PWD':$PATH' >> ~/.miga_modules cd .. ``` ## Krona If you followed the instructions for [conda](/master/part2/requirements/conda), you already have Krona. If you still need to install Krona, simply execute: ```bash curl -L \ https://github.com/marbl/Krona/archive/master.tar.gz | tar zx ( cd Krona-master/KronaTools && ./install.pl --prefix ../.. ) ``` # Installation ## Prerequisites If you simply want to access projects previously processed, you can skip this step. To install the prerequisites follow [system requirements](/master/part2/requirements). ## Getting MiGA To install MiGA itself, you'll just need: ```bash gem install miga-base ``` If necessary, you can use `sudo gem install miga-base` or `gem install --user miga-base` instead. ### Getting MiGA source If you want to get MiGA working from source instead of using the gem, you can use: ```bash # Get the source. Make sure you use --recursive, to clone submodules: git clone --recursive https://github.com/bio-miga/miga.git cd miga # You can use bundle to make sure you have the required gems, # or simply install them manually: bundle # And finally make MiGA available in the PATH. # This is not mandatory but it saves time and effort: echo "export PATH=\"$(pwd)/bin:\$PATH\"" >> ~/.bashrc source ~/.bashrc ``` ## Initializing MiGA To initialize MiGA for data processing, simply execute and follow the instructions: ```bash miga init ``` For additional initialization parameters, use `miga init -h`. ## Pre-loading Software If some of the prerequisites are not already available in your `$PATH`, you'll have to first tell MiGA how to find them. For that, create (or modify) a `bash` configuration file at `~/.miga_modules` loading any necessary packages. For example, if you're in a cluster architecture using `modules`, this is where you should load them. Note that some contents may already be present in this file from previous steps. Do not remove or modify those lines if you're not sure what they do. Here's an example of a configuration file for a single computer: ```bash #!/bin/bash # File: ~/.miga_modules # Enable software installed in this user export PATH="$HOME/apps/bin:$PATH" # Enable MyTaxa export PATH="$HOME/apps/MyTaxa:$PATH" ``` Here is an example of a configuration file for a cluster: ```bash #!/bin/bash # File: ~/.miga_modules shopt -s expand_aliases module purge module load gcc/4.9.0 module load ruby/2.1.5 module load R/3.3.2 module load prodigal/2.6.1 module load bedtools/2.21.0 module load scythe/0.993 module load fastqc/0.11.2 module load idba/1.1.1_kMSS module load hmmer/3.1b1 # Enable MyTaxa export PATH="$HOME/shared3/apps/MyTaxa:$PATH" # Workaround for broken FastQC in the cluster alias fastqc="perl $(which --skip-alias --skip-functions fastqc)" # Workaround for broken KronaTools in the cluster alias ktImportText="perl -I$HOME/shared3/apps/KronaTools-2.5/lib/ \ $HOME/shared3/bin/ktImportText" ``` Once you complete this file, rerun: ```bash miga init ``` # MiGA types ## MiGA types MiGA has predefined settings and analyses. In order to maximize simplicity and reproducibility while ensuring a wide array of use cases, MiGA uses presets called "types". There are different [types of projects](/master/part2/types#project-types) and [types of datasets](/master/part2/types#dataset-types): ### Project types When you create a project, the type of project defines which project-wide analyses are going to be executed (and how). The different types are: #### Mixed **Symbol**: `mixed`. A mixed collection of genomes, metagenomes, and viromes. This is the most basic type of project, with no support for any project-wide analyses. It is intended for projects that are only concerned with datasets preprocessing, *e.g.*, read trimming, assembly, etc. #### Genomes **Symbol**: `genomes`. A collection of genomes. This is the most typical type of project, storing a set of genomes from different taxonomic groups. It can be useful for anything from indexing a reference database, to create a collection of metagenomic bins, and anything in between. #### Clade **Symbol**: `clade`. A collection of closely-related genomes (ANI >= 90%). This is a project for a collection of genomes in the same species (or closely-related species) that require higher resolution but don't require support for a large distance range. #### Metagenomes **Symbol**: `metagenomes`. A collection of metagenomes and/or viromes. This is an experimental type, currently identical to [Mixed](/master/part2/types#mixed). ### Dataset types Once you have a project, the type of the datasets define which analyses are going to be executed for that particular entry (and how). The different types are: #### Genome **Symbol**: `genome`. The genome from an isolate. This is the most typical case, in which you have a genome (complete or draft) from a pure culture (excluding SAGs). #### Single-cell genome **Symbol**: `scgenome`. A Single-cell Amplified Genome (SAG). This is the particular case in which you are dealing with an amplified genome from a single cell. These datasets typically have very uneven coverage (resulting in very incomplete assemblies) and sometimes have contamination from external DNA. #### Population genome **Symbol**: `popgenome`. A population genome (including metagenomic bins). This is the type of dataset that includes sequences from different strains of the same species, such as metagenomic bins or metagenomes of highly enriched (but not pure) cultures. #### Metagenome **Symbol**: `metagenome`. A metagenome (excluding viromes). #### Virome **Symbol**: `virome`. A viral metagenome. ## Query vs reference datasets In addition to the dataset types, some analyses may differ depending on the status of a dataset as query or reference. Reference datasets are those that integrate the database of the project; *i.e.*, those that can be queried by analyses with other datasets like [distances](/master/part5/workflow#distances). In contrast, query datasets are more isolated: they can use data from other datasets (or the project), but don't get to form part of the project database. Defining query datasets is useful when, for example, you have a reference framework for taxonomy (formed by reference datasets) and want to find the best classification for a genome without affecting the project itself. By default, datasets are created as reference datasets. # Input data MiGA datasets can be created from three different points: [Raw reads](/master/part2/input#raw-reads), [Trimmed reads](/master/part2/input#trimmed-reads), and [Assemblies](/master/part2/input#assemblies). The input files can be added through the [CLI](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part3/cli.md) using `miga add` or any of the available [workflows](/master/part6). Files can also be added through the [Web](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part3/web.md) interface. ## Raw reads Raw (unprocessed) sequencing reads. MiGA can handle different sequencing technologies, but it has been optimized for short reads. * **Format**: FastQ, optionally gzipped (with .gz extension) * **Workflow step**: [Raw reads](/master/part5/workflow#raw-reads) ## Trimmed reads Sequencing reads already processed to remove low quality or other artifacts. MiGA can handle different sequencing technologies, but has been optimized for short reads. * **Format**: FastA, optionally gzipped (with .gz extension) * **Workflow step**: [Trimmed FastA](/master/part5/workflow#trimmed-fasta) ## Assemblies Assembled contigs/scaffolds. Ideally, but not necessarily, sequences longer than 1 Kbp. * **Format**: FastA, optionally gzipped (with .gz extension) * **Workflow step**: [Assembly](/master/part5/workflow#assembly) # Distances MiGA estimates distances (or similarities) between datasets using different techniques. Only genome-to-genome comparisons have been implemented, including the genomes of isolates, metagenome-assembled genomes, or single-cell amplified genomes. No metagenome-to-metagenome or genome-to-metagenome distances are currently available in MiGA. ## Hierarchical approach to distances For any given pair of genomes, MiGA attempts a hierarchical approach to identify the most appopriate metric of similarity: **1**. First, the genomes are compared using [hAAI](/master/part2/distances#haai). If this method is skipped, if it fails, or if the value is greater than 90%, MiGA continues to step 2. Otherwise, this value is used to estimate the AAI, both values are recorded, and the comparison ends. **2**. Next, MiGA compares genomes using [AAI](/master/part2/distances#aai). Whenever the AAI is 85% or higher, MiGA continues to step 3. Otherwise the comparison ends. **3**. Finally, MiGA estimates [ANI](/master/part2/distances#ani). ## hAAI **Heuristic Average Amino Acid Identity**. The hAAI is the average amino acid identity between the highly conserved proteins of two genomes, as identified by [essential genes](/master/part5/workflow#essential-genes). It is used to estimate AAI for distant pairs, but it loses resolution between close relatives. This metric is completely bypassed in projects of [type clade](/master/part2/types#clade) as well as projects with the [metadata field](/master/part5/metadata#projects) `haai_p=no`. This field also controls the Software used: `blast+` (default), `blast`, `blat`, or `diamond`. ## AAI **Average Amino Acid Identity**. The AAI is the average amino acid identity between all proteins of two genomes, as identified by [cds](/master/part5/workflow#cds). When running this analysis, the intermediate reciprocal best matches (RBMs) are also stored in projects of [type clade](/master/part2/types#clade). This feature can be turned off to save storage space or forced to be on in any project type using the [metadata field](/master/part5/metadata#projects) `aai_save_rbm=false` or `aai_save_rbm=true`, respectively. The Software used as a search engine can be controlled using the [metadata field](/master/part5/metadata#projects) `haai_p`: `blast+` (default), `blast`, `blat`, or `diamond`. [Workflows](/master/part6) use `blast+` by default, or `diamond` if the flag `--fast` is passed (whenever available). ## ANI **Average Nucleotide Identity**. The ANI is the average nucleotide identity between fragments of two genomes. The Software used as a search engine can be controlled using the [metadata field](/master/part5/metadata#projects) `haai_p`: `blast+` (default), `blast`, `blat`, or `fastani`. [Workflows](/master/part6) use `blast+` by default, or `fastani` if the flag `--fast` is passed (whenever available). ## SQLite3 schema The information on the different similarity metrics above is stored in SQLite3 database files. The general schema for [hAAI](/master/part2/distances#haai) and [AAI](/master/part2/distances#aai) is: ```sql CREATE TABLE aai( seq1 varchar(256), seq2 varchar(256), aai float, sd float, n int, omega int ); CREATE TABLE rbm( seq1 varchar(256), seq2 varchar(256), id1 varchar(256), id2 varchar(256), id float, evalue float, bitscore float ); ``` The `aai` table holds the metric values, including: the name of the genomes compared `seq1` and `seq2`, the AAI or hAAI as percentage `aai`, the standard deviation across proteins when available `sd`, the total number of RBMs `n`, and the smaller number of proteins from the two genomes `omega`. The `rbm` table holds the RBMs for [AAI](/master/part2/distances#aai) (whenever available if stored) and is always empty for [hAAI](/master/part2/distances#haai). The general schema for [ANI](/master/part2/distances#ani) is: ```sql CREATE TABLE ani( seq1 varchar(256), seq2 varchar(256), ani float, sd float, n int, omega int ); CREATE TABLE rbm( seq1 varchar(256), seq2 varchar(256), id1 int, id2 int, id float, evalue float, bitscore float ); CREATE TABLE regions( seq varchar(256), id int, source varchar(256), `start` int, `end` int ); ``` The `ani` table holds the metric values, including: the name of the genomes compared `seq1` and `seq2`, the ANI as percentage `ani`, the standard deviation across fragments when available `sd`, the total number of RBM fragments `n`, and the smaller number of fragments from the two genomes `omega`. The tables `rbm` and `regions` are always empty. # Clustering MiGA generates a clustering-based indexing of databases using ANI distances (for clade projects) or AAI distances (for all other projects). This indexing enables quick searching of databases with query genomes. ## General algorithm The AAI or ANI values are transformed to distances (1 - identity), and the all-vs-all distance matrix is used to generate a *k*-medoids partition (PAM: Partition Around Medoids). *k* is selected to simultaneously optimize for maximum Silhouette average width and minimum Silhouette negative area, between 2 and 100 (or the number of genomes minus 1, whichever is smaller). Once the partitions are defined, the same algorithm is applied recursively to each partition with 8 or more genomes. The resulting clustering-based indexing is used to speed-up query searches. In some cases, it can also be used as *de novo* typing scheme, in particular for ANI distances (clade projects). ## Genomospecies proposals In addition to the above clustering-based indexing, MiGA clusters genomes by Markov Clustering (MCL) using all ANI values above 95% as edges. The result is a collection of discrete genomospecies. The list of genomes per genomospecies is sorted by medoid-ranking, in which the first genome has the minimum average distance to all other genomes in the genomospecies. # Part III: Interfaces The core of MiGA is not as useful if we cannot interact with it, and there is a variety of interfaces for that. At the lowest level, the documented [API](https://github.com/bio-miga/miga/tree/1409226a6553bd36a66d2cf085b98080f3962f2b/manual/part3/api.md) is available for developers to communicate directly with MiGA, either through scripts or through the console. A convenient wrapper including most common tasks (and some not-so-common) is available as the MiGA [CLI](https://github.com/bio-miga/miga/tree/1409226a6553bd36a66d2cf085b98080f3962f2b/manual/part3/cli.md). Finally, a graphical interface allowing result browsing and querying of projects is available as [MiGA Web](https://github.com/bio-miga/miga/tree/1409226a6553bd36a66d2cf085b98080f3962f2b/manual/part3/web.md). # MiGA API # MiGA CLI # MiGA Web # Part IV: Deploying examples Now that your system is ready with MiGA, you may want to deploy some examples for a test run. Here you can find how to get the entire [RefSeq in MiGA](/master/part4/deploy-refseq) and how [build a clade collection](/master/part4/deploy-clade). For more details on running MiGA, see how to [launch daemons](/master/part4/daemons) and how to [setup MiGA in a cluster](/master/part4/cluster). # RefSeq in MiGA In this tutorial, we will create a genomes project including all the representative genomes available in RefSeq using MiGA alone. If you want to explore a more manual approach using `bash`, see the [RefSeq in MiGA using BASH example](https://github.com/bio-miga/miga/tree/46281f59136d6237b8629ae9f5017a63d9bfbe72/manual/part4/deploy-refseq-bash.md). ## 0. Initialize the project ```bash miga new -P RefSeq -t genomes cd RefSeq ``` ## 1. Download publicly available genomes **Re-running and updating**: If the following code fails at any point, for example due to a network interruption, you can simply re-run it, and it will take it from where it failed. ```bash miga ncbi_get -P . --reference -v ``` It is strongly recommended to use an [NCBI API Key](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/) to increase the number of allowed requests. Once you obtain one, you can pass it as an argument: ```bash miga ncbi_get -P . --reference --api-key ABCD123 -v ``` Or you can set it globally as an environmental variable before running `miga`: ```bash export NCBI_API_KEY=ABCD123 ``` ## 2. Launch the daemon Now that your data is ready, you can fire up the daemon to start processing the data. For additional details, see [launching daemons](/master/part4/daemons): ```bash miga daemon start -P . ``` # Build a clade collection In this tutorial, we will create a clade project including all the genomes available for a species in RefSeq as well as any additional genomes you may have using MiGA alone. If you want to explore a more manual approach using `bash`, see the [Build a clade collection using BASH example](https://github.com/bio-miga/miga/tree/0413f0c757dafd1c3f434abbb30b61514c2868d8/manual/part4/deploy-clade-bash.md). We will use *Escherichia coli* as the target species, but you can use any species (or any taxon) you want. ## 0. Initialize the project ```bash miga new -P E_coli -t clade cd E_coli ``` ## 1. Download publicly available genomes There are different stages of completeness defined in the NCBI Genome database, and you may want to include only some cases depending on your analysis. The stages (from higher to lower quality) are: 1. *Complete*: Genomes including all replicons in the organism(s) sequenced. 2. *Chromosome*: Genomes with complete chromosome (but missing other replicons). 3. *Scaffold*: Draft genomes with scaffold status. 4. *Contig*: Draft genomes with contig status. In this example, we'll skip the draft genomes. However, if you want all of them, simply use the `--all` option of `miga ncbi_get`. **Re-running and updating**: If the following code fails at any point, for example due to a network interruption, you can simply re-run it, and it will take it from where it failed. ```bash miga ncbi_get -T "Escherichia coli" -P . --complete --chromosome -v ``` Note that you can change the value of `-T` from `"Escherichia coli"` to any other species name, or even taxa of any rank such as genus or family. It is strongly recommended to use an [NCBI API Key](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/) to increase the number of allowed requests. Once you obtain one, you can pass it as an argument: ```bash miga ncbi_get -T "Escherichia coli" -P . --complete --chromosome --api-key ABCD123 -v ``` Or you can set it globally as an environmental variable before running `miga`: ```bash export NCBI_API_KEY=ABCD123 ``` ## 2. Add your own genomes If you have any unreleased genomes, you can simply add them to the same project to be processed together with those publicly available. You can initialize datasets at different points, see [input data](/master/part2/input). For the purposes of this tutorial, we'll assume that you have raw coupled reads from two sequencing lanes (1 and 2) in Gzipped FastQ files: ```bash # Unzip and/or concatenate input files # this is not necessary if your files are ready to use: gzip -d -c ~/some/file/d1_ACTG_L[12]_R1.fastq.gz > /tmp/sister.1.fastq gzip -d -c ~/some/file/d1_ACTG_L[12]_R2.fastq.gz > /tmp/sister.2.fastq # Register the dataset # change the dataset name MyDataset to whichever name you want: miga add -P . -D MyDataset -t genome -i trimmed_reads_paired /tmp/sister.[12].fastq ``` ## 3. Launch the daemon Now that your data is ready, you can fire up the daemon to start processing the data. For additional details, see [launching daemons](/master/part4/daemons): ```bash miga daemon start -P . ``` # Launching daemons MiGA Daemons are centralized processes in MiGA that define which jobs need to be executed, when, where, and how. All of this can be controlled by customizing the daemons: ## Customizing daemons ### Daemon JSON file When you initialize MiGA (`miga init`), the final step is generating a daemon JSON file stored in `~/.miga_daemon.json`. This file will control the type of daemon (see below) and the default configuration. In addition to this user-wide file, each MiGA project has a `daemon/daemon.json` file. Any variables defined here overwrite the user-wide file. ### `miga daemon` When a daemon is launched using `miga daemon`, several configuration variables can be overwritten using the different command-line flags. If undefined, the variables will be defined by the `daemon/daemon.json` file in the project, or by the daemon JSON file defined by `--json` if passed. Finally, and if a variable is still missing from there, it will default to the values in `~/.miga_daemon.json`. ### Workflows Daemons are also launched by workflows. This process is not manually controlled by the user, but it can also be controlled in a number of ways. First, the flags `--jobs` and `--threads` control the maximum number of jobs (`maxjobs`) and the number of CPUs per job (`ppn`), respectively. Next, a daemon JSON file can be defined using the flag `--daemon`. Finally, variables that are not defined by either method will default to the user-wide configuration in `~/.miga_daemon.json`. ## Daemon types MiGA currently supports three daemon modes, and they each have specific details: ### Local daemons Local daemons, or `bash` daemons, are the simplest mode. You can use this type of daemon if you are launching MiGA in a single computer. The most important consideration here is the **total number of available CPUs**. When you launch a MiGA daemon, the maximum number of jobs spawned will be determined by `--max-jobs` (or `maxjobs` in daemon JSON files, or `--jobs` in workflows), whereas the number of CPUs each job can use is determined by `--ppn` (or `ppn` in daemon JSON files, or `--threads` in workflows). The number of jobs times the number of CPUs per job should never exceed the number of CPUs available. For example, if you have 12 cores in your computer, the default configuration of 6 jobs and 2 CPUs per jobs could use up to 100% of the available CPUs. ### Remote daemons Remote daemons, or `ssh` daemons, are daemons that can communicate with other machines to launch tasks remotely using a login shell through SSH. The first consideration here is: the absolute path to the MiGA project and to the MiGA system **must** be the same in all nodes (launcher and running nodes). If this is not the case, remote executions will fail. Next, in this type of daemon the trickiest part is to configure the remote nodes correctly. This is, to correctly **define the node list file**. The node list file is a raw text file that contains the list of all remote nodes available (hostnames), one per line. It can be set as an environmental variable (*e.g.*, `$MIGA_NODELIST`) that will be read at execution or explicitly as a path to the file. Setting the node list file as a variable is useful when launching MiGA from a scheduler-controlled system. For example, when using Torque, this can be set to `$PBS_NODEFILE`. On the other hand, a fixed file can be a useful alternative if you have a defined set of nodes that are freely available to you (*e.g.*, a dedicated infrastructure). There are several ways to define the node list. First, it can be set using the command-line flag `--node-list` in `miga daemon` (no equivalent is available for workflows). If this is not defined, daemons will look for the value of `nodelist` in the daemon JSON files: first in `daemon/daemon.json` or in the file set by `--daemon` (for workflows) or `--json` (for `miga daemon`), and finally in `~/.miga_daemon.json`. If the node list is set to a variable (starting with `$`), it must be defined as an environmental varible. For example, you could execute `export MIGA_NODELIST=/path/to/file.txt` before running MiGA if the node list is set to `$MIGA_NODELIST`. Finally, **remote daemons ignore `maxjobs`** (or `--max-jobs` or `--jobs`). Instead, the maximum number of jobs is determined by the number of lines in the node list. This allows specifying how many jobs can be launched to each node. For example, if a given node is present three times in the node list, MiGA will run up to three jobs at the same time in that node. Be careful when directly using node files defined by schedulers. Some schedulers will list nodes on the basis of dedicated cores, which may be a problem if the number of threads is more than 1 (`--ppn`, `ppn`, or `--threads`). For example, consider the following PBS (Torque) script which you can use as a template (assuming that `nodelist` is `$MIGA_NODELIST` and `type` is `ssh` in `~/.miga_daemon.list`): ```bash #PBS -q my_nice_queue #PBS -l nodes=12:ppn=3 #PBS -l walltime=12:00:00 # Create the node list file # Change the '3' by the number of CPUs you are using (the value of ppn): awk 'NR % 3 == 0' < "$PBS_NODEFILE" > hosts.txt # Define the nodelist variable: export MIGA_NODELIST=hosts.txt # Run MiGA (change the '3' by the value of ppn): miga quality_wf -T 'Macondimonas' -o Macondimonas -v --threads 3 ``` ### Scheduler daemons Finally, daemons can also communicate directly with schedulers. Currently, MiGA supports: `qsub` (Torque), `msub` (Moab), and `slurm` (SLURM). If your HPCC runs a different scheduler and you'd like to see it added to this list, [please contact us](http://support.microbial-genomes.org). This is the preferred method when the flow of tasks is variable, because it can easily adapt to different task loads. For example, when maintaining a website that automatically processes incoming jobs (like [MiGA Online](http://microbial-genomes.org)), the daemons are low-resource tasks that launch as many or as few jobs as necessary. Importantly, this method tends to require large numbers of jobs submitted to the schedulers, and often wastes some requested resources since the resource use configuration does not adapt to each task, both of which may be against your HPCC policies. Therefore, when processing a defined task in HPCC, it is preferred to use either of the two modes above instead. For example, if you want to evaluate the quality of four genomes, a good option would be to launch a [local daemon](/master/part4/daemons#local-daemons) with a maximum of 4 jobs (`--max-jobs 4`). On the other hand, if you have hundreds of genomes to process, you could launch instead a [remote daemon](/master/part4/daemons#remote-daemons). # Setting up MiGA in a cluster MiGA is developed to run on clusters with a TORQUE scheduler. The administrator should install the [dependencies](/master/part2/requirements) and [MiGA-base](/master/part2/installation) on a networked file system. When you first initialize MiGA, the program will prompt the user for the type of daemon. For a cluster with TORQUE, the user can choose a qsub daemon. During the initialization, MiGA will also ask for the number of jobs to submit and the number of CPUs per job. After you have setup a MiGA project, you can create a job script to start daemon. The daemon requires to be alive until all the jobs for the have finished, so you should add a long-enough walltime. If you didn't give enough walltime for your job, the MiGA daemon will get killed by the system. In this case, you can restart the daemon again to finish processing the project. Example job script: ```bash #PBS -N MiGADaemon #PBS -l mem=2gb #PBS -l nodes=1:ppn2 #PBS -l walltime=48:00:00 cd $HOME/Path-to-project/ miga daemon start -P . ``` # Part V: Additional details # Advanced configuration MiGA allows flexible execution using a number of techniques, mainly setting [project metadata flags](/master/part5/metadata#project-flags), [dataset metadata flags](/master/part5/metadata#dataset-flags), and through the [daemon configuration](/master/part4/daemons). These are some examples of advanced behaviors that can be configured in MiGA ## Controlling many daemons at once The [MiGA CLI](https://github.com/bio-miga/miga/tree/f8e53391029bd74bab6003a79a53217b5d208ea7/manual/part3/miga-cli.md) includes support for mass-controlled daemons when several MiGA projects are in the same folder using the `miga lair` command. The MiGA Lair will find all the MiGA projects in a folder (or subfolders) and control all the corresponding daemons together: ```bash # Check the current status of all the daemons miga lair list -p /path/to/folder # Terminate all the daemons at once miga lair terminate -p /path/to/folder # Launch and daemonize a process to keep all daemons running miga lair start -p /path/to/folder # See more options miga lair -h ``` ## Sending an email when the project is ready If your project runs for a long time and you want to be notified by email when it's ready, you can use [project hooks](/master/part5/metadata#project-hooks): ```bash # First cd to the project folder cd /path/to/project # And open the MiGA Console miga c ``` In the MiGA Console: ```ruby # Set the hook "on_preprocessing_ready" to # execute a shell command ("run_cmd") consisting of # sending an email with the project path ("{{project}}") MiGA::Project.load('.').tap do |p| p.metadata[:on_preprocessing_ready] = [ ['run_cmd', 'echo \'Project ready: {{project}}\' | sendmail me@example.com'] ] end.save quit ``` Make sure to change `/path/to/project` to the project path and `me@example.com` to the email where you want to receive the notification. Also, note that depending on your `sendmail` configuration you might get that email in the spam folder (or flat-out rejected), so make sure to test `sendmail` first. # MiGA workflow ## MiGA Workflow This is the general overview of the MiGA workflow: ![The MiGA Workflow](/files/-LBP6DvubxUjSQkvMTSs) For each step, performed analyses may include the use of [external Software](/master/part5/external), and produce one or more result files (indexed in a hash). In most steps, different utilities from the [Enveomics Collection](/master/part5/external#enveomics-collection) are used in addition to the Software detailed below. Some files are mandatory to continue with the analysis (marked with *req*), some can be gzipped during or after the analysis (marked with *gz*), and some are directories (marked with *dir*). ### List of individual steps * [Dataset Results](/master/part5/workflow#dataset-results) * [Raw Reads](/master/part5/workflow#raw-reads) * [Trimmed Reads](/master/part5/workflow#trimmed-reads) * [Read Quality](/master/part5/workflow#read-quality) * [Trimmed FastA](/master/part5/workflow#trimmed-fasta) * [Assembly](/master/part5/workflow#assembly) * [CDS](/master/part5/workflow#cds) * [Essential Genes](/master/part5/workflow#essential-genes) * [SSU](/master/part5/workflow#ssu) * [MyTaxa](/master/part5/workflow#mytaxa) * [MyTaxa Scan](/master/part5/workflow#mytaxa-scan) * [Distances](/master/part5/workflow#distances) * [Taxonomy](/master/part5/workflow#taxonomy) * [Stats](/master/part5/workflow#stats) * [Project Results](/master/part5/workflow#project-results) * [hAAI Distances](/master/part5/workflow#haai-distances) * [AAI Distances](/master/part5/workflow#aai-distances) * [ANI Distances](/master/part5/workflow#ani-distances) * [Clade Finding](/master/part5/workflow#clade-finding) * [Subclades](/master/part5/workflow#subclades) * [OGS](/master/part5/workflow#ogs) * [Project Stats](/master/part5/workflow#project-stats) ## Dataset Results ### Raw Reads This step is never actually performed by MiGA, instead it serves as the entry point for raw reads input. Supported file keys: * **For single reads only** * `single` (*req*, *gz*): FastQ file containing the raw reads * **For paired-end reads only** * `pair1` (*req*, *gz*): FastQ file containing the raw forward reads * `pair2` (*req*, *gz*): FastQ file containing the raw reverse reads Statistics: * **For single reads only** * `reads`: Total number of reads * `length_average`: Average read length (in bp) * `length_standard_deviation`: Standard deviation of read length (in bp) * `g_c_content`: G+C content of all reads (in %) * `x_content`: Undetermined bases content of all reads (in %) * `a_t_skew`: A-T sequence skew across all reads (in %) * `g_c_skew`: G-C sequence skew across all reads (in %) * **For paired-end reads only** * `read_pairs`: Total number of read pairs * `forward_length_average`: Average forward read length (in bp) * `forward_length_standard_deviation`: Standard deviation of forward read length (in bp) * `forward_g_c_content`: G+C content of forward reads (in %) * `forward_x_content`: Undetermined bases content of forward reads (in %) * `forward_a_t_skew`: A-T sequence skew across forward reads (in %) * `forward_g_c_skew`: G-C sequence skew across forward reads (in %) * `reverse_length_average`, `reverse_length_standard_deviation`, * `reverse_g_c_content`: Same as above, for reverse reads * `reverse_x_content`: Undetermined bases content of reverse reads (in %) * `reverse_a_t_skew`: A-T sequence skew across reverse reads (in %) * `reverse_g_c_skew`: G-C sequence skew across reverse reads (in %) MiGA symbol: `raw_reads`. ### Trimmed Reads This is part of *Trimming & read quality* in the above diagram. In this step, MiGA trims reads by Phred quality score 20 (Q20) and minimum length of 50bp using [SolexaQA++](/master/part5/external#solexaqa), and clips potential adapter contamination using [Scythe](/master/part5/external#scythe) (reapplying the length filter). If the reads are paired, only pairs passing the filters are used. Supported file keys: * **For single reads only** * `single` (*req*, *gz*): FastQ file containing trimmed/clipped reads * **For paired-end reads only** * `pair1` (*req*, *gz*): FastQ file containing trimmed/clipped forward reads * `pair2` (*req*, *gz*): FastQ file containing trimmed/clipped reverse reads * `single` (*req*, *gz*): FastQ file containing trimmed/clipped reads with only one sister passing quality control * **For either type** * `trimming_summary`: Raw text file containing a summary of the trimmed sequences MiGA symbol: `trimmed_reads`. ### Read Quality This is a quality-control step included as part of *Trimming & read quality* in the diagram above. In this step, MiGA generates quality reports of the trimmed/clipped reads using [SolexaQA++](/master/part5/external#solexaqa) and [FastQC](/master/part5/external#fastqc). Supported file keys: * `solexaqa` (*dir*): Folder containing the SolexaQA++ quality-control summaries * `fastqc` (*dir*): Folder containing the FastQC quality-control analyses MiGA symbol: `read_quality`. ### Trimmed FastA This is the final step included in *Trimming & read quality* in the diagram above, in which MiGA generates FastA files with the trimmed/clipped reads. Supported file keys: * `coupled` (*req* for coupled reads, unless `pair1` and `pair2` exist): Interposed FastA file containing quality-checked paired reads. If this file doesn't exist, it is automatically generated from `pair1` and `pair2` * `single` (*req* for single reads, *gz* for coupled reads): FastA file with quality-checked single-end reads * `pair1` (*gz*): FastA file containing forward sisters of quality-checked paired-end reads * `pair2` (*gz*): FastA file containing reverse sisters of quality-checked paired-end reads Statistics: * `reads`: Total number of reads * `length_average`: Average read length (in bp) * `length_standard_deviation`: Standard deviation of read length (in bp) * `g_c_content`: G+C content of all reads (in %) * `x_content`: Undetermined bases content of all reads (in %) * `a_t_skew`: A-T sequence skew across all reads (in %) * `g_c_skew`: G-C sequence skew across all reads (in %) MiGA symbol: `trimmed_fasta`. ### Assembly In this step MiGA assembles trimmed FastA reads using [IDBA-UD](/master/part5/external#idba-ud). Supported file keys: * `largecontigs` (*req*): FastA file containing large contigs or scaffolds (>1Kbp) * `allcontigs`: FastA file containing all contigs or scaffolds (including large) * `assembly_data` (*dir*): Folder containing some intermediate files generated during the assembly Statistics: * `contigs`: Total number of (large) contigs * `n50`: N50 of (large) contigs (in bp) * `total_length`: Total length of large contigs (in bp) * `longest_sequence`: Length of the longest contig (in bp) * `n_content`: Undetermined bases content of large contigs (in %) * `g_c_content`: G+C content of large contigs (in %) * `x_content`: Undetermined bases content of large contigs (in %) * `a_t_skew`: A-T sequence skew across large contigs (in %) * `g_c_skew`: G-C sequence skew across large contigs (in %) MiGA symbol: `assembly`. ### CDS This step corresponds to *Gene prediction* in the diagram above. MiGA predicts coding sequences (putative genes and proteins) using [Prodigal](/master/part5/external#prodigal). Supported file keys: * `proteins` (*req*): FastA file containing translated protein sequences * `genes`: FastA file containing putative gene sequences * `gff3` (*gz*): GFF v3 file containing the coordinates of coding sequences This file is not required, but [MyTaxa](/master/part5/workflow#mytaxa) depends on it (or `gff2` or `tab`, whichever is available) * `gff2` (*gz*): GFF v2 file containing the coordinates of coding sequences This file is not produced by MiGA, but it's supported for backwards compatibility with earlier versions using MetaGeneMark * `tab` (*gz*): Tabular-delimited file containing the columns: gene ID, gene length, and contig ID. This file is not produced by MiGA, but it's supported to allow [MyTaxa](/master/part5/workflow#mytaxa) to run when more detailed information about the gene prediction is missing Statistics: * `predicted_proteins`: Total number of predicted proteins * `average_length`: Average length of predicted proteins (in aa) * `coding_density`: Coding density of the genome (in %) * `codon_table`: Optimal coding table (4 or 11) MiGA symbol: `cds`. ### Essential Genes In this step, MiGA uses `HMM.essential.rb` from the [Enveomics Collection](/master/part5/external#enveomics-collection) to identify a set of genes typically present in single-copy in Bacterial and Archaeal genomes. In this step, protein translations of those *essential* genes are extracted for other analyses in MiGA (*e.g.*, hAAI in [distances](/master/part5/workflow#distances)) or outside (*e.g.*, phylogeny or MLSA for [diversity analyses](/master/part1#diversity)). In addition, this step generates a report that can be used for quality control including estimations of completeness and contamination (for genomes) and median number of copies of single-copy genes (for metagenomes and viromes). Supported file keys: * `ess_genes` (*req*): FastA file containing all extracted protein translations from *essential* genes (.faa) or archived collection (proteins.tar.gz) * `collection` (*req*): Folder containing individual FastA files with protein translations from *essential* genes * `report` (*req*): Raw text report including derived statistics, as well as *essential* genes missing or detected in multiple copies (for genomes) or copy counts (for metagenomes and viromes) * `alignments`: Generated for all genomes (non-multi types). It contains the best matching protein for each detected model aligned to the model * `bac_report`: If present, this is the original report, and it indicates that a corrected report has been generated to accomodate particular features of the dataset Statistics: * **For metagenomes and viromes** * `mean_copies`: Average copy number across essential genes * `median_copies`: Median copy number across essential genes * **For genomes** * `completeness`: Estimated completeness of the genome, based on presence of essential genes (in %) * `contamination`: Estimated contamination of the genome, based on copy number of essential genes (in %) * `quality`: Completeness - 5 x Contamination MiGA symbol: `essential_genes`. ### SSU In this step, MiGA detects small-subunit rRNA genes (16S) using [Barrnap](/master/part5/external#barrnap) and extracts their sequences using [Bedtools](/master/part5/external#bedtools). Supported file keys: * `longest_ssu_gene` (*req*): FastA file containing the longest detected SSU gene * `gff` (*gz*): GFF v3 file containing the location of detected SSU genes * `all_ssu_genes` (*gz*): FastA file containing all the detected SSU genes Statistics: * `ssu`: Total number of detected SSU fragments * `complete_ssu`: Number of complete SSU loci * `max_length`: Length of the longest detected SSU fragment MiGA symbol: `ssu`. ### MyTaxa This step is only supported for metagenomes and viromes, and it requires the (optional) MyTaxa [requirements installed](/master/part2/requirements). In this step, the most likely taxonomic classification of each contig is identified using [MyTaxa](/master/part5/external#mytaxa), and a report is generated using [Krona](/master/part5/external#krona). Supported file keys: * `mytaxa` (*req*): Output generated by MyTaxa * `blast` (*gz*): BLAST against the reference genomes database * `mytaxain` (*gz*): Re-formatted BLAST used as input for MyTaxa * `nomytaxa`: If it exists, MiGA assumes no support for MyTaxa modules, and none of the above files are required * `species`: Profile of species composition (in permil) as raw tab-delimited text * `genus`: Profile of genus composition (in permil) as raw tab-delimited text * `phylum`: Profile of phylum composition (in permil) as raw tab-delimited text * `innominate`: List of innominate taxa (groups without a name but containing lower-rank classifications) as raw text * `kronain`: Raw-text list of taxa used as input for Krona * `krona`: HTML output produced by Krona MiGA symbol: `mytaxa`. ### MyTaxa Scan This step is only supported for genomes (dataset types genome, popgenome, and scgenome), and it requires the (optional) MyTaxa [requirements installed](/master/part2/requirements). In this step, the genomes are scanned in windows of ten genes. For each window, the taxonomic distribution is determined using [MyTaxa](/master/part5/external#mytaxa) and compared against the distribution for the entire genome. This is a quality-control step for manual curation. Supported file keys: * `mytaxa` (*req*): MyTaxa output * `report` (*req*): PDF file containing the graphic report * `regions_archive` (*gz*): Archived folder containing FastA files with the sequences of the genes in regions identified as abnormal * `nomytaxa`: If it exists, MiGA assumes no support for MyTaxa modules, and none of the above files are required Deprecated file keys: * `wintax`: Taxonomic distribution of each window * `blast` (*gz*): BLAST against the reference genomes database * `mytaxain` (*gz*): Re-formatted BLAST used as input for MyTaxa * `regions` (*dir*): Folder containing FastA files with the sequences of the genes in regions identified as abnormal * `gene_ids`: List of genes per window * `region_ids`: List of regions identified as abnormal MiGA symbol: `mytaxa_scan`. ### Distances This step is only supported for genomes ([dataset types](/master/part2/types#dataset-types) genome, popgenome, and scgenome). In this step, each dataset is compared against all other datasets in the project. If the dataset is a [reference dataset](/master/part2/types#query-vs-reference-datasets), it is compared against all other reference datasets in the project. If it's a query dataset, it is compared iteratively against medoids. For more details on the strategy used in this step, see the manual [section on distances](/master/part2/distances). Supported file keys: * **For reference datasets** * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * **For query datasets** * `aai_medoids` (*req* except for clades projects): Best hits among medoids at different hierarchical levels in the AAI indexing * `ani_medoids` (*req* for clades projects): Best hits among medoids at different hierarchical levels in the ANI indexing * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * `ref_tree`: Newick file with the Bio-NJ tree including queried medoids and the query dataset * `ref_tree_pdf`: PDF rendering of `ref_tree` * `intax`: Raw text result of the taxonomy test against the reference genome MiGA symbol: `distances`. ### Taxonomy This step is only supported for genomes ([dataset types](/master/part2/types#dataset-types) genome, popgenome, and scgenome) that are [reference datasets](/master/part2/types#query-vs-reference-datasets), in projects with a set reference project (`:ref_project` in metadata). In this step, MiGA compares the genome against a reference project using the query search method, and imports the resulting taxonomy with p-value below 0.05 (or whichever value is set as `:tax_pvalue` in metadata). Supported file keys: * `intax`: Raw text result of the taxonomy test against the reference genome * `aai_medoids` (*req* except for reference clades projects): Best hits among medoids at different hierarchical levels in the AAI indexing * `ani_medoids` (*req* for reference clades projects): Best hits among medoids at different hierarchical levels in the ANI indexing * `haai_db` (*req*): SQLite3 database containing hAAI values * `aai_db`: SQLite3 database containing AAI values * `ani_db`: SQLite3 database containing ANI values * `ref_tree`: Newick file with the Bio-NJ tree including queried medoids and the query dataset * `ref_tree_pdf`: PDF rendering of `ref_tree` Statistics: * `closest_relative`: Name of the reference dataset with highest AAI * `aai`: AAI to the closest relative * `domain_pvalue`, `phylum_pvalue`, `class_pvalue`, `order_pvalue`, `family_pvalue`, `genus_pvalue`, `species_pvalue`, `subspecies_pvalue`: Empirical p-values for classification at each rank with respect to the closest relative, based on the observed AAI MiGA symbol: `taxonomy` ### Stats In this step, MiGA traces back all the results of the dataset and estimates summary statistics. In addition, it cleans any stored values in the distances database including datasets no longer registered in the project. No supported file keys. MiGA symbol: `stats`. ## Project Results Once all datasets have been pre-processed (*i.e.*, once all the results above are available for all reference datasets), MiGA executes the following project-wide steps: ### hAAI Distances Consolidation of hAAI distances. Supported file keys: * `rdata` (*req*): Pairwise values in a `data.frame` for `R` * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of hAAI values as raw tab-delimited file MiGA symbol: `haai_distances`. ### AAI Distances Consolidation of AAI distances. Supported file keys: * `rdata` (*req*): Pairwise values in a `data.frame` for `R` * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of AAI values as raw tab-delimited file MiGA symbol: `aai_distances`. ### ANI Distances Consolidation of ANI distances. Supported file keys: * `rdata` (*req*): Pairwise values in a `data.frame` for `R` * `matrix` (*req*): Pairwise values in a raw tab-delimited file * `log` (*req*): List of datasets included in the matrix * `hist`: Histogram of ANI values as raw tab-delimited file MiGA symbol: `ani_distances`. ### Clade Finding This step is only supported for project types [genomes](/master/part2/types#genomes) and [clade](/master/part2/types#clade). In this step, MiGA attempts to identify clades at species level or above using a combination of ANI and AAI values. MiGA generates [AAI clades](/master/part2/clustering#aai-clades) in this step for [genomes projects](/master/part2/types#genomes). Clades proposed at AAI > 90% and ANI > 95% are formed using the Markov Clustering algorithm implemented in [MCL](/master/part5/external#mcl). Most distance manipulation and tree estimation and manipulation utilities use the R packages [Ape](/master/part5/external#ape) and [Vegan](/master/part5/external#vegan). Supported file keys: * `report` (*req* for `genomes`): PDF file including a graphic report for the clustering * `class_table` (*req* for `genomes`): Tab-delimited file containing the classification of all datasets in AAI clusters * `class_tree` (*req* for `genomes`): Newick file containing the classification of all datasets in AAI clusters as a dendrogram * `classif` (*req* for `genomes`): Tab-delimited file containing the highest-level classification of each dataset, the medoid of the cluster, and the AAI against the corresponding medoid * `medoids` (*req* for `genomes`): List of medoids per cluster * `aai_tree`: Bio-NJ tree based on AAI distances in Newick format * `proposal` (*req*): Proposed species-level clades in the project, based on `clades_ani95`. One line per proposed clade, with tab-delimited dataset names. Only clades with 5 or more members are included * `clades_aai90`: Clades formed at AAI > 90%. One clade per line, with comma-delimited dataset names * `clades_ani95`: Clades formed at ANI > 95%. One clade per line, with comma-delimited dataset names * `medoids_ani95`: List of `clades_ani95` datasets with the smallest ANI distance to all members of its own ANI95 clade. The list is in the same order MiGA symbol: `clade_finding`. ### Subclades This step is only supported for project type [clade](/master/part2/types#clade). In this step, MiGA attempts to identify clades below species level using ANI values. MiGA generates [ANI clades](/master/part2/clustering#ani-clades) in this step. Most distance manipulation and tree estimation and manipulation utilities use the R packages [Ape](/master/part5/external#ape) and [Vegan](/master/part5/external#vegan). Supported file keys: * `report` (*req*): PDF file including a graphic report for the clustering * `class_table` (*req*): Tab-delimited file containing the classification of all datasets in ANI clusters * `class_tree` (*req*): Newick file containing the classification of all datasets in ANI clusters as a dendrogram * `classif` (*req*): Tab-delimited file containing the highest-level classification of each dataset, the medoid of the cluster, and the ANI against the corresponding medoid * `medoids` (*req*): List of medoids per cluster * `ani_tree`: Bio-NJ tree based on AAI distances in Newick format MiGA symbol: `subclades`. ### OGS This step is only supported for project type [clade](/master/part2/types#clade). In this step, MiGA generates groups of orthology using reciprocal best matches between all pairs of datasets in the project. Groups are generated using [MCL](/master/part5/external#mcl) with pairs weighted by bit score. Once computed, MiGA uses the matrix of OGS to estimate summary and rarefied statistics. Supported file keys: * `ogs` (*req*): Matrix of orthology groups, as tab-delimited raw file * `stats` (*req*): Summary statistics in JSON format * `abc` (*gz*): When available, it includes all the individual RBM files in ABC format. This file is typically produced as intermediate result and removed before finishing, but can be maintained using `miga update -P . -m clean_ogs=false` in the project folder using the [CLI](https://github.com/bio-miga/miga/tree/e12cfe0225f651f8fc6e229e4b530a6fd6962c34/manual/part3/cli.md) * `core_pan`: Summary statistics of rarefied core-genome/pangenome sizes in tab-delimited format * `core_pan_plot`: Plot of rarefied core-genome/pangenome sizes in PDF MiGA symbol: `ogs`. ### Project Stats In this step, MiGA traces back all the results of the project and estimates summary statistics. Supported file keys: * `taxonomy_index` (*req*): Index of datasets per taxonomy in JSON format * `metadata_index` (*req*): Searchable index of datasets metadata as SQLite3 database MiGA symbol: `project_stats`. # Metadata ## All objects All metadata objects support the following fields: | Field | Supported values | Description | | --------: | :---------------------------------------: | ------------------- | | created\* | [Date](/master/part1/glossary#miga-dates) | Date of creation | | updated\* | [Date](/master/part1/glossary#miga-dates) | Date of last update | > **\*** Mandatory ## Projects The following metadata fields are recognized by different interfaces for **Projects**: ### Project Features Metadata with additional information and features about the project: | Field | Supported values | Description | | ----------: | :---------------------------------------: | --------------------- | | comments | String | Free-form comments | | description | String | Free-form description | | name\* | [Name](/master/part1/glossary#miga-names) | Name‡ | > **\*** Mandatory ### Project System Metadata Metadata entries automatically set by MiGA: | Field | Supported values | Description | | ---------: | :--------------: | ----------------------------------------- | | datasets\* | Array of String | List of datasets in the project | | type\* | String | [Type](/master/part2/types#project-types) | > **\*** Mandatory > > **‡** By default the base name of the project path ### Project Flags Metadata entries that trigger specific behaviors in MiGA: | Field | Supported values | Description | | -------------: | :--------------: | -------------------------------------------- | | ref\_project | Path | Project with reference taxonomy | | db\_proj\_dir | Path | Directory containing database projects | | tax\_pvalue | Float \[0,1] | Max p-value to transfer taxonomy (def: 0.05) | | aai\_p | String | Value of aai.rb -p° on AAI (def: blast+) | | haai\_p | String | Value of aai.rb -p° on hAAI (def: blast+) | | ani\_p | String | Value of ani.rb -p° on ANI (def: blast+) | | max\_try | Integer | Max number of task attempts (def: 10) | | aai\_save\_rbm | Boolean | Should RBMs be saved for OGS analysis? | | ogs\_identity | Float \[0,100] | Min RBM identity for OGS (def: 80) | | clean\_ogs | Boolean | If false, keeps ABC (clades only) | | run\_clades | Boolean | Should clades be estimated from distances? | | gsp\_ani | Float \[0,100] | ANI limit to propose gsp clades (def: 90) | | gsp\_aai | Float \[0,100] | AAI limit to propose gsp clades (def: 95) | | gsp\_metric | String | Metric to propose clades: `ani` (def), `aai` | | ess\_coll | String | Collection of essential genes to use+ | | min\_qual | Float (or 'no') | Min. genome quality (or no filter; def: 25) | > **°** By default: `blast+`. Other supported values: `blast`, `blat`, `diamond` (except for ANI), and `fastani` (only for ANI), `no` (only for hAAI). If using `diamond` and/or `fastani`, the corresponding software must be installed. **Important**: These defaults will change in v1.0 to: `blast+` for hAAI, `diamond` for AAI, and `fastani` for ANI. > > **+** One of: `dupont_2012` (default), or `lee_2019` ### Project Hooks Additionally, hooks can be defined for projects as arrays of arrays containing the action name and the arguments (if any). For example, one can define: ``` on_processing_ready: [ ['run_cmd', 'date > {{project}}/ALL_DONE.txt'], ['run_cmd', 'sendmail ...'] ] ``` or ``` on_add_dataset: [ ['run_cmd', 'echo {{object}} > {{project}}/LATEST_DATASET.txt'] ] ``` Supported events: * `on_create()`: When created * `on_load()`: When loaded * `on_save()`: When saved * `on_add_dataset(object)`: When a dataset is added, with name `object` * `on_unlink_dataset(object)`: When dataset with name `object` is unlinked * `on_result_ready(object)`: When any result is ready, with key `object` * `on_result_ready_{result}()`: When `result` is ready * `on_processing_ready()`: When processing is complete Supported hooks: * `run_lambda(lambda, args...)` * `run_cmd(cmd)` ## Datasets The following metadata fields are recognized by different interfaces for **Datasets**: ### Dataset Features Metadata with additional information and features about the dataset: | Field | Supported values | Description | | ----------------: | :--------------: | --------------------------------------- | | tax | MiGA::Taxonomy | Taxonomy of the dataset | | quality | String | Description of genome quality | | dprotologue | String | Taxonumber in the Digital Protologue DB | | ncbi\_tax\_id | String | Linking ID(s)‡ for NCBI Taxonomy | | ncbi\_nuccore | String | Linking ID(s)‡ for NCBI Nucleotide | | ncbi\_asm | String | Linking ID(s)‡ for NCBI Assembly | | ebi\_embl | String | Linking ID(s)‡ for EBI EMBL | | ebi\_ena | String | Linking ID(s)‡ for EBI ENA | | web\_assembly | String | URL to download assembly | | web\_assembly\_gz | String | URL to download gzipped assembly | | see\_also | String | Link(s)‡ in the format text:url | | is\_type | Boolean | If it is type material | | is\_ref\_type | Boolean | If it is reference material° | | type\_rel | String | Relationship to type material | | suspect | Array(String) | Flags indicating a suspect dataset | > **‡** Multiple values can be provided separated by commas or colons > > **°** This is not a valid type, but it represents the closest available dataset to material that is unavailable and unlikely to ever become available. See also [Federhen, 2015, NAR](https://doi.org/10.1093/nar/gku1127) ### Dataset System Metadata Metadata entries automatically set by MiGA: | Field | Supported values | Description | | -------------: | :--------------: | -------------------------------------------- | | type\* | String | [Type](/master/part2/types#dataset-types) | | ref | Boolean | [Reference](/master/part2/types#reference) | | inactive | Boolean | If auto-processing should stop | | metadata\_only | Boolean | Dataset with metadata but without input data | | status | String | Proc. status: complete, incomplete, inactive | | \_step | String | For internal control of processing | | \_try\_`step` | Integer | For internal control of processing | | ~~user~~ | String | Deprecated | > **\*** Mandatory ### Dataset Flags Metadata entries that trigger specific behaviors in MiGA: | Field | Supported values | Description | | ----------: | :--------------: | -------------------------------------- | | run\_`step` | Boolean | Forces running or not `step` | | db\_project | Path | Project to use as database | | dist\_req | Array of String | Run distances against these datasets\* | > **\*** When searching best-matching datasets, include these datasets even if they are not visited using the medoid tree ### Dataset Hooks Additionally, hooks can be defined for datasets as arrays of arrays containing the action name and the arguments. See above ([project hooks](/master/part5/metadata#project-hooks)) for examples. Supported events: * `on_load()`: When loaded * `on_save()`: When saved * `on_remove()`: When removed * `on_inactivate()`: When inactivated * `on_activate()`: When activated * `on_result_ready(object)`: When any result is ready, with key `object` * `on_result_ready_{result}()`: When `result` is ready * `on_preprocessing_ready()`: When preprocessing is complete Supported hooks: * `run_lambda(lambda, args...)` * `clear_run_counts()` * `run_cmd(cmd)` # External Software ## Enveomics Collection Rodriguez-R LM, and Konstantinidis KT (2016). The enveomics collection: a toolbox for specialized analyses of microbial genomes and metagenomes. [PeerJ Preprints 4:e1900v1](https://doi.org/10.7287/peerj.preprints.1900v1). ## NCBI BLAST+ Camacho C, Coulouris G, Avagyan V, Ma N, Papadopoulos J, Bealer K, and Madden TL (2009). BLAST+: architecture and applications. [BMC Bioinformatics 10:421](https://doi.org/10.1186/1471-2105-10-421). ## HMMer 3 Mistry J, Finn RD, Eddy SR, Bateman A, and Punta M (2013). Challenges in Homology Search: HMMER3 and Convergent Evolution of Coiled-Coil Regions. [Nucleic Acids Res 41:e121](https://doi.org/10.1093/nar/gkt263). ## SolexaQA++ Cox MP, Peterson DA, and Biggs PJ (2010). SolexaQA: At-a-glance quality assessment of Illumina second-generation sequencing data. [BMC Bioinformatics 11:485](https://doi.org/10.1186/1471-2105-11-485). ## Scythe [GitHub: vsbuffalo/scythe](https://github.com/vsbuffalo/scythe), by Vince Buffalo. ## FastQC [FastQC site](https://www.bioinformatics.babraham.ac.uk/projects/fastqc), by Simon Andrews. ## IDBA-UD Peng Y. Leung HC, Yiu SM, and Chin FY (2012). IDBA-UD: a de novo assembler for single-cell and metagenomic sequencing data with highly uneven depth. [Bioinformatics, 28:1420-1428](https://doi.org/10.1093/bioinformatics/bts174). ## Prodigal Hyatt D, Chen GL, Locascio PF, Land ML, Larimer FW, Hauser LJ (2010). Prodigal: prokaryotic gene recognition and translation initiation site identification. [BMC Bioinformatics, 11(1):119](https://doi.org/10.1186/1471-2105-11-119). ## Barrnap [GitHub: tseemann/barrnap](https://github.com/tseemann/barrnap), by Torsten Seemann. ## Bedtools [Bedtools site](http://bedtools.readthedocs.io/en/latest), by the Quinlan Laboratory. ## MyTaxa Luo C, Rodriguez-R LM, and Konstantinidis KT (2014). MyTaxa: an advanced taxonomic classifier for genomic and metagenomic sequences. [Nucleic Acids Res, 42(8):e73](https://doi.org/10.1093/nar/gku169). ## Krona Ondov BD, Bergman NH, and Phillippy AM (2011). Interactive metagenomic visualization in a Web browser. [BMC Bioinformatics 12:385](https://doi.org/10.1186/1471-2105-12-385). ## MCL Enright AJ, Van Dongen S, and Ouzounis CA (2002). An efficient algorithm for large-scale detection of protein families. [Nucleic Acids Res 30(7):1575-1584](https://doi.org/10.1093/nar/30.7.1575). ## Ape Paradis E, Claude J, and Strimmer K (2004). APE: Analyses of Phylogenetics and Evolution in R language. [Bioinformatics 20(2):289-290](https://doi.org/10.1093/bioinformatics/btg412). ## Vegan Oksanen J, Blanchet FG, Friendly M, Kindt R, Legendre P, McGlinn D, Minchin PR, O'Hara RB, Simpson GL, Solymos P, Stevens MHM, Szoecs E, and Wagner H (2017). Vegan: Community Ecology Package. [R package](https://CRAN.R-project.org/package=vegan). ## Diamond Buchfink B, XieC, and Huson DH (2015). Fast and Sensitive Protein Alignment using DIAMOND. [Nature Methods 12:59–60](https://doi.org/10.1093/10.1038/nmeth.3176). ## FastANI Jain C, Rodriguez-R LM, Phillippy AM, Konstantinidis KT, and Aluru S (2018). High throughput ANI analysis of 90K prokaryotic genomes reveals clear species boundaries. [Nature Comm 9:5114](https://doi.org/10.1038/s41467-018-07641-9). # Part VI: Workflows CLI Workflows automate common tasks in MiGA. Currently implemented workflows include: 1. [Quality](/master/part6/quality_wf) 2. [Dereplicate](/master/part6/derep_wf) 3. [Classify](/master/part6/classify_wf) 4. [Preprocess](/master/part6/preproc_wf) 5. [Index](/master/part6/index_wf) ## Using multiple workflows It is possible to concatenate workflows in the same project. First, run the first workflow as described in the documentation. For example: ```bash miga quality_wf -o my_project /path/to/genomes/*.fna ``` Next, execute any additional steps *without* specifying the input files, and using the same output directory. For example: ```bash miga classify_wf -o my_project miga rerep_wf -o my_project ``` In the examples above, input genomes will be processed to evaluate quality, next they'll be classified, and finally they'll be dereplicated. # Quality This workflow automates the evaluation of quality statistics on your genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs). To execute the workflow, run: ```bash miga quality_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga quality_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/master/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. In addition, if you pass the option `--mytaxa-scan`, you can expect the subdirectory `mytaxa_scan` including PDF reports for each input genome. # Dereplicate This workflow automates the dereplication of your collection of genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs). To execute the workflow, run: ```bash miga derep_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga derep_wf -h ``` Importantly, the dereplication can be performed on ANI (default) or AAI (passing the `--aai` flag) at a given threshold (by default 95%) that can be modified with the flag `--threshold`. Finally, the representative genomes can be selected to reflect the highest genome quality (default) or to be the most "central" genome in the clade in ANI or AAI space (passing the `--medoids` flag). ## Expected output Once your run is complete, you may expect the [standard summaries](/master/part6/summaries) for `cds`, `assembly`, and `essential_genes`, as well as a table (`genomospecies.tsv`) with three columns: (1) a clade name, (2) the name of the representative genome, and (3) the names of all the members in the clade separated by commas. Additionally you can expect the subdirectory `representatives` including assemblies (FastA files, nucleotides) of all representative genomes. This is the dereplicated set of genomes. # Classify This workflow automates the comparison of your genomes, Metagenome-Assembled Genomes (MAGs), or Single-Cell Amplified Genomes (SAGs) against a reference project for taxonomy. A reference project has to be a separate MiGA project with a database of your choice, taxonomically annotated, and fully indexed. If you want to use the pre-processed database provided by MiGA Online, run: ```bash miga download ``` To execute the workflow, run: ```bash miga classify_wf -o my_project path/to/mags/*.fasta ``` For additional options, run: ```bash miga classify_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/master/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `taxonomy`, as well as a summary classification table (`classification.tsv`) with two columns: (1) the genome name, and (2) the space-delimited classification of the genome with taxon names prefixed by the rank code and a colon. Something like: ``` NZ_CP010409_1 d:Bacteria p:Proteobacteria c:Gammaproteobacteria o:Xanthomonadales f:Xanthomonadaceae g:Xanthomonas s:Xanthomonas_sacchari NZ_CP016878_1 d:Bacteria p:Proteobacteria c:Gammaproteobacteria o:Xanthomonadales f:Xanthomonadaceae g:Xanthomonas s:Xanthomonas_hortoru [...] ``` # Preprocess This workflow automates the preprocessing of your genomic or metagenomic datasets. To execute the workflow, run: ```bash miga preproc_wf -o my_project -i raw_reads_single path/to/reads/*.fastq ``` Supported inputs include: * *raw\_reads\_single:* Single raw reads in a single FastQ file * *raw\_reads\_paired:* Paired raw reads in two FastQ files * *trimmed\_reads\_single:* Single trimmed reads in a single FastA file * *trimmed\_reads\_paired:* Paired trimmed reads in two FastA files * *trimmed\_reads\_interleaved:* Paired trimmed reads in a single FastA file * *assembly:* Assembled contigs or scaffolds in FastA format For additional options, run: ```bash miga preproc_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/master/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. Additionally all the intermediate files are preserved, including assemblies, predicted genes, and detected essential and ribosomal genes. # Index This workflow automates the indexing of your genomic collection. To execute the workflow, run: ```bash miga index_wf -o my_project path/to/genomes/*.fasta ``` For additional options, run: ```bash miga index_wf -h ``` ## Expected output Once your run is complete, you may expect the [standard summaries](/master/part6/summaries) for `cds`, `assembly`, `essential_genes`, and `ssu`. Additionally all the intermediate files are preserved, including assemblies, predicted genes, and detected essential and ribosomal genes. Importantly, all-vs-all comparisons are generated using Average Amino Acid Identity (AAI) and Average Nucleotide Identity (ANI), making this a queriable project. ## Indexing publicly available genomes It is also possible to use this workflow on genomes publicly available in NCBI. This enables taxonomic analysis, such as using this indexed project as a reference database for the [classification workflow](/master/part6/classify_wf). For example, to download and index all the genomes from the species **Xanthomonas vesicatoria** using Diamond for AAI and FastANI for ANI estimation (`--fast` flag), run: ```bash miga index_wf -o X_vesicatoria \ -T 'Xanthomonas vesicatoria' --project-type clade --fast -v ``` # Summaries All workflows generate summary tables, which are raw text tab-delimited files with information about the input datasets. The standard summaries include stats for: * `assembly.tsv`: [Assembly](/master/part5/workflow#assembly) * `cds.tsv`: [CDS](/master/part5/workflow#cds) * `essential_genes.tsv`: [Essential Genes](/master/part5/workflow#essential_genes) * `ssu.tsv`: [SSU](/master/part5/workflow#ssu) * `taxonomy.tsv`: [Taxonomy](/master/part5/workflow#taxonomy)