miga init), the final step is generating a daemon JSON file stored in
~/.miga_daemon.json. This file will control the type of daemon (see below) and the default configuration. In addition to this user-wide file, each MiGA project has a
daemon/daemon.jsonfile. Any variables defined here overwrite the user-wide file.
miga daemon, several configuration variables can be overwritten using the different command-line flags. If undefined, the variables will be defined by the
daemon/daemon.jsonfile in the project, or by the daemon JSON file defined by
--jsonif passed. Finally, and if a variable is still missing from there, it will default to the values in
--threadscontrol the maximum number of jobs (
maxjobs) and the number of CPUs per job (
ppn), respectively. Next, a daemon JSON file can be defined using the flag
--daemon. Finally, variables that are not defined by either method will default to the user-wide configuration in
bashdaemons, are the simplest mode. You can use this type of daemon if you are launching MiGA in a single computer.
maxjobsin daemon JSON files, or
--jobsin workflows), whereas the number of CPUs each job can use is determined by
ppnin daemon JSON files, or
--threadsin workflows). The number of jobs times the number of CPUs per job should never exceed the number of CPUs available. For example, if you have 12 cores in your computer, the default configuration of 6 jobs and 2 CPUs per jobs could use up to 100% of the available CPUs.
sshdaemons, are daemons that can communicate with other machines to launch tasks remotely using a login shell through SSH.
$MIGA_NODELIST) that will be read at execution or explicitly as a path to the file.
$PBS_NODEFILE. On the other hand, a fixed file can be a useful alternative if you have a defined set of nodes that are freely available to you (e.g., a dedicated infrastructure).
miga daemon(no equivalent is available for workflows). If this is not defined, daemons will look for the value of
nodelistin the daemon JSON files: first in
daemon/daemon.jsonor in the file set by
--daemon(for workflows) or
miga daemon), and finally in
~/.miga_daemon.json. If the node list is set to a variable (starting with
$), it must be defined as an environmental varible. For example, you could execute
export MIGA_NODELIST=/path/to/file.txtbefore running MiGA if the node list is set to
--jobs). Instead, the maximum number of jobs is determined by the number of lines in the node list. This allows specifying how many jobs can be launched to each node. For example, if a given node is present three times in the node list, MiGA will run up to three jobs at the same time in that node. Be careful when directly using node files defined by schedulers. Some schedulers will list nodes on the basis of dedicated cores, which may be a problem if the number of threads is more than 1 (
--threads). For example, consider the following PBS (Torque) script which you can use as a template (assuming that
slurm(SLURM). If your HPCC runs a different scheduler and you'd like to see it added to this list, please contact us.
--max-jobs 4). On the other hand, if you have hundreds of genomes to process, you could launch instead a remote daemon.