Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Last revision Both sides next revision | ||
|
namespace:doc-mach:soroban [2020/05/19 02:26] mickey [Directory organization for shared data] |
namespace:doc-mach:soroban [2020/05/19 02:30] mickey |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| ===== Directories for data shared among members of project MACH ===== | ===== Directories for data shared among members of project MACH ===== | ||
| - | ====== Internal page for MACH project members ====== | + | ==== Directory organization for shared data ==== |
| - | ====== ====== | + | * /bio/db |
| + | * Public biological sequence and related data. | ||
| + | * /data/mach/<user> | ||
| + | * Data by project MACH will be kept under | ||
| + | * <user> is a person who put the data / did the analysis. | ||
| + | * How MiSeq raw read data and analyzed data are organized will be desided among the users of the project. | ||
| - | [[:namespace:doc-mach:installed-software|List of software installed on soroban]] | ||
| - | [[:namespace:doc-mach:usage-soroban|Usage of 'soroban' server for non-MACH users]] | + | ==== Procedure to use /data/mach/<user> ==== |
| - | + | ||
| - | ===== Directories for data shared among members of project MACH ===== | + | |
| - | + | ||
| - | ==== Directory organization for shared data ==== * /bio/db ** Public biological sequence and related data. * Data by project MACH will be kept under /data/mach/* is a person who put the data / did the analysis. * How MiSeq raw read data and analyzed data are organized will be desided among the users of the project. | + | |
| - | + | ||
| - | ==== Procedure to start the use ==== | + | |
| + | <code> | ||
| cd /data/mach | cd /data/mach | ||
| - | + | mkdir <user> #e.g. mkdir mickey | |
| - | mkdir <user> (e.g. mkdir mickey) | + | </code> |
| - | + | ||
| - | ==== Note about priviledges of users ==== | + | |
| - | + | ||
| - | Highest priority: Usage for MACH project by the project members. | + | |
| - | + | ||
| - | ===== List of users ===== | + | |
| - | + | ||
| - | |**User** |**Family name** |**First name** |**Secondary groups** |**Note** | | + | |
| - | |bioadmin| | | |to install bioinformatic tools without using root by using su/sudo| | + | |
| - | |joaquin|Rilling|Joaquin|fuse, machanal|| | + | |
| - | |so|Fujiyoshi|So|fuse, machanal|| | + | |
| - | |mickey|Kawai|Mikihiko|fuse, machanal|| | + | |
| - | |fumito|Maruyama|Fumito|machanal| | | + | |
| - | |satoshi|Nagai|Satoshi|machanal| | | + | |
| - | |kyoko|Yarimizu|Kyoko|machanal| | | + | |
| - | |shoko|Ueki|Shoko|machanal| | | + | |
| - | + | ||
| - | ===== List of groups ===== | + | |
| - | + | ||
| - | |**Date created** |**Group name** |**Purpose** |Note| | + | |
| - | |20191001?|fuse|To mount basemount for getting raw gigabase-order raw sequence files of Illumina MiSeq from Illumina's BaseSpace|probably set by default setting of os| | + | |
| - | |20191001?|machanal|To share analyzed data among members of project MACH (e.g. at /data/mach/)|| | + | |
| - | |not yet|machdb| | | | + | |
| - | |not yet|machweb| | | | + | |
| - | |not yet|machmodel| | | | + | |
| - | + | ||
| - | ==== Assignment of users to a group ==== | + | |
| - | + | ||
| - | |**Group** |**Assigned user** | | + | |
| - | |machanal|joaquin| | + | |
| - | |machanal|so| | + | |
| - | |machanal|mickey| | + | |
| - | |machanal|fumito| | + | |
| - | |machanal|satoshi| | + | |
| - | |machanal|kyoko| | + | |
| - | |machanal|shoko| | + | |
| - | | | | | + | |
| - | |fuse|joaquin| | + | |
| - | |fuse|so| | + | |
| - | |fuse|mickey| | + | |
| - | + | ||
| - | ===== Directory organization ===== | + | |
| - | + | ||
| - | ==== For sharing data among project members ==== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/data/|775 (root-research)| |20TB| | + | |
| - | |/data/<project>/|2770 (<project>-research)|To keep data shared among users of the project| | | + | |
| - | |/data/<project>/<user>/|2750|To keep data analyzed by the user of the project| | | + | |
| - | + | ||
| - | * /data can be other path, for example, /home/proj | + | |
| - | * <user> belongs to a group, of which name for the <project> may be simply the same as <project>, but can be different. | + | |
| - | + | ||
| - | === List of projects === | + | |
| - | + | ||
| - | |**Project** |**Group** |**User** |**Last name** |**First name** | | + | |
| - | |mach|machanal|joaquin|Joaqui|Rilling| | + | |
| - | |mach|machanal|so|So|Fujiyoshi| | + | |
| - | |mach|machanal|mickey|Mikihiko|Kawai| | + | |
| - | |ao|name_ao|kei|Kei|Yoshimura| | + | |
| - | |ao|name_ao|fumiaki|Fumiaki|Morimoto| | + | |
| - | |ao|name_ao|xiaojun|Xiaojun|name| | + | |
| - | + | ||
| - | ===== For directories of tools and public databases of biological data ===== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/bio|775 (bioadmin-bioadmin)| |25TB| | + | |
| - | |/bio/bin| | | | | + | |
| - | |/bio/package| | | | | + | |
| - | |/bio/package/<tool>/<version>| | | | | + | |
| - | |/bio/db| | | | | + | |
| - | + | ||
| - | ===== Queues / Groups of user priviledge (SLURM) ===== | + | |
| - | + | ||
| - | See belows for basic usage of SLURM at UFRO-CEMCC servers [[https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages|https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages]] | + | |
| - | + | ||
| - | ===== ===== | + | |
| - | + | ||
| - | ===== ===== | + | |
| - | + | ||
| - | ==== Assignment of users to a group ==== | + | |
| - | + | ||
| - | |**Group** |**Assigned user** | | + | |
| - | |machanal|joaquin| | + | |
| - | |machanal|so| | + | |
| - | |machanal|mickey| | + | |
| - | |machanal|fumito| | + | |
| - | |machanal|satoshi| | + | |
| - | |machanal|kyoko| | + | |
| - | |machanal|shoko| | + | |
| - | |fuse|joaquin| | + | |
| - | |fuse|so| | + | |
| - | |fuse|mickey| | + | |
| - | + | ||
| - | ===== Directory organization ===== | + | |
| - | + | ||
| - | ==== For sharing data among project members ==== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** | | + | |
| - | |/data/|775 (root-research)| | | + | |
| - | |/data/<project>/|2770 (<project>-research)|To keep data shared among users of the project| | + | |
| - | |/data/<project>/<user>/|2750|To keep data analyzed by the user of the project| | + | |
| - | + | ||
| - | * /data can be other path, for example, /home/proj * <user> belongs to a group, of which name for the <project> may be simply the same as <project>, but can be different. | + | |
| - | + | ||
| - | === List of projects === | + | |
| - | + | ||
| - | |**Project** |**Group** |**User** | | + | |
| - | |mach|machanal|joaquin| | + | |
| - | |mach|machanal|so| | + | |
| - | |mach|machanal|mickey| | + | |
| - | + | ||
| - | ===== For directories of tools and public databases of biological data ===== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/bio|775 (bioadmin-bioadmin)| |25TB| | + | |
| - | |/bio/bin| | | | | + | |
| - | |/bio/db| | | | | + | |
| - | |/bio/package| | | | | + | |
| - | |/bio/package/<tool>/<version>| | | | | + | |
| - | + | ||
| - | ===== Queues / Groups of user priviledge (SLURM) ===== | + | |
| - | + | ||
| - | See belows for basic usage of SLURM at UFRO-CEMCC servers [[https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages|https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages]] | + | |
| - | + | ||
| - | ==== Groups of priviledges ==== | + | |
| - | + | ||
| - | |**Priority** |**Group name of priviledge** |**Purpose** |**Note** | | | + | |
| - | |High|mach|Usage for MACH project by the project members.|Distribute evenly among users (e.g. for biological analysis and for simulation) (by ticket, nice value etc. on SLURM)| | | + | |
| - | |machcolab|Coollaborative studies by MACH members| | | | | + | |
| - | |ufro-internal||Need to submit user form (e.g. Name, affiliation, purpose of usage, desired account if any)| | | | + | |
| - | |Low|external|Need to submit user form (e.g. Name, affiliation, purpose of usage, desired account if any)| | | | + | |
| - | + | ||
| - | ===== Limitation of initial resources for default user account ===== | + | |
| - | + | ||
| - | |**Queue name** |**Duration of one job** |**Memory per job** | | | | + | |
| - | |small|6hr| |medium|72hr| | + | |
| - | + | ||
| - | |**Group of priviledge** |**Number of simultaneous jobs** | | | | | + | |
| - | |default|e.g. 30?| | | | | + | |
| - | + | ||
| - | Submit an application form if one wants to request large resources, say, storage, memory per queue, number of similutaneously running queues/cpu. | + | |
| - | + | ||
| - | ====== Internal page for MACH project members ====== | + | |
| - | + | ||
| - | ====== ====== | + | |
| - | + | ||
| - | ===== General note about data that are/will be managed by members of project MACH ===== | + | |
| - | + | ||
| - | Roughly speaking, three types of data will be kept, public biological sequence and related data, MiSeq data and Analized data. | + | |
| - | + | ||
| - | * Public biological sequence and related data will be kept under /bio/db. | + | |
| - | * Data by project MACH will be kept under /data/mach/<user>, where <user> is a person who did the analysis. | + | |
| - | * How MiSeq raw read data and analyzed data are organized will be desided amont the users of the project. | + | |
| - | + | ||
| - | ==== Note about priviledges of users ==== | + | |
| - | + | ||
| - | Highest priority: Usage for MACH project by the project members. | + | |
| - | + | ||
| - | ===== List of users ===== | + | |
| - | + | ||
| - | |**User** |**Family name** |**First name** |**Secondary groups** |**Note** | | + | |
| - | |bioadmin| | | |to install bioinformatic tools without using root by using su/sudo| | + | |
| - | |joaquin|Rilling|Joaquin|fuse, machanal|| | + | |
| - | |so|Fujiyoshi|So|fuse, machanal|| | + | |
| - | |mickey|Kawai|Mikihiko|fuse, machanal|| | + | |
| - | |fumito|Maruyama|Fumito|machanal| | | + | |
| - | |satoshi|Nagai|Satoshi|machanal| | | + | |
| - | |kyoko|Yarimizu|Kyoko|machanal| | | + | |
| - | |shoko|Ueki|Shoko|machanal| | | + | |
| - | + | ||
| - | ===== List of groups ===== | + | |
| - | + | ||
| - | |**Date created** |**Group name** |**Purpose** |Note| | + | |
| - | |20191001?|fuse|To mount basemount for getting raw gigabase-order raw sequence files of Illumina MiSeq from Illumina's BaseSpace|probably set by default setting of os| | + | |
| - | |20191001?|machanal|To share analyzed data among members of project MACH (e.g. at /data/mach/)|| | + | |
| - | |not yet|machdb| | | | + | |
| - | |not yet|machweb| | | | + | |
| - | |not yet|machmodel| | | | + | |
| - | + | ||
| - | ==== Assignment of users to a group ==== | + | |
| - | + | ||
| - | |**Group** |**Assigned user** | | + | |
| - | |machanal|joaquin| | + | |
| - | |machanal|so| | + | |
| - | |machanal|mickey| | + | |
| - | |machanal|fumito| | + | |
| - | |machanal|satoshi| | + | |
| - | |machanal|kyoko| | + | |
| - | |machanal|shoko| | + | |
| - | | | | | + | |
| - | |fuse|joaquin| | + | |
| - | |fuse|so| | + | |
| - | |fuse|mickey| | + | |
| - | + | ||
| - | ===== Directory organization ===== | + | |
| - | + | ||
| - | ==== For sharing data among project members ==== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/data/|775 (root-research)| |20TB| | + | |
| - | |/data/<project>/|2770 (<project>-research)|To keep data shared among users of the project| | | + | |
| - | |/data/<project>/<user>/|2750|To keep data analyzed by the user of the project| | | + | |
| - | + | ||
| - | * /data can be other path, for example, /home/proj | + | |
| - | * <user> belongs to a group, of which name for the <project> may be simply the same as <project>, but can be different. | + | |
| - | + | ||
| - | === List of projects === | + | |
| - | + | ||
| - | |**Project** |**Group** |**User** |**Last name** |**First name** | | + | |
| - | |mach|machanal|joaquin|Joaqui|Rilling| | + | |
| - | |mach|machanal|so|So|Fujiyoshi| | + | |
| - | |mach|machanal|mickey|Mikihiko|Kawai| | + | |
| - | |ao|name_ao|kei|Kei|Yoshimura| | + | |
| - | |ao|name_ao|fumiaki|Fumiaki|Morimoto| | + | |
| - | |ao|name_ao|xiaojun|Xiaojun|name| | + | |
| - | + | ||
| - | ===== For directories of tools and public databases of biological data ===== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/bio|775 (bioadmin-bioadmin)| |25TB| | + | |
| - | |/bio/bin| | | | | + | |
| - | |/bio/package| | | | | + | |
| - | |/bio/package/<tool>/<version>| | | | | + | |
| - | |/bio/db| | | | | + | |
| - | + | ||
| - | ===== Queues / Groups of user priviledge (SLURM) ===== | + | |
| - | + | ||
| - | See belows for basic usage of SLURM at UFRO-CEMCC servers [[https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages|https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages]] | + | |
| - | + | ||
| - | ===== ===== | + | |
| - | + | ||
| - | ===== ===== | + | |
| - | + | ||
| - | ==== Assignment of users to a group ==== | + | |
| - | + | ||
| - | |**Group** |**Assigned user** | | + | |
| - | |machanal|joaquin| | + | |
| - | |machanal|so| | + | |
| - | |machanal|mickey| | + | |
| - | |machanal|fumito| | + | |
| - | |machanal|satoshi| | + | |
| - | |machanal|kyoko| | + | |
| - | |machanal|shoko| | + | |
| - | |fuse|joaquin| | + | |
| - | |fuse|so| | + | |
| - | |fuse|mickey| | + | |
| - | + | ||
| - | ===== Directory organization ===== | + | |
| - | + | ||
| - | ==== For sharing data among project members ==== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** | | + | |
| - | |/data/|775 (root-research)| | | + | |
| - | |/data/<project>/|2770 (<project>-research)|To keep data shared among users of the project| | + | |
| - | |/data/<project>/<user>/|2750|To keep data analyzed by the user of the project| | + | |
| - | + | ||
| - | * /data can be other path, for example, /home/proj * <user> belongs to a group, of which name for the <project> may be simply the same as <project>, but can be different. | + | |
| - | + | ||
| - | === List of projects === | + | |
| - | + | ||
| - | |**Project** |**Group** |**User** | | + | |
| - | |mach|machanal|joaquin| | + | |
| - | |mach|machanal|so| | + | |
| - | |mach|machanal|mickey| | + | |
| - | + | ||
| - | ===== For directories of tools and public databases of biological data ===== | + | |
| - | + | ||
| - | |**Path** |**Permission** |**Purpose** |**Size** | | + | |
| - | |/bio|775 (bioadmin-bioadmin)| |25TB| | + | |
| - | |/bio/bin| | | | | + | |
| - | |/bio/db| | | | | + | |
| - | |/bio/package| | | | | + | |
| - | |/bio/package/<tool>/<version>| | | | | + | |
| - | + | ||
| - | ===== Queues / Groups of user priviledge (SLURM) ===== | + | |
| - | + | ||
| - | See belows for basic usage of SLURM at UFRO-CEMCC servers [[https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages|https://cemcc.ufro.cl/wiki/doku.php?id=slurm#useful_reference_pages]] | + | |
| - | + | ||
| - | ==== Groups of priviledges ==== | + | |
| - | + | ||
| - | |**Priority** |**Group name of priviledge** |**Purpose** |**Note** | | | + | |
| - | |High|mach|Usage for MACH project by the project members.|Distribute evenly among users (e.g. for biological analysis and for simulation) (by ticket, nice value etc. on SLURM)| | | + | |
| - | |machcolab|Coollaborative studies by MACH members| | | | | + | |
| - | |ufro-internal||Need to submit user form (e.g. Name, affiliation, purpose of usage, desired account if any)| | | | + | |
| - | |Low|external|Need to submit user form (e.g. Name, affiliation, purpose of usage, desired account if any)| | | | + | |
| - | + | ||
| - | ===== Limitation of initial resources for default user account ===== | + | |
| - | + | ||
| - | | |**Disk quote per user** | | + | |
| - | |default|500GB| | + | |
| - | + | ||
| - | |**Queue name** |**Duration of one job** |**Memory per job** | | | | + | |
| - | |small|6hr| |medium|72hr| | + | |
| - | + | ||
| - | |**Group of priviledge** |**Number of simultaneous jobs** | | | | | + | |
| - | |default|e.g. 30?| | | | | + | |
| - | + | ||
| - | Submit an application form if one wants to request large resources, say, storage, memory per queue, number of similutaneously running queues/cpu. | + | |
| - | + | ||
| - | [[:namespace:doc-mach:meeting|MACH Meeting about soroban]] | + | |
| - | + | ||
| - | \\ | + | |
| - | + | ||
| - | + | ||
| - | ==== Procedure to start the use ==== | + | |
| - | + | ||
| - | cd /data/mach | + | |
| - | + | ||
| - | mkdir <user> (e.g. mkdir mickey) | + | |