Skip to content

Managing Storage Volumes Admin

Storage volumes in LabID represent the physical or cloud locations where group data libraries are stored. Proper configuration of these volumes is crucial for data organization and accessibility.

Adding a Local Data Volume

Each group should have at least one primary Storage volume. This primary volume is typically located on reliable local storage, ensuring data safety through backups or RAID configurations. Additionally, groups may have additional (non-primary) Storage volumes for different purposes, such as cloud storage.

Let's assume you want to add a primary volume for the group my_group located at /g/my_group/LabID/.

To add a new local Storage volume in the AdminUI:

  1. Navigate to Data Management > Storage volumes list page.
  2. Click Add Storage Volume (top right).
  3. Fill in the form:
    • Name: Provide a unique name. It's advisable to use a descriptive name, perhaps including the path, e.g., /g/my_group/.
    • Primary volume: Check this if it's the group's main storage. A primary volume must also be modifiable.
    • Modifiable: Check this if files on the volume can be directly edited (this is usually the case). Uncheck for read-only media like tapes.
    • Base path: Enter the exact absolute path to this LabID volume, e.g., /g/my_group/LabID/.
    • StorageBackend: Select LocalStorageBackend for local storage, or another appropriate backend.

On the local network server:

  1. Create the LabID library folder (e.g., /g/my_group/LabID/) with the user that runs the LabID server (e.g., labid_user).
    • Ensure this folder belongs to the correct unix group (e.g., my_group).
    • You might need to add labid_user to the my_group unix group.
  2. It's recommended to make this folder writable only by labid_user (e.g., chmod 751 /g/my_group/LabID/).
  3. Set the group sticky bit so sub-folders inherit the my_group unix group (e.g., chmod g+s /g/my_group/LabID/).
  4. Proceed to Defining User Dropboxes and Generating LabID Data Repository Structure.
  5. Ensure the generated user dropboxes are writable by the respective group users.
  6. Verify that the Data folder structure within the LabID library is write-protected for all users except labid_user.

Adding a Cloud Storage Volume

You can also integrate cloud-based storage solutions like S3-compatible services.

To add a new cloud Storage volume in the AdminUI:

  1. Navigate to Data Management > Storage volumes list page.
  2. Click Add cloud-storage-volume (top right).

  3. Fill in the form:

    • Name: A mandatory, unique, and meaningful name for the volume.
    • Description: A free text description of this item.
    • Base path: The base path or URL of the volume, e.g., s3://group1/.
    • Endpoint URL: The endpoint URL or host address of the volume, e.g., http://minio:9000 or https://s3.amazonaws.com.
    • StorageBackend: Select the appropriate backend for your cloud storage (e.g., S3 Storage).
    • Connection Parameters: Additional connection parameters compatible with Filesystem Spec. Do not include sensitive information here; use Credentials for that.
    • Credentials: Select pre-existing credentials from the drop-down list.
      • If you need to create new credentials:
        • Click the + icon next to the Credentials field.
        • A new form will appear. Fill in the following:
          • Name: A mandatory, meaningful name for the credential item.
          • Description: A free text description.
          • User: The account linked to the credential. If not linked, this is a non-personal credential record.
          • Access Key: The access key for the storage service.
          • Secret Access Key: The secret access key for the storage service.
          • Credential Type: The type of credential (e.g., AWS Credentials).
        • Save the new credential. It will then be available for selection.
    • Primary volume: always false for cloud volumes, they cannot be used as primary volumes.
    • Strict Folder Structure: also false for cloud volumes.
    • Modifiable: indicates if files can be modified on this volume, this is always true for cloud volumes.
    • Storage Time: If the volume has a limited store time, enter the duration here (0 means unlimited).
    • Storage Time Unit: Select the unit for the storage time (e.g., WEEK, MONTH, YEAR).

  4. Save the storage volume configuration.

Defining User Dropboxes

User dropboxes are specific folders within a group's storage volume where users can copy their data prior to data import in LabID.

To set up or ensure user dropboxes are correctly configured:

  1. Go to the Data Management > Storage volumes list page and select the group volume(s).
  2. In the action dropdown at the bottom of the list, select Setup User Dropboxes for selected group volumes. and click Go.

This process will generate any missing User_Dropboxes (sub)folders. It is safe to execute this command on an existing structure, as existing folders will be ignored.

Generating LabID Data Repository Structure (or Data Library)

The LabID Data Repository Structure refers to the standardized Data folder and its sub-folders within a group's volume, where processed and organized datasets are stored. We often refer to this as the Data Library.

To generate or verify this structure:

  1. Go to the Data Management > Storage volumes list page and select the group volume(s).
  2. In the action dropdown, select Setup the 'Data' (and derived) structure selected group volumes. and click Go.

This will create any missing Data (sub)folders. This command is also safe to run on an existing structure; it will not alter existing folders.