MinIO Object Storage Server¶

MinIO is a high-performance object storage suite, capable of storing unstructured data such as photos, videos, log files, backups, and container images.

The app deploys a standalone (single-node single drive) MinIO server instance, where the user can select the storage location directly on the UCloud file system.

Object Storage Volume¶

MinIO creates an object-based storage volume, which is a self-contained repository organized in buckets. Buckets have a flat structure where data is divided into discrete units called objects. The storage structure provides a unique identifier, that allows the objects to be found over a distributed filesystem, and metadata describing the data.

To create a new MinIO server, the user should mount an empty repository folder using the mandatory parameter Object storage volume. A hidden directory named .minio.sys, containing the object storage metadata, is then created inside the repository folder. This ensures that the server settings are recovered whenever the repository folder is mounted as an object storage volume.

As an example, consider a repository folder called minio-repo with the following structure:

minio-repo
├── data1
│   ├── file1
│   ├── file2
│   ├── file3
│   └── file4
├── data2
│   ├── image1.png
│   └── image2.png
└── data3
    └── doc.txt

In this case the storage volume contains three buckets with different file types. Buckets and files (objects) can be added and modified using the MinIO Console, the MinIO Client, or one of the MinIO Software Development Kits (SDK).

Note

In case a new object storage is created starting from a directory that is not empty, the top-level folders automatically become MinIO buckets, but the files are not converted into objects. They must be added manually to the object storage volume.

MinIO Console¶

The MinIO Console is launched together with the server and it can be accessed by clicking

Administrator login credentials are generated when the job starts and are printed in the app's standard output.

The MinIO Console provides an intuitive graphical user interface for managing buckets and objects. It looks like in the image below.

The user can perform the following operations from the MinIO Console:

Create/remove buckets
Enable bucket versioning
Enable bucket quota
Upload files and folders
Download files
Delete file and folders
Image/text file preview

In addition, the MinIO Console can be used for administration tasks like identity and access management, metrics and log monitoring, or server configuration.

MinIO Client¶

The mc MinIO Client command-line tool provides an alternative way to manage object-based storage volumes via the app's built-in terminal interface,

By default, the MinIO Client connects to the server with the alias name ucloud.

The following list provides a quick overview of using the mc command. For a more in-depth tutorial, check the official guide.

Test connection to the server¶

$ mc admin info ucloud

Tip

● localhost:9000
  Uptime: 27 minutes
  Version: 2023-02-27T18:10:45Z
  Network: 1/1 OK
  Drives: 1/1 OK
  Pool: 1

Pools:
  1st, Erasure sets: 1, Disks per erasure set: 1

1.2 GiB Used, 3 Buckets, 12 Objects, 20 Versions
1 drive online, 0 drives offline

List buckets and objects¶

$ mc ls --recursive ucloud

Tip

[2022-08-11 05:02:16 UTC] 5.4MiB STANDARD data1/file1
[2022-08-09 08:45:11 UTC] 11MiB STANDARD data1/file2
[2022-08-09 08:45:11 UTC] 11MiB STANDARD data1/file3
[2022-08-09 08:45:11 UTC] 16MiB STANDARD data1/file4
[2022-08-11 08:03:19 UTC] 1.2MiB STANDARD data2/image1.png
[2022-08-11 07:40:29 UTC] 1.2GiB STANDARD data2/image2.png
[2022-08-11 07:51:45 UTC] 29B STANDARD data3/doc.txt

Make a bucket¶

$ mc mb ucloud/data4

Tip

Bucket created successfully `ucloud/data4`.

Remove a bucket¶

$ mc rb ucloud/data4

Tip

Removed `ucloud/data4` successfully.

Copy objects¶

$ mc cp -r ucloud/data1 /work/

Move objects¶

$ mc mv ucloud/data1/file1 /work/
$ mc ls ucloud/data1/
$ mc mv /work/file1 ucloud/data1/
$ mc ls ucloud/data1/

Manage bucket versioning¶

$ mc version enable ucloud/data1

Tip

ucloud/data1 versioning is enabled

$ mc version suspend ucloud/data1

Tip

ucloud/data1 versioning is suspended

$ mc version info ucloud/data1

Tip

[2022-08-11 08:37:35 UTC] 5.4MiB STANDARD 93d9be13-c716-46bd-8c39-930fe9a5b3e5 v5 PUT file1
[2022-08-11 08:37:25 UTC] 0B STANDARD 33c5b6a5-9d04-4498-87ae-874803090a08 v4 DEL file1
[2022-08-11 05:02:16 UTC] 5.4MiB STANDARD 94386d9a-f68d-4cdd-9232-523f76390d47 v3 PUT file1
[2022-08-11 05:01:28 UTC] 0B STANDARD d2f19203-390f-47ea-ba90-552c527ffe15 v2 DEL file1
[2022-08-09 08:45:10 UTC] 5.4MiB STANDARD 329c44eb-6c96-4bd5-a241-57a2d8eca608 v1 PUT file1
[2022-08-09 08:45:11 UTC] 11MiB STANDARD e3ea200b-de26-48b7-9c2d-088b6b61d321 v1 PUT file2
[2022-08-09 08:45:11 UTC] 11MiB STANDARD 675dc696-2586-4741-819d-7fce1206d433 v1 PUT file3
[2022-08-09 08:45:11 UTC] 16MiB STANDARD c16c3cce-96f3-495f-bf21-7b2bfc2921a7 v1 PUT file4

Manage anonymous bucket policies¶

$ mc anonymous set public ucloud/data1

Tip

Access permission for `ucloud/data1` is set to `public`

$ mc anonymous list ucloud/data1

Tip

data1/* => readwrite

$ mc anonymous set upload ucloud/data2

Tip

Access permission for `ucloud/data2` is set to `upload`

$ mc anonymous list ucloud/data2

Tip

data2/* => writeonly

$ mc anonymous set download ucloud/data3

Tip

Access permission for `ucloud/data3` is set to `download`

$ mc anonymous list ucloud/data3

Tip

data3/* => readonly

Manage users¶

$ mc admin user add ucloud user1 user1password

Tip

Added user `user1` successfully.

$ mc admin user list ucloud

Tip

enabled user1

$ mc admin user info ucloud user1

Tip

AccessKey: user1
Status: enabled
PolicyName:
MemberOf:

$ mc admin user disable ucloud user1

Tip

Disabled user `user1` successfully.

$ mc admin user remove ucloud user1

Tip

Removed user `user1` successfully.

Manage groups¶

$ mc admin group add ucloud group12 user1 user2

Tip

Added members {user1,user2} to group group12 successfully.

$ mc admin group list ucloud

Tip

group12

$ mc admin group info ucloud group12

Tip

Group: group12
Status: enabled
Policy:
Members: user1,user2

$ mc admin group disable ucloud group12

Tip

Disabled group `group12` successfully.

$ mc admin group remove ucloud group12 user1 user2

Tip

Removed members {user1,user2} from group group12 successfully.

$ mc admin group remove ucloud group12

Tip

Removed group group12 successfully.

Manage canned policies¶

$ mc admin policy add ucloud getonly ./getonly.json

Tip

Added policy `getonly` successfully.

with the policy settings specified in the JSON file getonly.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

$ mc admin policy list ucloud

Tip

readwrite
writeonly
consoleAdmin
diagnostics
getonly
readonly

$ mc admin policy set ucloud getonly user=user1

Tip

Policy `getonly` is set on user `user1`

$ mc admin user info ucloud user1

Tip

AccessKey: user1
Status: enabled
PolicyName: getonly
MemberOf:

$ mc admin policy unset ucloud getonly user=user1

Tip

Policy `getonly` is unset on user `user1`

$ mc admin policy remove ucloud getonly

Tip

Removed policy `getonly` successfully.

Restart the server¶

$ mc admin service restart ucloud

Tip

Restart command successfully sent to `ucloud`. Type Ctrl-C to quit or wait to follow the status of the restart process.
...
Restarted `ucloud` successfully in 1 seconds

Shell aliases¶

Shell aliases can be used to override common Unix command-line tools. For example consider the following Bash script:

#!/bin/bash

tee -a ~/.bashrc << END
alias ls='mc ls'
alias cp='mc cp'
alias cat='mc cat'
alias mkdir='mc mb'
alias pipe='mc pipe'
alias find='mc find'
END

The script can be executed before starting the MinIO server using the optional Initialization parameter.

MinIO Object Sharing¶

The app leverages the MinIO multi-user feature to share data with external collaborators.

New user login accounts can be created by the administrator through the MinIO Console or the MinIO Client. The server can be configured to deny or allow access to buckets to each of these users.

To ensure remote connection, the app must be deployed with a public URL via the corresponding optional parameter. The console login URL and the API connection string are printed in the output logs, e.g.:

Console URL: https://app-myminio.cloud.sdu.dk/console/login

API connection string:
  $ mc alias set 'ucloud' 'https://app-myminio.cloud.sdu.dk' YWY4OGJmNDM1MDhhNzU4YTFjZDI0MmRh MmZkOTMzMjgyYTY3NTVmNDhlNGEzMThk

A MinIO client API must be installed on the remote host (see, e.g., instructions reported here) to connect to the MinIO server.

Pre-signed URLs¶

Once the API connection string is executed on the remote host, the mc share command can be used to generate pre-signed temporary URLs with integrated login credentials to download/upload objects from/to a MinIO bucket.

For example:

$ mc share download ucloud/data1/file1

Tip

URL: https://app-myminio.cloud.sdu.dk/data1/file1.txt
Expire: 7 days 0 hours 0 minutes 0 seconds
Share: https://app-myminio.cloud.sdu.dk/data1/file1.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MTAzOGQ4ZDYzM2QxODJlNDI4ZTMwYTA5%2F20240215%2Feu-north-0%2Fs3%2Faws4_request&X-Amz-Date=20240215T064116Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=d875156dbab0526a898a97b77f051e718d8cd886c1d2cc3c3cac004e4191d670

Note

Temporary sharable URLs do not work when generated from the MinIO Console.

MinIO SDK APIs¶

MinIO supports several SDK client APIs to access object storage volumes.

Some examples are reported below.

Go

package main

import (
    "log"

    "github.com/minio/minio-go/v7"
    "github.com/minio/minio-go/v7/pkg/credentials"
)

func main() {
    endpoint := "<PUBLIC_IP>:9000"
    accessKeyID := "USERNAME"
    secretAccessKey := "PASSWORD"
    useSSL := true

    // Initialize minio client object.
    minioClient, err := minio.New(endpoint, &minio.Options{
        Creds:  credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
        Secure: useSSL,
    })
    if err != nil {
        log.Fatalln(err)
    }

    log.Printf("%#v\n", minioClient) // minioClient is now setup
}

Java

MinioClient minioClient =
    MinioClient.builder()
        .endpoint("<PUBLIC_IP>:9000")
        .credentials("USERNAME", "PASSWORD")
        .build();

JavaScript

var Minio = require('minio')

var minioClient = new Minio.Client({
    endPoint: '<PUBLIC_IP>:9000',
    port: 9000,
    useSSL: true,
    accessKey: 'USERNAME',
    secretKey: 'PASSWORD'
});

Python

import os
from minio import Minio

client = Minio('<PUBLIC-IP>:9000',
               access_key='<USERNAME>',
               secret_key='<PASSWORD>',
               secure=False)

client.list_buckets()

For a more in-depth guide of the different APIs, check here.