Drives (FileCollection)

API: Stable

A FileCollection is an entrypoint to a user’s files

Rationale

📝 NOTE: This API follows the standard Resources API. We recommend that you have already read and understood the concepts described here.


This entrypoint allows the user to access all the files they have access to within a single project. It is important to note that a file collection is not the same as a directory! Common real-world examples of a file collection is listed below:

Name Typical path Comment
Home directory /home/$username/ The home folder is typically the main collection for a user
Work directory /work/$projectId/ The project 'home' folder
Scratch directory /scratch/$projectId/ Temporary storage for a project

The provider of storage manages a ‘database’ of these file collections and who they belong to. The file collections also play an important role in accounting and resource management. A file collection can have a quota attached to it and billing information is also stored in this object. Each file collection can be attached to a different product type, and as a result, can have different billing information attached to it. This is, for example, useful if a storage provider has both fast and slow tiers of storage, which is typically billed very differently.

All file collections additionally have a title. This title can be used for a user-friendly version of the folder. This title does not have to be unique, and can with great benefit choose to not reference who it belongs to. For example, if every user has exactly one home directory, then it would make sense to give this collection "Home" as its title.


📝 Provider Note: This is the API exposed to end-users. See the table below for other relevant APIs.

End-User Provider (Ingoing) Control (Outgoing)
FileCollections FileCollectionsProvider FileCollectionsControl

Table of Contents

1. Examples
Description
An example collection
Renaming a collection
2. Remote Procedure Calls
Name Description
browse Browses the catalog of available resources
retrieve Retrieve a single resource
retrieveProducts Retrieve product support for all accessible providers
search Searches the catalog of available resources
create Creates one or more resources
delete Deletes one or more resources
init Request (potential) initialization of resources
rename Request renaming of [`FileCollection`](/docs/reference/dk.sdu.cloud.file.orchestrator.api.FileCollection.md)
updateAcl Updates the ACL attached to a resource
3. Data Models
Name Description
FSCollectionSupport Declares which `FileCollection` operations are supported for a product
FSFileSupport Declares which file-level operations a product supports
FSProductStatsSupport Declares which stats a given product supports
FSSupport No description
FileCollection A `Resource` is the core data model used to synchronize tasks between UCloud and Provider.
FileCollection.Spec No description
FileCollection.Status Describes the current state of the `Resource`
FileCollection.Update Describes an update to the `Resource`
FileCollectionIncludeFlags No description
MemberFilesFilter Filter for member files.
FileCollectionsRenameRequestItem No description

Example: An example collection

Frequency of useCommon
Actors
  • An authenticated user (user)
Communication Flow: Kotlin
/* In this example we will see a simple collection. This collection models the 'home' directory of a user. */


/* 📝 NOTE: Collections are identified by a unique (UCloud provided) ID */

FileCollections.retrieve.call(
    ResourceRetrieveRequest(
        flags = FileCollectionIncludeFlags(
            filterCreatedAfter = null, 
            filterCreatedBefore = null, 
            filterCreatedBy = null, 
            filterIds = null, 
            filterMemberFiles = null, 
            filterProductCategory = null, 
            filterProductId = null, 
            filterProvider = null, 
            filterProviderIds = null, 
            hideProductCategory = null, 
            hideProductId = null, 
            hideProvider = null, 
            includeOthers = false, 
            includeProduct = false, 
            includeSupport = false, 
            includeUpdates = false, 
        ), 
        id = "54123", 
    ),
    user
).orThrow()

/*
FileCollection(
    createdAt = 1635151675465, 
    id = "54123", 
    owner = ResourceOwner(
        createdBy = "user", 
        project = null, 
    ), 
    permissions = ResourcePermissions(
        myself = listOf(Permission.ADMIN), 
        others = emptyList(), 
    ), 
    providerGeneratedId = null, 
    specification = FileCollection.Spec(
        product = ProductReference(
            category = "example-ssd", 
            id = "example-ssd", 
            provider = "example", 
        ), 
        title = "Home", 
    ), 
    status = FileCollection.Status(
        resolvedProduct = null, 
        resolvedSupport = null, 
    ), 
    updates = emptyList(), 
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------

# In this example we will see a simple collection. This collection models the 'home' directory of a user.

# 📝 NOTE: Collections are identified by a unique (UCloud provided) ID

# Authenticated as user
curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/collections/retrieve?includeOthers=false&includeUpdates=false&includeSupport=false&includeProduct=false&id=54123" 

# {
#     "id": "54123",
#     "specification": {
#         "title": "Home",
#         "product": {
#             "id": "example-ssd",
#             "category": "example-ssd",
#             "provider": "example"
#         }
#     },
#     "createdAt": 1635151675465,
#     "status": {
#         "resolvedSupport": null,
#         "resolvedProduct": null
#     },
#     "updates": [
#     ],
#     "owner": {
#         "createdBy": "user",
#         "project": null
#     },
#     "permissions": {
#         "myself": [
#             "ADMIN"
#         ],
#         "others": [
#         ]
#     },
#     "providerGeneratedId": null
# }
Communication Flow: Visual

Example: Renaming a collection

Frequency of useCommon
Actors
  • An authenticated user (user)
Communication Flow: Kotlin
/* In this example, we will see how a user can rename one of their collections. */


/* 📝 NOTE: Renaming must be supported by the provider */

FileCollections.retrieveProducts.call(
    Unit,
    user
).orThrow()

/*
SupportByProvider(
    productsByProvider = mapOf("ucloud" to listOf(ResolvedSupport(
        product = Product.Storage(
            allowAllocationRequestsFrom = AllocationRequestsGroup.ALL, 
            category = ProductCategoryId(
                id = "example-ssd", 
                name = "example-ssd", 
                provider = "example", 
            ), 
            chargeType = ChargeType.DIFFERENTIAL_QUOTA, 
            description = "Fast storage", 
            freeToUse = false, 
            hiddenInGrantApplications = false, 
            name = "example-ssd", 
            pricePerUnit = 1, 
            priority = 0, 
            productType = ProductType.STORAGE, 
            unitOfPrice = ProductPriceUnit.PER_UNIT, 
            version = 1, 
            balance = null, 
            id = "example-ssd", 
            maxUsableBalance = null, 
        ), 
        support = FSSupport(
            collection = FSCollectionSupport(
                aclModifiable = null, 
                usersCanCreate = true, 
                usersCanDelete = true, 
                usersCanRename = true, 
            ), 
            files = FSFileSupport(
                aclModifiable = false, 
                isReadOnly = false, 
                searchSupported = true, 
                sharesSupported = false, 
                streamingSearchSupported = false, 
                trashSupported = false, 
            ), 
            maintenance = null, 
            product = ProductReference(
                category = "example-ssd", 
                id = "example-ssd", 
                provider = "example", 
            ), 
            stats = FSProductStatsSupport(
                accessedAt = null, 
                createdAt = null, 
                modifiedAt = null, 
                sizeInBytes = null, 
                sizeIncludingChildrenInBytes = null, 
                unixGroup = null, 
                unixOwner = null, 
                unixPermissions = null, 
            ), 
        ), 
    ))), 
)
*/

/* As we can see, the provider does support the rename operation. We now look at our collections. */

FileCollections.browse.call(
    ResourceBrowseRequest(
        consistency = null, 
        flags = FileCollectionIncludeFlags(
            filterCreatedAfter = null, 
            filterCreatedBefore = null, 
            filterCreatedBy = null, 
            filterIds = null, 
            filterMemberFiles = null, 
            filterProductCategory = null, 
            filterProductId = null, 
            filterProvider = null, 
            filterProviderIds = null, 
            hideProductCategory = null, 
            hideProductId = null, 
            hideProvider = null, 
            includeOthers = false, 
            includeProduct = false, 
            includeSupport = false, 
            includeUpdates = false, 
        ), 
        itemsPerPage = null, 
        itemsToSkip = null, 
        next = null, 
        sortBy = null, 
        sortDirection = null, 
    ),
    user
).orThrow()

/*
PageV2(
    items = listOf(FileCollection(
        createdAt = 1635151675465, 
        id = "54123", 
        owner = ResourceOwner(
            createdBy = "user", 
            project = null, 
        ), 
        permissions = ResourcePermissions(
            myself = listOf(Permission.ADMIN), 
            others = emptyList(), 
        ), 
        providerGeneratedId = null, 
        specification = FileCollection.Spec(
            product = ProductReference(
                category = "example-ssd", 
                id = "example-ssd", 
                provider = "example", 
            ), 
            title = "Home", 
        ), 
        status = FileCollection.Status(
            resolvedProduct = null, 
            resolvedSupport = null, 
        ), 
        updates = emptyList(), 
    )), 
    itemsPerPage = 50, 
    next = null, 
)
*/

/* Using the unique ID, we can rename the collection */

FileCollections.rename.call(
    bulkRequestOf(FileCollectionsRenameRequestItem(
        id = "54123", 
        newTitle = "My Awesome Drive", 
    )),
    user
).orThrow()

/*
Unit
*/

/* The new title is observed when we browse the collections one more time */

FileCollections.browse.call(
    ResourceBrowseRequest(
        consistency = null, 
        flags = FileCollectionIncludeFlags(
            filterCreatedAfter = null, 
            filterCreatedBefore = null, 
            filterCreatedBy = null, 
            filterIds = null, 
            filterMemberFiles = null, 
            filterProductCategory = null, 
            filterProductId = null, 
            filterProvider = null, 
            filterProviderIds = null, 
            hideProductCategory = null, 
            hideProductId = null, 
            hideProvider = null, 
            includeOthers = false, 
            includeProduct = false, 
            includeSupport = false, 
            includeUpdates = false, 
        ), 
        itemsPerPage = null, 
        itemsToSkip = null, 
        next = null, 
        sortBy = null, 
        sortDirection = null, 
    ),
    user
).orThrow()

/*
PageV2(
    items = listOf(FileCollection(
        createdAt = 1635151675465, 
        id = "54123", 
        owner = ResourceOwner(
            createdBy = "user", 
            project = null, 
        ), 
        permissions = ResourcePermissions(
            myself = listOf(Permission.ADMIN), 
            others = emptyList(), 
        ), 
        providerGeneratedId = null, 
        specification = FileCollection.Spec(
            product = ProductReference(
                category = "example-ssd", 
                id = "example-ssd", 
                provider = "example", 
            ), 
            title = "My Awesome Drive", 
        ), 
        status = FileCollection.Status(
            resolvedProduct = null, 
            resolvedSupport = null, 
        ), 
        updates = emptyList(), 
    )), 
    itemsPerPage = 50, 
    next = null, 
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------

# In this example, we will see how a user can rename one of their collections.

# 📝 NOTE: Renaming must be supported by the provider

# Authenticated as user
curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/collections/retrieveProducts" 

# {
#     "productsByProvider": {
#         "ucloud": [
#             {
#                 "product": {
#                     "balance": null,
#                     "maxUsableBalance": null,
#                     "name": "example-ssd",
#                     "pricePerUnit": 1,
#                     "category": {
#                         "name": "example-ssd",
#                         "provider": "example"
#                     },
#                     "description": "Fast storage",
#                     "priority": 0,
#                     "version": 1,
#                     "freeToUse": false,
#                     "allowAllocationRequestsFrom": "ALL",
#                     "unitOfPrice": "PER_UNIT",
#                     "chargeType": "DIFFERENTIAL_QUOTA",
#                     "hiddenInGrantApplications": false,
#                     "productType": "STORAGE"
#                 },
#                 "support": {
#                     "product": {
#                         "id": "example-ssd",
#                         "category": "example-ssd",
#                         "provider": "example"
#                     },
#                     "stats": {
#                         "sizeInBytes": null,
#                         "sizeIncludingChildrenInBytes": null,
#                         "modifiedAt": null,
#                         "createdAt": null,
#                         "accessedAt": null,
#                         "unixPermissions": null,
#                         "unixOwner": null,
#                         "unixGroup": null
#                     },
#                     "collection": {
#                         "aclModifiable": null,
#                         "usersCanCreate": true,
#                         "usersCanDelete": true,
#                         "usersCanRename": true
#                     },
#                     "files": {
#                         "aclModifiable": false,
#                         "trashSupported": false,
#                         "isReadOnly": false,
#                         "searchSupported": true,
#                         "streamingSearchSupported": false,
#                         "sharesSupported": false
#                     },
#                     "maintenance": null
#                 }
#             }
#         ]
#     }
# }

# As we can see, the provider does support the rename operation. We now look at our collections.

curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/collections/browse?includeOthers=false&includeUpdates=false&includeSupport=false&includeProduct=false" 

# {
#     "itemsPerPage": 50,
#     "items": [
#         {
#             "id": "54123",
#             "specification": {
#                 "title": "Home",
#                 "product": {
#                     "id": "example-ssd",
#                     "category": "example-ssd",
#                     "provider": "example"
#                 }
#             },
#             "createdAt": 1635151675465,
#             "status": {
#                 "resolvedSupport": null,
#                 "resolvedProduct": null
#             },
#             "updates": [
#             ],
#             "owner": {
#                 "createdBy": "user",
#                 "project": null
#             },
#             "permissions": {
#                 "myself": [
#                     "ADMIN"
#                 ],
#                 "others": [
#                 ]
#             },
#             "providerGeneratedId": null
#         }
#     ],
#     "next": null
# }

# Using the unique ID, we can rename the collection

curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/collections/rename" -d '{
    "items": [
        {
            "id": "54123",
            "newTitle": "My Awesome Drive"
        }
    ]
}'


# {
# }

# The new title is observed when we browse the collections one more time

curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/collections/browse?includeOthers=false&includeUpdates=false&includeSupport=false&includeProduct=false" 

# {
#     "itemsPerPage": 50,
#     "items": [
#         {
#             "id": "54123",
#             "specification": {
#                 "title": "My Awesome Drive",
#                 "product": {
#                     "id": "example-ssd",
#                     "category": "example-ssd",
#                     "provider": "example"
#                 }
#             },
#             "createdAt": 1635151675465,
#             "status": {
#                 "resolvedSupport": null,
#                 "resolvedProduct": null
#             },
#             "updates": [
#             ],
#             "owner": {
#                 "createdBy": "user",
#                 "project": null
#             },
#             "permissions": {
#                 "myself": [
#                     "ADMIN"
#                 ],
#                 "others": [
#                 ]
#             },
#             "providerGeneratedId": null
#         }
#     ],
#     "next": null
# }
Communication Flow: Visual

Remote Procedure Calls

browse

API: Stable Auth: Users

Browses the catalog of available resources

Request Response Error
ResourceBrowseRequest<FileCollectionIncludeFlags> PageV2<FileCollection> CommonErrorMessage

retrieve

API: Stable Auth: Users

Retrieve a single resource

Request Response Error
ResourceRetrieveRequest<FileCollectionIncludeFlags> FileCollection CommonErrorMessage

retrieveProducts

API: Stable Auth: Users

Retrieve product support for all accessible providers

Request Response Error
Unit SupportByProvider<Product.Storage, FSSupport> CommonErrorMessage

This endpoint will determine all providers that which the authenticated user has access to, in the current workspace. A user has access to a product, and thus a provider, if the product is either free or if the user has been granted credits to use the product.

See also:

create

API: Stable Auth: Users

Creates one or more resources

Request Response Error
BulkRequest<FileCollection.Spec> BulkResponse<FindByStringId> CommonErrorMessage

delete

API: Stable Auth: Users

Deletes one or more resources

Request Response Error
BulkRequest<FindByStringId> BulkResponse<Unit> CommonErrorMessage

init

API: Stable Auth: Users

Request (potential) initialization of resources

Request Response Error
Unit Unit CommonErrorMessage

This request is sent by the client, if the client believes that initialization of resources might be needed. NOTE: This request might be sent even if initialization has already taken place. UCloud/Core does not check if initialization has already taken place, it simply validates the request.

rename

API: Stable Auth: Users

Request renaming of FileCollection

Request Response Error
BulkRequest<FileCollectionsRenameRequestItem> Unit CommonErrorMessage

updateAcl

API: Stable Auth: Users

Updates the ACL attached to a resource

Request Response Error
BulkRequest<UpdatedAcl> BulkResponse<Unit> CommonErrorMessage

Data Models

FSCollectionSupport

API: Stable

Declares which FileCollection operations are supported for a product

data class FSCollectionSupport(
    val aclModifiable: Boolean?,
    val usersCanCreate: Boolean?,
    val usersCanDelete: Boolean?,
    val usersCanRename: Boolean?,
)
Properties
aclModifiable: Boolean?
usersCanCreate: Boolean?
usersCanDelete: Boolean?
usersCanRename: Boolean?

FSFileSupport

API: Stable

Declares which file-level operations a product supports

data class FSFileSupport(
    val aclModifiable: Boolean?,
    val trashSupported: Boolean?,
    val isReadOnly: Boolean?,
    val searchSupported: Boolean?,
    val streamingSearchSupported: Boolean?,
    val sharesSupported: Boolean?,
)
Properties
aclModifiable: Boolean?
trashSupported: Boolean?
isReadOnly: Boolean?
searchSupported: Boolean? Declares support for the normal search endpoint

NOTE(Dan, 01/09/2022): For backwards compatibility, this is true by default, however, this will likely change to false in a later release. Providers should explicitly declare support for this endpoint for the time being.

streamingSearchSupported: Boolean? Declares support for the streamingSearch endpoint
sharesSupported: Boolean?

FSProductStatsSupport

API: Stable

Declares which stats a given product supports

data class FSProductStatsSupport(
    val sizeInBytes: Boolean?,
    val sizeIncludingChildrenInBytes: Boolean?,
    val modifiedAt: Boolean?,
    val createdAt: Boolean?,
    val accessedAt: Boolean?,
    val unixPermissions: Boolean?,
    val unixOwner: Boolean?,
    val unixGroup: Boolean?,
)
Properties
sizeInBytes: Boolean?
sizeIncludingChildrenInBytes: Boolean?
modifiedAt: Boolean?
createdAt: Boolean?
accessedAt: Boolean?
unixPermissions: Boolean?
unixOwner: Boolean?
unixGroup: Boolean?

FSSupport

API: Stable

data class FSSupport(
    val product: ProductReference,
    val stats: FSProductStatsSupport?,
    val collection: FSCollectionSupport?,
    val files: FSFileSupport?,
    val maintenance: Maintenance?,
)
Properties
product: ProductReference
stats: FSProductStatsSupport?
collection: FSCollectionSupport?
files: FSFileSupport?
maintenance: Maintenance?

FileCollection

API: Stable

A Resource is the core data model used to synchronize tasks between UCloud and Provider.

data class FileCollection(
    val id: String,
    val specification: FileCollection.Spec,
    val createdAt: Long,
    val status: FileCollection.Status,
    val updates: List<FileCollection.Update>,
    val owner: ResourceOwner,
    val permissions: ResourcePermissions?,
    val providerGeneratedId: String?,
)

For more information go here.

Properties
id: String A unique identifier referencing the `Resource`

The ID is unique across a provider for a single resource type.

specification: FileCollection.Spec
createdAt: Long Timestamp referencing when the request for creation was received by UCloud
status: FileCollection.Status Holds the current status of the `Resource`
updates: List<FileCollection.Update> Contains a list of updates from the provider as well as UCloud

Updates provide a way for both UCloud, and the provider to communicate to the user what is happening with their resource.

owner: ResourceOwner Contains information about the original creator of the `Resource` along with project association
permissions: ResourcePermissions? Permissions assigned to this resource

A null value indicates that permissions are not supported by this resource type.

providerGeneratedId: String?

FileCollection.Spec

API: Stable

data class Spec(
    val title: String,
    val product: ProductReference,
)
Properties
title: String
product: ProductReference A reference to the product which backs this `Resource`

FileCollection.Status

API: Stable

Describes the current state of the Resource

data class Status(
    val resolvedSupport: ResolvedSupport<Product.Storage, FSSupport>?,
    val resolvedProduct: Product.Storage?,
)

The contents of this field depends almost entirely on the specific Resource that this field is managing. Typically, this will contain information such as:

  • A state value. For example, a compute Job might be RUNNING

  • Key metrics about the resource.

  • Related resources. For example, certain Resources are bound to another Resource in a mutually exclusive way, this should be listed in the status section.

Properties
resolvedSupport: ResolvedSupport<Product.Storage, FSSupport>?
resolvedProduct: Product.Storage? The resolved product referenced by `product`.

This attribute is not included by default unless includeProduct is specified.


FileCollection.Update

API: Stable

Describes an update to the Resource

data class Update(
    val timestamp: Long,
    val status: String?,
)

Updates can optionally be fetched for a Resource. The updates describe how the Resource changes state over time. The current state of a Resource can typically be read from its status field. Thus, it is typically not needed to use the full update history if you only wish to know the current state of a Resource.

An update will typically contain information similar to the status field, for example:

  • A state value. For example, a compute Job might be RUNNING.

  • Change in key metrics.

  • Bindings to related Resources.

Properties
timestamp: Long A timestamp referencing when UCloud received this update
status: String? A generic text message describing the current status of the `Resource`

FileCollectionIncludeFlags

API: Stable

data class FileCollectionIncludeFlags(
    val filterMemberFiles: MemberFilesFilter?,
    val includeOthers: Boolean?,
    val includeUpdates: Boolean?,
    val includeSupport: Boolean?,
    val includeProduct: Boolean?,
    val filterCreatedBy: String?,
    val filterCreatedAfter: Long?,
    val filterCreatedBefore: Long?,
    val filterProvider: String?,
    val filterProductId: String?,
    val filterProductCategory: String?,
    val filterProviderIds: String?,
    val filterIds: String?,
    val hideProductId: String?,
    val hideProductCategory: String?,
    val hideProvider: String?,
)
Properties
filterMemberFiles: MemberFilesFilter?
includeOthers: Boolean?
includeUpdates: Boolean?
includeSupport: Boolean?
includeProduct: Boolean? Includes `specification.resolvedProduct`
filterCreatedBy: String?
filterCreatedAfter: Long?
filterCreatedBefore: Long?
filterProvider: String?
filterProductId: String?
filterProductCategory: String?
filterProviderIds: String? Filters by the provider ID. The value is comma-separated.
filterIds: String? Filters by the resource ID. The value is comma-separated.
hideProductId: String?
hideProductCategory: String?
hideProvider: String?

MemberFilesFilter

API: Stable

Filter for member files.

enum class MemberFilesFilter {
    SHOW_ONLY_MINE,
    SHOW_ONLY_MEMBER_FILES,
    DONT_FILTER_COLLECTIONS,
}

A member files collection must use the following format to be recognized: “Member Files: $username”

Properties
SHOW_ONLY_MINE Shows only the requesting user's personal member file along with all other collections
SHOW_ONLY_MEMBER_FILES Shows only the member file collections and hides all others
DONT_FILTER_COLLECTIONS Applies no filter and shows both normal collections and member files

FileCollectionsRenameRequestItem

API: Stable

data class FileCollectionsRenameRequestItem(
    val id: String,
    val newTitle: String,
)
Properties
id: String
newTitle: String