Files¶
Files in UCloud is a resource for storing, retrieving and organizing data in UCloud.
Rationale¶
📝 NOTE: This API follows the standard Resources API. We recommend that you have already read and understood the concepts described here.
The file-system of UCloud provide researchers with a way of storing large data-sets efficiently and securely. The file-system is one of UCloud’s core features and almost all other features, either directly or indirectly, interact with it. For example:
All interactions in UCloud (including files) are automatically audited
UCloud allows compute
Jobs
to consume UCloud files. Either through containerized workloads or virtual machines.Authorization and project management
Powerful file metadata system for data management
A file in UCloud (UFile
) closely follows the concept of a computer file you might already be familiar with.
The functionality of a file is mostly determined by its type
. The two most important types are the
DIRECTORY
and FILE
types. A
DIRECTORY
is a container of UFile
s. A directory can itself contain more
directories, which leads to a natural tree-like structure. FILE
s, also referred to as a
regular files, are data records which each contain a series of bytes.
📝 Provider Note: This is the API exposed to end-users. See the table below for other relevant APIs.
End-User | Provider (Ingoing) | Control (Outgoing) |
---|---|---|
Files |
FilesProvider |
FilesControl |
Table of Contents¶
1. Examples
2. Remote Procedure Calls
Name | Description |
---|---|
browse |
Browses the contents of a directory. |
retrieve |
Retrieves information about a single file. |
retrieveProducts |
Retrieve product support for all accessible providers |
search |
Searches the catalog of available resources |
streamingSearch |
Searches through the files of a user in all accessible files |
copy |
Copies a file from one path to another |
createDownload |
Creates a download session between the user and the provider |
createFolder |
Creates one or more folders |
createUpload |
Creates an upload session between the user and the provider |
delete |
Permanently deletes one or more files |
emptyTrash |
Permanently deletes all files from the selected trash folder thereby emptying it |
init |
Request (potential) initialization of resources |
move |
Move a file from one path to another |
trash |
Moves a file to the trash |
updateAcl |
Updates the ACL of a single file. |
3. Data Models
Name | Description |
---|---|
UFile |
A [`UFile`](/docs/reference/dk.sdu.cloud.file.orchestrator.api.UFile.md) is a resource for storing, retrieving and organizing data in UCloud |
UFileStatus |
General system-level stats about a file |
FileIconHint |
A hint to clients about which icon should be used in user-interfaces when representing a `UFile` |
FileType |
The type of a `UFile` |
UFileSpecification |
|
FileMetadataOrDeleted.Deleted |
Indicates that the metadata document has been deleted is no longer in use |
FileMetadataHistory |
No description |
FileMetadataOrDeleted |
No description |
FilesStreamingSearchResult |
No description |
FilesStreamingSearchResult.EndOfResults |
No description |
FilesStreamingSearchResult.Result |
No description |
FindByPath |
No description |
LongRunningTask |
No description |
LongRunningTask.Complete |
No description |
LongRunningTask.ContinuesInBackground |
No description |
UFileIncludeFlags |
No description |
UFileUpdate |
Describes an update to the `Resource` |
UploadProtocol |
No description |
UploadType |
No description |
WriteConflictPolicy |
A policy for how UCloud should handle potential naming conflicts for certain operations (e.g. copy) |
FilesCopyRequestItem |
No description |
FilesCreateDownloadRequestItem |
No description |
FilesCreateFolderRequestItem |
No description |
FilesCreateUploadRequestItem |
No description |
FilesMoveRequestItem |
No description |
FilesStreamingSearchRequest |
No description |
FilesCreateDownloadResponseItem |
No description |
FilesCreateUploadResponseItem |
No description |
Example: Renaming a file¶
Frequency of use | Common |
---|---|
Trigger | User-initiated action, typically though the user-interface |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.move.call(
bulkRequestOf(FilesMoveRequestItem(
conflictPolicy = WriteConflictPolicy.REJECT,
newId = "/123/my/new_file",
oldId = "/123/my/file",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(LongRunningTask.Complete()),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/move" -d '{
"items": [
{
"oldId": "/123/my/file",
"newId": "/123/my/new_file",
"conflictPolicy": "REJECT"
}
]
}'
# {
# "responses": [
# {
# "type": "complete"
# }
# ]
# }
Communication Flow: Visual
Example: Copying a file to itself¶
Frequency of use | Common |
---|---|
Trigger | User-initiated action, typically through the user-interface |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.copy.call(
bulkRequestOf(FilesCopyRequestItem(
conflictPolicy = WriteConflictPolicy.RENAME,
newId = "/123/my/file",
oldId = "/123/my/file",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(LongRunningTask.Complete()),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/copy" -d '{
"items": [
{
"oldId": "/123/my/file",
"newId": "/123/my/file",
"conflictPolicy": "RENAME"
}
]
}'
# {
# "responses": [
# {
# "type": "complete"
# }
# ]
# }
Communication Flow: Visual
Example: Uploading a file¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.createUpload.call(
bulkRequestOf(FilesCreateUploadRequestItem(
conflictPolicy = WriteConflictPolicy.REJECT,
id = "/123/folder",
supportedProtocols = listOf(UploadProtocol.CHUNKED),
type = UploadType.FILE,
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(FilesCreateUploadResponseItem(
endpoint = "https://provider.example.com/ucloud/example-provider/chunked",
protocol = UploadProtocol.CHUNKED,
token = "f1460d47e583653f7723204e5ff3f50bad91a658",
)),
)
*/
/* The user can now proceed to upload using the chunked protocol at the provided endpoint */
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/upload" -d '{
"items": [
{
"id": "/123/folder",
"type": "FILE",
"supportedProtocols": [
"CHUNKED"
],
"conflictPolicy": "REJECT"
}
]
}'
# {
# "responses": [
# {
# "endpoint": "https://provider.example.com/ucloud/example-provider/chunked",
# "protocol": "CHUNKED",
# "token": "f1460d47e583653f7723204e5ff3f50bad91a658"
# }
# ]
# }
# The user can now proceed to upload using the chunked protocol at the provided endpoint
Communication Flow: Visual
Example: Downloading a file¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.createDownload.call(
bulkRequestOf(FilesCreateDownloadRequestItem(
id = "/123/folder/file",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(FilesCreateDownloadResponseItem(
endpoint = "https://provider.example.com/ucloud/example-provider/download?token=d293435e94734c91394f17bb56268d3161c7f069",
)),
)
*/
/* The user can now download the file through normal HTTP(s) GET at the provided endpoint */
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/download" -d '{
"items": [
{
"id": "/123/folder/file"
}
]
}'
# {
# "responses": [
# {
# "endpoint": "https://provider.example.com/ucloud/example-provider/download?token=d293435e94734c91394f17bb56268d3161c7f069"
# }
# ]
# }
# The user can now download the file through normal HTTP(s) GET at the provided endpoint
Communication Flow: Visual
Example: Creating a folder¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.createFolder.call(
bulkRequestOf(FilesCreateFolderRequestItem(
conflictPolicy = WriteConflictPolicy.REJECT,
id = "/123/folder/a",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(LongRunningTask.Complete()),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/folder" -d '{
"items": [
{
"id": "/123/folder/a",
"conflictPolicy": "REJECT"
}
]
}'
# {
# "responses": [
# {
# "type": "complete"
# }
# ]
# }
Communication Flow: Visual
Example: Moving multiple files to trash¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.trash.call(
bulkRequestOf(FindByPath(
id = "/123/folder",
), FindByPath(
id = "/123/file",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(LongRunningTask.Complete(), LongRunningTask.Complete()),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/trash" -d '{
"items": [
{
"id": "/123/folder"
},
{
"id": "/123/file"
}
]
}'
# {
# "responses": [
# {
# "type": "complete"
# },
# {
# "type": "complete"
# }
# ]
# }
Communication Flow: Visual
Example: Emptying trash folder¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Post-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.trash.call(
bulkRequestOf(FindByPath(
id = "/home/trash",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(LongRunningTask.Complete()),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XPOST -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files/trash" -d '{
"items": [
{
"id": "/home/trash"
}
]
}'
# {
# "responses": [
# {
# "type": "complete"
# }
# ]
# }
Communication Flow: Visual
Example: Browsing the contents of a folder¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.browse.call(
ResourceBrowseRequest(
consistency = null,
flags = UFileIncludeFlags(
allowUnsupportedInclude = null,
filterByFileExtension = null,
filterCreatedAfter = null,
filterCreatedBefore = null,
filterCreatedBy = null,
filterHiddenFiles = false,
filterIds = null,
filterProductCategory = null,
filterProductId = null,
filterProvider = null,
filterProviderIds = null,
hideProductCategory = null,
hideProductId = null,
hideProvider = null,
includeMetadata = null,
includeOthers = false,
includePermissions = null,
includeProduct = false,
includeSizes = null,
includeSupport = false,
includeTimestamps = true,
includeUnixInfo = null,
includeUpdates = false,
path = null,
),
itemsPerPage = null,
itemsToSkip = null,
next = null,
sortBy = null,
sortDirection = null,
),
user
).orThrow()
/*
PageV2(
items = listOf(UFile(
createdAt = 1632903417165,
id = "/123/folder/file.txt",
owner = ResourceOwner(
createdBy = "user",
project = "f63919cd-60d3-45d3-926b-0246dcc697fd",
),
permissions = null,
specification = UFileSpecification(
collection = "123",
product = ProductReference(
category = "u1-cephfs",
id = "u1-cephfs",
provider = "ucloud",
),
),
status = UFileStatus(
accessedAt = null,
icon = null,
metadata = null,
modifiedAt = 1632903417165,
resolvedProduct = null,
resolvedSupport = null,
sizeInBytes = null,
sizeIncludingChildrenInBytes = null,
type = FileType.FILE,
unixGroup = null,
unixMode = null,
unixOwner = null,
),
updates = emptyList(),
providerGeneratedId = "/123/folder/file.txt",
)),
itemsPerPage = 50,
next = null,
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/browse?includeOthers=false&includeUpdates=false&includeSupport=false&includeProduct=false&includeTimestamps=true&filterHiddenFiles=false"
# {
# "itemsPerPage": 50,
# "items": [
# {
# "id": "/123/folder/file.txt",
# "specification": {
# "collection": "123",
# "product": {
# "id": "u1-cephfs",
# "category": "u1-cephfs",
# "provider": "ucloud"
# }
# },
# "createdAt": 1632903417165,
# "status": {
# "type": "FILE",
# "icon": null,
# "sizeInBytes": null,
# "sizeIncludingChildrenInBytes": null,
# "modifiedAt": 1632903417165,
# "accessedAt": null,
# "unixMode": null,
# "unixOwner": null,
# "unixGroup": null,
# "metadata": null,
# "resolvedSupport": null,
# "resolvedProduct": null
# },
# "owner": {
# "createdBy": "user",
# "project": "f63919cd-60d3-45d3-926b-0246dcc697fd"
# },
# "permissions": null,
# "updates": [
# ]
# }
# ],
# "next": null
# }
Communication Flow: Visual
Example: Retrieving a single file¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.retrieve.call(
ResourceRetrieveRequest(
flags = UFileIncludeFlags(
allowUnsupportedInclude = null,
filterByFileExtension = null,
filterCreatedAfter = null,
filterCreatedBefore = null,
filterCreatedBy = null,
filterHiddenFiles = false,
filterIds = null,
filterProductCategory = null,
filterProductId = null,
filterProvider = null,
filterProviderIds = null,
hideProductCategory = null,
hideProductId = null,
hideProvider = null,
includeMetadata = null,
includeOthers = false,
includePermissions = null,
includeProduct = false,
includeSizes = null,
includeSupport = false,
includeTimestamps = true,
includeUnixInfo = null,
includeUpdates = false,
path = null,
),
id = "/123/folder",
),
user
).orThrow()
/*
UFile(
createdAt = 1632903417165,
id = "/123/folder",
owner = ResourceOwner(
createdBy = "user",
project = "f63919cd-60d3-45d3-926b-0246dcc697fd",
),
permissions = null,
specification = UFileSpecification(
collection = "123",
product = ProductReference(
category = "u1-cephfs",
id = "u1-cephfs",
provider = "ucloud",
),
),
status = UFileStatus(
accessedAt = null,
icon = null,
metadata = null,
modifiedAt = 1632903417165,
resolvedProduct = null,
resolvedSupport = null,
sizeInBytes = null,
sizeIncludingChildrenInBytes = null,
type = FileType.DIRECTORY,
unixGroup = null,
unixMode = null,
unixOwner = null,
),
updates = emptyList(),
providerGeneratedId = "/123/folder",
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/retrieve?includeOthers=false&includeUpdates=false&includeSupport=false&includeProduct=false&includeTimestamps=true&filterHiddenFiles=false&id=/123/folder"
# {
# "id": "/123/folder",
# "specification": {
# "collection": "123",
# "product": {
# "id": "u1-cephfs",
# "category": "u1-cephfs",
# "provider": "ucloud"
# }
# },
# "createdAt": 1632903417165,
# "status": {
# "type": "DIRECTORY",
# "icon": null,
# "sizeInBytes": null,
# "sizeIncludingChildrenInBytes": null,
# "modifiedAt": 1632903417165,
# "accessedAt": null,
# "unixMode": null,
# "unixOwner": null,
# "unixGroup": null,
# "metadata": null,
# "resolvedSupport": null,
# "resolvedProduct": null
# },
# "owner": {
# "createdBy": "user",
# "project": "f63919cd-60d3-45d3-926b-0246dcc697fd"
# },
# "permissions": null,
# "updates": [
# ]
# }
Communication Flow: Visual
Example: Deleting a file permanently¶
Frequency of use | Common |
---|---|
Trigger | User initiated |
Pre-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.delete.call(
bulkRequestOf(FindByStringId(
id = "/123/folder",
)),
user
).orThrow()
/*
BulkResponse(
responses = listOf(Unit),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XDELETE -H "Authorization: Bearer $accessToken" -H "Content-Type: content-type: application/json; charset=utf-8" "$host/api/files" -d '{
"items": [
{
"id": "/123/folder"
}
]
}'
# {
# "responses": [
# {
# }
# ]
# }
Communication Flow: Visual
Example: Retrieving a list of products supported by accessible providers¶
Frequency of use | Common |
---|---|
Trigger | Typically triggered by a client to determine which operations are supported |
Pre-conditions |
|
Actors |
|
Communication Flow: Kotlin
Files.retrieveProducts.call(
Unit,
user
).orThrow()
/*
SupportByProvider(
productsByProvider = mapOf("ucloud" to listOf(ResolvedSupport(
product = Product.Storage(
allowAllocationRequestsFrom = AllocationRequestsGroup.ALL,
category = ProductCategoryId(
id = "u1-cephfs",
name = "u1-cephfs",
provider = "ucloud",
),
chargeType = ChargeType.DIFFERENTIAL_QUOTA,
description = "Storage provided by UCloud",
freeToUse = false,
hiddenInGrantApplications = false,
name = "u1-cephfs",
pricePerUnit = 1,
priority = 0,
productType = ProductType.STORAGE,
unitOfPrice = ProductPriceUnit.PER_UNIT,
version = 1,
balance = null,
id = "u1-cephfs",
maxUsableBalance = null,
),
support = FSSupport(
collection = FSCollectionSupport(
aclModifiable = false,
usersCanCreate = true,
usersCanDelete = true,
usersCanRename = true,
),
files = FSFileSupport(
aclModifiable = false,
isReadOnly = false,
searchSupported = true,
sharesSupported = true,
streamingSearchSupported = false,
trashSupported = true,
),
maintenance = null,
product = ProductReference(
category = "u1-cephfs",
id = "u1-cephfs",
provider = "ucloud",
),
stats = FSProductStatsSupport(
accessedAt = false,
createdAt = true,
modifiedAt = true,
sizeInBytes = true,
sizeIncludingChildrenInBytes = true,
unixGroup = true,
unixOwner = true,
unixPermissions = true,
),
),
))),
)
*/
Communication Flow: Curl
# ------------------------------------------------------------------------------------------------------
# $host is the UCloud instance to contact. Example: 'http://localhost:8080' or 'https://cloud.sdu.dk'
# $accessToken is a valid access-token issued by UCloud
# ------------------------------------------------------------------------------------------------------
# Authenticated as user
curl -XGET -H "Authorization: Bearer $accessToken" "$host/api/files/retrieveProducts"
# {
# "productsByProvider": {
# "ucloud": [
# {
# "product": {
# "balance": null,
# "maxUsableBalance": null,
# "name": "u1-cephfs",
# "pricePerUnit": 1,
# "category": {
# "name": "u1-cephfs",
# "provider": "ucloud"
# },
# "description": "Storage provided by UCloud",
# "priority": 0,
# "version": 1,
# "freeToUse": false,
# "allowAllocationRequestsFrom": "ALL",
# "unitOfPrice": "PER_UNIT",
# "chargeType": "DIFFERENTIAL_QUOTA",
# "hiddenInGrantApplications": false,
# "productType": "STORAGE"
# },
# "support": {
# "product": {
# "id": "u1-cephfs",
# "category": "u1-cephfs",
# "provider": "ucloud"
# },
# "stats": {
# "sizeInBytes": true,
# "sizeIncludingChildrenInBytes": true,
# "modifiedAt": true,
# "createdAt": true,
# "accessedAt": false,
# "unixPermissions": true,
# "unixOwner": true,
# "unixGroup": true
# },
# "collection": {
# "aclModifiable": false,
# "usersCanCreate": true,
# "usersCanDelete": true,
# "usersCanRename": true
# },
# "files": {
# "aclModifiable": false,
# "trashSupported": true,
# "isReadOnly": false,
# "searchSupported": true,
# "streamingSearchSupported": false,
# "sharesSupported": true
# },
# "maintenance": null
# }
# }
# ]
# }
# }
Communication Flow: Visual
Remote Procedure Calls¶
browse
¶
Browses the contents of a directory.
Request | Response | Error |
---|---|---|
ResourceBrowseRequest<UFileIncludeFlags> |
PageV2<UFile> |
CommonErrorMessage |
The results will be returned using the standard pagination API of UCloud. Consistency is slightly relaxed for this endpoint as it is typically hard to enforce for filesystems. Provider’s are heavily encouraged to try and find all files on the first request and return information about them in subsequent requests. For example, a client might list all file names in the initial request and use this list for all subsequent requests and retrieve additional information about the files. If the files no longer exist then the provider should simply not include these results.
retrieve
¶
Retrieves information about a single file.
Request | Response | Error |
---|---|---|
ResourceRetrieveRequest<UFileIncludeFlags> |
UFile |
CommonErrorMessage |
This file can be of any type. Clients can request additional information about the file using the
include*
flags of the request. Note that not all providers support all information. Clients can query
this information using files.collections.browse
or
files.collections.retrieve
with the includeSupport
flag.
retrieveProducts
¶
Retrieve product support for all accessible providers
Request | Response | Error |
---|---|---|
Unit |
SupportByProvider<Product.Storage, FSSupport> |
CommonErrorMessage |
This endpoint will determine all providers that which the authenticated user has access to, in the current workspace. A user has access to a product, and thus a provider, if the product is either free or if the user has been granted credits to use the product.
See also:
search
¶
Searches the catalog of available resources
Request | Response | Error |
---|---|---|
ResourceSearchRequest<UFileIncludeFlags> |
PageV2<UFile> |
CommonErrorMessage |
streamingSearch
¶
Searches through the files of a user in all accessible files
Request | Response | Error |
---|---|---|
FilesStreamingSearchRequest |
FilesStreamingSearchResult |
CommonErrorMessage |
This endpoint uses a specialized API for returning search results in a streaming fashion. In all other ways, this endpoint is identical to the normal search API.
This endpoint can be used instead of the normal search API as it will contact providers using the non-streaming version if they do not support it. In such a case, the core will retrieve multiple pages in order to stream in more content.
Clients should expect that this endpoint stops returning results after a given timeout. After which, it is no longer possible to request additional results.
copy
¶
Copies a file from one path to another
Request | Response | Error |
---|---|---|
BulkRequest<FilesCopyRequestItem> |
BulkResponse<LongRunningTask> |
CommonErrorMessage |
The file can be of any type. If a directory is chosen then this will recursively copy all of its children. This request might fail half-way through. This can potentially lead to a situation where a partial file is left on the file-system. It is left to the user to clean up this file.
This operation handles conflicts depending on the supplied WriteConflictPolicy
.
This is a long running task. As a result, this operation might respond with a status code which indicate that it will continue in the background. Progress of this job can be followed using the task API.
UCloud applied metadata will not be copied to the new file. File-system metadata (e.g. extended-attributes) may be moved, however this is provider dependant.
Errors:
Status Code | Description |
---|---|
400 Bad Request |
The operation couldn't be completed because of the write conflict policy |
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
Examples:
Example |
---|
Example of duplicating a file |
createDownload
¶
Creates a download session between the user and the provider
Request | Response | Error |
---|---|---|
BulkRequest<FilesCreateDownloadRequestItem> |
BulkResponse<FilesCreateDownloadResponseItem> |
CommonErrorMessage |
The returned endpoint will respond with a download to the user.
Errors:
Status Code | Description |
---|---|
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
Examples:
Example |
---|
Downloading a file |
createFolder
¶
Creates one or more folders
Request | Response | Error |
---|---|---|
BulkRequest<FilesCreateFolderRequestItem> |
BulkResponse<LongRunningTask> |
CommonErrorMessage |
This folder will automatically create parent directories if needed. This request may fail half-way through and leave the file-system in an inconsistent state. It is up to the user to clean this up.
Errors:
Status Code | Description |
---|---|
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
Examples:
Example |
---|
Creating a folder |
createUpload
¶
Creates an upload session between the user and the provider
Request | Response | Error |
---|---|---|
BulkRequest<FilesCreateUploadRequestItem> |
BulkResponse<FilesCreateUploadResponseItem> |
CommonErrorMessage |
An upload can be either a file or folder, if supported by the provider, and depending on the
UploadTypespecified
in the request. The desired path and a list of supported UploadProtocol
s
are also specified in the request. The latter is used by the provider to negotiate which protocol to use.
The response will contain an endpoint which is ready to accept the upload, as well as the chosen
UploadProtocol
and a unique token.
At the time of writing the default and preferred protocol is UploadProtocol.WEBSOCKET
.
Errors:
Status Code | Description |
---|---|
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
Examples:
Example |
---|
Uploading a file with the chunked protocol |
delete
¶
Permanently deletes one or more files
Request | Response | Error |
---|---|---|
BulkRequest<FindByStringId> |
BulkResponse<Unit> |
CommonErrorMessage |
This call will recursively delete files if needed. It is possible that a provider might fail to
completely delete the entire sub-tree. This can, for example, happen because of a crash or because the
file-system is unable to delete a given file. This will lead the file-system in an inconsistent state.
It is not guaranteed that the provider will be able to detect this error scenario. A client of the
API can check if the file has been deleted by calling retrieve
on the file.
emptyTrash
¶
Permanently deletes all files from the selected trash folder thereby emptying it
Request | Response | Error |
---|---|---|
BulkRequest<FindByPath> |
BulkResponse<LongRunningTask> |
CommonErrorMessage |
This operation acts as a permanent delete for users. Users will NOT be able to restore the file later, if needed.
Not all providers supports this endpoint. You can query files.collections.browse
or files.collections.retrieve
with the includeSupport
flag.
This is a long running task. As a result, this operation might respond with a status code which indicate that it will continue in the background. Progress of this job can be followed using the task API.
Errors:
Status Code | Description |
---|---|
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
400 Bad Request |
This operation is not supported by the provider |
Examples:
Example |
---|
Moving files to trash |
init
¶
Request (potential) initialization of resources
Request | Response | Error |
---|---|---|
Unit |
Unit |
CommonErrorMessage |
This request is sent by the client, if the client believes that initialization of resources might be needed. NOTE: This request might be sent even if initialization has already taken place. UCloud/Core does not check if initialization has already taken place, it simply validates the request.
move
¶
Move a file from one path to another
Request | Response | Error |
---|---|---|
BulkRequest<FilesMoveRequestItem> |
BulkResponse<LongRunningTask> |
CommonErrorMessage |
The file can be of any type. This request is also used for ‘renames’ of a file. This is simply
considered a move within a single directory. This operation handles conflicts depending on the supplied
WriteConflictPolicy
.
This is a long running task. As a result, this operation might respond with a status code which indicate that it will continue in the background. Progress of this job can be followed using the task API.
Errors:
Status Code | Description |
---|---|
400 Bad Request |
The operation couldn't be completed because of the write conflict policy |
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
Examples:
Example |
---|
Example of using move to rename a file |
trash
¶
Moves a file to the trash
Request | Response | Error |
---|---|---|
BulkRequest<FindByPath> |
BulkResponse<LongRunningTask> |
CommonErrorMessage |
This operation acts as a non-permanent delete for users. Users will be able to restore the file from trash later, if needed. It is up to the provider to determine if the trash should be automatically deleted and where this trash should be stored.
Not all providers supports this endpoint. You can query files.collections.browse
or files.collections.retrieve
with the includeSupport
flag.
This is a long running task. As a result, this operation might respond with a status code which indicate that it will continue in the background. Progress of this job can be followed using the task API.
Errors:
Status Code | Description |
---|---|
404 Not Found |
Either the oldPath or newPath exists or you lack permissions |
403 Forbidden |
You lack permissions to perform this operation |
400 Bad Request |
This operation is not supported by the provider |
Examples:
Example |
---|
Moving files to trash |
updateAcl
¶
Updates the ACL of a single file.
Request | Response | Error |
---|---|---|
BulkRequest<UpdatedAcl> |
BulkResponse<Unit> |
CommonErrorMessage |
⚠️ WARNING: No providers currently support this API. Instead use the
files.collections.updateAcl
endpoint.
Data Models¶
UFile
¶
A UFile
is a resource for storing, retrieving and organizing data in UCloud
data class UFile(
val id: String,
val specification: UFileSpecification,
val createdAt: Long,
val status: UFileStatus,
val owner: ResourceOwner,
val permissions: ResourcePermissions?,
val updates: List<UFileUpdate>,
val providerGeneratedId: String?,
)
A file in UCloud (UFile
) closely follows the concept of a computer file you might already be familiar with.
The functionality of a file is mostly determined by its type
. The two most important
types are the DIRECTORY
and FILE
types. A
DIRECTORY
is a container of UFile
s. A directory can itself contain more
directories, which leads to a natural tree-like structure. FILE
s, also referred to as a
regular files, are data records which each contain a series of bytes.
All files in UCloud have a name associated with them. This name uniquely identifies them within their directory. All files in UCloud belong to exactly one directory.
File operations must be able to reference the files on which they operate. In UCloud, these references are made through
the id
property, also known as a path. Paths use the tree-like structure of files to reference a file, it does so by
declaring which directories to go through, starting at the top, to reach the file we are referencing. This information
is serialized as a textual string, where each step of the path is separated by forward-slash /
(U+002F
). The path
must start with a single forward-slash, which signifies the root of the file tree. UCloud never users ‘relative’ file
paths, which some systems use.
All files in UCloud additionally have metadata associated with them. For this we differentiate between system-level metadata and user-defined metadata.
We have just covered two examples of system-level metadata, the id
(path) and
type
. UCloud additionally supports metadata such as general
stats about the files, such as file sizes. All files have a set of
permissions
associated with them, providers may optionally expose this information to
UCloud and the users.
User-defined metadata describe the contents of a file. All metadata is described by a template
(FileMetadataTemplate
), this template defines a document structure for the metadata. User-defined metadata
can be used for a variety of purposes, such as: Datacite metadata, sensitivity levels,
and other field specific metadata formats.
Properties
id
: String
A unique reference to a file
String
All files in UCloud have a name
associated with them. This name uniquely identifies them within their directory. All
files in UCloud belong to exactly one directory. A name
can be any textual string, for example: thesis-42.docx
.
However, certain restrictions apply to file name
s, see below for a concrete list of rules and recommendations.
The extension
of a file is typically used as a hint to clients how to treat a specific file. For example, an extension
might indicate that the file contains a video of a specific format. In UCloud, the file’s extension
is derived from
its name
. In UCloud, it is simply defined as the text immediately following, and not including, the last
period .
(U+002E
). The table below shows some examples of how UCloud determines the extension of a file:
File name |
Derived extension |
Comment |
---|---|---|
thesis-42.docx |
docx |
- |
thesis-43-final.tar |
tar |
- |
thesis-43-FINAL2.tar.gz |
gz |
Note that UCloud does not recognize tar as being part of the extension |
thesis |
Empty string | |
.ssh |
ssh |
'Hidden' files also have a surprising extension in UCloud |
File operations must be able to reference the files on which they operate. In UCloud, these references are made through
the path
property. Paths use the tree-like structure of files to reference a file, it does so by declaring which
directories to go through, starting at the top, to reach the file we are referencing. This information is serialized as
a textual string, where each step of the path is separated by forward-slash /
(U+002F
). The path must start with a
single forward-slash, which signifies the root of the file tree. UCloud never users ‘relative’ file paths, which some
systems use.
A path in UCloud is structured in such a way that they are unique across all providers and file systems. The figure below shows how a UCloud path is structured, and how it can be mapped to an internal file-system path.
Figure: At the top, a UCloud path along with the components of it. At the bottom, an example of an internal, provider specific, file-system path.
The figure shows how a UCloud path consists of four components:
The ‘Provider ID’ references the provider who owns and hosts the file
The product reference, this references the product that is hosting the
FileCollection
The
FileCollection
ID references the ID of the internal file collection. These are controlled by the provider and match the different types of file-systems they have available. A single file collection typically maps to a specific folder on the provider’s file-system.The internal path, which tells the provider how to find the file within the collection. Providers can typically pass this as a one-to-one mapping.
Rules of a file name
:
The
name
cannot be equal to.
(commonly interpreted to mean the current directory)The
name
cannot be equal to..
(commonly interpreted to mean the parent directory)The
name
cannot contain a forward-slash/
(U+002F
)Names are strictly unicode
UCloud will normalize a path which contain .
or ..
in a path’s step. It is normalized according to the comments
mentioned in rule 1 and 2.
Note that all paths in unicode are strictly unicode (rule 4). This is different from the unix standard. Unix file names can contain arbitrary binary data. (TODO determine how providers should handle this edge-case)
Additionally regarding file name
s, UCloud recommends to users the following:
Avoid the following file names:
Containing Windows reserved characters:
<
,>
,:
,"
,/
,|
,?
,*
,\
Any of the reserved file names in Windows:
AUX
COM1
,COM2
,COM3
,COM4
,COM5
,COM6
,COM7
,COM8
,COM9
CON
LPT1
,LPT2
,LPT3
,LPT4
,LPT5
,LPT6
,LPT7
,LPT8
,LPT9
NUL
PRN
Any of the above followed by an extension
Avoid ASCII control characters (decimal value 0-31 both inclusive)
Avoid Unicode control characters (e.g. right-to-left override)
Avoid line breaks, paragraph separators and other unicode separators which is typically interpreted as a line-break
Avoid binary names
UCloud will attempt to reject these for file operations initiated through the client, but it cannot ensure that these files do not appear regardless. This is due to the fact that the file systems are typically mounted directly by user-controlled jobs.
Rules of a file path
:
All paths must be absolute, that is they must start with
/
UCloud will normalize all path ‘steps’ containing either
.
or..
Additionally UCloud recommends to users the following regarding path
s:
Avoid long paths:
Older versions of Unixes report
PATH_MAX
as 1024Newer versions of Unixes report
PATH_MAX
as 4096Older versions of Windows start failing above 256 characters
specification
: UFileSpecification
UFileSpecification
createdAt
: Long
Timestamp referencing when the request for creation was received by UCloud
Long
status
: UFileStatus
Holds the current status of the `Resource`
UFileStatus
owner
: ResourceOwner
Contains information about the original creator of the `Resource` along with project association
ResourceOwner
permissions
: ResourcePermissions?
Permissions assigned to this resource
ResourcePermissions?
A null value indicates that permissions are not supported by this resource type.
updates
: List<UFileUpdate>
List<UFileUpdate>
providerGeneratedId
: String?
String?
UFileStatus
¶
General system-level stats about a file
data class UFileStatus(
val type: FileType?,
val icon: FileIconHint?,
val sizeInBytes: Long?,
val sizeIncludingChildrenInBytes: Long?,
val modifiedAt: Long?,
val accessedAt: Long?,
val unixMode: Int?,
val unixOwner: Int?,
val unixGroup: Int?,
val metadata: FileMetadataHistory?,
val resolvedSupport: ResolvedSupport<Product.Storage, FSSupport>?,
val resolvedProduct: Product.Storage?,
)
Properties
type
: FileType?
Which type of file this is, see `FileType` for more information.
FileType?
icon
: FileIconHint?
A hint to clients about which icon to display next to this file. See `FileIconHint` for details.
FileIconHint?
sizeInBytes
: Long?
The size of this file in bytes (Requires `includeSizes`)
Long?
sizeIncludingChildrenInBytes
: Long?
The size of this file and any child (Requires `includeSizes`)
Long?
modifiedAt
: Long?
The modified at timestamp (Requires `includeTimestamps`)
Long?
accessedAt
: Long?
The accessed at timestamp (Requires `includeTimestamps`)
Long?
unixMode
: Int?
The unix mode of a file (Requires `includeUnixInfo`
Int?
unixOwner
: Int?
The unix owner of a file as a UID (Requires `includeUnixInfo`)
Int?
unixGroup
: Int?
The unix group of a file as a GID (Requires `includeUnixInfo`)
Int?
metadata
: FileMetadataHistory?
User-defined metadata for this file. See `FileMetadataTemplate` for details.
FileMetadataHistory?
resolvedSupport
: ResolvedSupport<Product.Storage, FSSupport>?
ResolvedSupport<Product.Storage, FSSupport>?
resolvedProduct
: Product.Storage?
The resolved product referenced by `product`.
Product.Storage?
This attribute is not included by default unless includeProduct
is specified.
FileIconHint
¶
A hint to clients about which icon should be used in user-interfaces when representing a UFile
enum class FileIconHint {
DIRECTORY_STAR,
DIRECTORY_SHARES,
DIRECTORY_TRASH,
DIRECTORY_JOBS,
}
Properties
DIRECTORY_STAR
A directory containing 'starred' items
DIRECTORY_SHARES
A directory which contains items that are shared between users and projects
DIRECTORY_TRASH
A directory which contains items that have been 'trashed'
DIRECTORY_JOBS
A directory which contains items that are related to job results
FileType
¶
The type of a UFile
enum class FileType {
FILE,
DIRECTORY,
SOFT_LINK,
DANGLING_METADATA,
}
UFileSpecification
¶
__
data class UFileSpecification(
val collection: String,
val product: ProductReference,
)
Properties
collection
: String
String
product
: ProductReference
A reference to the product which backs this `Resource`
ProductReference
FileMetadataOrDeleted.Deleted
¶
Indicates that the metadata document has been deleted is no longer in use
data class Deleted(
val id: String,
val changeLog: String,
val createdAt: Long,
val createdBy: String,
val status: FileMetadataDocument.Status,
val type: String /* "deleted" */,
)
Properties
id
: String
String
changeLog
: String
Reason for this change
String
createdAt
: Long
Timestamp indicating when this change was made
Long
createdBy
: String
A reference to the user who made this change
String
status
: FileMetadataDocument.Status
FileMetadataDocument.Status
FileMetadataHistory
¶
data class FileMetadataHistory(
val templates: JsonObject,
val metadata: JsonObject,
)
FileMetadataOrDeleted
¶
sealed class FileMetadataOrDeleted {
abstract val createdAt: Long
abstract val createdBy: String
abstract val id: String
abstract val status: FileMetadataDocument.Status
class FileMetadataDocument : FileMetadataOrDeleted()
class Deleted : FileMetadataOrDeleted()
}
FilesStreamingSearchResult
¶
sealed class FilesStreamingSearchResult {
class EndOfResults : FilesStreamingSearchResult()
class Result : FilesStreamingSearchResult()
}
FilesStreamingSearchResult.EndOfResults
¶
data class EndOfResults(
val type: String /* "end_of_results" */,
)
Properties
type
: String /* "end_of_results" */
The type discriminator
String /* "end_of_results" */
FilesStreamingSearchResult.Result
¶
data class Result(
val batch: List<UFile>,
val type: String /* "result" */,
)
FindByPath
¶
data class FindByPath(
val id: String,
)
Properties
id
: String
String
LongRunningTask
¶
sealed class LongRunningTask {
class Complete : LongRunningTask()
class ContinuesInBackground : LongRunningTask()
}
LongRunningTask.Complete
¶
data class Complete(
val type: String /* "complete" */,
)
Properties
type
: String /* "complete" */
The type discriminator
String /* "complete" */
LongRunningTask.ContinuesInBackground
¶
data class ContinuesInBackground(
val taskId: String,
val type: String /* "continues_in_background" */,
)
UFileIncludeFlags
¶
data class UFileIncludeFlags(
val includeOthers: Boolean?,
val includeUpdates: Boolean?,
val includeSupport: Boolean?,
val includeProduct: Boolean?,
val includePermissions: Boolean?,
val includeTimestamps: Boolean?,
val includeSizes: Boolean?,
val includeUnixInfo: Boolean?,
val includeMetadata: Boolean?,
val filterCreatedBy: String?,
val filterCreatedAfter: Long?,
val filterCreatedBefore: Long?,
val filterProvider: String?,
val filterProductId: String?,
val filterProductCategory: String?,
val filterProviderIds: String?,
val filterByFileExtension: String?,
val path: String?,
val allowUnsupportedInclude: Boolean?,
val filterHiddenFiles: Boolean?,
val filterIds: String?,
val hideProductId: String?,
val hideProductCategory: String?,
val hideProvider: String?,
)
Properties
includeOthers
: Boolean?
Boolean?
includeUpdates
: Boolean?
Boolean?
includeSupport
: Boolean?
Boolean?
includeProduct
: Boolean?
Includes `specification.resolvedProduct`
Boolean?
includePermissions
: Boolean?
Boolean?
includeTimestamps
: Boolean?
Boolean?
includeSizes
: Boolean?
Boolean?
includeUnixInfo
: Boolean?
Boolean?
includeMetadata
: Boolean?
Boolean?
filterCreatedBy
: String?
String?
filterCreatedAfter
: Long?
Long?
filterCreatedBefore
: Long?
Long?
filterProvider
: String?
String?
filterProductId
: String?
String?
filterProductCategory
: String?
String?
filterProviderIds
: String?
Filters by the provider ID. The value is comma-separated.
String?
filterByFileExtension
: String?
String?
path
: String?
Path filter
String?
allowUnsupportedInclude
: Boolean?
Determines if the request should succeed if the underlying system does not support this data.
Boolean?
This value is true
by default
filterHiddenFiles
: Boolean?
Determines if dot files should be hidden from the result-set
Boolean?
filterIds
: String?
Filters by the resource ID. The value is comma-separated.
String?
hideProductId
: String?
String?
hideProductCategory
: String?
String?
hideProvider
: String?
String?
UFileUpdate
¶
Describes an update to the Resource
data class UFileUpdate(
val timestamp: Long,
val status: String?,
)
Updates can optionally be fetched for a Resource
. The updates describe how the Resource
changes state over time.
The current state of a Resource
can typically be read from its status
field. Thus, it is typically not needed to
use the full update history if you only wish to know the current state of a Resource
.
An update will typically contain information similar to the status
field, for example:
A state value. For example, a compute
Job
might beRUNNING
.Change in key metrics.
Bindings to related
Resource
s.
UploadProtocol
¶
enum class UploadProtocol {
CHUNKED,
WEBSOCKET,
}
Properties
CHUNKED
WEBSOCKET
UploadType
¶
enum class UploadType {
FILE,
FOLDER,
}
Properties
FILE
FOLDER
WriteConflictPolicy
¶
A policy for how UCloud should handle potential naming conflicts for certain operations (e.g. copy)
enum class WriteConflictPolicy {
RENAME,
REJECT,
REPLACE,
MERGE_RENAME,
}
Properties
RENAME
UCloud should handle the conflict by renaming the file
REJECT
UCloud should fail the request entirely
REPLACE
UCloud should replace the existing file
MERGE_RENAME
"Attempt to merge the results
This will result in the merging of folders. Concretely this means that directory conflicts will be resolved by
re-using the existing directory. If there any file conflicts in the operation then this will act identical to RENAME
.
Note: This mode is not supported for all operations. "
FilesCopyRequestItem
¶
data class FilesCopyRequestItem(
val oldId: String,
val newId: String,
val conflictPolicy: WriteConflictPolicy,
)
FilesCreateDownloadRequestItem
¶
data class FilesCreateDownloadRequestItem(
val id: String,
)
Properties
id
: String
String
FilesCreateFolderRequestItem
¶
data class FilesCreateFolderRequestItem(
val id: String,
val conflictPolicy: WriteConflictPolicy,
)
FilesCreateUploadRequestItem
¶
data class FilesCreateUploadRequestItem(
val id: String,
val type: UploadType,
val supportedProtocols: List<UploadProtocol>,
val conflictPolicy: WriteConflictPolicy,
)
Properties
id
: String
String
type
: UploadType
UploadType
supportedProtocols
: List<UploadProtocol>
List<UploadProtocol>
conflictPolicy
: WriteConflictPolicy
WriteConflictPolicy
FilesMoveRequestItem
¶
data class FilesMoveRequestItem(
val oldId: String,
val newId: String,
val conflictPolicy: WriteConflictPolicy,
)
FilesStreamingSearchRequest
¶
data class FilesStreamingSearchRequest(
val flags: UFileIncludeFlags,
val query: String,
val currentFolder: String?,
)
FilesCreateDownloadResponseItem
¶
data class FilesCreateDownloadResponseItem(
val endpoint: String,
)
Properties
endpoint
: String
String
FilesCreateUploadResponseItem
¶
data class FilesCreateUploadResponseItem(
val endpoint: String,
val protocol: UploadProtocol,
val token: String,
)