Atom feed of this document
 
 
 

 8.2.1. Creating a Dynamic Large Object

Objects that are larger than 5 GB must be segmented into smaller segment objects before you upload them. You then upload the segment objects as you would any other object. You create a manifest object that tells Cloud Files how to find the segments that make up the large object. The segments remain individually addressable, but retrieving the manifest object streams all the segments, concatenated. There is no limit to the number of segments that can be a part of a single large object. Dynamic large objects rely on the eventual consistency model.

[Note]Note

In this context, the eventual consistency model means that although you have uploaded a segment object, it might not appear in the container list immediately. If you download the manifest before the segment object appears in the container, the object will not be part of the content returned in response to a GET request.

To ensure that the download works correctly, you must upload all the object segments to the same container and ensure that each object name is prefixed in such a way that the names sort in the order in which they should be concatenated. You also create and upload a manifest file. The manifest file is simply a zero-byte file with the extra X-Object-Manifest: container/prefix header, where container is the container that the object segments are in and prefix is the common prefix for all the segments. The container and common prefix must be UTF-8 encoded and URL-encoded in the X-Object-Manifest header.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.

[Note]Note

The segments are deletable by the user at any time. If a segment is deleted by mistake, a dynamic large object, having no way of knowing the segment was ever there, ignores the deleted file, and the user is returned an incomplete file.

The following examples show how to upload a segment of a large object, the next segment of a large object, and the manifest.

 

Example 8.3. Upload a Segment of a Large Object Request

PUT /v1/MossoCloudFS_0672d7fa-9f85-4a81-a3ab-adb66a880123/MyContainer/MyObject HTTP/1.1
Host: storage.clouddrive.com
X-Auth-Token: f064c46a782c444cb4ba4b6434288f7c
ETag: 8a964ee2a5e88be344f36c22562a6486
Content-Length: 1

 

Example 8.4. Upload a Segment of a Large Object Response

s

No response body is returned. A status code of 201 (Created) indicates a successful write. Status code 411 (Length Required) indicates that the Content-Length header is missing. If the MD5 checksum calculated by the storage system does not match the optionally supplied ETag value, a 422 (Unprocessable Entity) status code is returned.

You can continue uploading segments as this example shows, prior to uploading the manifest.

 

Example 8.5. Upload the Next Segment of the Large Object Request

PUT /v1/MossoCloudFS_0672d7fa-9f85-4a81-a3ab-adb66a880123/MyContainer/MyObject HTTP/1.1
Host: storage.clouddrive.com
X-Auth-Token: f064c46a782c444cb4ba4b6434288f7c
ETag: 8a964ee2a5e88be344f36c22562a6486
Content-Length: 1

 

Example 8.6. Upload the Next Segment of the Large Object Response

w

Next, upload the manifest that you created that indicates the container in which the object segments reside. Note that uploading additional segments after the manifest is created causes the concatenated object to be that much larger, but you do not need to re-create the manifest file for subsequent additional segments.

 

Example 8.7. Upload Manifest Request

PUT /v1/MossoCloudFS_0672d7fa-9f85-4a81-a3ab-adb66a880123/MyContainer/MyObject HTTP/1.1
Host: storage.clouddrive.com
X-Auth-Token: f064c46a782c444cb4ba4b6434288f7c
Content-Length: 0
X-Object-Manifest: container/prefix/object/segments

 

Example 8.8. Upload Manifest Response

[...]

A GET request to the manifest object returns the concatenation of the objects from the manifest.

When you perform a GET or HEAD request on the manifest, the response's Content-Type is the same as the Content-Type that was set during the PUT request that created the manifest. You can easily change the Content-Type by reissuing the PUT request.

[Note]Note

The ETag in the response for a GET or HEAD on the manifest file is the MD5 sum of the concatenated string of ETags for each of the segments in the manifest. Usually, the ETag is the MD5 sum of the contents of the object, and that holds true for each segment independently. But it is not meaningful to generate such an ETag for the manifest itself, so this method was chosen to at least offer change detection.



loading table of contents...