8.2. Creating large objects

The content of an object cannot be larger than 5 GB, by default. However, you can store a larger amount of content by using the following types of objects:

  • Segment objects: You divide your content into pieces and upload each piece into its own object, which is a segment object.

  • Manifest objects: You create a manifest object that "points to" the segment objects.

Segment objects do not have any special features and can be created, updated, downloaded, and deleted. However, a manifest object is special—when you download, the system concatenates the contents of the segment objects and returns the concatenation in the response body of the request. This behavior extends to the response headers returned by GET and HEAD operations. The value of the Content-Length header is the total size of all segment objects, and the value of the ETag header is calculated by taking the ETag value of each segment, concatenating them together, and then returning the MD5 checksum of the result.

[Note]Note

If you use a manifest object as the source in a COPY operation, the new object is a "normal" object (not segmented). If the total size of the source segment objects exceeds 5 GB, the COPY operation fails. However, you can make a duplicate of the manifest object. This new object can be larger than 5 GB.

Following are the types of manifest objects:

  • Dynamic large objects: The manifest object has no content. However, it has the X-Object-Manifest metadata header. The value of this header is container/prefix , where container is the name of the container where the segment objects are stored and prefix is a string that all the segment objects have in common.

  • Static large objects: The manifest object content is an ordered list of the names of the segment objects in JSON format.

Although both types of manifest objects have similar behavior, there are differences as explained in the following table.

Table 8.1. Comparison of static and dynamic large objects
Feature Static large object Dynamic large object
End-to-end integrity Assured. The list of segments includes the MD5 checksum (ETag) of each segment. You cannot upload the manifest object if the ETag in the list differs from the segment object already uploaded. If a segment is somehow lost, an attempt to download the manifest object results in an error. Not assured. The eventual consistency model means that although you have uploaded a segment object, it might not appear in the container list immediately. If you download the manifest before the object appears in the container, the object will not be part of the content returned in response to a GET request.
Upload order The segment objects must be uploaded before the manifest object. You can upload manifest and segment objects in any order. We recommend that you upload the manifest object after the segments in case a premature download of the manifest occurs. However, this is not enforced.
Removal or addition of segment objects You cannot add or remove segment objects from the manifest. However, you can create a completely new manifest object of the same name with a different manifest list. You can upload new segment objects or remove existing segments—the names must simply match the <prefix> supplied in the X-Object-Manifest header.
Segment object size and number Segment objects must be at least 1 MB in size, by default. The final segment object can be any size. By default, a maximum of 1000 segments are supported. Segment objects can be of any size.
Segment object container name The manifest list includes the container name of each object (that is, segment objects might be in different containers). All segment objects must be in the same container.
Manifest object metadata The object has the X-Static-Large-Object metadata header set to True. You do not set this metadata directly. Instead the system sets it when you use a PUT operation on a static manifest object. The X-Object-Manifest header value is container/prefix, indicating where the segment objects are located. You supply this request header in the PUT operation
Making a copy of the manifest object To make a copy of the manifest object, include the ?multipart- manifest=get query string with the COPY operation. The new object contains the same manifest as the original. The segment objects are not copied. Instead, both the original and new manifest objects share the same set of segment objects. The COPY operation does not create a manifest object. To duplicate a manifest object, use the GET operation to read the value of X-Object-Manifest and use this value in the X-Object-Manifest request header in a PUT operation. This creates a new manifest object that shares the same set of segment objects as the original manifest object.


loading table of contents...