Use Swiftly to upload an image

For Cloud Files to provide high availability and data resiliency, we limit all files
to 5 GB in size, which is smaller than most
VM images and many other files that people might want to store. To get
around this limit, you can split any Large Object—any file larger than 5 GB—
into segments bound together by a manifest.

There are two types of manifest objects in Cloud Files: Dynamic Large Objects
and Static Large Objects. For image uploads, we recommend that you use a Static Large Object.

With a Static Large Object, the manifest is an explicit listing of the
size, MD5 checksum, and location of each segment that makes up the Large
Object. The Swiftly® tool divides your local file into segments, uploads
the segments in parallel, and automatically creates a manifest for you.
This article describes how to use Swiftly to upload a file larger than 5 GB
to Cloud Files as a Static Large Object.

For more information about Cloud Files Large Objects, see the
Cloud Files Developer documentation.

Note: If you prefer to use the API to manage large files, see
Use the API to manage large files,
and if you prefer Swift, see Use Swift to manage large files.

Prerequisites

This article assumes you have already properly prepared an image for import into
the public cloud and have already installed the Swiftly client. If you need help
with either of these prerequisites, see the following articles:

Set environment variables

In a Bash shell, set the following environment variables:

CF_USERNAME=       # your Rackspace Cloud username
CF_API_KEY=        # your Rackspace Cloud API key
CF_REGION=         # Three character region code for your upload location (for example, `ORD`)
SOURCEFILE=        # the local file you are uploading
CONTAINER=         # the container in Cloud Files where the image should go
IMAGEFILENAME=     # the name you want to call the image in Cloud Files
SWFLY_SEG_BYTES=134217728
SWFLY_CONCURRENCY=20

To view information about viewing your Rackspace API key, see
View and reset your API key.

Ensure that the container into which you want to upload your object
already exists in the appropriate region in Cloud Files. (You can create
it in the Cloud Control Panel, if necessary.)

As mentioned earlier, Swiftly automatically segments your image file and
uploads the segments in parallel. The two SWFLY environment variables
listed in the preceding example control this.

SWFLY_SEG_BYTES specifies the size, in bytes, that Swiftly uses
for each segment (except for the last segment, which could
be smaller). The value shown in the preceding example is 128 MB,
expressed in bytes. We recommend that you use this value. You can
experiment with different values to see if you get better
performance, but avoid using a smaller value (certainly no smaller
than 100 MB) and avoid a value larger than 1 GB.
SWFLY_CONCURRENCY specifies the maximum number of parallel threads
that Swiftly uses to upload the object. The author of Swiftly suggests
the value shown in the example. You can experiment with the value, but
your parallel uploads might saturate your network card and actually
slow down the overall file transfer if you set it too high.

Invoke Swiftly

After you set the environment variables, invoke Swiftly from the command
line to perform the upload, as shown in the following example:

Note: Consider using a screen session to perform this task. If
you're not familiar with the GNU Screen program, you can find an
introduction in Install the Swiftly client for Cloud Files.

swiftly
  --auth-url=https://identity.api.rackspacecloud.com/v2.0
  --auth-user=$CF_USERNAME
  --auth-key=$CF_API_KEY
  --region=$CF_REGION
  --concurrency=$SWFLY_CONCURRENCY
  put
    --segment-size=s${SWFLY_SEG_BYTES}
    --input=$SOURCEFILE
  ${CONTAINER}/${IMAGEFILENAME}

If you are importing a file from a cloud server already in the
Rackspace Cloud, add the --snet option to the command so that the
system transfers the file over the internal cloud network. Additionally,
if you want Swiftly to notify you about what it's doing as it uploads
your image file, you can add the --verbose option. If you add these
options, your invocation looks similar to the following example:

swiftly
  --auth-url=https://identity.api.rackspacecloud.com/v2.0
  --auth-user=$CF_USERNAME
  --auth-key=$CF_API_KEY
  --region=$CF_REGION
  --snet
  --verbose
  --concurrency=$SWFLY_CONCURRENCY
  put
    --segment-size=s${SWFLY_SEG_BYTES}
    --input=$SOURCEFILE
  ${CONTAINER}/${IMAGEFILENAME}

Note that the Swiftly invocation contains the following line:

--segment-size=s${SWFLY_SEG_BYTES}

The s after the equals sign tells Swiftly to create a Static Large
Object. As mentioned earlier, we highly recommend that you upload your
image as a Static Large Object.

Check your upload

Suppose that you used the following the environment variable settings
you used for your upload:

CF_REGION="DFW"
SOURCEFILE="my-awesome-image.vhd"
CONTAINER="uploaded-images"
IMAGEFILENAME="my-custom-image.vhd"

And this was the image you uploaded:

$ ls -l
total 2524008
-rw-rw-r-- 1 joeuser joeuser 2584576512 Apr 24 03:01 my-awesome-image.vhd

To check your upload, view the manifest for the Static Large Object that
was created in Cloud Files:

# get the manifest
swiftly
 --auth-url=https://identity.api.rackspacecloud.com/v2.0
 --auth-user=$CF_USERNAME
 --auth-key=$CF_API_KEY
 --region=$CF_REGION
 get
 --query=multipart-manifest=get
 --output=my-manifest.json
 ${CONTAINER}/${IMAGEFILENAME}

# look at the manifest
$ cat my-manifest.json | python -m json.tool
[
    {
        "bytes": 134217728,
        "content_type": "application/octet-stream",
        "hash": "bc5dc9c7f93b214e648e3ce2b9ee4bd1",
        "last_modified": "2014-04-24T03:46:16.000000",
        "name": "/uploaded-images_segments/my-custom-image.vhd/1398308466.19/2584576512/00000000"
    },
    {
        "bytes": 134217728,
        "content_type": "application/octet-stream",
        "hash": "c4a2dbe171bd60a3a23198baa916879c",
        "last_modified": "2014-04-24T03:46:23.000000",
        "name": "/uploaded-images_segments/my-custom-image.vhd/1398308466.19/2584576512/00000001"
    },
# etc
    {
        "bytes": 134217728,
        "content_type": "application/octet-stream",
        "hash": "9acffa882c4bf8beb3025e856f6e9d01",
        "last_modified": "2014-04-24T03:47:52.000000",
        "name": "/uploaded-images_segments/my-custom-image.vhd/1398308466.19/2584576512/00000018"
    },
    {
        "bytes": 34439680,
        "content_type": "application/octet-stream",
        "hash": "dd594916413c2e1ef05875606b813528",
        "last_modified": "2014-04-24T03:47:55.000000",
        "name": "/uploaded-images_segments/my-custom-image.vhd/1398308466.19/2584576512/00000019"
    }
]

Alternatively, you can look at your Cloud Files account in the Cloud Control Panel.
The following example shows the Containers page in the
Control Panel, which displays the containers in the DFW region.

The uploaded-images container existed before the upload, and
the invocation to Swiftly identified it as the container to hold the
manifest. The size of this container is small because it contains only
the Static Large Object manifest file, which is a text file in JSON
format. (The content of this file is shown in the preceding example.)
The uploaded-images_segments container, which Swiftly created,
contains the actual image data divided into segment files. The size of
this container is 2.41 GB, which is the size of the uploaded image.

The uploaded-images container holds the manifest file, and the
uploaded-images_segments container holds the parts that merge
to make the image file. Each segment file ends with a number
sequence identifying the segment, such as 00000000, 00000001,
00000002, and so on.

Note that while the manifest in uploaded-images depends on the image
segments in uploaded-images_segments, Cloud Files won't prevent you
from deleting segments. If you delete any of those segments, you get
an error when you attempt to download the Static Large Object that
comprises the image file.

If you look inside the uploaded-images container, the image file that
the manifest describes displays rather than the manifest itself. In this
example, the display shows my-custom-image.vhd. The file size Cloud Files
displays in the container view reflects the overall size of the
Large Static Object because that's the size of the object you'd get if
you downloaded the file. The actual storage used in that container is
the few kilobytes used by the manifest file because the system stores
the actual image data in the uploaded-images_segments container.

Summary

To download the image from the example above, request the
object named my-custom-image.vhd from the uploaded-images
container in the DFW region of your Cloud Files account.
To import the example image using Cloud Images, create an
import task, specifying the value for import_from as
uploaded-images/my-custom-image.vhd
Do not delete any of the segments in the uploaded-images_segments
container, or you corrupt your image.
Swiftly takes care of dividing your image file into segments,
uploading the segments to their own container, and creating the
Static Large Object manifest automatically in the container you
requested. But it's important to know what it's doing
and how your data is stored so that you don't corrupt your image
by mistake.

Use the Feedback tab to make any comments or ask questions. You can also start a conversation with us.