Use Cloud Backup to recover from a Bad Actor attack

The article shows you how to recover backups from partial deletion of
your assets at Rackspace. The Related comments section at the end of this
article discusses how to recover from complete asset deletion.

Occasionally, bad actors attempt to destroy your cloud assets, such as files,
websites, databases, and so on. The bad actor might be a foreign attacker who stole
cloud account authentication info, or it might be a disgruntled employee with
access to company assets or any similar bad actor. Attacks like this might
cripple or kill a company, and the ability to recover backups makes
the difference between whether the company survives the attack or not.

Note: It should go without saying that you are responsible for protecting
the credentials that allow access to your cloud assets. This article outlines
how to recover if your credentials were compromised.

Symptoms of an attack

Company assets, such as servers, backups, or cloud files, suddenly begin
disappearing as fast as the bad actor can destroy them.

Solution

Note: This article addresses the recovery of only those assets related to
Cloud Backup (CBU).

For each intact backup configuration (config) container in Cloud Files, you
can recover backups for that backup config.

The following steps provide a summary of the solution:

1. Save or recover as much from associated Cloud Files
   containers as possible.
2. The Rackspace CBU support team must revert any machine agents and backup
   configs that have been deleted for this account.
3. Do a cross-site restore from the recovered backups to new servers.

The following sections describe the preceding steps in detail.

Step 1

Time is of the essence in this step. You need to stop the damage before it
becomes total. Most importantly, stop the bad actor from deleting any
more Cloud Files assets. These should be protected first.

After you discover that assets are being destroyed, contact support to
immediately halt access to the account and reset credentials. If you have
monitoring and alerts in place to immediately notify stakeholders of
malfunctioning assets, you have an advantage in detecting these attacks.

One advantage of Cloud Files in this scenario is that there is not an easy way
to do bulk deletes through the web interface without some high-powered
utilities that aren't available through our web interface. So deleting
these files can be slow enough that you can interrupt the attacker before he
finishes.

The more Cloud Files assets that can be saved or restored before recovery starts,
the more backups you can salvage.

Step 2

Revert any valid machine agents and/or backup configs that have been deleted.

Note: The Rackspace CBU Operations Engineering (OpsEng) or support teams
must perform this step.

If you have any offsite backups of the Cloud Files containers for the backup
configs in question (see Related comments at the end of this article), you
should restore them to their original locations at this time. For this step to
be successful, you must have at least one undeleted (or restored) Cloud Files
container for at least one backup configuration.

When you request help for this step, reference the Cloud Backup support wiki article,
Cloud Backup - Bad Actor Attack, in your comments on the ticket. The public
can't view the article, but Cloud Backup support can access it. Only Rackspace Support
can perform the steps described in that wiki article. They must revert the
deleted assets (machine agents and backup configurations) in your account.

Step 3

For any machine agent and backup config that Support successfully restored, use
Cloud Backup to do a cross-site restore to a new server.

Related comments

As mentioned earlier, it should go without saying that you are responsible for
protecting the credentials that allow access to your cloud assets. But sometimes
the good guys become the bad guys, or in spite of your best efforts, attackers
manage to defeat your protective measures.

The instructions in this article describe a strategy to recover from the scenario
where a bad actor compromised those credentials and destroyed some, or all, of
the assets in your account. The recovery process isn't easy, but losing all your
data is worse.

To significantly increase the chance of recovering assets deleted by a bad actor, you
can choose to have an offsite copy of one Cloud Files container for each backup
configuration that you want to protect. (Some hints for how to find these
containers are detailed below.) In this context, offsite refers to a
copy of Cloud Files containers on media which is not on Rackspace infrastructure
(which would, of course, be accessible via the stolen credentials). You can
restore these files to their original locations in Cloud Files and then use
them to create cross-site restores to new servers.

Important: For this solution to provide meaningful protection, you must
have separation of access. Because we assume bad actors in this scenario have
full access to your Rackspace assets through the stolen credentials, you must
make sure that those with access (stolen or otherwise) to the Rackspace asset
credentials do not have access to the offsite copies of the containers, and
vice versa. You can do this with policies like physically locking up the offsite
copies and giving the person who performs the offsite backups read-only access
to Cloud Files. So, for instance, your system administrator should not have
access to the offsite backup flash drive, and your secretary should have
read-only access to Cloud Files.

It is not an easy process to identify the containers in Cloud Files that hold
the artifacts necessary to restore backups. Open each machine agent in the
Cloud Backup Systems list. View the Agent Configuration for each agent. This
Agent Configuration is a large JSON document. The JSON has a section named
"BackupConfigurations". In that section there is an array of records that contain
backup configuration details. In each record is an item named "VolumeUri". This is
the private address which the API uses to store the artifacts necessary to restore
a backup. There is one such record for each backup configured for each agent,
and there may be multiple backups configured for a given agent. This means there
might be multiple VolumeUri addresses for any one agent. At the end of each address
is a section that starts with "zDO_NOT_DELETE_CloudBackup_v2_0" and which has a
GUID-formatted number appended to it. Search Cloud Files for a container name that
matches this z-name. This is the container that holds the artifacts that Cloud
Backup uses to restore files for this backup configuration. Save this entire
container in your off-site copy.

The Rackspace CBU team currently does not have a tool that can bulk-copy Cloud
Files containers, but there are third-party tools (such as swiftly or rclone)
that might be used for this. If we create a tool specifically for this purpose
in the future, we'll update this article.