Aliases: How to create backup jobs for one or more AWS accounts
Creating your alias - what you need to know
Before you can create backup jobs for your S3 buckets you'll first need to create an alias.
An alias represents an AWS credentials keypair. An AWS keypair consists of the following:
- Access Key
- Secret Key
To create an alias follow these steps:
- Enter your access key
- Enter your secret key
- Click the "Create Alias" button
You will then need to password protect your alias
Note: We do NOT store alias, password or any other sensitive information on our servers.
All information pertaining to your credentials, passwords, aliases and backup jobs are stored on your computer in the BucketBacker APPDATA folder.
All sensitive information like secret keys and passwords are stored in encrypted format.
Once you create an alias BucketBacker will then be able to retrieve the buckets for that AWS keypair
along with any backup jobs you may have created.
Reasons you may wish to create multiple aliases.
- You wish to backup buckets for multiple AWS accounts
- You wish to logically separate your backup jobs
Please note: You may wish to use AWS Identity and Access Management (IAM) for your alias' keypair.
Learn more about AWS Identity and Access Management
Backup Jobs - What you need to know
Backup jobs consist of line items
- Backup jobs consist of one or more line items.
- Each line item represents a specific bucket you wish to back up.
- Each backup job can contain up to 20 line items.
- The job list allows you to Add, Rename or Delete a backup job.
- Once a backup job is selected the "Line Item Wizard" allows you to add, edit or delete line items.
The job list lets you create, rename and delete jobs
Creating a new backup job workflow:
- Create a new job using the "New" button
Rename/Delete a backup job workflow:
- Rename or delete a backup job using the, "Edit" button
Please note that if you are renaming a backup job that has been previously scheduled in the Windows Task Scheduler you need to do three things:
- Delete the old task with the old job name from the Windows Task Scheduler
- Create a trigger for the new scheduled job in the Windows Task Scheduler (right-click the job and select properties)
Line Items: How to add buckets to your backup job
- Select a backup job from the jobs list
At this point the "Line Item Wizard" is displayed.
For more information about the Line Item Wizard click the link below
Global Backup Bucket - What you need to know
- The global backup bucket is a bucket dedicated to storing your backups
- The global backup bucket applies to all backup jobs for a single alias.
- The global backup bucket can store unlimited backups.
Because AWS S3 has a hard limit of 100 buckets per account we have decided this is the best way to store your backups.
BucketBacker will copy your S3 objects to the global backup bucket in folder format with self-describing names
How backup jobs are named and stored in the global backup bucket
The folders stored here are self-descriptive based on the following paradigm:
Source bucket name + "-" + backup type
In the example below three line items are created to backup the bucket, "aaa_qa_snappy"
Each line item specifies a different backup type.
In this example the backup types selected are as follows
Thus the folder names containing your backups will become:
AWS Bucket Naming Rules for the Global Backup Bucket
Rules for Bucket Naming
The following is excerpted directly from the AWS website.
We recommend that all bucket names comply with DNS naming conventions. These conventions are enforced in all regions except for the US Standard region.
If you use the AWS management console, bucket names must be DNS compliant in all regions.
DNS-compliant bucket names allow customers to benefit from new features and operational improvements, as well as providing support for virtual-host style access to buckets. While the US Standard region currently allows non-compliant DNS bucket naming, we are moving to the same DNS-compliant bucket naming convention for the US Standard region in the coming months. This will ensure a single, consistent naming approach for Amazon S3 buckets. The rules for DNS-compliant bucket names are:
Bucket names must be at least 3 and no more than 63 characters long.
Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.
Bucket names must not be formatted as an IP address (e.g., 192.168.5.4).
The following examples are valid bucket names:
The following examples are invalid bucket names:
|Invalid Bucket Name||Comment|
|Bucket name cannot start with a period (.).|
|Bucket name cannot end with a period (.).|
|There can be only one period between labels.|
Challenges with Non–DNS-Compliant Bucket Names
The US Standard region currently allows more relaxed standards for bucket
naming, which can result in a bucket name that is not DNS-compliant. For
MyAWSBucket is a valid bucket name, even though it
contains uppercase letters. If you try to access this bucket by using a
http://MyAWSBucket.s3.amazonaws.com/yourobject), the URL
resolves to the bucket
myawsbucket and not the bucket
MyAWSBucket. In response, Amazon S3 will return a "bucket not found"
To avoid this problem, we recommend as a best practice that you always use DNS-compliant bucket names regardless of the region in which you create the bucket. For more information about virtual-hosted–style access to your buckets, see Virtual Hosting of Buckets.
The rules for bucket names in the US Standard region allow bucket names to be as long as 255 characters, and bucket names can contain any combination of uppercase letters, lowercase letters, numbers, periods (.), hyphens (-), and underscores (_).
Line Item Wizard - What you need to know
Note: Once you master the Line Item Wizard the rest is child's play
Note: You should understand the following topics before reading this section
What is a line item?
- A line item represents a single bucket that you wish to backup.
- Backup jobs can contain one or more line items.
- The Line Item Wizard makes adding line items to your backup jobs very easy
How does the Line Item Wizard work?
The Line Item Wizard has three pages
- Page 1: Backup Job line items - View, add, edit and delete line items
- Page 2: Configure Details - Select the line item source bucket and backup type
- Page 3: Include/Exclude bucket folders
Creating a line item workflow
The easiest way to understand line items is to create one
In this example we are going to create a single line item to backup a single bucket.
- We will backup a bucket named, "aaa_qa_snappy".
- We will make the backup type: Synchronized
- We will only backup one folder in the bucket named, "aaaaaaa2"
- Click the Save button
You have just created a new line item
The following documentation is available for the individual wizard pages
Backup Types (Cumulative, Synchronized & Snapshot)
The three types of backups available to you
Each backup type explained
A cumulative backup will copy the latest version of any object to your backup folder.
If you delete an object in the source bucket it will NOT be deleted in the backup folder.
This means that your backup folder will have a copy of every file you've ever backed up.
This is contrasted with the "Synchronized" backup which will DELETE any object in the backup folder that is not found in the source bucket.
Note: This will be saved to your global backup bucket in the following format
<source_bucket>-cumulative as illustrated below where the source bucket is, "aaa_qa_snappy"
A synchronized backup will copy the latest version of any object to your backup folder.
It will also delete any object in your backup folder that was deleted from the source bucket.
This is ideal for when you wish to maintain a perfect copy of a bucket.
Note: This will be saved to your global backup bucket in the following format
<source_bucket>-synchronized as illustrated below where the source bucket is, "aaa_qa_snappy"
A snapshot backup will create a timestamped folder to keep a copy of the source bucket.
This is ideal when you want to take a picture in time of your bucket.
Note: This will be saved to your global backup bucket in the following format
<source_bucket>-<snapshot> Note: If you drill into the folder you will find subfolders that are timestamped as illustrated below
Include/Exclude bucket folders (optional)
The difference between "Include" and "Exclude explained"
If you only wish to back up specific folders you will want to select the "Include" option
- Only the folders you select for inclusion will be backed up
- If you also wish to backup root level objects for the bucket be sure to check the, "Include root level files" checkbox as illustrated below.
If you only wish to exclude certain folders you will want to select the "Exclude" option
- Everything BUT the folder(s) you exclude will be backed up
Pro-tips: How to know which one to choose
Exclude candidate: If you have folders in your buckets that fall into the following categories you may wish to EXCLUDE them from backups
- Folders already backed with an extremely large number of objects
- Folders already backed up whose contents never change
By excluding these types of folders your backups run faster and you save money
Include candidate: If you have folders in your buckets that meet the following criteria
- Only a few folders change frequently
- Only a few folders need frequent snapshots
By including these types of folders your backups run faster and you save money
Real world examples
We have certain buckets containing folders with JSON files that never change. These objects number in the hundreds of thousands.
However, we still wish to backup the non-static objects in these buckets frequently. We run cumulative, synchronized and snapshot backups for these buckets and we exclude our folders containing the hundreds of thousands of JSON files that never change and are already backed up.
We have certain buckets containing folders used to run static websites. These files rarely change so less frequent backups are appropriate.
However, some of these buckets have folders containing user generated data that changes constantly and we wish to back those folders up frequently.
In this case we run cumulative, synchronized and snapshot backups just for those folders.
Run/Schedule backup jobs
Manually run a backup job
How to manually run or schedule your backup jobs
To instantly test your backup jobs simply click the, "Manually Run Backup Job Now" button as illustrated below.
Windows® Scheduler Automated Backup JobsWindows Task Scheduler documentation
How to create Set & Forget scheduled backup jobs
Question: When do I create scheduled jobs?
Answer: Anytime after you create a new backup job you can use the Create/Overwrite Scheduled Job button to create a new task in the Windows Task Scheduler
Please note: Normally, you only have to create a scheduled job once. As soon as you have scheduled your job as a task in the Windows Task Scheduler any line item changes
you make to the backup job will automatically take effect the next time the backup job runs with two exceptions noted below:
- If you are a subscriber to the Email Notification Service and wish to change the email address for a previously scheduled job you will have to recreate the scheduled job and add a new trigger.
- If you rename a backup job you will have to recreate the scheduled job and add a new trigger.
To instantly schedule backup jobs simply click the, "Create/Overwrite Scheduled Job" button in the ""Run/Schedule Backup Job" wizard as illustrated below.
Clicking this button will automatically create a task in the Windows® Task Scheduler.
All BucketBacker backup job tasks are placed in a folder called, "BucketBacker" in the "Task Scheduler Libary" as illustrated below.
Your job will appear in the tasks list and will be named using the following convention:
Configuring the task to run on schedule:
- Right-click the task and select "Properties"
Next you will need to make changes to the following tabs of the "Properties" window.
Select the "General" tab
- Select, "Run whether user is logged on or not"
- Check, "Run with highest privileges"
Select the "Triggers" tab
- Click the "New..." button of the "Triggers" tab
- Configure the "New Trigger" form
- Save the trigger
Select the "Settings" tab
- Change the dropdown FROM
- Do not start a new instance
- Stop the existing instance
- Click the OK button of the Properties Wizard to finalize your settings
Congratulations! You have now created a Set & Forget BucketBacker backup job.
Email Notification Service Premium Subscription
This premium service is available free during your two week trial period.
To use this feature simply enter an email address in Step 1.
Each schedule backup job can notify a different email address
Please note: The email notification address is assigned on a job by job basis.
If you wish to change the notification email address for a pre-existing scheduled job you will need to perform the following steps.
- Re-create the scheduled task by clicking the, "Create/Overwrite Scheduled Job" button
- Re-create the trigger for the scheduled task in Windows Task Scheduler
For more information about creating schedule tasks and triggers please view the following topic: Scheduled Backups
Email Notification Service - Benefits of a premium subscription
- Peace of mind: every time a backup job runs you will receive a completed job notification straight to your inbox
- Error Reporting: If your backup job encounters any errors (e.g. detection of corrupt S3 objects, missing bucket, connectivity issues) you will receive an error notification
- Archiving: completed job notifications will be archived in the AppData logs next to their XML counterparts
- Insight: completed job notifications are highly detailed and help you ensure that your jobs are both properly configured and running without issue
View sample completed job notification as a web page
A word about Email Notification Service security
We do not store your job notifications on our servers after they are emailed.
All job notifications are immediately deleted from our servers after they are successfully emailed to you.
BucketBacker scaling and resource allocation
BucketBacker is a multithreaded application, capable of running unlimited backup jobs simultaneously and in parallel.
This feature may overload your hardware resources and throughput (internet connection) if you do not carefully consider the load on your computer and verify your resources are sufficient for the successful completion of your backup jobs.
It is your responsibility to ensure that BucketBacker is installed on a machine (including virtual machines) capable of handling the load you intend to place upon it.
To illustrate an overload scenario:
You have 200 buckets with one million objects in each - If you attempt to create 200 backup jobs and run them simultaneously on a computer with insufficient RAM, CPU cores and available internet throughput, you will DEFINITELY overload your hardware and internet connection. At this point BucketBacker will begin to malfunction and write errors to your log files.
It is your responsibility to intelligently provision your scheduled backup jobs to avoid over-utilizing your resources as there is a direct correlation between the capacity of your resources (bandwidth/throughput, CPU cores and RAM) and your ability to run multiple large backup jobs simultaneously.
It is your responsibility to test your scheduled backup jobs to ensure that you are not overloading your resources. BucketBacker writes completion and error log files in XML format to your APPDATA folder, allowing detailed analysis of backup job status. It is your responsibility to review your log files to ensure backup jobs ran as intended.
BucketBacker also offers a premium email notification service that proactively pushes your log files, in HTML format, directly to your inbox.