Skip to main content

Python CLI To Sync Rackspace Cloud Files

Tags

Python logo

I have always been on the lookout for a tool that I can use to synchronize files between two Rackspace Cloud Containers.  I have looked high and low for a suitable tool that syncs files from an origin container to a destination container whether that destination container exists or not.  Finally, I decided that if I want something like this then I was just going to have to roll up my sleeves and create the tool myself.  Today I wrote a Python command line tool that extends the Rackspace Python SDK, pyrax, and uses the SDK to take the input of an origin container and syncs it to a destination container.  This sync from the origin container to the destination container is done destructively and does not try and merge the files with the destination container, but deletes the files in the destionation container if they exist and uploads a complete set of new files from the origin container.

The complete tool is up on my Github account if you want to fork it or send suggestions my way about improvements.  Otherwise, below is a copy of the Container Sync Script.  All comments and suggestions are welcome!

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# 5-2-2016
# Matt Eaton Agnosticdev
# 
# The purpose of this script is to sync objects from one container to the other.
#
# This script will take the input of the origin container and also the destination container.
#
# Then this script will delete the destination container if present, create it and
# then copy files recursively from the origin container to the destination container.
#
# Pyrax source was very helpful for this script:
# https://github.com/rackspace/pyrax
#
from __future__ import print_function
 
import os
import pyrax
 
print("*****************************************************\n")
print("Welcome to the Rackspace Container Sync Script\n")
print("*****************************************************")
print("What region will both of your containers be in?")
container_region = raw_input("ORD [1]:, DFW [2]:, IAD [3]: ")
container_region = container_region.strip()
 
# Make sure the input is a valid container region
if container_region == '1':
	container_region = "ORD"
elif container_region == '2':
	container_region = "DFW"
elif container_region == '3':
	container_region = "IAD"
else:
	exit("Container input was not recognized.  Exiting")
 
# Setup and parse input file with credentials
pyrax.set_setting("identity_type", "rackspace")
creds_file = os.path.expanduser(".rackspace_config")
pyrax.set_credential_file(creds_file, container_region)
cf = pyrax.cloudfiles
 
# Make sure that when cloud files is initialized that the regions match
if not cf.region_name == container_region:
	exit("Something went wrong with Racksapce initialization.  Exiting")
 
# Get the container name 
origin_container_name = raw_input("What is the name of the origin container? ")
origin_container_name = origin_container_name.strip()
 
# Make sure container_name is not empty
if not origin_container_name:
    exit("Origin container cannot be blank.  Exiting")
 
# Print to the user the origin container name
print("OK, the origin container name is: " + origin_container_name + "\n")
 
# Get the name of the container where you would likes files copied
destination_container_name = raw_input("What is the name of your destination container? ")
destination_container_name = destination_container_name.strip()
 
# Make sure the destination container name is not empty
if not destination_container_name:
	exit("Destination container name cannot be blank.  Exiting")
 
# Print to the user the destination container name
print("OK, the destination container name is: " + destination_container_name + "\n")
 
# Warn the user that we will delete the destination container if set
print("Warning! This script will delete your destination container if it exists.")
deletion_check = raw_input("Are you OK with this? [y/n] ")
deletion_check = deletion_check.strip().lower()
 
print("Deletion check " + deletion_check)
 
# Make sure the user knows we plan to delete the destination container
if not deletion_check == 'y':
    exit("OK, no problem.  Container is not deleted.  Exiting")
else:
	print("OK, the " + destination_container_name + " will now attempt to be deleted if it exists.")
 
# Attempt to get reference to the destination container to see if we need to delete it
try:
    dest_container_ref = cf.get_container(destination_container_name)
 
    # Alert the user the destination container is being deleted
    print("Looks like " + destination_container_name + " exists.  Deleting now...")
    cf.delete(dest_container_ref, del_objects=True)
except:
    # Handle the error of an empty container being requested
    print("Destination container does not exist.")
 
 
print("Creating " + destination_container_name + " now.")
dest_container_ref = cf.create_container(destination_container_name)
 
# Alert the user that creating the container was successful
print(destination_container_name + " successfully created.")
 
# Make sure that the origin container exists before moving on
try:
    # Try to get reference to the origin container
    origin_container_ref = cf.get_container(origin_container_name)
except:
	# Remove the newly created destination container and exit
	print("Origin container must exist to copy objects.")
	print("Cleaning up by deleting the destination container")
	cf.delete(dest_container_ref, del_objects=True)
	exit("Exiting script.")
 
# Get all of the objects in the origin container
print("Getting all objects in the origin container.  This could take awhile...")
objects = origin_container_ref.get_objects(full_listing=True)
 
i = 0
for obj in objects:
	try:
	    cf.copy_object(origin_container_ref, obj, dest_container_ref, new_obj_name=obj.name, content_type=None)
	    print("Object: " + obj.name + " -- successfully transfered!")
	    i+=1
	except:
		print("There was an error that occurred!")
		print(str(i) + " files successfully transferred to " + destination_container_name)
 
print()
print("*****************************************************")
print()
print("File sync completed successfully!")
print(str(i) + " files synced from " + origin_container_name + " to " + destination_container_name)
print()
print("*****************************************************")
exit("Script Exiting")

Member for

3 years 9 months
Matt Eaton

Long time mobile team lead with a love for network engineering, security, IoT, oss, writing, wireless, and mobile.  Avid runner and determined health nut living in the greater Chicagoland area.