Running 1,266 km (786 miles):
Cycling 3865 km (2401 miles):
Let’s see if I can improve for next year
Running 1,266 km (786 miles):
Cycling 3865 km (2401 miles):
Let’s see if I can improve for next year
It seems that integrating Swift and Keystone together present some challenges to people and this is absolutely normal as there is a lot of changes going on. This is my attempt to document how everything is plugged together.
I am not going to explain how a middleware is supposed to work as this is nicely documented on Wikipedia :
http://en.wikipedia.org/wiki/Middleware
or how the auth middlewares works on Swift :
http://swift.openstack.org/development_auth.html
or even how this is plugged inside Keystone :
http://keystone.openstack.org/middleware_architecture.html
At first let’s get some of the wordings right :
Now that you keep this in mind let’s walk-though how a request will
look like.
At first your user connect to keystone and says this is my username for this
tenant and here is the secret/api key, give me the endpoints for the
services and add a token to it. This will look like this in curl :
curl -s -d '{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "password"}}}' -H 'Content-type: application/json' http://localhost:5000/v2.0/tokens |
If successfully authenticated you get back in Json those public/internal urls
for swift so you are able to connect, here is some part of the replied request :
{
"endpoints": [
{
"adminURL": "http://localhost:8080/",
"internalURL": "http://localhost:8080/v1/AUTH_2",
"publicURL": "http://localhost:8080/v1/AUTH_2",
"region": "RegionOne"
}
],
"name": "swift",
"type": "object-store"
}
[...]
"token": {
"expires": "2011-11-24T12:35:56",
"id": "ea29dae7-4c54-4e80-98e1-9f886acb389a",
"tenant": {
"id": "2",
"name": "demo"
}
},
So now the clients is going to get the publicURL (or can be internal) with the token and able to give request to swift with it. Let’s take the simple request which list the container, this is a basic GET on the account :
curl -v -H 'X-Auth-Token: ea29dae7-4c54-4e80-98e1-9f886acb389a' http://localhost:8080/v1/AUTH_2 |
which should come back by a 20* http code if that work.
What’s happening here is that when you connect to swift it will pass it to the middleware to make sure we are able to have access with that token.
The middleware will take that token connect to keystone admin url with the admin token and pass that user token to be validated. The query looks like this in curl :
curl -H 'X-Auth-Token: 7XX' http://localhost:35357/v2.0/tokens/ea29dae7-4c54-4e80-98e1-9f886acb389a |
note: localhost:35357 is the keystone admin url and 7XX is the admin token set in the configuration of the middleware.
if successful keystone will come back with a reply that look like this :
{ "access": { "token": { "expires": "2011-11-24T12:35:56", "id": "ea29dae7-4c54-4e80-98e1-9f886acb389a", "tenant": { "id": "2", "name": "demo" } }, "user": { "id": "2", "roles": [ { "id": "2", "name": "Member", "tenantId": "2" }, { "id": "5", "name": "SwiftOperator", "tenantId": "2" }, ], "username": "demo" } } } |
Let’s step back before more Curl command and understand a thing about Swift, a user of an account in Swift by default don’t have any rights at all but there is one user in that account whose able to give ACL on containers for other users. In swift keystone middleware we call it an Operator.
The way the middleware knows which user is able to be admin on an account is by using the roles matching to whatever configuration we have on the middleware setting called :
keystone_swift_operator_roles = Admin, SwiftOperator
since this user is part the SwiftOperator then it has access and he’s allowed to do whatever he wants for that account like creating containers or giving ACL to other users.
So let’s say we have a user called demo2 which is part of the demo account and have only the role Member to it and not SwiftOperator by default as we say before he will not be able to do much.
But if demo user give access to the group/role Memeber to a container via acl then demo2 will be able to do stuff on it.
We can all have fun with bunch of curl commands but since swift 1.4.7 the swift CLI tool have support for the auth server version 2.0 and allow you to connect to keystone for auth so we are going to use that instead.
Let first create a testcontainer and upload a file into it with our ‘operator’ user :
swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer |
now let’s give access to the Member group for that container on reading :
swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer -r Member |
and now if we try to read that file directly with demo2 it will be allowed :
swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo2 -K password download testcontainer etc/issue -o- |
Hope this make things a bit more clears how everything works, in the next part I am going to explain how the config files and packages will look like for installing keystone and swift.
I have done lately quite a bit of work with python-novaclient the (nova/keystone) openstack client. I often experiment it with ipython in the console.
There is a nice debugging facility in novaclient which you can see while using –debug argument on the command line and if you wanted to use it with ipython you could have that at the beginning of your session :
This would give you the details of the session showing you the REST requests and responses including the headers. It even show you the curl commands that you can use on the command line to experiment with it.
I have released a python binding to Rackspace CloudDNS here which allow you to create/update/delete domains and records. It’s available on github
https://github.com/rackspace/python-clouddns/
The binding is pretty simple and have unfortunately no documentation (or even tests) but you can figure out most of it from here :
https://github.com/rackspace/python-clouddns/blob/master/tests/t.py
I will be very welcoming pull request that add a bit of documentation.
A lot of our customers in Rackspace cloud has been asking how to mass edit firewalls of servers when you have multiple servers without doing it manually.
Part of my cloudservers-api-demo I have written a simple firewall scripts abstracting the Operating System firewall software to allow/enable/disable the firewall and ports/networks.
The script has been kept very simple by design and currently allow only to :
enable the firewall
disable the firewall
allow or disallow a port or a network
see firewall status
A management server under Ubuntu maverick.
A supported Operating System for clients which includes :
Debian.
Ubuntu.
RHEL.
Fedora
My patched python-cloudservers library (see below for installs).
Your SSH key installed on all VM for root users.
After you have kicked a VM with a Ubuntu maverick and connected to it as root you want first execute intall some prereq packages :
apt-get update && apt-get -y install python-stdeb git |
checkout my python-cloudservers library :
git clone git://github.com/chmouel/python-cloudservers.git |
after being checked-out you will go into the python-cloudservers directory which has just been created and do this :
cd python-cloudservers/ python setup.py install |
this should automatically install all the dependences.
Now you can install my api-demo which include the firewall script :
cd ../ git clone git://github.com/chmouel/cloudservers-api-demo |
You need to configure some environemnt variable first which keep information about your rackspace account.
edit your ~/.bashrc (or /etc/environement if you want to make it global) and configure those variable :
export RCLOUD_DATACENTER=UK export UK_RCLOUD_USER="MY_USERNAME" export UK_RCLOUD_KEY="MY_API_KEY" export UK_RCLOUD_AURL="https://lon.auth.api.rackspacecloud.com/v1.0"
or for the US you would have :
export RCLOUD_DATACENTER=US export UK_RCLOUD_USER="MY_USERNAME" export UK_RCLOUD_KEY="MY_API_KEY" export UK_RCLOUD_AURL="https://auth.api.rackspacecloud.com/v1.0"
source the ~/.bashrc or relog into your account to have those accounts set-up you can test it to see if that works by going to :
~/cloudservers-api-demo/python
and launch the command :
./list-servers.py
to test if this is working properly (it should list your servers for your DATACENTER)
you are now basically ready to mass update firewall on all servers.
Let’s say you have two web servers named web1 and web2 and two db servers named db1 and db2 and you would like to allow the 80 port on the web servers and 3306 port on the db servers.
You would have to go to this directory :
~/cloudservers-api-demo/firewall/
and first execute this command to see the help/usages :
./fw-control.py --help
so let’s say to enable the firewall on all the web and db server first you can do :
./fw-control.py -s "web db" enable
it will connect and enable the firewall on all the servers which match the name web and db.
now let’s say we want to enable port 80 on the web :
./fw-control.py -s "web" allow port 80
if you log into the servers you can check with
iptables -L -n
that it it has been enabled properly.
This is simple enough for you to modify the script to your liking to make it more modular for your specific environement.
Currently in Rackspace-Cloud when you are shutting-down your Cloud Servers you are still paying for it.
The reason is that when the Cloud Server is shut-down your CloudServer is still sitting on the hyper-visor and still use resources on the Cloud and then get you billed for it.
There is a way to get around it by having the CloudServer stored as an image into CloudFiles.
The caveat with this solution is that every-time you are creating a server out of the stored image you are getting a new IP and in certain cases you would need to make a change in your application with the new IP.
If you only use domain names instead of IP in your application you are not dependent of the IP change, to update the domain with the new IP after creating the VM you can either :
– Have a dynamic DNS or ‘Cloud DNS’ updated just after you created your server out of the image.
– Have a script going into your server and update the IP directly in /etc/hosts.
In programming words this is the steps you would do. I am using the python-nova binding which allow you to connect to RackSpace Cloud.
At first I am going to create an object which we are going to authenticate
import novaclient cx = novaclient.OpenStack(USERNAME, API_KEY) |
or for the UK :
import novaclient cx = novaclient.OpenStack("USERNAME", "API_KEY", 'https://lon.auth.api.rackspacecloud.com/v1.0') |
cx is going to be the object from where we can do things on it. Let’s first find the server server that we want, assuming your server is called test you would get the server like this :
server = cx.servers.find(name='test') |
The variable ‘server’ contain our server ‘object’ and we can get its ID out of it :
server_id = server.id |
We got the function cx.images.create to create an image from a server which accept as first argument the image name and the second the server id we just got. this would start the creation of the image :
cx.images.create("backup_server", server_id) |
The server has started to get backed-up into your Cloud Files account, you can see it directly into the “My Server Images” tab of Hosting => Cloud Servers section :
You can now delete the server since it’s ‘backuped’ into cloud files ;
server.delete() |
At this time you are not billed for your Cloud Servers anymore and only for the storage usage in Cloud Files.
When you want to restore the image as a server, you would first get the id of your image :
image = cx.images.find(name='backup-test') image_id = image.id |
and create the server out of this image :
CNX.servers.create(image=image_id, flavor=1, name="test", ) |
The flavor argument is the type of image you want, 1 the minimal 256M flavor. The full list is :
In [14]: for x in cx.flavors.list(): ....: print x.id, '-', x.name ....: ....: 1 - 256 server 2 - 512 server 3 - 1GB server 4 - 2GB server 5 - 4GB server 6 - 8GB server 7 - 15.5GB server |
When the server has created it should be exactly the same as what you have before created in image. You can now run a script using SSH with SSH keys to log into servers and do adjustment with the new IP.
Sometime ago I wrote a FTP proxy to RackSpace Cloud Files which expose Rackspace Cloud Files as a FTP server acting as a proxy.
Thanks to the OpenSource community a user on github took it and add support OpenStack and all the latest features available in Cloud Files.
It is now pretty robust and works pretty well via nautilus even with the pseudo hierarchical folder feature. The fun part here is that it allow you to effectively have a Cloud Drive where you can easily store your files/backup from your Linux desktop via nautilus built-in ftp support.
I have made a video that show how it works :
I have just uploaded python-cloudfiles to pypi available here
This make things easy to add as a dependence of your project like you can have something like this in your setup.py :
requirements = ['python-cloudfiles']
and it will automatically download it as part of the dependence with easy_install or pip.
cool kids on latest debian/ubuntu can do stuff like this (from python-stdeb package) :
pypi-install python-cloudfiles
which would automatically download the tarball from pypi and install it as a packages (like the way it should be for prod machine!)
If you have a virtualenv environment you can easily do a (needs python-pip package) :
pip -E /usr/local/myvirtualenvroot install python-cloudfiles
and magic would be done to get you on latest python-cloudfiles.
As a bonus side you can browse online the python-cloudfiles library :
http://packages.python.org/python-cloudfiles/
[Update] This has been renamed back to python-cloudfiles please update your setup.py or scripts.
Fireuploader is a Firefox addon that gives you a nice GUI to upload files via your firefox browser.
I have made a special version of the extensions to make it works with the RackSpace Cloud UK.
Install the addon from here :
http://www.chmouel.com/pub/firefox_universal_uploader__fireuploader_-0.4.5-fx+mz+ukcf.xpi
Allow the website by clicking Allow in the yellow bar on the top as seen in this screenshot :
Click on Install Now and restart Firefox.
After the Firefox browser is restarted you can go in Tools => Fireuploader and choose Rackspace Cloud UK in the dropdown list :
and click on Manage Account :
enter your UK username and UK API Key and “Save Password” if you like and it should log you into your UK Cloud :
One of the last library I didn’t documented in my earlier post was php-cloudfiles. You need to have at least the version 1.7.6 released to have support to different auth_server and when you have that you can do it like this to get access to cloud files via the library :
< ?php require_once("cloudfiles.php"); # Allow override by environment variable $USER = "MY_API_USERNAME"; $API_KEY = "MY_API_KEY"; $auth = new CF_Authentication($USER, $API_KEY, NULL, UK_AUTHURL); $auth->authenticate(); ?> |