Middleware

Account Quotas

Bulk Operations (Delete and Archive Auto Extraction)

Middleware that will perform many operations on a single request.

Extract Archive

Expand tar files into a Swift account. Request must be a PUT with the query parameter ?extract-archive=format specifying the format of archive file. Accepted formats are tar, tar.gz, and tar.bz2.

For a PUT to the following url:

/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz

UPLOAD_PATH is where the files will be expanded to. UPLOAD_PATH can be a container, a pseudo-directory within a container, or an empty string. The destination of a file in the archive will be built as follows:

/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH

Where FILE_PATH is the file name from the listing in the tar file.

If the UPLOAD_PATH is an empty string, containers will be auto created accordingly and files in the tar that would not map to any container (files in the base directory) will be ignored.

Only regular files will be uploaded. Empty directories, symlinks, etc will not be uploaded.

Content Type

If the content-type header is set in the extract-archive call, Swift will assign that content-type to all the underlying files. The bulk middleware will extract the archive file and send the internal files using PUT operations using the same headers from the original request (e.g. auth-tokens, content-Type, etc.). Notice that any middleware call that follows the bulk middleware does not know if this was a bulk request or if these were individual requests sent by the user.

In order to make Swift detect the content-type for the files based on the file extension, the content-type in the extract-archive call should not be set. Alternatively, it is possible to explicitly tell Swift to detect the content type using this header:

X-Detect-Content-Type: true

For example:

curl -X PUT http://127.0.0.1/v1/AUTH_acc/cont/$?extract-archive=tar
 -T backup.tar
 -H "Content-Type: application/x-tar"
 -H "X-Auth-Token: xxx"
 -H "X-Detect-Content-Type: true"

Assigning Metadata

The tar file format (1) allows for UTF-8 key/value pairs to be associated with each file in an archive. If a file has extended attributes, then tar will store those as key/value pairs. The bulk middleware can read those extended attributes and convert them to Swift object metadata. Attributes starting with “user.meta” are converted to object metadata, and “user.mime_type” is converted to Content-Type.

For example:

setfattr -n user.mime_type -v "application/python-setup" setup.py
setfattr -n user.meta.lunch -v "burger and fries" setup.py
setfattr -n user.meta.dinner -v "baked ziti" setup.py
setfattr -n user.stuff -v "whee" setup.py

Will get translated to headers:

Content-Type: application/python-setup
X-Object-Meta-Lunch: burger and fries
X-Object-Meta-Dinner: baked ziti

The bulk middleware will handle xattrs stored by both GNU and BSD tar (2). Only xattrs user.mime_type and user.meta.* are processed. Other attributes are ignored.

Notes:

(1) The POSIX 1003.1-2001 (pax) format. The default format on GNU tar 1.27.1 or later.

(2) Even with pax-format tarballs, different encoders store xattrs slightly differently; for example, GNU tar stores the xattr “user.userattribute” as pax header “SCHILY.xattr.user.userattribute”, while BSD tar (which uses libarchive) stores it as “LIBARCHIVE.xattr.user.userattribute”.

Response

The response from bulk operations functions differently from other Swift responses. This is because a short request body sent from the client could result in many operations on the proxy server and precautions need to be made to prevent the request from timing out due to lack of activity. To this end, the client will always receive a 200 OK response, regardless of the actual success of the call. The body of the response must be parsed to determine the actual success of the operation. In addition to this the client may receive zero or more whitespace characters prepended to the actual response body while the proxy server is completing the request.

The format of the response body defaults to text/plain but can be either json or xml depending on the Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml. An example body is as follows:

{"Response Status": "201 Created",
 "Response Body": "",
 "Errors": [],
 "Number Files Created": 10}

If all valid files were uploaded successfully the Response Status will be 201 Created. If any files failed to be created the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In both cases the response body will specify the number of files successfully uploaded and a list of the files that failed.

There are proxy logs created for each file (which becomes a subrequest) in the tar. The subrequest’s proxy log will have a swift.source set to “EA” the log’s content length will reflect the unzipped size of the file. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the unexpanded size of the tar.gz).

Bulk Delete

Will delete multiple objects or containers from their account with a single request. Responds to POST requests with query parameter ?bulk-delete set. The request url is your storage url. The Content-Type should be set to text/plain. The body of the POST request will be a newline separated list of url encoded objects to delete. You can delete 10,000 (configurable) objects per request. The objects specified in the POST request body must be URL encoded and in the form:

/container_name/obj_name

or for a container (which must be empty at time of delete):

/container_name

The response is similar to extract archive as in every response will be a 200 OK and you must parse the response body for actual results. An example response is:

{"Number Not Found": 0,
 "Response Status": "200 OK",
 "Response Body": "",
 "Errors": [],
 "Number Deleted": 6}

If all items were successfully deleted (or did not exist), the Response Status will be 200 OK. If any failed to delete, the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In all cases the response body will specify the number of items successfully deleted, not found, and a list of those that failed. The return body will be formatted in the way specified in the request’s Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml.

There are proxy logs created for each object or container (which becomes a subrequest) that is deleted. The subrequest’s proxy log will have a swift.source set to “BD” the log’s content length of 0. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the list of objects/containers to be deleted).

swift.common.middleware.bulk.get_response_body(data_format, data_dict, error_list)

Returns a properly formatted response body according to format. Handles json and xml, otherwise will return text/plain. Note: xml response does not include xml declaration. :params data_format: resulting format :params data_dict: generated data about results. :params error_list: list of quoted filenames that failed

CatchErrors

CNAME Lookup

CNAME Lookup Middleware

Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.

This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.

class swift.common.middleware.cname_lookup.CNAMELookupMiddleware(app, conf)

Bases: object

CNAME Lookup Middleware

See above for a full description.

Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.
swift.common.middleware.cname_lookup.lookup_cname(domain)

Given a domain, returns its DNS CNAME mapping and DNS ttl.

Parameters:domain – domain to query on
Returns:(ttl, result)

Container Quotas

Container Sync Middleware

Cross Domain Policies

class swift.common.middleware.crossdomain.CrossDomainMiddleware(app, conf, *args, **kwargs)

Bases: object

Cross domain middleware used to respond to requests for cross domain policy information.

If the path is /crossdomain.xml it will respond with an xml cross domain policy document. This allows web pages hosted elsewhere to use client side technologies such as Flash, Java and Silverlight to interact with the Swift API.

To enable this middleware, add it to the pipeline in your proxy-server.conf file. It should be added before any authentication (e.g., tempauth or keystone) middleware. In this example ellipsis (...) indicate other middleware you may have chosen to use:

[pipeline:main]
pipeline =  ... crossdomain ... authtoken ... proxy-server

And add a filter section, such as:

[filter:crossdomain]
use = egg:swift#crossdomain
cross_domain_policy = <allow-access-from domain="*.example.com" />
    <allow-access-from domain="www.example.com" secure="false" />

For continuation lines, put some whitespace before the continuation text. Ensure you put a completely blank line to terminate the cross_domain_policy value.

The cross_domain_policy name/value is optional. If omitted, the policy defaults as if you had specified:

cross_domain_policy = <allow-access-from domain="*" secure="false" />
GET(req)

Returns a 200 response with cross domain policy information

Discoverability

Swift will by default provide clients with an interface providing details about the installation. Unless disabled (i.e expose_info=false in Proxy Server Configuration), a GET request to /info will return configuration data in JSON format. An example response:

{"swift": {"version": "1.11.0"}, "staticweb": {}, "tempurl": {}}

This would signify to the client that swift version 1.11.0 is running and that staticweb and tempurl are available in this installation.

There may be administrator-only information available via /info. To retrieve it, one must use an HMAC-signed request, similar to TempURL. The signature may be produced like so:

swift-temp-url GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g

Domain Remap

Domain Remap Middleware

Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.

container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object

account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object

Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. When none match, the default reseller prefix is used. When no default reseller prefix is configured, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.

Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).

Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.

class swift.common.middleware.domain_remap.DomainRemapMiddleware(app, conf)

Bases: object

Domain Remap Middleware

See above for a full description.

Parameters:
  • app – The next WSGI filter or app in the paste.deploy chain.
  • conf – The configuration dict for the middleware.

Dynamic Large Objects

DLO support centers around a user specified filter that matches segments and concatenates them together in object listing order. Please see the DLO docs for Dynamic Large Objects further details.

Encryption

Encryption middleware should be deployed in conjunction with the Keymaster middleware.

FormPost

GateKeeper

Healthcheck

class swift.common.middleware.healthcheck.HealthCheckMiddleware(app, conf)

Bases: object

Healthcheck middleware used for monitoring.

If the path is /healthcheck, it will respond 200 with “OK” as the body.

If the optional config parameter “disable_path” is set, and a file is present at that path, it will respond 503 with “DISABLED BY FILE” as the body.

DISABLED(req)

Returns a 503 response with “DISABLED BY FILE” in the body.

GET(req)

Returns a 200 response with “OK” in the body.

Keymaster

Keymaster middleware should be deployed in conjunction with the Encryption middleware.

KeystoneAuth

List Endpoints

Memcache

class swift.common.middleware.memcache.MemcacheMiddleware(app, conf)

Bases: object

Caching middleware that manages caching in swift.

Name Check (Forbidden Character Filter)

Created on February 27, 2012

A filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.

Place early in the proxy-server pipeline after the left-most occurrence of the proxy-logging middleware (if present) and before the final proxy-logging middleware (if present) or the proxy-serer app itself, e.g.:

[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging name_check cache ratelimit tempauth sos proxy-logging proxy-server

[filter:name_check]
use = egg:swift#name_check
forbidden_chars = '"`<>
maximum_length = 255

There are default settings for forbidden_chars (FORBIDDEN_CHARS) and maximum_length (MAX_LENGTH)

The filter returns HTTPBadRequest if path is invalid.

@author: eamonn-otoole

Object Versioning

Proxy Logging

Ratelimit

Recon

Server Side Copy

Static Large Objects

Please see the SLO docs for Static Large Objects further details.

StaticWeb

TempAuth

TempURL

XProfile

Profiling middleware for Swift Servers.

The current implementation is based on eventlet aware profiler.(For the future, more profilers could be added in to collect more data for analysis.) Profiling all incoming requests and accumulating cpu timing statistics information for performance tuning and optimization. An mini web UI is also provided for profiling data analysis. It can be accessed from the URL as below.

Index page for browse profile data:

http://SERVER_IP:PORT/__profile__

List all profiles to return profile ids in json format:

http://SERVER_IP:PORT/__profile__/
http://SERVER_IP:PORT/__profile__/all

Retrieve specific profile data in different formats:

http://SERVER_IP:PORT/__profile__/PROFILE_ID?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/current?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/all?format=[default|json|csv|ods]

Retrieve metrics from specific function in json format:

http://SERVER_IP:PORT/__profile__/PROFILE_ID/NFL?format=json
http://SERVER_IP:PORT/__profile__/current/NFL?format=json
http://SERVER_IP:PORT/__profile__/all/NFL?format=json

NFL is defined by concatenation of file name, function name and the first
line number.
e.g.::
    account.py:50(GETorHEAD)
or with full path:
    opt/stack/swift/swift/proxy/controllers/account.py:50(GETorHEAD)

A list of URL examples:

http://localhost:8080/__profile__    (proxy server)
http://localhost:6200/__profile__/all    (object server)
http://localhost:6201/__profile__/current    (container server)
http://localhost:6202/__profile__/12345?format=json    (account server)

The profiling middleware can be configured in paste file for WSGI servers such as proxy, account, container and object servers. Please refer to the sample configuration files in etc directory.

The profiling data is provided with four formats such as binary(by default), json, csv and odf spreadsheet which requires installing odfpy library.

sudo pip install odfpy

There’s also a simple visualization capability which is enabled by using matplotlib toolkit. it is also required to be installed if you want to use it to visualize statistic data.

sudo apt-get install python-matplotlib