Bases: object
Middleware that will do many operations on a single request.
Extract Archive:
Expand tar files into a swift account. Request must be a PUT with the query parameter ?extract-archive=format specifying the format of archive file. Accepted formats are tar, tar.gz, and tar.bz2.
For a PUT to the following url:
/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz
UPLOAD_PATH is where the files will be expanded to. UPLOAD_PATH can be a container, a pseudo-directory within a container, or an empty string. The destination of a file in the archive will be built as follows:
/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH
Where FILE_PATH is the file name from the listing in the tar file.
If the UPLOAD_PATH is an empty string, containers will be auto created accordingly and files in the tar that would not map to any container (files in the base directory) will be ignored.
Only regular files will be uploaded. Empty directories, symlinks, etc will not be uploaded.
The response from bulk operations functions differently from other swift responses. This is because a short request body sent from the client could result in many operations on the proxy server and precautions need to be made to prevent the request from timing out due to lack of activity. To this end, the client will always receive a 200 OK response, regardless of the actual success of the call. The body of the response must be parsed to determine the actual success of the operation. In addition to this the client may receive zero or more whitespace characters prepended to the actual response body while the proxy server is completing the request.
The format of the response body defaults to text/plain but can be either json or xml depending on the Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml. An example body is as follows:
If all valid files were uploaded successfully the Response Status will be 201 Created. If any files failed to be created the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In both cases the response body will specify the number of files successfully uploaded and a list of the files that failed.
There are proxy logs created for each file (which becomes a subrequest) in the tar. The subrequest’s proxy log will have a swift.source set to “EA” the log’s content length will reflect the unzipped size of the file. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the unexpanded size of the tar.gz).
Bulk Delete:
Will delete multiple objects or containers from their account with a single request. Responds to POST requests with query parameter ?bulk-delete set. The request url is your storage url. The Content-Type should be set to text/plain. The body of the POST request will be a newline separated list of url encoded objects to delete. You can delete 10,000 (configurable) objects per request. The objects specified in the POST request body must be URL encoded and in the form:
/container_name/obj_name
or for a container (which must be empty at time of delete)
/container_name
The response is similar to extract archive as in every response will be a 200 OK and you must parse the response body for actual results. An example response is:
If all items were successfully deleted (or did not exist), the Response Status will be 200 OK. If any failed to delete, the response code corresponds to the subrequest’s error. Possible codes are 400, 401, 502 (on server errors), etc. In all cases the response body will specify the number of items successfully deleted, not found, and a list of those that failed. The return body will be formatted in the way specified in the request’s Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml.
There are proxy logs created for each object or container (which becomes a subrequest) that is deleted. The subrequest’s proxy log will have a swift.source set to “BD” the log’s content length of 0. If double proxy-logging is used the leftmost logger will not have a swift.source set and the content length will reflect the size of the payload sent to the proxy (the list of objects/containers to be deleted).
Checks if the container exists and if not try to create it. :params container_path: an unquoted path to a container to be created :returns: True if created container, False if container exists :raises: CreateContainerError when unable to create container
Will populate objs_to_delete with data from request input. :params req: a Swob request :returns: a list of the contents of req.body when separated by newline. :raises: HTTPException on failures
A generator that can be assigned to a swob Response’s app_iter which, when iterated over, will delete the objects specified in request body. Will occasionally yield whitespace while request is being processed. When the request is completed will yield a response body that can be parsed to determine success. See above documentation for details.
Params req: | a swob Request |
---|---|
Params objs_to_delete: | |
a list of dictionaries that specifies the objects to be deleted. If None, uses self.get_objs_to_delete to query request. |
A generator that can be assigned to a swob Response’s app_iter which, when iterated over, will extract and PUT the objects pulled from the request body. Will occasionally yield whitespace while request is being processed. When the request is completed will yield a response body that can be parsed to determine success. See above documentation for details.
Params req: | a swob Request |
---|---|
Params compress_type: | |
specifying the compression type of the tar. Accepts ‘’, ‘gz’, or ‘bz2’ |
Returns a properly formatted response body according to format. Handles json and xml, otherwise will return text/plain. Note: xml response does not include xml declaration. :params data_format: resulting format :params data_dict: generated data about results. :params error_list: list of quoted filenames that failed
CNAME Lookup Middleware
Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.
This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.
Bases: object
CNAME Lookup Middleware
See above for a full description.
Parameters: |
|
---|
Given a domain, returns its DNS CNAME mapping and DNS ttl.
Parameters: | domain – domain to query on |
---|---|
Returns: | (ttl, result) |
Bases: object
Cross domain middleware used to respond to requests for cross domain policy information.
If the path is /crossdomain.xml it will respond with an xml cross domain policy document. This allows web pages hosted elsewhere to use client side technologies such as Flash, Java and Silverlight to interact with the Swift API.
To enable this middleware, add it to the pipeline in your proxy-server.conf file. It should be added before any authentication (e.g., tempauth or keystone) middleware. In this example ellipsis (...) indicate other middleware you may have chosen to use:
[pipeline:main]
pipeline = ... crossdomain ... authtoken ... proxy-server
And add a filter section, such as:
[filter:crossdomain]
use = egg:swift#crossdomain
cross_domain_policy = <allow-access-from domain="*.example.com" />
<allow-access-from domain="www.example.com" secure="false" />
For continuation lines, put some whitespace before the continuation text. Ensure you put a completely blank line to terminate the cross_domain_policy value.
The cross_domain_policy name/value is optional. If omitted, the policy defaults as if you had specified:
cross_domain_policy = <allow-access-from domain="*" secure="false" />
Returns a 200 response with cross domain policy information
Swift will by default provide clients with an interface providing details about the installation. Unless disabled (i.e expose_info=false in Proxy Server Configuration), a GET request to /info will return configuration data in JSON format. An example response:
{"swift": {"version": "1.11.0"}, "staticweb": {}, "tempurl": {}}
This would signify to the client that swift version 1.11.0 is running and that staticweb and tempurl are available in this installation.
There may be administrator-only information available via /info. To retrieve it, one must use an HMAC-signed request, similar to TempURL. The signature may be produced like so:
swift-temp-url GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g
Domain Remap Middleware
Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.
container.account.storageurl/object gets translated to container.account.storageurl/path_root/account/container/object
account.storageurl/path_root/container/object gets translated to account.storageurl/path_root/account/container/object
Browsers can convert a host header to lowercase, so check that reseller prefix on the account is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. When none match, the default reseller prefix is used. When no default reseller prefix is configured, any request with an account prefix not in that list will be ignored by this middleware. reseller_prefixes defaults to ‘AUTH’.
Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what swift requires and are not explicitly checked. Simply put, the this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).
Also note that using container sync with remapped domain names is not advised. With container sync, you should use the true storage end points as sync destinations.
Bases: object
Domain Remap Middleware
See above for a full description.
Parameters: |
|
---|
Bases: object
Healthcheck middleware used for monitoring.
If the path is /healthcheck, it will respond 200 with “OK” as the body.
If the optional config parameter “disable_path” is set, and a file is present at that path, it will respond 503 with “DISABLED BY FILE” as the body.
Returns a 503 response with “DISABLED BY FILE” in the body.
Returns a 200 response with “OK” in the body.
Bases: object
Caching middleware that manages caching in swift.
Created on February 27, 2012
A filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.
Place early in the proxy-server pipeline after the left-most occurrence of the proxy-logging middleware (if present) and before the final proxy-logging middleware (if present) or the proxy-serer app itself, e.g.:
[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging name_check cache ratelimit tempauth sos proxy-logging proxy-server
[filter:name_check]
use = egg:swift#name_check
forbidden_chars = '"`<>
maximum_length = 255
There are default settings for forbidden_chars (FORBIDDEN_CHARS) and maximum_length (MAX_LENGTH)
The filter returns HTTPBadRequest if path is invalid.
@author: eamonn-otoole
Profiling middleware for Swift Servers.
The current implementation is based on eventlet aware profiler.(For the future, more profilers could be added in to collect more data for analysis.) Profiling all incoming requests and accumulating cpu timing statistics information for performance tuning and optimization. An mini web UI is also provided for profiling data analysis. It can be accessed from the URL as below.
Index page for browse profile data:
http://SERVER_IP:PORT/__profile__
List all profiles to return profile ids in json format:
http://SERVER_IP:PORT/__profile__/
http://SERVER_IP:PORT/__profile__/all
Retrieve specific profile data in different formats:
http://SERVER_IP:PORT/__profile__/PROFILE_ID?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/current?format=[default|json|csv|ods]
http://SERVER_IP:PORT/__profile__/all?format=[default|json|csv|ods]
Retrieve metrics from specific function in json format:
http://SERVER_IP:PORT/__profile__/PROFILE_ID/NFL?format=json
http://SERVER_IP:PORT/__profile__/current/NFL?format=json
http://SERVER_IP:PORT/__profile__/all/NFL?format=json
NFL is defined by concatenation of file name, function name and the first
line number.
e.g.::
account.py:50(GETorHEAD)
or with full path:
opt/stack/swift/swift/proxy/controllers/account.py:50(GETorHEAD)
A list of URL examples:
http://localhost:8080/__profile__ (proxy server)
http://localhost:6000/__profile__/all (object server)
http://localhost:6001/__profile__/current (container server)
http://localhost:6002/__profile__/12345?format=json (account server)
The profiling middleware can be configured in paste file for WSGI servers such as proxy, account, container and object servers. Please refer to the sample configuration files in etc directory.
The profiling data is provided with four formats such as binary(by default), json, csv and odf spreadsheet which requires installing odfpy library.
sudo pip install odfpy
There’s also a simple visualization capability which is enabled by using matplotlib toolkit. it is also required to be installed if you want to use it to visualize statistic data.
sudo apt-get install python-matplotlib