Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.
Warning
If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.
swift.common.bufferedhttp.
BufferedHTTPConnection
(host, port=None, strict=None, timeout=<object object>, source_address=None)¶Bases: httplib.HTTPConnection
HTTPConnection class that uses BufferedHTTPResponse
putrequest
(method, url, skip_host=0, skip_accept_encoding=0)¶Send a request to the server.
Parameters: |
|
---|
response_class
¶alias of BufferedHTTPResponse
swift.common.bufferedhttp.
BufferedHTTPResponse
(sock, debuglevel=0, strict=0, method=None)¶Bases: httplib.HTTPResponse
HTTPResponse class that buffers reading of headers
nuke_from_orbit
()¶Terminate the socket with extreme prejudice.
Closes the underlying socket regardless of whether or not anyone else has references to it. Use this when you are certain that nobody else you care about has a reference to this socket.
swift.common.bufferedhttp.
http_connect
(ipaddr, port, device, partition, method, path, headers=None, query_string=None, ssl=False)¶Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
swift.common.bufferedhttp.
http_connect_raw
(ipaddr, port, method, path, headers=None, query_string=None, ssl=False)¶Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
swift.common.constraints.
check_account_format
(req, name)¶Validate that the header contains valid account or container name.
Parameters: |
|
---|---|
Returns: | A properly encoded account name or container name |
Raises: | HTTPPreconditionFailed – if account header is not well formatted. |
swift.common.constraints.
check_container_format
(req, name)¶Validate that the header contains valid account or container name.
Parameters: |
|
---|---|
Returns: | A properly encoded account name or container name |
Raises: | HTTPPreconditionFailed – if account header is not well formatted. |
swift.common.constraints.
check_delete_headers
(request)¶Check that ‘x-delete-after’ and ‘x-delete-at’ headers have valid values. Values should be positive integers and correspond to a time greater than the request timestamp.
If the ‘x-delete-after’ header is found then its value is used to compute an ‘x-delete-at’ value which takes precedence over any existing ‘x-delete-at’ header.
Parameters: | request – the swob request object |
---|---|
Raises: | HTTPBadRequest in case of invalid values |
Returns: | the swob request object |
swift.common.constraints.
check_dir
(root, drive)¶Verify that the path to the device is a directory and is a lesser constraint that is enforced when a full mount_check isn’t possible with, for instance, a VM using loopback or partitions.
Parameters: |
|
---|---|
Returns: | full path to the device |
Raises: | ValueError – if drive fails to validate |
swift.common.constraints.
check_drive
(root, drive, mount_check)¶Validate the path given by root and drive is a valid existing directory.
Parameters: |
|
---|---|
Returns: | full path to the device |
Raises: | ValueError – if drive fails to validate |
swift.common.constraints.
check_float
(string)¶Helper function for checking if a string can be converted to a float.
Parameters: | string – string to be verified as a float |
---|---|
Returns: | True if the string can be converted to a float, False otherwise |
swift.common.constraints.
check_metadata
(req, target_type)¶Check metadata sent in the request headers. This should only check that the metadata in the request given is valid. Checks against account/container overall metadata should be forwarded on to its respective server to be checked.
Parameters: |
|
---|---|
Returns: | HTTPBadRequest with bad metadata otherwise None |
swift.common.constraints.
check_mount
(root, drive)¶Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidentally filling up the root partition.
Parameters: |
|
---|---|
Returns: | full path to the device |
Raises: | ValueError – if drive fails to validate |
swift.common.constraints.
check_name_format
(req, name, target_type)¶Validate that the header contains valid account or container name.
Parameters: |
|
---|---|
Returns: | A properly encoded account name or container name |
Raises: | HTTPPreconditionFailed – if account header is not well formatted. |
swift.common.constraints.
check_object_creation
(req, object_name)¶Check to ensure that everything is alright about an object to be created.
Parameters: |
|
---|---|
Returns: | HTTPRequestEntityTooLarge – the object is too large |
Returns: | HTTPLengthRequired – missing content-length header and not a chunked request |
Returns: | HTTPBadRequest – missing or bad content-type header, or bad metadata |
Returns: | HTTPNotImplemented – unsupported transfer-encoding header value |
swift.common.constraints.
check_utf8
(string)¶Validate if a string is valid UTF-8 str or unicode and that it does not contain any null character.
Parameters: | string – string to be validated |
---|---|
Returns: | True if the string is valid utf-8 str or unicode and contains no null characters, False otherwise |
swift.common.constraints.
reload_constraints
()¶Parse SWIFT_CONF_FILE and reset module level global constraint attrs, populating OVERRIDE_CONSTRAINTS AND EFFECTIVE_CONSTRAINTS along the way.
swift.common.constraints.
valid_api_version
(version)¶Checks if the requested version is valid.
Currently Swift only supports “v1” and “v1.0”.
swift.common.constraints.
valid_timestamp
(request)¶Helper function to extract a timestamp from requests that require one.
Parameters: | request – the swob request object |
---|---|
Returns: | a valid Timestamp instance |
Raises: | HTTPBadRequest – on missing or invalid X-Timestamp |
swift.common.container_sync_realms.
ContainerSyncRealms
(conf_path, logger)¶Bases: object
Loads and parses the container-sync-realms.conf, occasionally checking the file’s mtime to see if it needs to be reloaded.
clusters
(realm)¶Returns a list of clusters for the realm.
endpoint
(realm, cluster)¶Returns the endpoint for the cluster in the realm.
get_sig
(request_method, path, x_timestamp, nonce, realm_key, user_key)¶Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for the information given.
Parameters: |
|
---|---|
Returns: | hexdigest str of the HMAC-SHA1 for the request. |
key
(realm)¶Returns the key for the realm.
key2
(realm)¶Returns the key2 for the realm.
realms
()¶Returns a list of realms.
reload
()¶Forces a reload of the conf file.
Internal client library for making calls directly to the servers rather than through the proxy.
swift.common.direct_client.
DirectClientException
(stype, method, node, part, path, resp, host=None)¶swift.common.direct_client.
direct_delete_account
(node, part, account, conn_timeout=5, response_timeout=15, headers=None)¶swift.common.direct_client.
direct_delete_container
(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None)¶Delete container directly from the container server.
Parameters: |
|
---|---|
Raises: | ClientException – HTTP DELETE request failed |
swift.common.direct_client.
direct_delete_container_object
(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶swift.common.direct_client.
direct_delete_object
(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶Delete object directly from the object server.
Parameters: |
|
---|---|
Raises: | ClientException – HTTP DELETE request failed |
swift.common.direct_client.
direct_get_account
(node, part, account, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15, end_marker=None, reverse=None)¶Get listings directly from the account server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of containers) The response headers will HeaderKeyDict. |
swift.common.direct_client.
direct_get_container
(node, part, account, container, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15, end_marker=None, reverse=None, headers=None)¶Get container listings directly from the container server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of objects) The response headers will be a HeaderKeyDict. |
swift.common.direct_client.
direct_get_object
(node, part, account, container, obj, conn_timeout=5, response_timeout=15, resp_chunk_size=None, headers=None)¶Get object directly from the object server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, the object’s contents) The response headers will be a HeaderKeyDict. |
Raises: | ClientException – HTTP GET request failed |
swift.common.direct_client.
direct_get_suffix_hashes
(node, part, suffixes, conn_timeout=5, response_timeout=15, headers=None)¶Get suffix hashes directly from the object server.
Parameters: |
|
---|---|
Returns: | dict of suffix hashes |
Raises: | ClientException – HTTP REPLICATE request failed |
swift.common.direct_client.
direct_head_container
(node, part, account, container, conn_timeout=5, response_timeout=15)¶Request container information directly from the container server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers in a HeaderKeyDict |
Raises: | ClientException – HTTP HEAD request failed |
swift.common.direct_client.
direct_head_object
(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶Request object information directly from the object server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers in a HeaderKeyDict |
Raises: | ClientException – HTTP HEAD request failed |
swift.common.direct_client.
direct_post_object
(node, part, account, container, name, headers, conn_timeout=5, response_timeout=15)¶Direct update to object metadata on object server.
Parameters: |
|
---|---|
Raises: | ClientException – HTTP POST request failed |
swift.common.direct_client.
direct_put_container
(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None, contents=None, content_length=None, chunk_size=65535)¶Make a PUT request to a container server.
Parameters: |
|
---|---|
Raises: | ClientException – HTTP PUT request failed |
swift.common.direct_client.
direct_put_container_object
(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶swift.common.direct_client.
direct_put_object
(node, part, account, container, name, contents, content_length=None, etag=None, content_type=None, headers=None, conn_timeout=5, response_timeout=15, chunk_size=65535)¶Put object directly from the object server.
Parameters: |
|
---|---|
Returns: | etag from the server response |
Raises: | ClientException – HTTP PUT request failed |
swift.common.direct_client.
gen_headers
(hdrs_in=None, add_ts=True)¶Get the headers ready for a request. All requests should have a User-Agent string, but if one is passed in don’t over-write it. Not all requests will need an X-Timestamp, but if one is passed in do not over-write it.
Parameters: |
|
---|---|
Returns: | HeaderKeyDict based on headers and ready for the request |
swift.common.direct_client.
retry
(func, *args, **kwargs)¶Helper function to retry a given function a number of times.
Parameters: |
|
---|---|
Returns: | result of func |
Raises: | ClientException – all retries failed |
swift.common.exceptions.
APIVersionError
¶swift.common.exceptions.
ChunkReadError
¶swift.common.exceptions.
ChunkReadTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.exceptions.
ChunkWriteTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.exceptions.
ClientException
(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=None, http_reason='', http_device='', http_response_content='', http_headers=None)¶Bases: exceptions.Exception
swift.common.exceptions.
ConnectionTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.exceptions.
DiskFileBadMetadataChecksum
¶swift.common.exceptions.
DiskFileCollision
¶swift.common.exceptions.
DiskFileDeleted
(metadata=None)¶swift.common.exceptions.
DiskFileError
¶swift.common.exceptions.
DiskFileExpired
(metadata=None)¶swift.common.exceptions.
DiskFileNoSpace
¶swift.common.exceptions.
DiskFileNotExist
¶swift.common.exceptions.
DiskFileNotOpen
¶swift.common.exceptions.
DiskFileQuarantined
¶swift.common.exceptions.
DiskFileXattrNotSupported
¶swift.common.exceptions.
DriveNotMounted
¶swift.common.exceptions.
DuplicateDeviceError
¶swift.common.exceptions.
EmptyRingError
¶swift.common.exceptions.
EncryptionException
¶swift.common.exceptions.
FileNotFoundError
¶swift.common.exceptions.
InsufficientStorage
¶swift.common.exceptions.
InvalidAccountInfo
¶swift.common.exceptions.
InvalidPidFileException
¶Bases: exceptions.Exception
swift.common.exceptions.
InvalidTimestamp
¶swift.common.exceptions.
LinkIterError
¶swift.common.exceptions.
ListingIterError
¶swift.common.exceptions.
ListingIterNotAuthorized
(aresp)¶swift.common.exceptions.
ListingIterNotFound
¶swift.common.exceptions.
LockTimeout
(seconds=None, msg=None)¶swift.common.exceptions.
MessageTimeout
(seconds=None, msg=None)¶Bases: eventlet.timeout.Timeout
swift.common.exceptions.
MimeInvalid
¶swift.common.exceptions.
MultiphasePUTNotSupported
¶swift.common.exceptions.
PathNotDir
¶Bases: exceptions.OSError
swift.common.exceptions.
PermissionError
¶swift.common.exceptions.
PutterConnectError
(status=None)¶Bases: exceptions.Exception
swift.common.exceptions.
RangeAlreadyComplete
¶swift.common.exceptions.
ReplicationException
¶Bases: exceptions.Exception
swift.common.exceptions.
ReplicationLockTimeout
(seconds=None, msg=None)¶swift.common.exceptions.
ResponseTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.exceptions.
RingBuilderError
¶swift.common.exceptions.
RingLoadError
¶swift.common.exceptions.
RingValidationError
¶swift.common.exceptions.
SegmentError
¶swift.common.exceptions.
ShortReadError
¶swift.common.exceptions.
SuffixSyncError
¶swift.common.exceptions.
SwiftException
¶Bases: exceptions.Exception
swift.common.exceptions.
UnPicklingError
¶swift.common.exceptions.
UnknownSecretIdError
¶swift.common.manager.
Manager
(servers, run_dir='/var/run/swift')¶Bases: object
Main class for performing commands on groups of servers.
Parameters: | servers – list of server names as strings |
---|
force_reload
(**kw)¶alias for reload
get_command
(cmd)¶Find and return the decorated method named like cmd
Parameters: | cmd – the command to get, a string, if not found raises UnknownCommandError |
---|
kill
(**kw)¶stop a server (no error if not running)
list_commands
()¶Get all publicly accessible commands
Returns: | a list of string tuples (cmd, help), the method names who are decorated as commands |
---|
no_daemon
(**kw)¶start a server interactively
no_wait
(**kw)¶spawn server and return immediately
once
(**kw)¶start server and run one pass on supporting daemons
reload
(**kw)¶graceful shutdown then restart on supporting servers
restart
(**kw)¶stops then restarts server
run_command
(cmd, **kwargs)¶Find the named command and run it
Parameters: | cmd – the command name to run |
---|
shutdown
(**kw)¶allow current requests to finish on supporting servers
start
(**kw)¶starts a server
status
(**kw)¶display status of tracked pids for server
stop
(**kw)¶stops a server
swift.common.manager.
Server
(server, run_dir='/var/run/swift')¶Bases: object
Manage operations on a server or group of servers of similar type
Parameters: | server – name of server |
---|
conf_files
(**kwargs)¶Get conf files for this server
Param: | number, if supplied will only lookup the nth server |
---|---|
Returns: | list of conf files |
get_conf_file_name
(pid_file)¶Translate pid_file to a corresponding conf_file
Parameters: | pid_file – a pid_file for this server, a string |
---|---|
Returns: | the conf_file for this pid_file |
get_pid_file_name
(conf_file)¶Translate conf_file to a corresponding pid_file
Parameters: | conf_file – an conf_file for this server, a string |
---|---|
Returns: | the pid_file for this conf_file |
get_running_pids
(**kwargs)¶Get running pids
Returns: | a dict mapping pids (ints) to pid_files (paths) |
---|
interact
(**kwargs)¶wait on spawned procs to terminate
iter_pid_files
(**kwargs)¶Generator, yields (pid_file, pids)
kill_running_pids
(**kwargs)¶Kill running pids
Parameters: | graceful – if True, attempt SIGHUP on supporting servers |
---|---|
Returns: | a dict mapping pids (ints) to pid_files (paths) |
launch
(**kwargs)¶Collect conf files and attempt to spawn the processes for this server
pid_files
(**kwargs)¶Get pid files for this server
Param: | number, if supplied will only lookup the nth server |
---|---|
Returns: | list of pid files |
signal_pids
(sig, **kwargs)¶Send a signal to pids for this server
Parameters: | sig – signal to send |
---|---|
Returns: | a dict mapping pids (ints) to pid_files (paths) |
spawn
(conf_file, once=False, wait=True, daemon=True, additional_args=None, **kwargs)¶Launch a subprocess for this server.
Parameters: |
|
---|---|
Returns: | the pid of the spawned process |
status
(pids=None, **kwargs)¶Display status of server
Param: | pids, if not supplied pids will be populated automatically |
---|---|
Param: | number, if supplied will only lookup the nth server |
Returns: | 1 if server is not running, 0 otherwise |
stop
(**kwargs)¶Send stop signals to pids for this server
Returns: | a dict mapping pids (ints) to pid_files (paths) |
---|
wait
(**kwargs)¶wait on spawned procs to start
swift.common.manager.
UnknownCommandError
¶Bases: exceptions.Exception
swift.common.manager.
command
(func)¶Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.
Parameters: | func – function to make public |
---|
swift.common.manager.
kill_group
(pid, sig)¶Send signal to process group
: param pid: process id : param sig: signal to send
swift.common.manager.
safe_kill
(pid, sig, name)¶Send signal to process and check process name
: param pid: process id : param sig: signal to send : param name: name to ensure target process
swift.common.manager.
setup_env
()¶Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp
swift.common.manager.
watch_server_pids
(server_pids, interval=1, **kwargs)¶Monitor a collection of server pids yielding back those pids that aren’t responding to signals.
Parameters: | server_pids – a dict, lists of pids [int,…] keyed on Server objects |
---|
Why our own memcache client? By Michael Barton
python-memcached doesn’t use consistent hashing, so adding or removing a memcache server from the pool invalidates a huge percentage of cached items.
If you keep a pool of python-memcached client objects, each client object has its own connection to every memcached server, only one of which is ever in use. So you wind up with n * m open sockets and almost all of them idle. This client effectively has a pool for each server, so the number of backend connections is hopefully greatly reduced.
python-memcache uses pickle to store things, and there was already a huge stink about Swift using pickles in memcache (http://osvdb.org/show/osvdb/86581). That seemed sort of unfair, since nova and keystone and everyone else use pickles for memcache too, but it’s hidden behind a “standard” library. But changing would be a security regression at this point.
Also, pylibmc wouldn’t work for us because it needs to use python sockets in order to play nice with eventlet.
Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:
http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt
swift.common.memcached.
MemcacheConnPool
(server, size, connect_timeout)¶Bases: eventlet.pools.Pool
Connection pool for Memcache Connections
The server parameter can be a hostname, an IPv4 address, or an IPv6
address with an optional port. See
swift.common.utils.parse_socket_string()
for details.
create
()¶Generate a new pool item. In order for the pool to function, either this method must be overriden in a subclass or the pool must be constructed with the create argument. It accepts no arguments and returns a single instance of whatever thing the pool is supposed to contain.
In general, create()
is called whenever the pool exceeds its
previous high-water mark of concurrently-checked-out-items. In other
words, in a new pool with min_size of 0, the very first call
to get()
will result in a call to create()
. If the first
caller calls put()
before some other caller calls get()
,
then the first item will be returned, and create()
will not be
called a second time.
get
()¶Return an item from the pool, when one is available. This may cause the calling greenthread to block.
swift.common.memcached.
MemcacheConnectionError
¶Bases: exceptions.Exception
swift.common.memcached.
MemcachePoolTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.memcached.
MemcacheRing
(servers, connect_timeout=0.3, io_timeout=2.0, pool_timeout=1.0, tries=3, allow_pickle=False, allow_unpickle=False, max_conns=2)¶Bases: object
Simple, consistent-hashed memcache client.
decr
(key, delta=1, time=0)¶Decrements a key which has a numeric value by delta. Calls incr with -delta.
Parameters: |
|
---|---|
Returns: | result of decrementing |
Raises: |
delete
(key)¶Deletes a key/value pair from memcache.
Parameters: | key – key to be deleted |
---|
get
(key)¶Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.
Parameters: | key – key |
---|---|
Returns: | value of the key in memcache |
get_multi
(keys, server_key)¶Gets multiple values from memcache for the given keys.
Parameters: |
|
---|---|
Returns: | list of values |
incr
(key, delta=1, time=0)¶Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.
Parameters: |
|
---|---|
Returns: | result of incrementing |
Raises: |
set
(key, value, serialize=True, time=0, min_compress_len=0)¶Set a key/value pair in memcache
Parameters: |
|
---|
set_multi
(mapping, server_key, serialize=True, time=0, min_compress_len=0)¶Sets multiple key/value pairs in memcache.
Parameters: |
|
---|---|
Min_compress_len: | |
minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it |
swift.common.memcached.
sanitize_timeout
(timeout)¶Sanitize a timeout value to use an absolute expiration time if the delta is greater than 30 days (in seconds). Note that the memcached server translates negative values to mean a delta of 30 days in seconds (and 1 additional second), client beware.
Implementation of WSGI Request and Response objects.
This library has a very similar API to Webob. It wraps WSGI request environments and response values into objects that are more friendly to interact with.
Why Swob and not just use WebOb? By Michael Barton
We used webob for years. The main problem was that the interface wasn’t stable. For a while, each of our several test suites required a slightly different version of webob to run, and none of them worked with the then-current version. It was a huge headache, so we just scrapped it.
This is kind of a ton of code, but it’s also been a huge relief to not have to scramble to add a bunch of code branches all over the place to keep Swift working every time webob decides some interface needs to change.
swift.common.swob.
Accept
(headerval)¶Bases: object
Wraps a Request’s Accept header as a friendly object.
Parameters: | headerval – value of the header as a str |
---|
best_match
(options)¶Returns the item from “options” that best matches the accept header. Returns None if no available options are acceptable to the client.
Parameters: | options – a list of content-types the server can respond with |
---|---|
Raises: | ValueError – if the header is malformed |
swift.common.swob.
HTTPException
(*args, **kwargs)¶Bases: swift.common.swob.Response
, exceptions.Exception
swift.common.swob.
HeaderEnvironProxy
(environ)¶Bases: _abcoll.MutableMapping
A dict-like object that proxies requests to a wsgi environ, rewriting header keys to environ keys.
For example, headers[‘Content-Range’] sets and gets the value of headers.environ[‘HTTP_CONTENT_RANGE’]
keys
() → list of D's keys¶swift.common.swob.
Match
(headerval)¶Bases: object
Wraps a Request’s If-[None-]Match header as a friendly object.
Parameters: | headerval – value of the header as a str |
---|
swift.common.swob.
Range
(headerval)¶Bases: object
Wraps a Request’s Range header as a friendly object. After initialization, “range.ranges” is populated with a list of (start, end) tuples denoting the requested ranges.
If there were any syntactically-invalid byte-range-spec values, the constructor will raise a ValueError, per the relevant RFC:
“The recipient of a byte-range-set that includes one or more syntactically invalid byte-range-spec values MUST ignore the header field that includes that byte-range-set.”
According to the RFC 2616 specification, the following cases will be all considered as syntactically invalid, thus, a ValueError is thrown so that the range header will be ignored. If the range value contains at least one of the following cases, the entire range is considered invalid, ValueError will be thrown so that the header will be ignored.
Every syntactically valid range will be added into the ranges list even when some of the ranges may not be satisfied by underlying content.
Parameters: | headerval – value of the header as a str |
---|
ranges_for_length
(length)¶This method is used to return multiple ranges for a given length which should represent the length of the underlying content. The constructor method __init__ made sure that any range in ranges list is syntactically valid. So if length is None or size of the ranges is zero, then the Range header should be ignored which will eventually make the response to be 200.
If an empty list is returned by this method, it indicates that there are unsatisfiable ranges found in the Range header, 416 will be returned.
if a returned list has at least one element, the list indicates that there is at least one range valid and the server should serve the request with a 206 status code.
The start value of each range represents the starting position in the content, the end value represents the ending position. This method purposely adds 1 to the end number because the spec defines the Range to be inclusive.
The Range spec can be found at the following link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1
Parameters: | length – length of the underlying content |
---|
swift.common.swob.
Request
(environ)¶Bases: object
WSGI Request object.
accept
¶Retrieve and set the accept property in the WSGI environ, as a Accept object
acl
¶Get and set the swob.ACL property in the WSGI environment
blank
(path, environ=None, headers=None, body=None, **kwargs)¶Create a new request object with the given parameters, and an environment otherwise filled in with non-surprising default values.
Parameters: |
|
---|
body
¶Get and set the request body str
body_file
¶Get and set the wsgi.input property in the WSGI environment
call_application
(application)¶Calls the application with this request’s environment. Returns the status, headers, and app_iter for the response as a tuple.
Parameters: | application – the WSGI application to call |
---|
content_length
¶Retrieve and set the content-length header as an int
copy_get
()¶Makes a copy of the request, converting it to a GET.
get_response
(application)¶Calls the application with this request’s environment. Returns a Response object that wraps up the application’s result.
Parameters: | application – the WSGI application to call |
---|
host
¶Get and set the HTTP_HOST property in the WSGI environment
host_url
¶Get url for request/response up to path
if_match
¶Retrieve and set the if-match property in the WSGI environ, as a Match object
if_modified_since
¶Retrieve and set the if-modified-since header as a datetime, set it with a datetime, int, or str
if_none_match
¶Retrieve and set the if-none-match property in the WSGI environ, as a Match object
if_unmodified_since
¶Retrieve and set the if-unmodified-since header as a datetime, set it with a datetime, int, or str
message_length
()¶Properly determine the message length for this request. It will return an integer if the headers explicitly contain the message length, or None if the headers don’t contain a length. The ValueError exception will be raised if the headers are invalid.
Raises: |
|
---|
method
¶Get and set the REQUEST_METHOD property in the WSGI environment
params
¶Provides QUERY_STRING parameters as a dictionary
path
¶Provides the full path of the request, excluding the QUERY_STRING
path_info
¶Get and set the PATH_INFO property in the WSGI environment
path_info_pop
()¶Takes one path portion (delineated by slashes) from the path_info, and appends it to the script_name. Returns the path segment.
path_qs
¶The path of the request, without host but with query string.
query_string
¶Get and set the QUERY_STRING property in the WSGI environment
range
¶Retrieve and set the range property in the WSGI environ, as a Range object
referer
¶Get and set the HTTP_REFERER property in the WSGI environment
referrer
¶Get and set the HTTP_REFERER property in the WSGI environment
remote_addr
¶Get and set the REMOTE_ADDR property in the WSGI environment
remote_user
¶Get and set the REMOTE_USER property in the WSGI environment
script_name
¶Get and set the SCRIPT_NAME property in the WSGI environment
split_path
(minsegs=1, maxsegs=None, rest_with_last=False)¶Validate and split the Request’s path.
Examples:
['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters: |
|
---|---|
Returns: | list of segments with a length of maxsegs (non-existent segments will return as None) |
Raises: | ValueError – if given an invalid path |
str_params
¶Provides QUERY_STRING parameters as a dictionary
swift_entity_path
¶Provides the (native string) account/container/object path, sans API version.
This can be useful when constructing a path to send to a backend server, as that path will need everything after the “/v1”.
url
¶Provides the full url of the request
user_agent
¶Get and set the HTTP_USER_AGENT property in the WSGI environment
swift.common.swob.
Response
(body=None, status=200, headers=None, app_iter=None, request=None, conditional_response=False, conditional_etag=None, **kw)¶Bases: object
WSGI Response object.
__call__
(env, start_response)¶Respond to the WSGI request.
Warning
This will translate any relative Location header value to an absolute URL using the WSGI environment’s HOST_URL as a prefix, as RFC 2616 specifies.
However, it is quite common to use relative redirects, especially when it is difficult to know the exact HOST_URL the browser would have used when behind several CNAMEs, CDN services, etc. All modern browsers support relative redirects.
To skip over RFC enforcement of the Location header value,
you may set env['swift.leave_relative_location'] = True
in the WSGI environment.
absolute_location
()¶Attempt to construct an absolute location.
accept_ranges
¶Retrieve and set the accept-ranges header
app_iter
¶Retrieve and set the response app_iter
body
¶Retrieve and set the Response body str
charset
¶Retrieve and set the response charset
conditional_etag
¶The conditional_etag keyword argument for Response will allow the conditional match value of a If-Match request to be compared to a non-standard value.
This is available for Storage Policies that do not store the client object data verbatim on the storage nodes, but still need support conditional requests.
It’s most effectively used with X-Backend-Etag-Is-At which would define the additional Metadata key(s) where the original ETag of the clear-form client request data may be found.
content_length
¶Retrieve and set the content-length header as an int
content_range
¶Retrieve and set the content-range header
content_type
¶Retrieve and set the response Content-Type header
etag
¶Retrieve and set the response Etag header
fix_conditional_response
()¶You may call this once you have set the content_length to the whole object length and body or app_iter to reset the content_length properties on the request.
It is ok to not call this method, the conditional response will be maintained for you when you __call__ the response.
host_url
¶Get url for request/response up to path
last_modified
¶Retrieve and set the last-modified header as a datetime, set it with a datetime, int, or str
location
¶Retrieve and set the location header
status
¶Retrieve and set the Response status, e.g. ‘200 OK’
www_authenticate
()¶Construct a suitable value for WWW-Authenticate response header
If we have a request and a valid-looking path, the realm is the account; otherwise we set it to ‘unknown’.
swift.common.swob.
StatusMap
¶Bases: object
A dict-like object that returns HTTPException subclasses/factory functions where the given key is the status code.
swift.common.swob.
WsgiBytesIO
(buf='')¶Bases: StringIO.StringIO
This class adds support for the additional wsgi.input methods defined on eventlet.wsgi.Input to the BytesIO class which would otherwise be a fine stand-in for the file-like object in the WSGI environment.
swift.common.swob.
wsgify
(func)¶A decorator for translating functions which take a swob Request object and return a Response object into WSGI callables. Also catches any raised HTTPExceptions and treats them as a returned Response.
Miscellaneous utility functions for use with Swift.
swift.common.utils.
ATTRIBUTES_RE
= <_sre.SRE_Pattern object>¶Regular expression to match form attributes.
swift.common.utils.
CloseableChain
(*iterables)¶Bases: object
Like itertools.chain, but with a close method that will attempt to invoke its sub-iterators’ close methods, if any.
swift.common.utils.
ContextPool
(size=1000)¶Bases: eventlet.greenpool.GreenPool
GreenPool subclassed to kill its coros when it gets gc’ed
swift.common.utils.
Everything
¶Bases: object
A container that contains everything. If “e” is an instance of Everything, then “x in e” is true for all x.
swift.common.utils.
GreenAsyncPile
(size_or_pool)¶Bases: object
Runs jobs in a pool of green threads, and the results can be retrieved by using this object as an iterator.
This is very similar in principle to eventlet.GreenPile, except it returns results as they become available rather than in the order they were launched.
Correlating results with jobs (if necessary) is left to the caller.
spawn
(func, *args, **kwargs)¶Spawn a job in a green thread on the pile.
waitall
(timeout)¶Wait timeout seconds for any results to come in.
Parameters: | timeout – seconds to wait for results |
---|---|
Returns: | list of results accrued in that time |
waitfirst
(timeout)¶Wait up to timeout seconds for first result to come in.
Parameters: | timeout – seconds to wait for results |
---|---|
Returns: | first item to come back, or None |
swift.common.utils.
GreenAsyncPileWaitallTimeout
(seconds=None, exception=None)¶Bases: eventlet.timeout.Timeout
swift.common.utils.
GreenthreadSafeIterator
(unsafe_iterable)¶Bases: object
Wrap an iterator to ensure that only one greenthread is inside its next() method at a time.
This is useful if an iterator’s next() method may perform network IO, as that may trigger a greenthread context switch (aka trampoline), which can give another greenthread a chance to call next(). At that point, you get an error like “ValueError: generator already executing”. By wrapping calls to next() with a mutex, we avoid that error.
swift.common.utils.
InputProxy
(wsgi_input)¶Bases: object
File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.
read
(*args, **kwargs)¶Pass read request to the underlying file-like object and add bytes read to total.
readline
(*args, **kwargs)¶Pass readline request to the underlying file-like object and add bytes read to total.
swift.common.utils.
InvalidHashPathConfigError
¶Bases: exceptions.ValueError
swift.common.utils.
LRUCache
(maxsize=1000, maxtime=3600)¶Bases: object
Decorator for size/time bound memoization that evicts the least recently used members.
swift.common.utils.
LogAdapter
(logger, server)¶Bases: logging.LoggerAdapter
, object
A Logger like object which performs some reformatting on calls to
exception()
. Can be used to store a threadlocal transaction id and
client ip.
exception
(msg, *args, **kwargs)¶Delegate an exception call to the underlying logger, after adding contextual information from this adapter instance.
notice
(msg, *args, **kwargs)¶Convenience function for syslog priority LOG_NOTICE. The python logging lvl is set to 25, just above info. SysLogHandler is monkey patched to map this log lvl to the LOG_NOTICE syslog priority.
process
(msg, kwargs)¶Add extra info to message
set_statsd_prefix
(prefix)¶The StatsD client prefix defaults to the “name” of the logger. This method may override that default with a specific value. Currently used in the proxy-server to differentiate the Account, Container, and Object controllers.
statsd_delegate
()¶Factory to create methods which delegate to methods on self.logger.statsd_client (an instance of StatsdClient). The created methods conditionally delegate to a method whose name is given in ‘statsd_func_name’. The created delegate methods are a no-op when StatsD logging is not configured.
Parameters: | statsd_func_name – the name of a method on StatsdClient. |
---|
swift.common.utils.
LogLevelFilter
(level=10)¶Bases: object
Drop messages for the logger based on level.
This is useful when dependencies log too much information.
Parameters: | level – All messages at or below this level are dropped (DEBUG < INFO < WARN < ERROR < CRITICAL|FATAL) Default: DEBUG |
---|
swift.common.utils.
ModifiedParseResult
¶Bases: urlparse.ParseResult
Parse results class for urlparse.
swift.common.utils.
NR_ioprio_set
()¶Give __NR_ioprio_set value for your system.
swift.common.utils.
NullLogger
¶Bases: object
A no-op logger for eventlet wsgi.
swift.common.utils.
OverrideOptions
(devices, partitions, policies)¶Bases: tuple
devices
¶Alias for field number 0
partitions
¶Alias for field number 1
policies
¶Alias for field number 2
swift.common.utils.
PipeMutex
¶Bases: object
Mutex using a pipe. Works across both greenlets and real threads, even at the same time.
acquire
(blocking=True)¶Acquire the mutex.
If called with blocking=False, returns True if the mutex was acquired and False if it wasn’t. Otherwise, blocks until the mutex is acquired and returns True.
This lock is recursive; the same greenthread may acquire it as many times as it wants to, though it must then release it that many times too.
close
()¶Close the mutex. This releases its file descriptors.
You can’t use a mutex after it’s been closed.
release
()¶Release the mutex.
swift.common.utils.
PrefixLoggerAdapter
(logger, extra)¶Bases: swift.common.utils.SwiftLoggerAdapter
Adds an optional prefix to all its log messages. When the prefix has not been set, messages are unchanged.
exception
(*a, **kw)¶Delegate an exception call to the underlying logger, after adding contextual information from this adapter instance.
process
(msg, kwargs)¶Process the logging message and keyword arguments passed in to a logging call to insert contextual information. You can either manipulate the message itself, the keyword args or both. Return the message and kwargs modified (or not) to suit your needs.
Normally, you’ll only need to override this one method in a LoggerAdapter subclass for your specific needs.
swift.common.utils.
RateLimitedIterator
(iterable, elements_per_second, limit_after=0, ratelimit_if=<function <lambda>>)¶Bases: object
Wrap an iterator to only yield elements at a rate of N per second.
Parameters: |
|
---|
swift.common.utils.
ShardRange
(name, timestamp, lower=MinBound, upper=MaxBound, object_count=0, bytes_used=0, meta_timestamp=None, deleted=False, state=None, state_timestamp=None, epoch=None)¶Bases: object
A ShardRange encapsulates sharding state related to a container including lower and upper bounds that define the object namespace for which the container is responsible.
Shard ranges may be persisted in a container database. Timestamps associated with subsets of the shard range attributes are used to resolve conflicts when a shard range needs to be merged with an existing shard range record and the most recent version of an attribute should be persisted.
Parameters: |
|
---|
copy
(timestamp=None, **kwargs)¶Creates a copy of the ShardRange.
Parameters: | timestamp – (optional) If given, the returned ShardRange will have all of its timestamps set to this value. Otherwise the returned ShardRange will have the original timestamps. |
---|---|
Returns: | an instance of ShardRange |
entire_namespace
()¶Returns True if the ShardRange includes the entire namespace, False otherwise.
from_dict
(params)¶Return an instance constructed using the given dict of params. This method is deliberately less flexible than the class __init__() method and requires all of the __init__() args to be given in the dict of params.
Parameters: | params – a dict of parameters |
---|---|
Returns: | an instance of this class |
includes
(other)¶Returns True if this namespace includes the whole of the other namespace, False otherwise.
Parameters: | other – an instance of ShardRange |
---|
increment_meta
(object_count, bytes_used)¶Increment the object stats metadata by the given values and update the meta_timestamp to the current time.
Parameters: |
|
---|---|
Raises: | ValueError – if |
make_path
(shards_account, root_container, parent_container, timestamp, index)¶Returns a path for a shard container that is valid to use as a name
when constructing a ShardRange
.
Parameters: |
|
---|---|
Returns: | a string of the form <account_name>/<container_name> |
overlaps
(other)¶Returns True if the ShardRange namespace overlaps with the other ShardRange’s namespace.
Parameters: | other – an instance of ShardRange |
---|
resolve_state
(state)¶Given a value that may be either the name or the number of a state return a tuple of (state number, state name).
Parameters: | state – Either a string state name or an integer state number. |
---|---|
Returns: | A tuple (state number, state name) |
Raises: | ValueError – if state is neither a valid state name nor a
valid state number. |
set_deleted
(timestamp=None)¶Mark the shard range deleted and set timestamp to the current time.
Parameters: | timestamp – optional timestamp to set; if not given the current time will be set. |
---|---|
Returns: | True if the deleted attribute or timestamp was changed, False otherwise |
update_meta
(object_count, bytes_used, meta_timestamp=None)¶Set the object stats metadata to the given values and update the meta_timestamp to the current time.
Parameters: |
|
---|---|
Raises: | ValueError – if |
update_state
(state, state_timestamp=None)¶Set state to the given value and optionally update the state_timestamp to the given time.
Parameters: |
|
---|---|
Returns: | True if the state or state_timestamp was changed, False otherwise |
swift.common.utils.
Spliterator
(source_iterable)¶Bases: object
Takes an iterator yielding sliceable things (e.g. strings or lists) and yields subiterators, each yielding up to the requested number of items from the source.
>>> si = Spliterator(["abcde", "fg", "hijkl"])
>>> ''.join(si.take(4))
"abcd"
>>> ''.join(si.take(3))
"efg"
>>> ''.join(si.take(1))
"h"
>>> ''.join(si.take(3))
"ijk"
>>> ''.join(si.take(3))
"l" # shorter than requested; this can happen with the last iterator
swift.common.utils.
StreamingPile
(size)¶Bases: swift.common.utils.GreenAsyncPile
Runs jobs in a pool of green threads, spawning more jobs as results are retrieved and worker threads become available.
When used as a context manager, has the same worker-killing properties as
ContextPool
.
asyncstarmap
(func, args_iter)¶This is the same as itertools.starmap()
, except that func is
executed in a separate green thread for each item, and results won’t
necessarily have the same order as inputs.
swift.common.utils.
SwiftLogFormatter
(fmt=None, datefmt=None, max_line_length=0)¶Bases: logging.Formatter
Custom logging.Formatter will append txn_id to a log message if the record has one and the message does not. Optionally it can shorten overly long log lines.
format
(record)¶Format the specified record as text.
The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.
swift.common.utils.
SwiftLoggerAdapter
(logger, extra)¶Bases: logging.LoggerAdapter
A logging.LoggerAdapter subclass that also passes through StatsD method calls.
Like logging.LoggerAdapter, you have to subclass this and override the process() method to accomplish anything useful.
swift.common.utils.
ThreadSafeSysLogHandler
(address=('localhost', 514), facility=1, socktype=None)¶Bases: logging.handlers.SysLogHandler
createLock
()¶Acquire a thread lock for serializing access to the underlying I/O.
swift.common.utils.
Timestamp
(timestamp, offset=0, delta=0)¶Bases: object
Internal Representation of Swift Time.
The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like “1402464677.04188”
To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like “1402464677.04188_0000000000000000” - the portion after the underscore is the offset and is a formatted hexadecimal integer.
The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset.
The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it’s considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it’s offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset.
swift.common.utils.
affinity_key_function
(affinity_str)¶Turns an affinity config value into a function suitable for passing to sort(). After doing so, the array will be sorted with respect to the given ordering.
For example, if affinity_str is “r1=1, r2z7=2, r2z8=2”, then the array will be sorted with all nodes from region 1 (r1=1) first, then all the nodes from region 2 zones 7 and 8 (r2z7=2 and r2z8=2), then everything else.
Note that the order of the pieces of affinity_str is irrelevant; the priority values are what comes after the equals sign.
If affinity_str is empty or all whitespace, then the resulting function will not alter the ordering of the nodes.
Parameters: | affinity_str – affinity config value, e.g. “r1z2=3” or “r1=1, r2z1=2, r2z2=2” |
---|---|
Returns: | single-argument function |
Raises: | ValueError – if argument invalid |
swift.common.utils.
affinity_locality_predicate
(write_affinity_str)¶Turns a write-affinity config value into a predicate function for nodes. The returned value will be a 1-arg function that takes a node dictionary and returns a true value if it is “local” and a false value otherwise. The definition of “local” comes from the affinity_str argument passed in here.
For example, if affinity_str is “r1, r2z2”, then only nodes where region=1 or where (region=2 and zone=2) are considered local.
If affinity_str is empty or all whitespace, then the resulting function will consider everything local
Parameters: | write_affinity_str – affinity config value, e.g. “r1z2” or “r1, r2z1, r2z2” |
---|---|
Returns: | single-argument function, or None if affinity_str is empty |
Raises: | ValueError – if argument invalid |
swift.common.utils.
audit_location_generator
(devices, datadir, suffix='', mount_check=True, logger=None)¶Given a devices path and a data directory, yield (path, device, partition) for all files in that directory
Parameters: |
|
---|
swift.common.utils.
backward
(f, blocksize=4096)¶A generator returning lines from a file starting with the last line, then the second last line, etc. i.e., it reads lines backwards. Stops when the first line (if any) is read. This is useful when searching for recent activity in very large files.
Parameters: |
|
---|
swift.common.utils.
cache_from_env
(env, allow_none=False)¶Get memcache connection pool from the environment (which had been previously set by the memcache middleware
Parameters: | env – wsgi environment dict |
---|---|
Returns: | swift.common.memcached.MemcacheRing from environment |
swift.common.utils.
capture_stdio
(logger, **kwargs)¶Log unhandled exceptions, close stdio, capture stdout and stderr.
param logger: Logger object to use
swift.common.utils.
closing_if_possible
(*args, **kwds)¶Like contextlib.closing(), but doesn’t crash if the object lacks a close() method.
PEP 333 (WSGI) says: “If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request[.]” This function makes that easier.
swift.common.utils.
compute_eta
(start_time, current_value, final_value)¶Compute an ETA. Now only if we could also have a progress bar…
Parameters: |
|
---|---|
Returns: | ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
swift.common.utils.
config_auto_int_value
(value, default)¶Returns default if value is None or ‘auto’. Returns value as an int or raises ValueError otherwise.
swift.common.utils.
config_fallocate_value
(reserve_value)¶Returns fallocate reserve_value as an int or float. Returns is_percent as a boolean. Returns a ValueError on invalid fallocate value.
swift.common.utils.
config_positive_int_value
(value)¶Returns positive int value if it can be cast by int() and it’s an integer > 0. (not including zero) Raises ValueError otherwise.
swift.common.utils.
config_read_prefixed_options
(conf, prefix_name, defaults)¶Read prefixed options from configuration
Parameters: |
|
---|---|
Returns: | a dict containing the options |
swift.common.utils.
config_read_reseller_options
(conf, defaults)¶Read reseller_prefix option and associated options from configuration
Reads the reseller_prefix option, then reads options that may be associated with a specific reseller prefix. Reads options such that an option without a prefix applies to all reseller prefixes unless an option has an explicit prefix.
Parameters: |
|
---|---|
Returns: | tuple of an array of reseller prefixes and a dict of option values |
swift.common.utils.
config_true_value
(value)¶Returns True if the value is either True or a string in TRUE_VALUES. Returns False otherwise.
swift.common.utils.
csv_append
(csv_string, item)¶Appends an item to a comma-separated string.
If the comma-separated string is empty/None, just returns item.
swift.common.utils.
decode_timestamps
(encoded, explicit=False)¶Parses a string of the form generated by encode_timestamps and returns a tuple of the three component timestamps. If explicit is False, component timestamps that are not explicitly encoded will be assumed to have zero delta from the previous component and therefore take the value of the previous component. If explicit is True, component timestamps that are not explicitly encoded will be returned with value None.
swift.common.utils.
distribute_evenly
(items, num_buckets)¶Distribute items as evenly as possible into N buckets.
swift.common.utils.
document_iters_to_http_response_body
(ranges_iter, boundary, multipart, logger)¶Takes an iterator of range iters and turns it into an appropriate HTTP response body, whether that’s multipart/byteranges or not.
This is almost, but not quite, the inverse of request_helpers.http_response_to_document_iters(). This function only yields chunks of the body, not any headers.
Parameters: |
|
---|
swift.common.utils.
document_iters_to_multipart_byteranges
(ranges_iter, boundary)¶Takes an iterator of range iters and yields a multipart/byteranges MIME document suitable for sending as the body of a multi-range 206 response.
See document_iters_to_http_response_body for parameter descriptions.
swift.common.utils.
drop_buffer_cache
(fd, offset, length)¶Drop ‘buffer’ cache for the given range of the given file.
Parameters: |
|
---|
swift.common.utils.
drop_privileges
(user, call_setsid=True)¶Sets the userid/groupid of the current process, get session leader, etc.
Parameters: | user – User name to change privileges to |
---|
swift.common.utils.
dump_recon_cache
(cache_dict, cache_file, logger, lock_timeout=2, set_owner=None)¶Update recon cache values
Parameters: |
|
---|
swift.common.utils.
encode_timestamps
(t1, t2=None, t3=None, explicit=False)¶Encode up to three timestamps into a string. Unlike a Timestamp object, the encoded string does NOT used fixed width fields and consequently no relative chronology of the timestamps can be inferred from lexicographic sorting of encoded timestamp strings.
i.e. if t1 = t2 = t3 then just the string representation of t1 is returned, otherwise the time offsets for t2 and t3 are appended. If explicit is True then the offsets for t2 and t3 are always appended even if zero.
Note: any offset value in t1 will be preserved, but offsets on t2 and t3 are not preserved. In the anticipated use cases for this method (and the inverse decode_timestamps method) the timestamps passed as t2 and t3 are not expected to have offsets as they will be timestamps associated with a POST request. In the case where the encoding is used in a container objects table row, t1 could be the PUT or DELETE time but t2 and t3 represent the content type and metadata times (if different from the data file) i.e. correspond to POST timestamps. In the case where the encoded form is used in a .meta file name, t1 and t2 both correspond to POST timestamps.
swift.common.utils.
eventlet_monkey_patch
()¶Install the appropriate Eventlet monkey patches.
swift.common.utils.
expand_ipv6
(address)¶Expand ipv6 address. :param address: a string indicating valid ipv6 address :returns: a string indicating fully expanded ipv6 address
swift.common.utils.
extract_swift_bytes
(content_type)¶Parameters: | content_type – a content-type string |
---|---|
Returns: | a tuple of (content-type, swift_bytes or None) |
swift.common.utils.
fallocate
(fd, size, offset=0)¶Pre-allocate disk space for a file.
This function can be disabled by calling disable_fallocate(). If no suitable C function is available in libc, this function is a no-op.
Parameters: |
|
---|
swift.common.utils.
fdatasync
(fd)¶Sync modified file data to disk.
Parameters: | fd – file descriptor |
---|
swift.common.utils.
find_shard_range
(item, ranges)¶Find a ShardRange in given list of shard_ranges
whose namespace
contains item
.
Parameters: |
|
---|---|
Returns: | the ShardRange whose namespace contains |
swift.common.utils.
fs_has_free_space
(fs_path, space_needed, is_percent)¶Check to see whether or not a filesystem has the given amount of space free. Unlike fallocate(), this does not reserve any space.
Parameters: |
|
---|---|
Returns: | True if the filesystem has at least that much free space, False otherwise |
Raises: | OSError – if fs_path does not exist |
swift.common.utils.
fsync
(fd)¶Sync modified file data and metadata to disk.
Parameters: | fd – file descriptor |
---|
swift.common.utils.
fsync_dir
(dirpath)¶Sync directory entries to disk.
Parameters: | dirpath – Path to the directory to be synced. |
---|
swift.common.utils.
get_db_files
(db_path)¶Given the path to a db file, return a sorted list of all valid db files that actually exist in that path’s dir. A valid db filename has the form:
<hash>[_<epoch>].db
where <hash> matches the <hash> part of the given db_path as would be
parsed by parse_db_filename()
.
Parameters: | db_path – Path to a db file that does not necessarily exist. |
---|---|
Returns: | List of valid db files that do exist in the dir of the
db_path . This list may be empty. |
swift.common.utils.
get_expirer_container
(x_delete_at, expirer_divisor, acc, cont, obj)¶Returns an expiring object container name for given X-Delete-At and a/c/o.
swift.common.utils.
get_hmac
(request_method, path, expires, key, digest=<built-in function openssl_sha1>, ip_range=None)¶Returns the hexdigest string of the HMAC (see RFC 2104) for the request.
Parameters: |
|
---|---|
Returns: | hexdigest str of the HMAC for the request using the specified digest algorithm. |
swift.common.utils.
get_hub
()¶Checks whether poll is available and falls back on select if it isn’t.
Note about epoll:
Review: https://review.openstack.org/#/c/18806/
There was a problem where once out of every 30 quadrillion connections, a coroutine wouldn’t wake up when the client closed its end. Epoll was not reporting the event or it was getting swallowed somewhere. Then when that file descriptor was re-used, eventlet would freak right out because it still thought it was waiting for activity from it in some other coro.
Another note about epoll: it’s hard to use when forking. epoll works like so:
- create an epoll instance: efd = epoll_create(…)
- register file descriptors of interest with epoll_ctl(efd,
- EPOLL_CTL_ADD, fd, …)
- wait for events with epoll_wait(efd, …)
If you fork, you and all your child processes end up using the same epoll instance, and everyone becomes confused. It is possible to use epoll and fork and still have a correct program as long as you do the right things, but eventlet doesn’t do those things. Really, it can’t even try to do those things since it doesn’t get notified of forks.
In contrast, both poll() and select() specify the set of interesting file descriptors with each call, so there’s no problem with forking.
As eventlet monkey patching is now done before call get_hub() in wsgi.py if we use ‘import select’ we get the eventlet version, but since version 0.20.0 eventlet removed select.poll() function in patched select (see: http://eventlet.net/doc/changelog.html and https://github.com/eventlet/eventlet/commit/614a20462).
We use eventlet.patcher.original function to get python select module to test if poll() is available on platform.
swift.common.utils.
get_log_line
(req, res, trans_time, additional_info)¶Make a line for logging that matches the documented log line format for backend servers.
Parameters: |
|
---|---|
Returns: | a properly formatted line for logging. |
swift.common.utils.
get_logger
(conf, name=None, log_to_console=False, log_route=None, fmt='%(server)s: %(message)s')¶Get the current system logger using config settings.
Log config and defaults:
log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
log_max_line_length = 0
log_udp_host = (disabled)
log_udp_port = logging.handlers.SYSLOG_UDP_PORT
log_address = /dev/log
log_statsd_host = (disabled)
log_statsd_port = 8125
log_statsd_default_sample_rate = 1.0
log_statsd_sample_rate_factor = 1.0
log_statsd_metric_prefix = (empty-string)
Parameters: |
|
---|
swift.common.utils.
get_md5_socket
()¶Get an MD5 socket file descriptor. One can MD5 data with it by writing it to the socket with os.write, then os.read the 16 bytes of the checksum out later.
NOTE: It is the caller’s responsibility to ensure that os.close() is called on the returned file descriptor. This is a bare file descriptor, not a Python object. It doesn’t close itself.
swift.common.utils.
get_policy_index
(req_headers, res_headers)¶Returns the appropriate index of the storage policy for the request from a proxy server
Parameters: |
|
---|---|
Returns: | string index of storage policy, or None |
swift.common.utils.
get_redirect_data
(response)¶Extract a redirect location from a response’s headers.
Parameters: | response – a response |
---|---|
Returns: | a tuple of (path, Timestamp) if a Location header is found, otherwise None |
Raises: | ValueError – if the Location header is found but a X-Backend-Redirect-Timestamp is not found, or if there is a problem with the format of etiher header |
swift.common.utils.
get_swift_info
(admin=False, disallowed_sections=None)¶Returns information about the swift cluster that has been previously registered with the register_swift_info call.
Parameters: |
|
---|---|
Returns: | dictionary of information about the swift cluster. |
swift.common.utils.
get_time_units
(time_amount)¶Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)
Parameters: | time_amount – length of time in seconds |
---|---|
Returns: | A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
swift.common.utils.
get_valid_utf8_str
(str_or_unicode)¶Get valid parts of utf-8 str from str, unicode and even invalid utf-8 str
Parameters: | str_or_unicode – a string or an unicode which can be invalid utf-8 |
---|
swift.common.utils.
get_zero_indexed_base_string
(base, index)¶This allows the caller to make a list of things with indexes, where the first item (zero indexed) is just the bare base string, and subsequent indexes are appended ‘-1’, ‘-2’, etc.
e.g.:
'lock', None => 'lock'
'lock', 0 => 'lock'
'lock', 1 => 'lock-1'
'object', 2 => 'object-2'
Parameters: |
|
---|
swift.common.utils.
hash_path
(account, container=None, object=None, raw_digest=False)¶Get the canonical hash for an account/container/object
Parameters: |
|
---|---|
Returns: | hash string |
swift.common.utils.
human_readable
(value)¶Returns the number in a human readable format; for example 1048576 = “1Mi”.
swift.common.utils.
is_valid_ip
(ip)¶Return True if the provided ip is a valid IP-address
swift.common.utils.
is_valid_ipv4
(ip)¶Return True if the provided ip is a valid IPv4-address
swift.common.utils.
is_valid_ipv6
(ip)¶Returns True if the provided ip is a valid IPv6-address
swift.common.utils.
ismount
(path)¶Test whether a path is a mount point. This will catch any exceptions and translate them into a False return value Use ismount_raw to have the exceptions raised instead.
swift.common.utils.
ismount_raw
(path)¶Test whether a path is a mount point. Whereas ismount will catch any exceptions and just return False, this raw version will not catch exceptions.
This is code hijacked from C Python 2.6.8, adapted to remove the extra lstat() system call.
swift.common.utils.
item_from_env
(env, item_name, allow_none=False)¶Get a value from the wsgi environment
Parameters: |
|
---|---|
Returns: | the value from the environment |
swift.common.utils.
iter_multipart_mime_documents
(wsgi_input, boundary, read_chunk_size=4096)¶Given a multi-part-mime-encoded input file object and boundary, yield file-like objects for each part. Note that this does not split each part into headers and body; the caller is responsible for doing that if necessary.
Parameters: |
|
---|---|
Returns: | A generator of file-like objects for each part. |
Raises: | MimeInvalid – if the document is malformed |
swift.common.utils.
last_modified_date_to_timestamp
(last_modified_date_str)¶Convert a last modified date (like you’d get from a container listing, e.g. 2014-02-28T23:22:36.698390) to a float.
swift.common.utils.
link_fd_to_path
(fd, target_path, dirs_created=0, retries=2, fsync=True)¶Creates a link to file descriptor at target_path specified. This method does not close the fd for you. Unlike rename, as linkat() cannot overwrite target_path if it exists, we unlink and try again.
Attempts to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
Parameters: |
|
---|
swift.common.utils.
list_from_csv
(comma_separated_str)¶Splits the str given and returns a properly stripped list of the comma separated values.
swift.common.utils.
load_libc_function
(func_name, log_error=True, fail_if_missing=False, errcheck=False)¶Attempt to find the function in libc, otherwise return a no-op func.
Parameters: |
|
---|
swift.common.utils.
load_recon_cache
(cache_file)¶Load a recon cache file. Treats missing file as empty.
swift.common.utils.
lock_file
(*args, **kwds)¶Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
swift.common.utils.
lock_parent_directory
(filename, timeout=10)¶Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
swift.common.utils.
lock_path
(*args, **kwds)¶Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.
Parameters: |
|
---|---|
Raises: |
|
swift.common.utils.
make_db_file_path
(db_path, epoch)¶Given a path to a db file, return a modified path whose filename part has the given epoch.
A db filename takes the form <hash>[_<epoch>].db
; this method replaces
the <epoch>
part of the given db_path
with the given epoch
value, or drops the epoch part if the given epoch
is None
.
Parameters: |
|
---|---|
Returns: | A modified path to a db file. |
Raises: | ValueError – if the |
swift.common.utils.
makedirs_count
(path, count=0)¶Same as os.makedirs() except that this method returns the number of new directories that had to be created.
Also, this does not raise an error if target directory already exists. This behaviour is similar to Python 3.x’s os.makedirs() called with exist_ok=True. Also similar to swift.common.utils.mkdirs()
swift.common.utils.
maybe_multipart_byteranges_to_document_iters
(app_iter, content_type)¶Takes an iterator that may or may not contain a multipart MIME document as well as content type and returns an iterator of body iterators.
Parameters: |
|
---|
swift.common.utils.
md5_hash_for_file
(fname)¶Get the MD5 checksum of a file.
Parameters: | fname – path to file |
---|---|
Returns: | MD5 checksum, hex encoded |
swift.common.utils.
mime_to_document_iters
(input_file, boundary, read_chunk_size=4096)¶Takes a file-like object containing a multipart MIME document and returns an iterator of (headers, body-file) tuples.
Parameters: |
|
---|
swift.common.utils.
mkdirs
(path)¶Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.
Parameters: | path – path to create |
---|
swift.common.utils.
modify_priority
(conf, logger)¶Modify priority by nice and ionice.
swift.common.utils.
multipart_byteranges_to_document_iters
(input_file, boundary, read_chunk_size=4096)¶Takes a file-like object containing a multipart/byteranges MIME document (see RFC 7233, Appendix A) and returns an iterator of (first-byte, last-byte, length, document-headers, body-file) 5-tuples.
Parameters: |
|
---|
swift.common.utils.
normalize_delete_at_timestamp
(timestamp)¶Format a timestamp (string or numeric) into a standardized xxxxxxxxxx (10) format.
Note that timestamps less than 0000000000 are raised to 0000000000 and values greater than November 20th, 2286 at 17:46:39 UTC will be capped at that date and time, resulting in no return value exceeding 9999999999.
This cap is because the expirer is already working through a sorted list of strings that were all a length of 10. Adding another digit would mess up the sort and cause the expirer to break from processing early. By 2286, this problem will need to be fixed, probably by creating an additional .expiring_objects account to work from with 11 (or more) digit container names.
Parameters: | timestamp – unix timestamp |
---|---|
Returns: | normalized timestamp as a string |
swift.common.utils.
normalize_timestamp
(timestamp)¶Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx (10.5) format.
Note that timestamps using values greater than or equal to November 20th, 2286 at 17:46 UTC will use 11 digits to represent the number of seconds.
Parameters: | timestamp – unix timestamp |
---|---|
Returns: | normalized timestamp as a string |
swift.common.utils.
override_bytes_from_content_type
(listing_dict, logger=None)¶Takes a dict from a container listing and overrides the content_type, bytes fields if swift_bytes is set.
swift.common.utils.
pairs
(item_list)¶Returns an iterator of all pairs of elements from item_list.
Parameters: | item_list – items (no duplicates allowed) |
---|
swift.common.utils.
parse_content_disposition
(header)¶Given the value of a header like: Content-Disposition: form-data; name=”somefile”; filename=”test.html”
Return data like (“form-data”, {“name”: “somefile”, “filename”: “test.html”})
Parameters: | header – Value of a header (the part after the ‘: ‘). |
---|---|
Returns: | (value name, dict) of the attribute data parsed (see above). |
swift.common.utils.
parse_content_range
(content_range)¶Parse a content-range header into (first_byte, last_byte, total_size).
See RFC 7233 section 4.2 for details on the header format, but it’s basically “Content-Range: bytes ${start}-${end}/${total}”.
Parameters: | content_range – Content-Range header value to parse, e.g. “bytes 100-1249/49004” |
---|---|
Returns: | 3-tuple (start, end, total) |
Raises: | ValueError – if malformed |
swift.common.utils.
parse_content_type
(content_type)¶Parse a content-type and its parameters into values. RFC 2616 sec 14.17 and 3.7 are pertinent.
Examples:
'text/plain; charset=UTF-8' -> ('text/plain', [('charset, 'UTF-8')])
'text/plain; charset=UTF-8; level=1' ->
('text/plain', [('charset, 'UTF-8'), ('level', '1')])
Parameters: | content_type – content_type to parse |
---|---|
Returns: | a tuple containing (content type, list of k, v parameter tuples) |
swift.common.utils.
parse_db_filename
(filename)¶Splits a db filename into three parts: the hash, the epoch, and the extension.
>>> parse_db_filename("ab2134.db")
('ab2134', None, '.db')
>>> parse_db_filename("ab2134_1234567890.12345.db")
('ab2134', '1234567890.12345', '.db')
Parameters: | filename – A db file basename or path to a db file. |
---|---|
Returns: | A tuple of (hash , epoch, extension). epoch may be None. |
Raises: | ValueError – if filename is not a path to a file. |
swift.common.utils.
parse_mime_headers
(doc_file)¶Takes a file-like object containing a MIME document and returns a HeaderKeyDict containing the headers. The body of the message is not consumed: the position in doc_file is left at the beginning of the body.
This function was inspired by the Python standard library’s http.client.parse_headers.
Parameters: | doc_file – binary file-like object containing a MIME document |
---|---|
Returns: | a swift.common.swob.HeaderKeyDict containing the headers |
swift.common.utils.
parse_options
(parser=None, once=False, test_args=None)¶Parse standard swift server/daemon options with optparse.OptionParser.
Parameters: |
|
---|---|
Returns: | Tuple of (config, options); config is an absolute path to the config file, options is the parser options as a dictionary. |
Raises: | SystemExit – First arg (CONFIG) is required, file must exist |
swift.common.utils.
parse_override_options
(**kwargs)¶Figure out which policies, devices, and partitions we should operate on, based on kwargs.
If ‘override_policies’ is already present in kwargs, then return that value. This happens when using multiple worker processes; the parent process supplies override_policies=X to each child process.
Otherwise, in run-once mode, look at the ‘policies’ keyword argument. This is the value of the “–policies” command-line option. In run-forever mode or if no –policies option was provided, an empty list will be returned.
The procedures for devices and partitions are similar.
Returns: | a named tuple with fields “devices”, “partitions”, and “policies”. |
---|
swift.common.utils.
parse_socket_string
(socket_string, default_port)¶Given a string representing a socket, returns a tuple of (host, port). Valid strings are DNS names, IPv4 addresses, or IPv6 addresses, with an optional port. If an IPv6 address is specified it must be enclosed in [], like [::1] or [::1]:11211. This follows the accepted prescription for IPv6 host literals.
Examples:
server.org
server.org:1337
127.0.0.1:1337
[::1]:1337
[::1]
swift.common.utils.
public
(func)¶Decorator to declare which methods are publicly accessible as HTTP requests
Parameters: | func – function to make public |
---|
swift.common.utils.
punch_hole
(fd, offset, length)¶De-allocate disk space in the middle of a file.
Parameters: |
|
---|
swift.common.utils.
put_recon_cache_entry
(cache_entry, key, item)¶Update a recon cache entry item.
If item
is an empty dict then any existing key
in cache_entry
will be deleted. Similarly if item
is a dict and any of its values are
empty dicts then the corrsponsing key will be deleted from the nested dict
in cache_entry
.
We use nested recon cache entries when the object auditor runs in parallel or else in ‘once’ mode with a specified subset of devices.
Parameters: |
|
---|
swift.common.utils.
quorum_size
(n)¶quorum size as it applies to services that use ‘replication’ for data integrity (Account/Container services). Object quorum_size is defined on a storage policy basis.
Number of successful backend requests needed for the proxy to consider the client request successful.
swift.common.utils.
quote
(value, safe='/')¶Patched version of urllib.quote that encodes utf-8 strings before quoting
swift.common.utils.
random
() → x in the interval [0, 1).¶swift.common.utils.
ratelimit_sleep
(running_time, max_rate, incr_by=1, rate_buffer=5)¶Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.
Parameters: |
|
---|
swift.common.utils.
readconf
(conf_path, section_name=None, log_name=None, defaults=None, raw=False)¶Read config file(s) and return config items as a dict
Parameters: |
|
---|---|
Returns: | dict of config items |
Raises: |
|
swift.common.utils.
register_swift_info
(name='swift', admin=False, **kwargs)¶Registers information about the swift cluster to be retrieved with calls to get_swift_info.
Parameters: |
|
---|---|
Raises: | ValueError – if name or any of the keys in kwargs has “.” in it |
swift.common.utils.
reiterate
(iterable)¶Consume the first item from an iterator, then re-chain it to the rest of the iterator. This is useful when you want to make sure the prologue to downstream generators have been executed before continuing.
Parameters: | iterable – an iterable object |
---|
swift.common.utils.
remove_directory
(path)¶Wrapper for os.rmdir, ENOENT and ENOTEMPTY are ignored
Parameters: | path – first and only argument passed to os.rmdir |
---|
swift.common.utils.
remove_file
(path)¶Quiet wrapper for os.unlink, OSErrors are suppressed
Parameters: | path – first and only argument passed to os.unlink |
---|
swift.common.utils.
renamer
(old, new, fsync=True)¶Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
The containing directory of ‘new’ and of all newly created directories are fsync’d by default. This _will_ come at a performance penalty. In cases where these additional fsyncs are not necessary, it is expected that the caller of renamer() turn it off explicitly.
Parameters: |
|
---|
swift.common.utils.
replace_partition_in_path
(path, part_power)¶Takes a full path to a file and a partition power and returns the same path, but with the correct partition number. Most useful when increasing the partition power.
Parameters: |
|
---|---|
Returns: | Path with re-computed partition power |
swift.common.utils.
replication
(func)¶Decorator to declare which methods are accessible for different type of servers:
Parameters: | func – function to mark accessible for replication |
---|
swift.common.utils.
round_robin_iter
(its)¶Takes a list of iterators, yield an element from each in a round-robin fashion until all of them are exhausted. :param its: list of iterators
swift.common.utils.
rsync_ip
(ip)¶Transform ip string to an rsync-compatible form
Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.
Parameters: | ip – an ip string (ipv4 or ipv6) |
---|---|
Returns: | a string ip address |
swift.common.utils.
rsync_module_interpolation
(template, device)¶Interpolate devices variables inside a rsync module template
Parameters: |
|
---|---|
Returns: | a string with all variables replaced by device attributes |
swift.common.utils.
search_tree
(root, glob_match, ext='', exts=None, dir_ext=None)¶Look in root, for any files/dirs matching glob, recursively traversing any found directories looking for files ending with ext
Parameters: |
|
---|---|
Returns: | list of full paths to matching files, sorted |
swift.common.utils.
set_swift_dir
(swift_dir)¶Sets the directory from which swift config files will be read. If the given directory differs from that already set then the swift.conf file in the new directory will be validated and storage policies will be reloaded from the new swift.conf file.
Parameters: | swift_dir – non-default directory to read swift.conf from |
---|
swift.common.utils.
sockaddr_alg
¶Bases: _ctypes.Structure
swift.common.utils.
split_path
(path, minsegs=1, maxsegs=None, rest_with_last=False)¶Validate and split the given HTTP request path.
Examples:
['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters: |
|
---|---|
Returns: | list of segments with a length of maxsegs (non-existent segments will return as None) |
Raises: | ValueError – if given an invalid path |
swift.common.utils.
storage_directory
(datadir, partition, name_hash)¶Get the storage directory
Parameters: |
|
---|---|
Returns: | Storage directory |
swift.common.utils.
streq_const_time
(s1, s2)¶Constant-time string comparison.
Params s1: | the first string |
---|---|
Params s2: | the second string |
Returns: | True if the strings are equal. |
This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.
swift.common.utils.
strict_b64decode
(value, allow_line_breaks=False)¶Validate and decode Base64-encoded data.
The stdlib base64 module silently discards bad characters, but we often want to treat them as an error.
Parameters: |
|
---|---|
Returns: | the decoded data |
Raises: | ValueError – if |
swift.common.utils.
timing_stats
(**dec_kwargs)¶Returns a decorator that logs timing events or errors for public methods in swift’s wsgi server controllers, based on response code.
swift.common.utils.
unlink_older_than
(path, mtime)¶Remove any file in a given path that was last modified before mtime.
Parameters: |
|
---|
swift.common.utils.
unlink_paths_older_than
(filepaths, mtime)¶Remove any files from the given list that were last modified before mtime.
Parameters: |
|
---|
swift.common.utils.
urlparse
(url)¶urlparse augmentation. This is necessary because urlparse can’t handle RFC 2732 URLs.
Parameters: | url – URL to parse. |
---|
swift.common.utils.
validate_device_partition
(device, partition)¶Validate that a device and a partition are valid and won’t lead to directory traversal when used.
Parameters: |
|
---|---|
Raises: | ValueError – if given an invalid device or partition |
swift.common.utils.
validate_sync_to
(value, allowed_sync_hosts, realms_conf)¶Validates an X-Container-Sync-To header value, returning the validated endpoint, realm, and realm_key, or an error string.
Parameters: |
|
---|---|
Returns: | A tuple of (error_string, validated_endpoint, realm, realm_key). The error_string will None if the rest of the values have been validated. The validated_endpoint will be the validated endpoint to sync to. The realm and realm_key will be set if validation was done through realms_conf. |
swift.common.utils.
whataremyips
(bind_ip=None)¶Get “our” IP addresses (“us” being the set of services configured by one *.conf file). If our REST listens on a specific address, return it. Otherwise, if listen on ‘0.0.0.0’ or ‘::’ return all addresses, including the loopback.
Parameters: | bind_ip (str) – Optional bind_ip from a config file; may be IP address or hostname. |
---|---|
Returns: | list of Strings of ip addresses |
swift.common.utils.
write_file
(path, contents)¶Write contents to file at path
Parameters: |
|
---|
swift.common.utils.
write_pickle
(obj, dest, tmp=None, pickle_protocol=0)¶Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location
Parameters: |
|
---|
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.