Container

Container

Container Auditor

class swift.container.auditor.ContainerAuditor(conf, logger=None)

Bases: swift.common.daemon.Daemon

Audit containers.

container_audit(path)

Audits the given container path

Parameters:path – the path to a container db
run_forever(*args, **kwargs)

Run the container audit until stopped.

run_once(*args, **kwargs)

Run the container audit once.

swift.container.auditor.random() → x in the interval [0, 1).

Container Backend

Pluggable Back-ends for Container Server

class swift.container.backend.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False, skip_commits=False, force_db_file=False)

Bases: swift.common.db.DatabaseBroker

Encapsulates working with a container database.

Note that this may involve multiple on-disk DB files if the container becomes sharded:

  • _db_file is the path to the legacy container DB name, i.e. <hash>.db. This file should exist for an initialised broker that has never been sharded, but will not exist once a container has been sharded.
  • db_files is a list of existing db files for the broker. This list should have at least one entry for an initialised broker, and should have two entries while a broker is in SHARDING state.
  • db_file is the path to whichever db is currently authoritative for the container. Depending on the container’s state, this may not be the same as the db_file argument given to __init__(), unless force_db_file is True in which case db_file is always equal to the db_file argument given to __init__().
  • pending_file is always equal to _db_file extended with .pending, i.e. <hash>.db.pending.
classmethod create_broker(device_path, part, account, container, logger=None, epoch=None, put_timestamp=None, storage_policy_index=None)

Create a ContainerBroker instance. If the db doesn’t exist, initialize the db file.

Parameters:
  • device_path – device path
  • part – partition number
  • account – account name string
  • container – container name string
  • logger – a logger instance
  • epoch – a timestamp to include in the db filename
  • put_timestamp – initial timestamp if broker needs to be initialized
  • storage_policy_index – the storage policy index
Returns:

a swift.container.backend.ContainerBroker instance

create_container_info_table(conn, put_timestamp, storage_policy_index)

Create the container_info table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code. Also creates the container_stat view.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
  • storage_policy_index – storage policy index
create_object_table(conn)

Create the object table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:conn – DB connection object
create_policy_stat_table(conn, storage_policy_index=0)

Create policy_stat table.

Parameters:
  • conn – DB connection object
  • storage_policy_index – the policy_index the container is being created with
create_shard_range_table(conn)

Create the shard_range table which is specific to the container DB.

Parameters:conn – DB connection object
db_contains_type = 'object'
db_epoch
db_file

Get the path to the primary db file for this broker. This is typically the db file for the most recent sharding epoch. However, if no db files exist on disk, or if force_db_file was True when the broker was constructed, then the primary db file is the file passed to the broker constructor.

Returns:A path to a db file; the file does not necessarily exist.
db_files

Gets the cached list of valid db files that exist on disk for this broker.

The cached list may be refreshed by calling
reload_db_files().
Returns:A list of paths to db files ordered by ascending epoch; the list may be empty.
db_reclaim_timestamp = 'created_at'
db_type = 'container'
delete_object(name, timestamp, storage_policy_index=0)

Mark an object deleted.

Parameters:
  • name – object name to be deleted
  • timestamp – timestamp when the object was marked as deleted
  • storage_policy_index – the storage policy index for the object
empty()

Check if container DB is empty.

This method uses more stringent checks on object count than is_deleted(): this method checks that there are no objects in any policy; if the container is in the process of sharding then both fresh and retiring databases are checked to be empty; if a root container has shard ranges then they are checked to be empty.

Returns:True if the database has no active objects, False otherwise
enable_sharding(epoch)

Updates this broker’s own shard range with the given epoch, sets its state to SHARDING and persists it in the DB.

Parameters:epoch – a Timestamp
Returns:the broker’s updated own shard range.
find_shard_ranges(shard_size, limit=-1, existing_ranges=None)

Scans the container db for shard ranges. Scanning will start at the upper bound of the any existing_ranges that are given, otherwise at ShardRange.MIN. Scanning will stop when limit shard ranges have been found or when no more shard ranges can be found. In the latter case, the upper bound of the final shard range will be equal to the upper bound of the container namespace.

This method does not modify the state of the db; callers are responsible for persisting any shard range data in the db.

Parameters:
  • shard_size – the size of each shard range
  • limit – the maximum number of shard points to be found; a negative value (default) implies no limit.
  • existing_ranges – an optional list of existing ShardRanges; if given, this list should be sorted in order of upper bounds; the scan for new shard ranges will start at the upper bound of the last existing ShardRange.
Returns:

a tuple; the first value in the tuple is a list of dicts each having keys {‘index’, ‘lower’, ‘upper’, ‘object_count’} in order of ascending ‘upper’; the second value in the tuple is a boolean which is True if the last shard range has been found, False otherwise.

get_all_shard_range_data()

Returns a list of all shard range data, including own shard range and deleted shard ranges.

Returns:A list of dict representations of a ShardRange.
get_brokers()

Return a list of brokers for component dbs. The list has two entries while the db state is sharding: the first entry is a broker for the retiring db with skip_commits set to True; the second entry is a broker for the fresh db with skip_commits set to False. For any other db state the list has one entry.

Returns:a list of ContainerBroker
get_db_state()

Returns the current state of on disk db files.

get_db_version(conn)
get_info()

Get global data for the container.

Returns:dict with keys: account, container, created_at, put_timestamp, delete_timestamp, status_changed_at, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, x_container_sync_point2, and storage_policy_index, db_state.
get_info_is_deleted()

Get the is_deleted status and info for the container.

Returns:a tuple, in the form (info, is_deleted) info is a dict as returned by get_info and is_deleted is a boolean.
get_misplaced_since(start, count)

Get a list of objects which are in a storage policy different from the container’s storage policy.

Parameters:
  • start – last reconciler sync point
  • count – maximum number of entries to get
Returns:

list of dicts with keys: name, created_at, size, content_type, etag, storage_policy_index

get_objects(limit=None, marker='', end_marker='', include_deleted=None, since_row=None)

Returns a list of objects, including deleted objects, in all policies. Each object in the list is described by a dict with keys {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’, ‘storage_policy_index’}.

Parameters:
  • limit – maximum number of entries to get
  • marker – if set, objects with names less than or equal to this value will not be included in the list.
  • end_marker – if set, objects with names greater than or equal to this value will not be included in the list.
  • include_deleted – if True, include only deleted objects; if False, include only undeleted objects; otherwise (default), include both deleted and undeleted objects.
  • since_row – include only items whose ROWID is greater than the given row id; by default all rows are included.
Returns:

a list of dicts, each describing an object.

get_own_shard_range(no_default=False)

Returns a shard range representing this broker’s own shard range. If no such range has been persisted in the broker’s shard ranges table then a default shard range representing the entire namespace will be returned.

The returned shard range will be updated with the current object stats for this broker and a meta timestamp set to the current time. For these values to be persisted the caller must merge the shard range.

Parameters:no_default – if True and the broker’s own shard range is not found in the shard ranges table then None is returned, otherwise a default shard range is returned.
Returns:an instance of ShardRange
get_policy_stats()
get_reconciler_sync()
get_replication_info()

Get information about the DB required for replication.

Returns:dict containing keys from get_info plus max_row and metadata
Note:: get_info’s <db_contains_type>_count is translated to just
“count” and metadata is the raw string.
get_shard_ranges(marker=None, end_marker=None, includes=None, reverse=False, include_deleted=False, states=None, include_own=False, exclude_others=False, fill_gaps=False)

Returns a list of persisted shard ranges.

Parameters:
  • marker – restricts the returned list to shard ranges whose namespace includes or is greater than the marker value.
  • end_marker – restricts the returned list to shard ranges whose namespace includes or is less than the end_marker value.
  • includes – restricts the returned list to the shard range that includes the given value; if includes is specified then marker and end_marker are ignored.
  • reverse – reverse the result order.
  • include_deleted – include items that have the delete marker set
  • states – if specified, restricts the returned list to shard ranges that have the given state(s); can be a list of ints or a single int.
  • include_own – boolean that governs whether the row whose name matches the broker’s path is included in the returned list. If True, that row is included, otherwise it is not included. Default is False.
  • exclude_others – boolean that governs whether the rows whose names do not match the broker’s path are included in the returned list. If True, those rows are not included, otherwise they are included. Default is False.
  • fill_gaps – if True, insert own shard range to fill any gaps in at the tail of other shard ranges.
Returns:

a list of instances of swift.common.utils.ShardRange

get_shard_usage()

Get the aggregate object stats for all shard ranges in states ACTIVE, SHARDING or SHRINKING.

Returns:a dict with keys {bytes_used, object_count}
get_sharding_sysmeta(key=None)

Returns sharding specific info from the broker’s metadata.

Parameters:key – if given the value stored under key in the sharding info will be returned.
Returns:either a dict of sharding info or the value stored under key in that dict.
has_multiple_policies()
is_own_shard_range(shard_range)
is_reclaimable(now, reclaim_age)

Check if the broker abstraction is empty, and has been marked deleted for at least a reclaim age.

is_root_container()

Returns True if this container is a root container, False otherwise.

A root container is a container that is not a shard of another container.

is_sharded()
list_objects_iter(limit, marker, end_marker, prefix, delimiter, path=None, storage_policy_index=0, reverse=False, include_deleted=False, since_row=None, transform_func=None, all_policies=False)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • end_marker – end marker query
  • prefix – prefix query
  • delimiter – delimiter for query
  • path – if defined, will set the prefix and delimiter based on the path
  • storage_policy_index – storage policy index for query
  • reverse – reverse the result order.
  • include_deleted – if True, include only deleted objects; if False (default), include only undeleted objects; otherwise, include both deleted and undeleted objects.
  • since_row – include only items whose ROWID is greater than the given row id; by default all rows are included.
  • transform_func – an optional function that if given will be called for each object to get a transformed version of the object to include in the listing; should have same signature as _transform_record(); defaults to _transform_record().
  • all_policies – if True, include objects for all storage policies ignoring any value given for storage_policy_index
Returns:

list of tuples of (name, created_at, size, content_type, etag, deleted)

make_tuple_for_pickle(record)

Turn this db record dict into the format this service uses for pending pickles.

merge_items(item_list, source=None)

Merge items into the object table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’, ‘storage_policy_index’, ‘ctype_timestamp’, ‘meta_timestamp’}
  • source – if defined, update incoming_sync with the source
merge_shard_ranges(shard_ranges)

Merge shard ranges into the shard range table.

Parameters:shard_ranges – a shard range or a list of shard ranges; each shard range should be an instance of ShardRange or a dict representation of a shard range having SHARD_RANGE_KEYS.
path
put_object(name, timestamp, size, content_type, etag, deleted=0, storage_policy_index=0, ctype_timestamp=None, meta_timestamp=None)

Creates an object in the DB with its metadata.

Parameters:
  • name – object name to be created
  • timestamp – timestamp of when the object was created
  • size – object size
  • content_type – object content-type
  • etag – object etag
  • deleted – if True, marks the object as deleted and sets the deleted_at timestamp to timestamp
  • storage_policy_index – the storage policy index for the object
  • ctype_timestamp – timestamp of when content_type was last updated
  • meta_timestamp – timestamp of when metadata was last updated
reload_db_files()

Reloads the cached list of valid on disk db files for this broker.

remove_objects(lower, upper, max_row=None)

Removes object records in the given namespace range from the object table.

Note that objects are removed regardless of their storage_policy_index.

Parameters:
  • lower – defines the lower bound of object names that will be removed; names greater than this value will be removed; names less than or equal to this value will not be removed.
  • upper – defines the upper bound of object names that will be removed; names less than or equal to this value will be removed; names greater than this value will not be removed. The empty string is interpreted as there being no upper bound.
  • max_row – if specified only rows less than or equal to max_row will be removed
reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats, available with container’s get_info.

Parameters:
  • put_timestamp – put_timestamp to update
  • delete_timestamp – delete_timestamp to update
  • object_count – object_count to update
  • bytes_used – bytes_used to update
classmethod resolve_shard_range_states(states)

Given a list of values each of which may be the name of a state, the number of a state, or an alias, return the set of state numbers described by the list.

The following alias values are supported: ‘listing’ maps to all states that are considered valid when listing objects; ‘updating’ maps to all states that are considered valid for redirecting an object update.

Parameters:states – a list of values each of which may be the name of a state, the number of a state, or an alias
Returns:a set of integer state numbers, or None if no states are given
Raises:ValueError – if any value in the given list is neither a valid state nor a valid alias
root_account
root_container
root_path
set_sharded_state()

Unlink’s the broker’s retiring DB file.

Returns:True if the retiring DB was successfully unlinked, False otherwise.
set_sharding_state()

Creates and initializes a fresh DB file in preparation for sharding a retiring DB. The broker’s own shard range must have an epoch timestamp for this method to succeed.

Returns:True if the fresh DB was successfully created, False otherwise.
set_sharding_sysmeta(key, value)

Updates the broker’s metadata metadata stored under the given key prefixed with a sharding specific namespace.

Parameters:
  • key – metadata key in the sharding metadata namespace.
  • value – metadata value
set_storage_policy_index(policy_index, timestamp=None)

Update the container_stat policy_index and status_changed_at.

set_x_container_sync_points(sync_point1, sync_point2)
sharding_initiated()

Returns True if a broker has shard range state that would be necessary for sharding to have been initiated, False otherwise.

sharding_required()

Returns True if a broker has shard range state that would be necessary for sharding to have been initiated but has not yet completed sharding, False otherwise.

storage_policy_index
update_reconciler_sync(point)
swift.container.backend.merge_shards(shard_data, existing)

Compares shard_data with existing and updates shard_data with any items of existing that take precedence over the corresponding item in shard_data.

Parameters:
  • shard_data – a dict representation of shard range that may be modified by this method.
  • existing – a dict representation of shard range.
Returns:

True if shard data has any item(s) that are considered to take precedence over the corresponding item in existing

swift.container.backend.update_new_item_from_existing(new_item, existing)

Compare the data and meta related timestamps of a new object item with the timestamps of an existing object record, and update the new item with data and/or meta related attributes from the existing record if their timestamps are newer.

The multiple timestamps are encoded into a single string for storing in the ‘created_at’ column of the objects db table.

Parameters:
  • new_item – A dict of object update attributes
  • existing – A dict of existing object attributes
Returns:

True if any attributes of the new item dict were found to be newer than the existing and therefore not updated, otherwise False implying that the updated item is equal to the existing.

Container Replicator

Container Server

Container Reconciler

Container Sharder

Container Sync

Container Updater

class swift.container.updater.ContainerUpdater(conf, logger=None)

Bases: swift.common.daemon.Daemon

Update container information in account listings.

container_report(node, part, container, put_timestamp, delete_timestamp, count, bytes, storage_policy_index)

Report container info to an account server.

Parameters:
  • node – node dictionary from the account ring
  • part – partition the account is on
  • container – container name
  • put_timestamp – put timestamp
  • delete_timestamp – delete timestamp
  • count – object count in the container
  • bytes – bytes used in the container
  • storage_policy_index – the policy index for the container
container_sweep(path)

Walk the path looking for container DBs and process them.

Parameters:path – path to walk
get_account_ring()

Get the account ring. Load it if it hasn’t been yet.

get_paths()

Get paths to all of the partitions on each drive to be processed.

Returns:a list of paths
process_container(dbfile)

Process a container, and update the information in the account.

Parameters:dbfile – container DB to process
run_forever(*args, **kwargs)

Run the updater continuously.

run_once(*args, **kwargs)

Run the updater once.

swift.container.updater.random() → x in the interval [0, 1).
Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

swift 2.21.1.dev15