swift.container.auditor.
ContainerAuditor
(conf, logger=None)¶Bases: swift.common.daemon.Daemon
Audit containers.
container_audit
(path)¶Audits the given container path
Parameters: | path – the path to a container db |
---|
run_forever
(*args, **kwargs)¶Run the container audit until stopped.
run_once
(*args, **kwargs)¶Run the container audit once.
swift.container.auditor.
random
() → x in the interval [0, 1).¶Pluggable Back-ends for Container Server
swift.container.backend.
ContainerBroker
(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False, skip_commits=False, force_db_file=False)¶Bases: swift.common.db.DatabaseBroker
Encapsulates working with a container database.
Note that this may involve multiple on-disk DB files if the container becomes sharded:
_db_file
is the path to the legacy container DB name, i.e.<hash>.db
. This file should exist for an initialised broker that has never been sharded, but will not exist once a container has been sharded.db_files
is a list of existing db files for the broker. This list should have at least one entry for an initialised broker, and should have two entries while a broker is in SHARDING state.db_file
is the path to whichever db is currently authoritative for the container. Depending on the container’s state, this may not be the same as thedb_file
argument given to__init__()
, unlessforce_db_file
is True in which casedb_file
is always equal to thedb_file
argument given to__init__()
.pending_file
is always equal to_db_file
extended with.pending
, i.e.<hash>.db.pending
.
create_broker
(device_path, part, account, container, logger=None, epoch=None, put_timestamp=None, storage_policy_index=None)¶Create a ContainerBroker instance. If the db doesn’t exist, initialize the db file.
Parameters: |
|
---|---|
Returns: | a |
create_container_info_table
(conn, put_timestamp, storage_policy_index)¶Create the container_info table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code. Also creates the container_stat view.
Parameters: |
|
---|
create_object_table
(conn)¶Create the object table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.
Parameters: | conn – DB connection object |
---|
create_policy_stat_table
(conn, storage_policy_index=0)¶Create policy_stat table.
Parameters: |
|
---|
create_shard_range_table
(conn)¶Create the shard_range table which is specific to the container DB.
Parameters: | conn – DB connection object |
---|
db_contains_type
= 'object'¶db_epoch
¶db_file
¶Get the path to the primary db file for this broker. This is typically
the db file for the most recent sharding epoch. However, if no db files
exist on disk, or if force_db_file
was True when the broker was
constructed, then the primary db file is the file passed to the broker
constructor.
Returns: | A path to a db file; the file does not necessarily exist. |
---|
db_files
¶Gets the cached list of valid db files that exist on disk for this broker.
reload_db_files()
.Returns: | A list of paths to db files ordered by ascending epoch; the list may be empty. |
---|
db_reclaim_timestamp
= 'created_at'¶db_type
= 'container'¶delete_object
(name, timestamp, storage_policy_index=0)¶Mark an object deleted.
Parameters: |
|
---|
empty
()¶Check if container DB is empty.
This method uses more stringent checks on object count than
is_deleted()
: this method checks that there are no objects in any
policy; if the container is in the process of sharding then both fresh
and retiring databases are checked to be empty; if a root container has
shard ranges then they are checked to be empty.
Returns: | True if the database has no active objects, False otherwise |
---|
enable_sharding
(epoch)¶Updates this broker’s own shard range with the given epoch, sets its state to SHARDING and persists it in the DB.
Parameters: | epoch – a Timestamp |
---|---|
Returns: | the broker’s updated own shard range. |
find_shard_ranges
(shard_size, limit=-1, existing_ranges=None)¶Scans the container db for shard ranges. Scanning will start at the
upper bound of the any existing_ranges
that are given, otherwise
at ShardRange.MIN
. Scanning will stop when limit
shard ranges
have been found or when no more shard ranges can be found. In the
latter case, the upper bound of the final shard range will be equal to
the upper bound of the container namespace.
This method does not modify the state of the db; callers are responsible for persisting any shard range data in the db.
Parameters: |
|
---|---|
Returns: | a tuple; the first value in the tuple is a list of dicts each having keys {‘index’, ‘lower’, ‘upper’, ‘object_count’} in order of ascending ‘upper’; the second value in the tuple is a boolean which is True if the last shard range has been found, False otherwise. |
get_all_shard_range_data
()¶Returns a list of all shard range data, including own shard range and deleted shard ranges.
Returns: | A list of dict representations of a ShardRange. |
---|
get_brokers
()¶Return a list of brokers for component dbs. The list has two entries
while the db state is sharding: the first entry is a broker for the
retiring db with skip_commits
set to True
; the second entry is
a broker for the fresh db with skip_commits
set to False
. For
any other db state the list has one entry.
Returns: | a list of ContainerBroker |
---|
get_db_state
()¶Returns the current state of on disk db files.
get_db_version
(conn)¶get_info
()¶Get global data for the container.
Returns: | dict with keys: account, container, created_at, put_timestamp, delete_timestamp, status_changed_at, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, x_container_sync_point2, and storage_policy_index, db_state. |
---|
get_info_is_deleted
()¶Get the is_deleted status and info for the container.
Returns: | a tuple, in the form (info, is_deleted) info is a dict as returned by get_info and is_deleted is a boolean. |
---|
get_misplaced_since
(start, count)¶Get a list of objects which are in a storage policy different from the container’s storage policy.
Parameters: |
|
---|---|
Returns: | list of dicts with keys: name, created_at, size, content_type, etag, storage_policy_index |
get_objects
(limit=None, marker='', end_marker='', include_deleted=None, since_row=None)¶Returns a list of objects, including deleted objects, in all policies. Each object in the list is described by a dict with keys {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’, ‘storage_policy_index’}.
Parameters: |
|
---|---|
Returns: | a list of dicts, each describing an object. |
get_own_shard_range
(no_default=False)¶Returns a shard range representing this broker’s own shard range. If no such range has been persisted in the broker’s shard ranges table then a default shard range representing the entire namespace will be returned.
The returned shard range will be updated with the current object stats for this broker and a meta timestamp set to the current time. For these values to be persisted the caller must merge the shard range.
Parameters: | no_default – if True and the broker’s own shard range is not found in the shard ranges table then None is returned, otherwise a default shard range is returned. |
---|---|
Returns: | an instance of ShardRange |
get_policy_stats
()¶get_reconciler_sync
()¶get_replication_info
()¶Get information about the DB required for replication.
Returns: | dict containing keys from get_info plus max_row and metadata |
---|
get_shard_ranges
(marker=None, end_marker=None, includes=None, reverse=False, include_deleted=False, states=None, include_own=False, exclude_others=False, fill_gaps=False)¶Returns a list of persisted shard ranges.
Parameters: |
|
---|---|
Returns: | a list of instances of |
get_shard_usage
()¶Get the aggregate object stats for all shard ranges in states ACTIVE, SHARDING or SHRINKING.
Returns: | a dict with keys {bytes_used, object_count} |
---|
get_sharding_sysmeta
(key=None)¶Returns sharding specific info from the broker’s metadata.
Parameters: | key – if given the value stored under key in the sharding
info will be returned. |
---|---|
Returns: | either a dict of sharding info or the value stored under
key in that dict. |
get_sharding_sysmeta_with_timestamps
()¶Returns sharding specific info from the broker’s metadata with timestamps.
Parameters: | key – if given the value stored under key in the sharding
info will be returned. |
---|---|
Returns: | a dict of sharding info with their timestamps. |
has_multiple_policies
()¶is_own_shard_range
(shard_range)¶is_reclaimable
(now, reclaim_age)¶Check if the broker abstraction is empty, and has been marked deleted for at least a reclaim age.
is_root_container
()¶Returns True if this container is a root container, False otherwise.
A root container is a container that is not a shard of another container.
is_sharded
()¶list_objects_iter
(limit, marker, end_marker, prefix, delimiter, path=None, storage_policy_index=0, reverse=False, include_deleted=False, since_row=None, transform_func=None, all_policies=False)¶Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.
Parameters: |
|
---|---|
Returns: | list of tuples of (name, created_at, size, content_type, etag, deleted) |
make_tuple_for_pickle
(record)¶Turn this db record dict into the format this service uses for pending pickles.
merge_items
(item_list, source=None)¶Merge items into the object table.
Parameters: |
|
---|
merge_shard_ranges
(shard_ranges)¶Merge shard ranges into the shard range table.
Parameters: | shard_ranges – a shard range or a list of shard ranges; each
shard range should be an instance of
ShardRange or a dict representation of
a shard range having SHARD_RANGE_KEYS . |
---|
path
¶put_object
(name, timestamp, size, content_type, etag, deleted=0, storage_policy_index=0, ctype_timestamp=None, meta_timestamp=None)¶Creates an object in the DB with its metadata.
Parameters: |
|
---|
reload_db_files
()¶Reloads the cached list of valid on disk db files for this broker.
remove_objects
(lower, upper, max_row=None)¶Removes object records in the given namespace range from the object table.
Note that objects are removed regardless of their storage_policy_index.
Parameters: |
|
---|
reported
(put_timestamp, delete_timestamp, object_count, bytes_used)¶Update reported stats, available with container’s get_info.
Parameters: |
|
---|
resolve_shard_range_states
(states)¶Given a list of values each of which may be the name of a state, the number of a state, or an alias, return the set of state numbers described by the list.
The following alias values are supported: ‘listing’ maps to all states that are considered valid when listing objects; ‘updating’ maps to all states that are considered valid for redirecting an object update.
Parameters: | states – a list of values each of which may be the name of a state, the number of a state, or an alias |
---|---|
Returns: | a set of integer state numbers, or None if no states are given |
Raises: | ValueError – if any value in the given list is neither a valid state nor a valid alias |
root_account
¶root_container
¶root_path
¶set_sharded_state
()¶Unlink’s the broker’s retiring DB file.
Returns: | True if the retiring DB was successfully unlinked, False otherwise. |
---|
set_sharding_state
()¶Creates and initializes a fresh DB file in preparation for sharding a retiring DB. The broker’s own shard range must have an epoch timestamp for this method to succeed.
Returns: | True if the fresh DB was successfully created, False otherwise. |
---|
set_sharding_sysmeta
(key, value)¶Updates the broker’s metadata metadata stored under the given key prefixed with a sharding specific namespace.
Parameters: |
|
---|
set_storage_policy_index
(policy_index, timestamp=None)¶Update the container_stat policy_index and status_changed_at.
set_x_container_sync_points
(sync_point1, sync_point2)¶sharding_initiated
()¶Returns True if a broker has shard range state that would be necessary for sharding to have been initiated, False otherwise.
sharding_required
()¶Returns True if a broker has shard range state that would be necessary for sharding to have been initiated but has not yet completed sharding, False otherwise.
storage_policy_index
¶update_reconciler_sync
(point)¶swift.container.backend.
merge_shards
(shard_data, existing)¶Compares shard_data
with existing
and updates shard_data
with
any items of existing
that take precedence over the corresponding item
in shard_data
.
Parameters: |
|
---|---|
Returns: | True if |
swift.container.backend.
update_new_item_from_existing
(new_item, existing)¶Compare the data and meta related timestamps of a new object item with the timestamps of an existing object record, and update the new item with data and/or meta related attributes from the existing record if their timestamps are newer.
The multiple timestamps are encoded into a single string for storing in the ‘created_at’ column of the objects db table.
Parameters: |
|
---|---|
Returns: | True if any attributes of the new item dict were found to be newer than the existing and therefore not updated, otherwise False implying that the updated item is equal to the existing. |
swift.container.updater.
ContainerUpdater
(conf, logger=None)¶Bases: swift.common.daemon.Daemon
Update container information in account listings.
container_report
(node, part, container, put_timestamp, delete_timestamp, count, bytes, storage_policy_index)¶Report container info to an account server.
Parameters: |
|
---|
container_sweep
(path)¶Walk the path looking for container DBs and process them.
Parameters: | path – path to walk |
---|
get_account_ring
()¶Get the account ring. Load it if it hasn’t been yet.
get_paths
()¶Get paths to all of the partitions on each drive to be processed.
Returns: | a list of paths |
---|
process_container
(dbfile)¶Process a container, and update the information in the account.
Parameters: | dbfile – container DB to process |
---|
run_forever
(*args, **kwargs)¶Run the updater continuously.
run_once
(*args, **kwargs)¶Run the updater once.
swift.container.updater.
random
() → x in the interval [0, 1).¶Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.