Pluggable On-Disk Back-end APIs

The internal REST API used between the proxy server and the account, container and object server is almost identical to public Swift REST API, but with a few internal extensions (for example, update an account with a new container).

The pluggable back-end APIs for the three REST API servers (account, container, object) abstracts the needs for servicing the various REST APIs from the details of how data is laid out and stored on-disk.

The APIs are documented in the reference implementations for all three servers. For historical reasons, the object server backend reference implementation module is named diskfile, while the account and container server backend reference implementation modules are named appropriately.

This API is still under development and not yet finalized.

Back-end API for Account Server REST APIs

Pluggable Back-end for Account Server

class swift.account.backend.AccountBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False)

Encapsulates working with an account database.

create_account_stat_table(conn, put_timestamp)

Create account_stat table which is specific to the account DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
create_container_table(conn)

Create container table which is specific to the account DB.

Parameters:conn – DB connection object
create_policy_stat_table(conn)

Create policy_stat table which is specific to the account DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:conn – DB connection object
empty()

Check if the account DB is empty.

Returns:True if the database has no active containers.
get_info()

Get global data for the account.

Returns:dict with keys: account, created_at, put_timestamp, delete_timestamp, status_changed_at, container_count, object_count, bytes_used, hash, id
get_policy_stats(do_migrations=False)

Get global policy stats for the account.

Parameters:do_migrations – boolean, if True the policy stat dicts will always include the ‘container_count’ key; otherwise it may be omitted on legacy databases until they are migrated.
Returns:dict of policy stats where the key is the policy index and the value is a dictionary like {‘object_count’: M, ‘bytes_used’: N, ‘container_count’: L}
is_status_deleted()

Only returns true if the status field is set to DELETED.

list_containers_iter(limit, marker, end_marker, prefix, delimiter, reverse=False)

Get a list of containers sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • end_marker – end marker query
  • prefix – prefix query
  • delimiter – delimiter for query
  • reverse – reverse the result order.
Returns:

list of tuples of (name, object_count, bytes_used, 0)

merge_items(item_list, source=None)

Merge items into the container table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘put_timestamp’, ‘delete_timestamp’, ‘object_count’, ‘bytes_used’, ‘deleted’, ‘storage_policy_index’}
  • source – if defined, update incoming_sync with the source
put_container(name, put_timestamp, delete_timestamp, object_count, bytes_used, storage_policy_index)

Create a container with the given attributes.

Parameters:
  • name – name of the container to create
  • put_timestamp – put_timestamp of the container to create
  • delete_timestamp – delete_timestamp of the container to create
  • object_count – number of objects in the container
  • bytes_used – number of bytes used by the container
  • storage_policy_index – the storage policy for this container

Back-end API for Container Server REST APIs

Pluggable Back-ends for Container Server

class swift.container.backend.ContainerBroker(db_file, timeout=25, logger=None, account=None, container=None, pending_timeout=None, stale_reads_ok=False)

Encapsulates working with a container database.

create_container_info_table(conn, put_timestamp, storage_policy_index)

Create the container_info table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code. Also creates the container_stat view.

Parameters:
  • conn – DB connection object
  • put_timestamp – put timestamp
  • storage_policy_index – storage policy index
create_object_table(conn)

Create the object table which is specific to the container DB. Not a part of Pluggable Back-ends, internal to the baseline code.

Parameters:conn – DB connection object
create_policy_stat_table(conn, storage_policy_index=0)

Create policy_stat table.

Parameters:
  • conn – DB connection object
  • storage_policy_index – the policy_index the container is being created with
delete_object(name, timestamp, storage_policy_index=0)

Mark an object deleted.

Parameters:
  • name – object name to be deleted
  • timestamp – timestamp when the object was marked as deleted
  • storage_policy_index – the storage policy index for the object
empty()

Check if container DB is empty.

Returns:True if the database has no active objects, False otherwise
get_info()

Get global data for the container.

Returns:dict with keys: account, container, created_at, put_timestamp, delete_timestamp, status_changed_at, object_count, bytes_used, reported_put_timestamp, reported_delete_timestamp, reported_object_count, reported_bytes_used, hash, id, x_container_sync_point1, x_container_sync_point2, and storage_policy_index.
get_info_is_deleted()

Get the is_deleted status and info for the container.

Returns:a tuple, in the form (info, is_deleted) info is a dict as returned by get_info and is_deleted is a boolean.
get_misplaced_since(start, count)

Get a list of objects which are in a storage policy different from the container’s storage policy.

Parameters:
  • start – last reconciler sync point
  • count – maximum number of entries to get
Returns:

list of dicts with keys: name, created_at, size, content_type, etag, storage_policy_index

list_objects_iter(limit, marker, end_marker, prefix, delimiter, path=None, storage_policy_index=0, reverse=False)

Get a list of objects sorted by name starting at marker onward, up to limit entries. Entries will begin with the prefix and will not have the delimiter after the prefix.

Parameters:
  • limit – maximum number of entries to get
  • marker – marker query
  • end_marker – end marker query
  • prefix – prefix query
  • delimiter – delimiter for query
  • path – if defined, will set the prefix and delimiter based on the path
  • storage_policy_index – storage policy index for query
  • reverse – reverse the result order.
Returns:

list of tuples of (name, created_at, size, content_type, etag)

merge_items(item_list, source=None)

Merge items into the object table.

Parameters:
  • item_list – list of dictionaries of {‘name’, ‘created_at’, ‘size’, ‘content_type’, ‘etag’, ‘deleted’, ‘storage_policy_index’, ‘ctype_timestamp’, ‘meta_timestamp’}
  • source – if defined, update incoming_sync with the source
put_object(name, timestamp, size, content_type, etag, deleted=0, storage_policy_index=0, ctype_timestamp=None, meta_timestamp=None)

Creates an object in the DB with its metadata.

Parameters:
  • name – object name to be created
  • timestamp – timestamp of when the object was created
  • size – object size
  • content_type – object content-type
  • etag – object etag
  • deleted – if True, marks the object as deleted and sets the deleted_at timestamp to timestamp
  • storage_policy_index – the storage policy index for the object
  • ctype_timestamp – timestamp of when content_type was last updated
  • meta_timestamp – timestamp of when metadata was last updated
reported(put_timestamp, delete_timestamp, object_count, bytes_used)

Update reported stats, available with container’s get_info.

Parameters:
  • put_timestamp – put_timestamp to update
  • delete_timestamp – delete_timestamp to update
  • object_count – object_count to update
  • bytes_used – bytes_used to update
set_storage_policy_index(policy_index, timestamp=None)

Update the container_stat policy_index and status_changed_at.

swift.container.backend.update_new_item_from_existing(new_item, existing)

Compare the data and meta related timestamps of a new object item with the timestamps of an existing object record, and update the new item with data and/or meta related attributes from the existing record if their timestamps are newer.

The multiple timestamps are encoded into a single string for storing in the ‘created_at’ column of the objects db table.

Parameters:
  • new_item – A dict of object update attributes
  • existing – A dict of existing object attributes
Returns:

True if any attributes of the new item dict were found to be newer than the existing and therefore not updated, otherwise False implying that the updated item is equal to the existing.

Back-end API for Object Server REST APIs