megstore.indexed.msgpack module

class megstore.indexed.msgpack.IndexedMsgpackReader(*args, **kwargs)

Bases: BaseIndexedReader[T]

Random reading for msgpack files

If fd_idx is None, attempt to build an index from fp_msgpack to support random reading

__init__(*args, **kwargs)

Parameters:

• fp_data – File object

• fp_index_path – Corresponding index file path, default is None

• close_fileobj_when_close – When True, close stream on exit, default is False

• index_file_mode – Index file mode, default is ‘’rb’’

• index_build_callback – Optional callback during index building, receives each item as parameter

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

property atomic: bool

Return True if the file-like object is atomic.

batch_get(index: slice) → IterableValue[VT]

classmethod build_index(fp_data: BinaryIO, fp_index: BinaryIO, index_build_callback: Callable[[Any], None] | None = None)

Build index and write to index file

Parameters:

• fp_data – Data stream to build index from

• fp_index – Index file stream to write to

• index_build_callback – Optional callback during index building

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

count()

Get total number of values in file.

Returns:

Total number of values

get(index: int) → T

Get item at specified index

Parameters:

index – Item index

Returns:

The item at the index

property mode: str

property name: Any

class megstore.indexed.msgpack.IndexedMsgpackWriter(fp_msgpack: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)

Bases: BaseWriter[T], Countable

Used to write msgpack streams with index support

__init__(fp_msgpack: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)

Parameters:

• fp_msgpack – msgpack stream, needs to be seekable, if it’s an s3 address, can use megstore.utils.smart_limited_seekable_open to open

• fp_index – msgpack index stream

• close_fileobj_when_close – When True, will close the stream on exit; when False, will not close the stream

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

append(value: T)

property atomic: bool

Return True if the file-like object is atomic.

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

commit()

Write already added values to msgpack stream

count() → int

extend(values: Iterable[VT])

property mode: str

property name: Any

tell() → int

megstore.indexed.msgpack.indexed_msgpack_open(path: str, mode: str = 'r', *, index_path: str | None = None, open_func: Callable[[str, str], BinaryIO] | None = None) → IndexedMsgpackReader | IndexedMsgpackWriter | IndexedMsgpackHandler

Open an indexed msgpack file

Note

Currently indexed_msgpack_open does not support opening an s3_url in a mode, users need to manually pass an open_func that supports rb+ mode, such as megfile.s3_cached_open / megfile.s3_memory_open provided by megfile.

Parameters:

• path – msgpack file path

• mode – Opening mode, supports read r, write w, append a and read-write w+ / a+, default r

• index_path – Index file path, default is None

• open_func – Open function for msgpack file stream Default uses smart_open with limited_seekable=True

Raises:

ValueError – Invalid mode

Returns:

Returns megstore.IndexedMsgpackReader when mode is r, returns megstore.IndexedMsgpackWriter when mode is w