megstore.indexed.txt module

class megstore.indexed.txt.IndexedTxtReader(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None, errors: str = 'strict')[source]

Bases: BaseIndexedReader[str]

Random reading for txt files

If fd_idx is None, attempt to build an index from fp_jsonline to support random reading

__init__(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None, errors: str = 'strict')[source]
Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path, default is None

  • close_fileobj_when_close – When True, close stream on exit, default is False

  • index_file_mode – Index file mode, default is ‘’rb’’

  • index_build_callback – Optional callback during index building, receives each item as parameter

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

property atomic: bool

Return True if the file-like object is atomic.

batch_get(index: slice) → IterableValue[VT]
classmethod build_index(fp_data: BinaryIO, fp_index: BinaryIO, index_build_callback: Callable[[Any], None] | None = None)

Build index and write to index file

Parameters:

  • fp_data – Data stream to build index from

  • fp_index – Index file stream to write to

  • index_build_callback – Optional callback during index building

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

count()

Get total number of values in file.

Returns:

Total number of values

get(index: int) → T

Get item at specified index

Parameters:

index – Item index

Returns:

The item at the index

property mode: str
property name: Any
class megstore.indexed.txt.IndexedTxtWriter(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)[source]

Bases: BaseIndexedWriter[str]

Used to write txt streams with index support

__init__(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)
Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path

  • close_fileobj_when_close – When True, close stream on exit; when False, will not close stream

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

append(value: T)

Add a record

Parameters:

value – Record to be added

property atomic: bool

Return True if the file-like object is atomic.

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

commit()

Write already added values to jsonline stream

extend(values: Iterable[VT])
property mode: str
property name: Any
megstore.indexed.txt.indexed_txt_open(path: str, mode: str = 'r', *, index_path: str | None = None, open_func: ~typing.Callable[[str, str], ~typing.BinaryIO] = <function smart_open>, index_build_callback: ~typing.Callable[[~typing.Any], None] | None = None, errors: str = 'strict') → IndexedTxtReader | IndexedTxtWriter[source]

Open an indexed txt file

Note

When opening mode is r, if .idx file doesn’t exist, rebuild using txt file. (.idx file corruption will not trigger rebuild, will report error) When opening mode is a, if .idx file doesn’t exist, consider .idx file as empty, equivalent to w mode.

Parameters:

  • path – txt file path

  • mode – Opening mode, supports read (r), write (w), and append (a), default r

  • index_path – Index file path, default is None

  • open_func – Open function for txt file stream Default uses smart_open

  • index_build_callback – Callback function for building index

  • errors – errors parameter for decode

Raises:

ValueError – Invalid mode

Returns:

Returns megstore.IndexedTxtReader when mode is r, Returns megstore.IndexedTxtWriter when mode is w or a