megstore.indexed.txt module

class megstore.indexed.txt.IndexedTxtReader(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None, errors: str = 'strict')[source]

Bases: BaseIndexedReader[str]

Random reading for txt files

If fd_idx is None, attempt to build an index from fp_jsonline to support random reading

__init__(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None, errors: str = 'strict')[source]
Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path, default is None

  • close_fileobj_when_close – When True, close stream on exit, default is False

  • index_file_mode – Index file mode, default is ‘’rb’’

  • index_build_callback – Optional callback during index building, receives each item as parameter

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

property atomic: bool

Return True if the file-like object is atomic.

batch_get(index: slice) → IterableValue[VT]
classmethod build_index(fp_data: BinaryIO, fp_index: BinaryIO, index_build_callback: Callable[[Any], None] | None = None)

Build index and write to index file

Parameters:

  • fp_data – Data stream to build index from

  • fp_index – Index file stream to write to

  • index_build_callback – Optional callback during index building

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

count()

Get total number of values in file.

Returns:

Total number of values

get(index: int) → T

Get item at specified index

Parameters:

index – Item index

Returns:

The item at the index

property mode: str
property name: Any
class megstore.indexed.txt.IndexedTxtWriter(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False, buffer_size: int = 0)[source]

Bases: BaseIndexedWriter[str]

Used to write txt streams with index support

__init__(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False, buffer_size: int = 0) → None

Initialize the indexed writer.

Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path

  • append_mode – Whether to append to an existing stream

  • close_fileobj_when_close – When True, close stream on exit; when False, will not close stream

  • buffer_size – Number of records to buffer before writing them to the data and index streams; 0 disables buffering, default is 0

Raises:

ValueError – If buffer_size is less than 0

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

append(value: T) → None

Add a record

Parameters:

value – Record to be added

property atomic: bool

Return True if the file-like object is atomic.

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

commit() → None

Write all added values to the data and index streams.

extend(values: Iterable[T]) → None

Add multiple records.

Parameters:

values – Records to be added

property mode: str
property name: Any
megstore.indexed.txt.indexed_txt_open(path: str, mode: str = 'r', *, index_path: str | None = None, open_func: ~typing.Callable[[str, str], ~typing.BinaryIO] = <function smart_open>, index_build_callback: ~typing.Callable[[~typing.Any], None] | None = None, errors: str = 'strict', buffer_size: int = 0) → IndexedTxtReader | IndexedTxtWriter[source]

Open an indexed txt file

Note

When opening mode is r, if .idx file doesn’t exist, rebuild using txt file. (.idx file corruption will not trigger rebuild, will report error) When opening mode is a, if .idx file doesn’t exist, consider .idx file as empty, equivalent to w mode.

Parameters:

  • path – txt file path

  • mode – Opening mode, supports read (r), write (w), and append (a), default r

  • index_path – Index file path, default is None

  • open_func – Open function for txt file stream Default uses smart_open

  • index_build_callback – Callback function for building index

  • errors – errors parameter for decode

  • buffer_size – Number of records buffered by a writer before writing to the data and index files; 0 disables buffering, default is 0

Raises:

ValueError – Invalid mode

Returns:

Returns megstore.IndexedTxtReader when mode is r, Returns megstore.IndexedTxtWriter when mode is w or a