megstore.indexed.jsonline module

class megstore.indexed.jsonline.IndexedJsonlineReader(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None)[source]

Bases: BaseIndexedReader[T]

Random reading for jsonline files

If fd_idx is None, attempt to build an index from fp_jsonline to support random reading

__init__(fp_data: BinaryIO, fp_index_path: str | None = None, *, close_fileobj_when_close: bool = False, index_file_mode: str = 'rb', index_build_callback: Callable[[Any], None] | None = None) → None
Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path, default is None

  • close_fileobj_when_close – When True, close stream on exit, default is False

  • index_file_mode – Index file mode, default is ‘’rb’’

  • index_build_callback – Optional callback during index building, receives each item as parameter

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

property atomic: bool

Return True if the file-like object is atomic.

batch_get(index: slice) → IterableValue[VT]
classmethod build_index(fp_data: BinaryIO, fp_index: BinaryIO, index_build_callback: Callable[[Any], None] | None = None)

Build index and write to index file

Parameters:

  • fp_data – Data stream to build index from

  • fp_index – Index file stream to write to

  • index_build_callback – Optional callback during index building

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

count()

Get total number of values in file.

Returns:

Total number of values

get(index: int) → T

Get item at specified index

Parameters:

index – Item index

Returns:

The item at the index

property mode: str
property name: Any
class megstore.indexed.jsonline.IndexedJsonlineWriter(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)[source]

Bases: BaseIndexedWriter[T]

Used to write jsonline streams with index support

__init__(fp_data: BinaryIO, fp_index_path: str, *, append_mode: bool = False, close_fileobj_when_close: bool = False)
Parameters:

  • fp_data – File object

  • fp_index_path – Corresponding index file path

  • close_fileobj_when_close – When True, close stream on exit; when False, will not close stream

abort() → bool

Abort the file-like object without saving.

This method has no effect if the file is already closed.

append(value: T)

Add a record

Parameters:

value – Record to be added

property atomic: bool

Return True if the file-like object is atomic.

close() → None

Flush and close the file-like object.

This method has no effect if the file is already closed.

property closed: bool

Return True if the file-like object is closed.

commit()

Write already added values to jsonline stream

extend(values: Iterable[VT])
property mode: str
property name: Any
megstore.indexed.jsonline.indexed_jsonline_open(path: str, mode: str = 'r', *, index_path: str | None = None, open_func: ~typing.Callable[[str, str], ~typing.BinaryIO] = <function smart_open>, index_build_callback: ~typing.Callable[[~typing.Any], None] | None = None) → IndexedJsonlineReader | IndexedJsonlineWriter[source]

Open an indexed jsonline file

Note

When opening mode is r, if .idx file doesn’t exist, rebuild using jsonline file. (.idx file corruption will not trigger rebuild, will report error) When opening mode is a, if .idx file doesn’t exist, consider .idx file as empty, equivalent to w mode.

Parameters:

  • path – jsonline file path

  • mode – Opening mode, supports read r, write w, and append a, default r

  • index_path – Index file path, default is None

  • open_func – Open function for jsonline file stream Default uses smart_open

  • index_build_callback – Callback function for building index

Raises:

ValueError – Invalid mode

Returns:

Returns IndexedJsonlineReader when mode is r, Returns IndexedJsonlineWriter when mode is w or a