Getting Started

megstore - Common Formats. Uncommon Speed.

• Docs: http://megvii-research.github.io/megstore

megstore is a Python library that provides a unified interface for file operations across various file formats. It aims to simplify file handling, while also enhancing performance and reliability.

Why megstore

• Faster file read and write operations, the random read performance is also very fast with index support.

• Low memory usage with streaming read and write support.

• Supports popular file formats and easy to use.

Quick Start

Installation

pip3 install megstore

# for msgpack support
pip3 install 'megstore[msgpack]'

Examples

• indexed jsonline

from megstore import indexed_jsonline_open

with indexed_jsonline_open("data.jsonl", "w") as writer:
    writer.append({"key": "value"})
    writer.append({"number": 123})

with indexed_jsonline_open("data.jsonl", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for item in reader:
        print(item)

• indexed msgpack

from megstore import indexed_msgpack_open

with indexed_msgpack_open("data.msg", "w") as writer:
    writer.append({"key": "value"})
    writer.append({"number": 123})

with indexed_msgpack_open("data.msg", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for item in reader:
        print(item)

• indexed txt

from megstore import indexed_txt_open

with indexed_txt_open("data.txt", "w") as writer:
    writer.append("Hello, World!")
    writer.append("This is a test.")

with indexed_txt_open("data.txt", "r") as reader:
    second_item = reader[1]
    second_to_last_items = reader[1:]
    total_count = len(reader)
    for line in reader:
        print(line)

How to Contribute

• We welcome everyone to contribute code to the megstore project, but the contributed code needs to meet the following conditions as much as possible:

  You can submit code even if the code doesn’t meet conditions. The project members will evaluate and assist you in making code changes

  • Code format: Your code needs to pass code format check. megstore uses ruff as lint tool

  • Static check: Your code needs complete type hint. megstore uses pytype as static check tool. If pytype failed in static check, use # pytype: disable=XXX to disable the error and please tell us why you disable it.

  • Test: Your code needs complete unit test coverage. megstore uses pyfakefs and moto as local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctness

• You can help to improve megstore in many ways:

  • Write code.

  • Improve documentation.

  • Report or investigate bugs and issues.

  • If you find any problem or have any improving suggestion, submit a new issuse as well. We will reply as soon as possible and evaluate whether to adopt.

  • Review pull requests.

  • Star megstore repo.

  • Recommend megstore to your friends.

  • Any other form of contribution is welcomed.