Getting Started

aiomegfile - Asyncio implementation of megfile

Docs: http://megvii-research.github.io/aiomegfile

aiomegfile is asyncio implementation of megfile.

Support Protocols

fs(local filesystem)
s3

Support more in the future.

Installation

pip3 install aiomegfile

Quick Start

Smart methods and SmartPath are the same as megfile, but all methods are async. You can use aiomegfile to do file operations in an async way.

Functional Interface Example

import asyncio
from aiomegfile import smart_open

async def main():
    async with smart_open('s3://bucket/key', 'r') as f:
        content = await f.read()
        print(content)

if __name__ == '__main__':
    asyncio.run(main())

SmartPath Interface

SmartPath has a similar interface with pathlib.Path, but async.

import asyncio
from aiomegfile import SmartPath

async def main():
    path = SmartPath('s3://bucket/key')
    content = await path.read_text()
    print(content)

if __name__ == '__main__':
    asyncio.run(main())

Quick Start

Smart methods and SmartPath are the same as megfile, but all methods are async. You can use aiomegfile to do file operations in an async way.

Functional Interface Example

import asyncio
from aiomegfile import smart_open

async def main():
    async with smart_open('s3://bucket/key', 'r') as f:
        content = await f.read()
        print(content)

if __name__ == '__main__':
    asyncio.run(main())

SmartPath Interface

SmartPath has a similar interface with pathlib.Path, but async.

import asyncio
from aiomegfile import SmartPath

async def main():
    path = SmartPath('s3://bucket/key')
    content = await path.read_text()
    print(content)

if __name__ == '__main__':
    asyncio.run(main())

CLI

Install the CLI extras:

pip3 install "aiomegfile[cli]"

Example usage:

amf ls ./data
amf ls s3://bucket/prefix -l
amf cp -r ./data s3://bucket/backup
amf sync ./data s3://bucket/backup --progress-bar

For a wider protocol matrix and advanced configuration, see the megfile docs: http://megvii-research.github.io/megfile

How to Contribute

We welcome everyone to contribute code to the aiomegfile project, but the contributed code needs to meet the following conditions as much as possible:
You can submit code even if the code doesn’t meet conditions. The project members will evaluate and assist you in making code changes
- Code format: Your code needs to pass code format check. aiomegfile uses ruff as lint tool
- Static check: Your code needs complete type hint. aiomegfile uses pytype as static check tool. If pytype failed in static check, use # pytype: disable=XXX to disable the error and please tell us why you disable it.
- Test: Your code needs complete unit test coverage. aiomegfile uses pyfakefs and moto as local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctness
You can help to improve aiomegfile in many ways:
- Write code.
- Improve documentation.
- Report or investigate bugs and issues.
- If you find any problem or have any improving suggestion, submit a new issue as well. We will reply as soon as possible and evaluate whether to adopt.
- Review pull requests.
- Star aiomegfile repo.
- Recommend aiomegfile to your friends.
- Any other form of contribution is welcomed.