CHANGELOG
3.1.3 - 2024.08.23
feat
support retrying S3 request with error codes of
Timeout
andRequestTimeout
.
3.1.2 - 2024.08.12
perf
add retry logic for S3 error
429
fix
fix the deadlock issue of prefetch reader after fork
3.1.1 - 2024.07.24
perf
smart_cache
inw
mode can generate cache path when paramcache_path
is Nonereader and writer support more types like
numpy.uint64
for some params, e.g.offset
param inseek
supportnumpy.uint64
type
3.1.0.post2 - 2024.07.22
fix
fix
block_size
overwritten by defaultmin_block_size
ins3_open
3.1.0.post1 - 2024.07.15
fix
fix wrong name of cli
3.1.0 - 2024.07.09
breaking change
remove
megfile.utils.cachedproperty
modify the parameter order of
s3_download
ands3_upload
change param
cookie
tooffset
in someseek
functions
perf
optimize the type annotations
FileLike
is subclass ofio.IOBase
andtyping.IO
chore
use
pyproject.toml
instead ofsetup.py
3.0.6.post1 - 2024.06.21
fix
add cli extras require, run
pip install 'megfile[cli]'
to install cli requirements
3.0.6 - 2024.06.12
feat
support more parameters of
requests
likeheaders
inHttpPath
.support
overwrite
parameter incopy
,sync
andmove
functions, default isTrue
.cli command
mv
,cp
,sync
support--skip
option, can skip existed files.
perf
http request will check body size and retry when body is incomplete.
3.0.5 - 2024.05.28
feat
auto decode http content when response with
Content-Encoding
header
perf
add s3 retry code, retry request when catch response code
499
add environments for setup retry times:
MEGFILE_MAX_RETRY_TIMES
,MEGFILE_S3_MAX_RETRY_TIMES
,MEGFILE_HTTP_MAX_RETRY_TIMES
,MEGFILE_HDFS_MAX_RETRY_TIMES
,MEGFILE_SFTP_MAX_RETRY_TIMES
. default is10
3.0.4 - 2024.05.16
feat
Support more official S3 configurations. New env:
AWS_ENDPOINT_URL_S3
,AWS_ENDPOINT_URL
.
3.0.3 - 2024.05.10
perf
optimize read and write performance for s3, http and hdfs
3.0.2.post1 - 2024.04.25
fix
fix
smart_glob
bug of s3 path with profile name
3.0.2 - 2024.04.24
feat
support
-f
/--follow
intail
command
fix
fix
smart_copy
bug in the function registered byatexit
3.0.1 - 2024.03.08
feat
command
ll
show file count
perf
support new err for sftp to retry
3.0.0.post1 - 2024.01.22
fix
fix symlink paths in command
ls
andll
3.0.0 - 2024.01.22
breaking change
python3.6 and python3.7 is no longer supported
chore
support python3.12
perf
command
ls
andll
support symlink files
2.2.10.post1 - 2024.01.04
fix
fix callback in smart_copy
sync command show error log correctly
2.2.10 - 2023.12.12
feat
support some new environments for config
2.2.9.post5 - 2023.11.26
fix
add connect errors to retry list
2.2.9.post4 - 2023.11.17
fix
The file object returned by
sftp_open
supports thename
property.
2.2.9.post3 - 2023.11.09
fix
Reduce unnecessary requests in the s3 scan.
2.2.9.post2 - 2023.11.08
fix
fix the issue where
s3_upload
ands3_download
are not retrying correctly.
2.2.9.post1 - 2023.11.02
fix
fix
is_symlink
inStatResult
from scan funcs
2.2.9 - 2023.10.31
feat
cli support
ll
command
perf
optimize error message of scan funcs
when dst in cli
mv
andcp
is endswith/
, will put file into the dst dir even if it doesn’t exist
fix
fix sync when empty dir
fix the return value of the callback for smart_copy when the path is sftp
2.2.8.post2 - 2023.10.25
fix
fix sync when empty dir
2.2.8.post1 - 2023.10.20
fix
fix sftp custom port not working error
fix http retry error when upload file
2.2.8 - 2023.10.17
feat
support hdfs protocol
fix
fix
megfile config
command when config file’s parent dir not exists
2.2.7 - 2023.10.08
fix
exit 1 when cli had error
feat
cli add
config
commandcli support
--debug
parameter
2.2.6 - 2023.09.22
fix
fix s3 client cache with profile name
fix s3 error info of endpoint url in s3 open
perf
optimize concurrent SFTP connections
optimize ls command with glob path
optimize sync command
add retry error
2.2.5.post1 - 2023.09.19
fix
fix s3 error info of endpoint url
fix
s3_open
when s3 path with custom profile namefix
SmartPath
extract protocol when path is inherited fromPurePath
2.2.5 - 2023.09.15
perf
Increase
connect_timeout
config in s3perf http open when web api support
Accept-Range
but not bigger than block size.add retry for more errors
fix
fix
SftpPath.parts
fix s3
ProfileNotFound
error when profile_name not in the config file
2.2.4.post1 - 2023.09.04
fix
fix sftp readlink
2.2.4 - 2023.09.04
perf
perf prefetch reader
setup s3 connect timeout
fix
prevent http Range header out-of-range in http prefetch reader
2.2.3 - 2023.08.25
feat
add
force
param in sync methods for copy file forcibleadd
-f
/--force
inrefile sync
check same file in copy, and raise
SameFileError
perf
Set the timeout for SSH connections.
2.2.2 - 2023.08.18
fix
fix
s3_access
write permission’s check
feat
support
AWS_S3_ADDRESSING_STYLE
env for setting s3 addressing style
perf
perf some s3 api
2.2.1.post1 - 2023.08.14
fix
remove unuseful print
fix smart copy from http to local
support all s3 other config for s3 profile mode
2.2.1 - 2023.08.07
fix
fix tqdm unit divisor to
1024
feat
all open func support
encoding
anderrors
parametersadd
HttpPrefetchReader
for perf http open
2.2.0.post1 - 2023.08.03
fix
change retry log leval to info
fix
megfile head
command error from http pathfix retry not work in prefetch_reader
fix upload and download when local file path with protocol, like
file:///data/test.txt
2.2.0 - 2023.08.01
breaking change
sftp path protocol change for supporting relative path
- new protocol:
‒ sftp://[username[:password]@]hostname[:port]//absolute_file_path ‒ sftp://[username[:password]@]hostname[:port]/relative_file_path
feat
cli
Add
megfile to
command, can write content from stdin to a fileAdd
megfile head
andmegfile tail
command
Add http and stdio methods in
__init__.py
, now you can from megfile import them
2.1.4 - 2023.07.21
feat
add
SmartCache
, andsmart_cache
support more protocol
2.1.3.post1 - 2023.07.13
fix
fix the cleanup behavior of ThreadLocal after forking
2.1.3 - 2023.07.11
feat
add
http_exists
2.1.2 - 2023.07.07
feat
sync
command with-g
support sync files concurrentlysync
command add-w
/--worker
for concurrent worker’s count, default 8
fix
fix sftp error when mkdir concurrently
fix
sftp_download
andsftp_upload
‘scallback
parameter
2.1.1.post2 - 2023.07.07
fix
fix sftp client error when multi threads
fix
HttpPath
method’s parameter
2.1.1.post1 - 2023.07.04
fix
fix dst_path check in SftpPath’s methods
2.1.1 - 2023.07.03
fix
fix
ls -r
not display directoriesfix
SftpPath.cwd
fix sftp exec command method’s return data
2.1.0 - 2023.06.26
feat
smart_sync
will raiseIsADirectoryError
when src_path is a file and dst_path is a directory
fix
fix
sftp_upload
,sftp_download
,sftp_copy
path check, when path is not sftp or is dir, will raise errorsftp_copy
makedir ifdst_path
‘s dir not exist
2.0.7 - 2023.06.16
perf
retry ConnectionError in sftp retry
2.0.6.post1 - 2023.06.16
fix
fix sftp retry bug when catch EOFError
fix the bug from new
urlsplit
in py3.11.4fix the path list’s order returned by fs glob, now return path list in ascending alphabetical order
2.0.6 - 2023.06.13
fix
fix sftp connect timeout after long time
perf
smart_sync
and other sync methods will ignore same filessmart_sync
will raiseFileNotFound
error when src_path is not exist
2.0.5.post1 - 2023.05.11
fix
fix
SftpPath.rename
error log
2.0.5 - 2023.05.11
feat
support python 3.11
perf
cli support s3 log
fix
fix
is_dir
andis_file
ofSftpPath
when file not found
2.0.4 - 2023.04.12
feat
s3 path support custom profile name, like
s3[+profile_name]://bucket/key
remove
smart-open
from requirementssmart_sync
supportmap_func
parameter for concurrentadd
smart_concat
perf
reduce the number of
s3_open
‘s requests
2.0.3 - 2023.03.22
feat
add smart_lstat
smart_scandir support ‘with’ operate
fix
fix smart_sync error when file name in dir is empty str
fix stat properties default value
fix smart_load_content when path is not fs or s3
2.0.2 - 2023.03.13
support s3 endpoint env: AWS_ENDPOINT
2.0.1 - 2023.03.01
cli
megfile ls
support glob pathmegfile cp
,megfile mv
andmegfile sync
support-g
,--progress-bar
perf err output
megfile sync
support glob path
close ssh connection before process exit
2.0.0 - 2023.02.10
path classes align with
pathlib.Path
methods(
glob
,iglob
,glob_stat
,resolve
,home
,cwd
,readlink
) in all path classes return path objectmethods(
glob
,iglob
,glob_stat
) in all path classes addpattern
parameter. Functions(likesmart_glob
,s3_glob
) not change.relative_to
‘s parameterother
in all path classes change to*other
FSPath.parts
align withpathlib.Path.parts
, return valueparts[0]
will not befile://
any more.mkdir
in all path classes add parameters(mode=0o777
,parents=False
), Functions(likesmart_makedirs
) not change.
change
s3_symlink
,S3Path.symlink
,s3_rename
,S3Path.rename
parameter name, changesrc_url
,dst_url
tosrc_path
,dst_path
change
fs_stat
,FSPath.stat
,s3_stat
,S3Path.stat
parameter name, changefollowlinks
tofollow_symlinks
FileEntry
add methodinode
StatResult
add properties(st_mode
,st_ino
,st_dev
,st_nlink
,st_uid
,st_gid
,st_size
,st_atime
,st_mtime
,st_ctime
,st_atime_ns
,st_mtime_ns
,st_ctime_ns
)support sftp protocol
1.0.2 - 2022.09.22
remove
smart_getmd5_by_paths
methodretry when catch
botocore.exceptions.ResponseStreamingError
remove
followlinks
parameter in rename, move, remove; make behavior same as standard libraryfix
smart_rename
bug, when rename file cross platform or device
1.0.1 - 2022.08.04
fix open mode with + in different order
sort
smart_getmd5_by_paths
parameter paths
1.0.0 - 2022.07.25
refactor code
add
smart_getmd5_by_paths
change of symlink’s parameters position
0.1.2 - 2022.04.26
handle s3 remove file errors
support s3 symlink
0.1.1 - 2022.01.14
fix smart api bug
0.1.0 - 2022.01.14
update get_md5, s3 use etag and support dir
fix py35 test about moto
add fs symlink support
support python 3.10
0.0.11 - 2021.12.08
smart_open
support read and write pipe
0.0.10 - 2021.11.29
add info log about environ OSS_ENDPOINT and oss config file
smart_getsize
andsmart_getmtime
support httpupdate cli cp and mv, make them like cp and mv in linux
fix sed warning in macOS
add some test code
add error code callback to _patch_make_request
Generate cache_path automatically
0.0.9 - 2021.10.11
megfile.s3
retries when server returns 500 - 503remove
megfile.lib.fakefs
0.0.8 - 2021.09.15
megfile.s3.s3_memory_open
support ab / rb+ / wb+ / ab+ modemegfile.s3.s3_open
support ab / rb+ / wb+ / ab+ mode (by using s3_memory_open)Speed up
s3_glob
Accept
s3.endpoint_url
in aws config file
0.0.7 - 2021.09.06
[Breaking] Rename
megfile.interfaces.MegfilePathLike
tomegfile.interfaces.PathLike
Fix ungloblize
0.0.6 - 2021.09.01
[Breaking] Rename
megfile.s3.MEGFILE_MD5_HEADER
tomegfile.s3.content_md5_header
[Breaking] Remove
megfile.lib.get_image_size
,megfile.smart.smart_load_image_metadata
andmegfile.smart.IMAGE_EXTNAMES
0.0.5 - 2021.08.31
Refactor
process_local
/thread_local
, remove dependency onmultiprocessing.utils.register_after_fork
0.0.4 - 2021.08.29
Speed up
s3_glob
0.0.3 - 2021.08.24
First release of
megfile