CHANGELOG
4.0.2 - 2024.12.20
feat
add env
MEGFILE_LOG_LEVEL
for set log leveladd
--log-level
in cli for set log level
fix
fix
max_buffer_size
not work when is0
insmart_open
4.0.1 - 2024.12.12
feat
support kubernetes canonical form quantity in environment variable
support add host key for sftp
perf
use
head_object
to get file size in prefetch reader when no cache
fix
fix
smart_load_content
bug when only end is not None
4.0.0.post1 - 2024.12.09
fix s3 part upload size being too small caused by autoscaling block size
4.0.0 - 2024.12.06
breaking change
Dropped support for Python 3.8; added support for Python 3.13.
Removed the deprecated methods
is_link
andget_protocol
fromBasePath
.The
__del__
method inFilelike
now closes file handles.Removed
BaseURIPath
.- Changed the default SFTP missing host key policy from
auto
toreject
. You can set the default policy using the
MEGFILE_SFTP_HOST_KEY_POLICY
environment.
Available values:auto
,reject
,warning
.
- Changed the default SFTP missing host key policy from
Environment variable changes:
Removed:
MEGFILE_BLOCK_SIZE
MEGFILE_MAX_BUFFER_SIZE
MEGFILE_BLOCK_CAPACITY
MEGFILE_MIN_BLOCK_SIZE
MEGFILE_MAX_BLOCK_SIZE
Added:
MEGFILE_READER_BLOCK_SIZE
MEGFILE_READER_MAX_BUFFER_SIZE
MEGFILE_WRITER_BLOCK_SIZE
MEGFILE_WRITER_MAX_BUFFER_SIZE
Updated
MEGFILE_MAX_WORKERS
default value from32
to8
.
Updated all
open
method parameters:Removed
min_block_size
andmax_block_size
.Replaced
forward_ratio: Optional[float]
byblock_forward: Optional[int]
.Renamed
max_concurrency
tomax_workers
.
Updated
__init__
parameters for allReader
classes inheriting fromBasePrefetchReader
:Replaced
block_capacity: int
bymax_buffer_size: int
.
Updated
S3BufferedWriter.__init__
parameters:Removed
max_block_size
.
perf
Introduced
TransferConfig
for S3 to allow user-configurable upload and download settings.By default, writing larger files to S3 is now supported. Refer to the configuration docs for details.
Reduced the number of S3 requests during
smart_sync
.
feat
Added the
MEGFILE_WRITER_BLOCK_AUTOSCALE
environment variable to enable S3 block autoscaling. Default istrue
. However, if you setMEGFILE_WRITER_BLOCK_SIZE
, default will befalse
.
3.1.7 - 2024.12.12
fix
fix
smart_load_content
bug when only end is not None
3.1.6.post1 - 2024.11.18
fix
support ali oss response
strip auth headers on redirect
3.1.6 - 2024.11.14
fix
fix bugs of alias paths in
megfile ls
commandsupport
InvalidRange
error in tos oss
feat
add new env
AWS_S3_VERIFY
for setting s3 verify, default istrue
add new env
AWS_S3_REDIRECT
for turning on supporting http redirect, default isfalse
3.1.5 - 2024.11.05
feat
support
AWS_SESSION_TOKEN
env
3.1.4 - 2024.11.01
feat
add
megfile config alias
command for adding custom protocol aliasglob path support using glob syntax within curly braces but does not support nesting.
3.1.3 - 2024.08.23
feat
support retrying S3 request with error codes of
Timeout
andRequestTimeout
.
3.1.2 - 2024.08.12
perf
add retry logic for S3 error
429
fix
fix the deadlock issue of prefetch reader after fork
3.1.1 - 2024.07.24
perf
smart_cache
inw
mode can generate cache path when paramcache_path
is Nonereader and writer support more types like
numpy.uint64
for some params, e.g.offset
param inseek
supportnumpy.uint64
type
3.1.0.post2 - 2024.07.22
fix
fix
block_size
overwritten by defaultmin_block_size
ins3_open
3.1.0.post1 - 2024.07.15
fix
fix wrong name of cli
3.1.0 - 2024.07.09
breaking change
remove
megfile.utils.cachedproperty
modify the parameter order of
s3_download
ands3_upload
change param
cookie
tooffset
in someseek
functions
perf
optimize the type annotations
FileLike
is subclass ofio.IOBase
andtyping.IO
chore
use
pyproject.toml
instead ofsetup.py
3.0.6.post1 - 2024.06.21
fix
add cli extras require, run
pip install 'megfile[cli]'
to install cli requirements
3.0.6 - 2024.06.12
feat
support more parameters of
requests
likeheaders
inHttpPath
.support
overwrite
parameter incopy
,sync
andmove
functions, default isTrue
.cli command
mv
,cp
,sync
support--skip
option, can skip existed files.
perf
http request will check body size and retry when body is incomplete.
3.0.5 - 2024.05.28
feat
auto decode http content when response with
Content-Encoding
header
perf
add s3 retry code, retry request when catch response code
499
add environments for setup retry times:
MEGFILE_MAX_RETRY_TIMES
,MEGFILE_S3_MAX_RETRY_TIMES
,MEGFILE_HTTP_MAX_RETRY_TIMES
,MEGFILE_HDFS_MAX_RETRY_TIMES
,MEGFILE_SFTP_MAX_RETRY_TIMES
. default is10
3.0.4 - 2024.05.16
feat
Support more official S3 configurations. New env:
AWS_ENDPOINT_URL_S3
,AWS_ENDPOINT_URL
.
3.0.3 - 2024.05.10
perf
optimize read and write performance for s3, http and hdfs
3.0.2.post1 - 2024.04.25
fix
fix
smart_glob
bug of s3 path with profile name
3.0.2 - 2024.04.24
feat
support
-f
/--follow
intail
command
fix
fix
smart_copy
bug in the function registered byatexit
3.0.1 - 2024.03.08
feat
command
ll
show file count
perf
support new err for sftp to retry
3.0.0.post1 - 2024.01.22
fix
fix symlink paths in command
ls
andll
3.0.0 - 2024.01.22
breaking change
python3.6 and python3.7 is no longer supported
chore
support python3.12
perf
command
ls
andll
support symlink files
2.2.10.post1 - 2024.01.04
fix
fix callback in smart_copy
sync command show error log correctly
2.2.10 - 2023.12.12
feat
support some new environments for config
2.2.9.post5 - 2023.11.26
fix
add connect errors to retry list
2.2.9.post4 - 2023.11.17
fix
The file object returned by
sftp_open
supports thename
property.
2.2.9.post3 - 2023.11.09
fix
Reduce unnecessary requests in the s3 scan.
2.2.9.post2 - 2023.11.08
fix
fix the issue where
s3_upload
ands3_download
are not retrying correctly.
2.2.9.post1 - 2023.11.02
fix
fix
is_symlink
inStatResult
from scan funcs
2.2.9 - 2023.10.31
feat
cli support
ll
command
perf
optimize error message of scan funcs
when dst in cli
mv
andcp
is endswith/
, will put file into the dst dir even if it doesn’t exist
fix
fix sync when empty dir
fix the return value of the callback for smart_copy when the path is sftp
2.2.8.post2 - 2023.10.25
fix
fix sync when empty dir
2.2.8.post1 - 2023.10.20
fix
fix sftp custom port not working error
fix http retry error when upload file
2.2.8 - 2023.10.17
feat
support hdfs protocol
fix
fix
megfile config
command when config file’s parent dir not exists
2.2.7 - 2023.10.08
fix
exit 1 when cli had error
feat
cli add
config
commandcli support
--debug
parameter
2.2.6 - 2023.09.22
fix
fix s3 client cache with profile name
fix s3 error info of endpoint url in s3 open
perf
optimize concurrent SFTP connections
optimize ls command with glob path
optimize sync command
add retry error
2.2.5.post1 - 2023.09.19
fix
fix s3 error info of endpoint url
fix
s3_open
when s3 path with custom profile namefix
SmartPath
extract protocol when path is inherited fromPurePath
2.2.5 - 2023.09.15
perf
Increase
connect_timeout
config in s3perf http open when web api support
Accept-Range
but not bigger than block size.add retry for more errors
fix
fix
SftpPath.parts
fix s3
ProfileNotFound
error when profile_name not in the config file
2.2.4.post1 - 2023.09.04
fix
fix sftp readlink
2.2.4 - 2023.09.04
perf
perf prefetch reader
setup s3 connect timeout
fix
prevent http Range header out-of-range in http prefetch reader
2.2.3 - 2023.08.25
feat
add
force
param in sync methods for copy file forcibleadd
-f
/--force
inrefile sync
check same file in copy, and raise
SameFileError
perf
Set the timeout for SSH connections.
2.2.2 - 2023.08.18
fix
fix
s3_access
write permission’s check
feat
support
AWS_S3_ADDRESSING_STYLE
env for setting s3 addressing style
perf
perf some s3 api
2.2.1.post1 - 2023.08.14
fix
remove unuseful print
fix smart copy from http to local
support all s3 other config for s3 profile mode
2.2.1 - 2023.08.07
fix
fix tqdm unit divisor to
1024
feat
all open func support
encoding
anderrors
parametersadd
HttpPrefetchReader
for perf http open
2.2.0.post1 - 2023.08.03
fix
change retry log leval to info
fix
megfile head
command error from http pathfix retry not work in prefetch_reader
fix upload and download when local file path with protocol, like
file:///data/test.txt
2.2.0 - 2023.08.01
breaking change
sftp path protocol change for supporting relative path
- new protocol:
‒ sftp://[username[:password]@]hostname[:port]//absolute_file_path ‒ sftp://[username[:password]@]hostname[:port]/relative_file_path
feat
cli
Add
megfile to
command, can write content from stdin to a fileAdd
megfile head
andmegfile tail
command
Add http and stdio methods in
__init__.py
, now you can from megfile import them
2.1.4 - 2023.07.21
feat
add
SmartCache
, andsmart_cache
support more protocol
2.1.3.post1 - 2023.07.13
fix
fix the cleanup behavior of ThreadLocal after forking
2.1.3 - 2023.07.11
feat
add
http_exists
2.1.2 - 2023.07.07
feat
sync
command with-g
support sync files concurrentlysync
command add-w
/--worker
for concurrent worker’s count, default 8
fix
fix sftp error when mkdir concurrently
fix
sftp_download
andsftp_upload
‘scallback
parameter
2.1.1.post2 - 2023.07.07
fix
fix sftp client error when multi threads
fix
HttpPath
method’s parameter
2.1.1.post1 - 2023.07.04
fix
fix dst_path check in SftpPath’s methods
2.1.1 - 2023.07.03
fix
fix
ls -r
not display directoriesfix
SftpPath.cwd
fix sftp exec command method’s return data
2.1.0 - 2023.06.26
feat
smart_sync
will raiseIsADirectoryError
when src_path is a file and dst_path is a directory
fix
fix
sftp_upload
,sftp_download
,sftp_copy
path check, when path is not sftp or is dir, will raise errorsftp_copy
makedir ifdst_path
‘s dir not exist
2.0.7 - 2023.06.16
perf
retry ConnectionError in sftp retry
2.0.6.post1 - 2023.06.16
fix
fix sftp retry bug when catch EOFError
fix the bug from new
urlsplit
in py3.11.4fix the path list’s order returned by fs glob, now return path list in ascending alphabetical order
2.0.6 - 2023.06.13
fix
fix sftp connect timeout after long time
perf
smart_sync
and other sync methods will ignore same filessmart_sync
will raiseFileNotFound
error when src_path is not exist
2.0.5.post1 - 2023.05.11
fix
fix
SftpPath.rename
error log
2.0.5 - 2023.05.11
feat
support python 3.11
perf
cli support s3 log
fix
fix
is_dir
andis_file
ofSftpPath
when file not found
2.0.4 - 2023.04.12
feat
s3 path support custom profile name, like
s3[+profile_name]://bucket/key
remove
smart-open
from requirementssmart_sync
supportmap_func
parameter for concurrentadd
smart_concat
perf
reduce the number of
s3_open
‘s requests
2.0.3 - 2023.03.22
feat
add smart_lstat
smart_scandir support ‘with’ operate
fix
fix smart_sync error when file name in dir is empty str
fix stat properties default value
fix smart_load_content when path is not fs or s3
2.0.2 - 2023.03.13
support s3 endpoint env: AWS_ENDPOINT
2.0.1 - 2023.03.01
cli
megfile ls
support glob pathmegfile cp
,megfile mv
andmegfile sync
support-g
,--progress-bar
perf err output
megfile sync
support glob path
close ssh connection before process exit
2.0.0 - 2023.02.10
path classes align with
pathlib.Path
methods(
glob
,iglob
,glob_stat
,resolve
,home
,cwd
,readlink
) in all path classes return path objectmethods(
glob
,iglob
,glob_stat
) in all path classes addpattern
parameter. Functions(likesmart_glob
,s3_glob
) not change.relative_to
‘s parameterother
in all path classes change to*other
FSPath.parts
align withpathlib.Path.parts
, return valueparts[0]
will not befile://
any more.mkdir
in all path classes add parameters(mode=0o777
,parents=False
), Functions(likesmart_makedirs
) not change.
change
s3_symlink
,S3Path.symlink
,s3_rename
,S3Path.rename
parameter name, changesrc_url
,dst_url
tosrc_path
,dst_path
change
fs_stat
,FSPath.stat
,s3_stat
,S3Path.stat
parameter name, changefollowlinks
tofollow_symlinks
FileEntry
add methodinode
StatResult
add properties(st_mode
,st_ino
,st_dev
,st_nlink
,st_uid
,st_gid
,st_size
,st_atime
,st_mtime
,st_ctime
,st_atime_ns
,st_mtime_ns
,st_ctime_ns
)support sftp protocol
1.0.2 - 2022.09.22
remove
smart_getmd5_by_paths
methodretry when catch
botocore.exceptions.ResponseStreamingError
remove
followlinks
parameter in rename, move, remove; make behavior same as standard libraryfix
smart_rename
bug, when rename file cross platform or device
1.0.1 - 2022.08.04
fix open mode with + in different order
sort
smart_getmd5_by_paths
parameter paths
1.0.0 - 2022.07.25
refactor code
add
smart_getmd5_by_paths
change of symlink’s parameters position
0.1.2 - 2022.04.26
handle s3 remove file errors
support s3 symlink
0.1.1 - 2022.01.14
fix smart api bug
0.1.0 - 2022.01.14
update get_md5, s3 use etag and support dir
fix py35 test about moto
add fs symlink support
support python 3.10
0.0.11 - 2021.12.08
smart_open
support read and write pipe
0.0.10 - 2021.11.29
add info log about environ OSS_ENDPOINT and oss config file
smart_getsize
andsmart_getmtime
support httpupdate cli cp and mv, make them like cp and mv in linux
fix sed warning in macOS
add some test code
add error code callback to _patch_make_request
Generate cache_path automatically
0.0.9 - 2021.10.11
megfile.s3
retries when server returns 500 - 503remove
megfile.lib.fakefs
0.0.8 - 2021.09.15
megfile.s3.s3_memory_open
support ab / rb+ / wb+ / ab+ modemegfile.s3.s3_open
support ab / rb+ / wb+ / ab+ mode (by using s3_memory_open)Speed up
s3_glob
Accept
s3.endpoint_url
in aws config file
0.0.7 - 2021.09.06
[Breaking] Rename
megfile.interfaces.MegfilePathLike
tomegfile.interfaces.PathLike
Fix ungloblize
0.0.6 - 2021.09.01
[Breaking] Rename
megfile.s3.MEGFILE_MD5_HEADER
tomegfile.s3.content_md5_header
[Breaking] Remove
megfile.lib.get_image_size
,megfile.smart.smart_load_image_metadata
andmegfile.smart.IMAGE_EXTNAMES
0.0.5 - 2021.08.31
Refactor
process_local
/thread_local
, remove dependency onmultiprocessing.utils.register_after_fork
0.0.4 - 2021.08.29
Speed up
s3_glob
0.0.3 - 2021.08.24
First release of
megfile