Common Configuration
Environment configurations
All size unit is bytes, and support kubernetes canonical form quantity like 200Mi
.
MEGFILE_READER_BLOCK_SIZE
: default block size of read operate, unit is bytes, default is8Mi
MEGFILE_READER_MAX_BUFFER_SIZE
: max read buffer size, unit is bytes, default is128Mi
MEGFILE_WRITER_BLOCK_SIZE
:default block size of write operate, unit is bytes, default is
8Mi
In S3, the block size automatically increases with the amount of data written if you don’t set
MEGFILE_WRITER_BLOCK_SIZE
and don’t setMEGFILE_WRITER_BLOCK_AUTOSCALE
to false. The largest file size you can write under these conditions is500Gi
. If you need to write a larger file to S3, you should set a larger block size. Note that AWS S3’s multipart upload supports a maximum of 10,000 parts, so the maximum supported file size isMEGFILE_WRITE_BLOCK_SIZE
* 10,000.
MEGFILE_WRITER_MAX_BUFFER_SIZE
: max write buffer size, unit is bytes, default is128Mi
MEGFILE_WRITER_BLOCK_AUTOSCALE
: whether to automatically increase the block size; the default istrue
. However, if you setMEGFILE_WRITER_BLOCK_SIZE
, it will be set tofalse
.MEGFILE_MAX_WORKERS
: max threads will be used, default is8
MEGFILE_MAX_RETRY_TIMES
: default max retry times when catch error which may fix by retry, default is10
.