From 379e0bcf782de5208b9644dbf76b48c1866d6777 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 1 Nov 2023 15:56:11 +0800 Subject: [PATCH 01/13] version to v2.13.0 --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c05bf5f2..c3ba80f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# 2.13.0 +## 更新 +1. batchstat 和 batchsign 支持结果输出到文件 +2. get / qdwonload / qdownlaod2 当指定下载域名则仅使用此域名进行下载 +3. rput / qupload / qupload2 支持顺序读 + + # 2.12.0 ## 新增 1. qdownload / qdownload2 / get 下载命令支持使用源站域名进行下载,用户可不对 bucket 绑定域名(qshell 内部自动查询源站域名 ,用户不用设置) From faea8bb545c1fbf57b2b4b519436270fb1cd2420 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 1 Nov 2023 16:23:09 +0800 Subject: [PATCH 02/13] version to v2.13.0 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3ba80f1..d8f2051f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # 2.13.0 ## 更新 1. batchstat 和 batchsign 支持结果输出到文件 -2. get / qdwonload / qdownlaod2 当指定下载域名则仅使用此域名进行下载 +2. get / qdownload / qdownload2 当指定下载域名则仅使用此域名进行下载 3. rput / qupload / qupload2 支持顺序读 From c63bfdf146390ca2bfe8b416a9d3c836450ee935 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Tue, 7 Nov 2023 19:35:21 +0800 Subject: [PATCH 03/13] change qdownload / qdownload2 / get doc --- docs/get.md | 5 ++++- docs/qdownload.md | 7 +++---- docs/qdownload2.md | 5 +++-- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/get.md b/docs/get.md index 42be3487..35b5b359 100644 --- a/docs/get.md +++ b/docs/get.md @@ -23,9 +23,12 @@ $ qshell get --doc - Bucket:空间名。 【必选】 - Key:存储空间中的文件名字。 【必选】 +## 注: +1. 使用 bucket 绑定的源站域名和七牛源站域名下载资源,这部分下载产生的流量会生成存储源站下载流量的计费,请注意,这部分计费不在七牛 CDN 免费 10G 流量覆盖范围,具体域名使用参考:--domain 选项。 + # 选项 - -o/--outfile:保存在本地的文件路径;不指定,保存在当前文件夹,文件名使用存储空间中的名字【可选】 -- --domain:指定下载请求的域名,当指定了下载域名则仅使用此下载域名进行下载;默认为空,此时 qshell 下载使用域名的优先级:1.bucket 绑定的域名(qshell 内部查询,无需配置) 2. 源站域名(qshell 内部查询,无需配置),当优先级高的域名下载失败后会尝试使用优先级低的域名进行下载。【可选】 +- --domain:指定下载请求的域名,当指定了下载域名则仅使用此下载域名进行下载;默认为空,此时 qshell 下载使用域名的优先级:1.bucket 绑定的 CDN 域名(qshell 内部查询,无需配置) 2.bucket 绑定的源站域名(qshell 内部查询,无需配置) 3. 七牛源站域名(qshell 内部查询,无需配置),当优先级高的域名下载失败后会尝试使用优先级低的域名进行下载。【可选】 - --get-file-api: 当存储服务端支持 getfile 接口时才有效。【可选】 - --public:空间是否为公开空间;为 `true` 时为公有空间,公有空间下载时不会对下载 URL 进行签名,可以提升 CDN 域名性能,默认为 `false`(私有空间)【可选】 - --check-size: 下载后检测本地文件和服务端文件 size 的一致性。【可选】 diff --git a/docs/qdownload.md b/docs/qdownload.md index 1e91767c..8680a16e 100644 --- a/docs/qdownload.md +++ b/docs/qdownload.md @@ -2,12 +2,11 @@ `qdownload` 可以将七牛空间中的文件同步到本地磁盘中。支持只同步带特定前缀或者后缀的文件,也支持在本地备份路径不变的情况下进行增量同步。 需要额外指出的是,将文件同步到本地都是走的七牛存储源站的流量而不是 `CDN` 的流量,因为 `CDN` 通常情况下会认为大量的文件下载操作是非法访问从而进行限制。 -注: +## 注: +- 使用 bucket 绑定的源站域名和七牛源站域名下载资源,这部分下载产生的流量会生成存储源站下载流量的计费,请注意,这部分计费不在七牛 CDN 免费 10G 流量覆盖范围,具体域名使用参考配置:domain 。 - `Key` 中的 `/` 会被当做路径处理,也即任何以 `/` 结尾的 `Key` 均会被当做文件夹处理。 - 如果使用的是 CDN 域名,且 CDN 域名开启了图片优化中的图片自动瘦身功能时,下载文件的信息和七牛服务端记录的文件信息不一致,此时下载不要使用 --check-size 和 --check-hash 选项,否则下载会失败。 -### 注:【该功能默认需要计费,如果希望享受 10G 的免费流量,请自行将 domain 参数设置为 CDN 域名,如不设置,需支付源站流量费用,无法减免!!!】 - 本工具批量下载文件支持多文件并发下载,另外还支持单个文件的断点下载。除此之外,也可以支持指定前缀或者后缀的文件同步,注意这里的前缀只能指定一个,但是后缀可以指定多个,多个后缀直接使用英文的逗号(,)分隔。 # 格式 @@ -69,7 +68,7 @@ $ qshell qdownload --doc - save_path_handler:指定一个回调函数;在构建文件的保存路径时,优先使用此选项进行构建,如果不配置则使用 $dest_dir + $文件分割符 + $Key 方式进行构建。文档下面有常用场景实例。此函数通过 Go 语言的模板实现,函数验证使用 func 命令,具体语法可参考 func 命令说明,handler 使用方式下方有示例可供参考 【可选】 - check_size:下载后检测本地文件和服务端文件 size 的一致性,默认为 `false`。【可选】 - check_hash:是否验证 hash,如果开启可能会耗费较长时间,默认为 `false` 【可选】 -- domain:指定下载请求的域名,当指定了下载域名则仅使用此下载域名进行下载;默认为空,此时 qshell 下载使用域名的优先级:1.bucket 绑定的域名(qshell 内部查询,无需配置) 2. 源站域名(qshell 内部查询,无需配置),当优先级高的域名下载失败后会尝试使用优先级低的域名进行下载。【该功能默认需要计费,如果希望享受 10G 的免费流量,请自行配置此参数为 CDN 域名,如不配置,需支付源站流量费用,无法减免!!!】 【可选】 +- domain:指定下载请求的域名,当指定了下载域名则仅使用此下载域名进行下载;默认为空,此时 qshell 下载使用域名的优先级:1.bucket 绑定的 CDN 域名(qshell 内部查询,无需配置) 2.bucket 绑定的源站域名(qshell 内部查询,无需配置) 3. 七牛源站域名(qshell 内部查询,无需配置),当优先级高的域名下载失败后会尝试使用优先级低的域名进行下载。【可选】 - referer:如果下载请求域名配置了域名白名单防盗链,需要指定一个允许访问的 referer 地址;默认为空 【可选】 - public:空间是否为公开空间;为 `true` 时为公有空间,公有空间下载时不会对下载 URL 进行签名,可以提升 CDN 域名性能,默认为 `false`(私有空间)【可选】 - enable_slice: 是否开启切片下载,需要注意 `slice_file_size_threshold` 切片阈值选项的配置,只有开启切片下载,并且下载的文件大小大于切片阈值方会启动切片下载。默认不开启。【可选】 diff --git a/docs/qdownload2.md b/docs/qdownload2.md index 74050d3c..3a2a3bf1 100644 --- a/docs/qdownload2.md +++ b/docs/qdownload2.md @@ -1,10 +1,11 @@ # 简介 -`qdownload2` 功能和 `qdownload` 一致,不过 `qdownload2` 通过命令行的方式来指定各个需要的参数,例如: +`qdownload2` 功能和 `qdownload` 一致,选项也是一一对应,不过 `qdownload2` 通过命令行的方式来指定各个需要的参数,例如: ``` qshell qdownload2 --dest-dir=/home/jemy/temp --bucket=test ``` -注: +## 注: +- 使用 bucket 绑定的源站域名和七牛源站域名下载资源,这部分下载产生的流量会生成存储源站下载流量的计费,请注意,这部分计费不在七牛 CDN 免费 10G 流量覆盖范围,具体域名使用参考 qdownload 命令的同功能配置:domain 。 - `Key` 中的 `/` 会被当做路径处理,也即任何以 `/` 结尾的 `Key` 均会被当做文件夹处理。 - 如果使用的是 CDN 域名,且 CDN 域名开启了图片优化中的图片自动瘦身功能时,下载文件的信息和七牛服务端记录的文件信息不一致,此时下载不要使用 --check-size 和 --check-hash 选项,否则下载会失败。 From b01034ee170e8cc164cdbcadfc588c809eb254ce Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Tue, 7 Nov 2023 21:15:24 +0800 Subject: [PATCH 04/13] handle download range --- iqshell/storage/object/download/downloader.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/iqshell/storage/object/download/downloader.go b/iqshell/storage/object/download/downloader.go index 759c88b3..107416f3 100644 --- a/iqshell/storage/object/download/downloader.go +++ b/iqshell/storage/object/download/downloader.go @@ -294,7 +294,7 @@ func downloadTempFileWithDownloader(dl downloader, fInfo *fileInfo, info *Downlo // 检查 fromBytes 和 fileSize,fromBytes 不能 > fileSize if info.RangeFromBytes > 0 { - if info.RangeFromBytes > info.FileSize || info.RangeFromBytes > info.RangeToBytes { + if info.RangeFromBytes > info.FileSize || (info.RangeFromBytes > info.RangeToBytes && info.RangeToBytes > 0) { errorDesc := "download, check fromBytes error: fromBytes bigger than file size, should remove temp file and retry." log.Warning(errorDesc) if e := fInfo.cleanTempFile(); e != nil { From 9f1f5931530778c4892d031e2c130092493360af Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Fri, 24 Nov 2023 18:05:03 +0800 Subject: [PATCH 05/13] add archive ir --- cmd/asyncfetch.go | 5 +++-- cmd/bucket.go | 5 +++-- cmd/rs.go | 11 +++++++---- cmd/rsbatch.go | 5 +++-- cmd/upload.go | 8 ++++---- docs/abfetch.md | 2 +- docs/batchchlifecycle.md | 14 ++++++++------ docs/batchchtype.md | 4 ++-- docs/chlifecycle.md | 15 +++++++++------ docs/chtype.md | 4 ++-- docs/fput.md | 2 +- docs/listbucket2.md | 2 +- docs/qupload.md | 2 +- docs/qupload2.md | 2 +- docs/rput.md | 4 ++-- docs/sync.md | 2 +- iqshell/storage/bucket/list.go | 2 +- iqshell/storage/bucket/operations/list.go | 2 +- iqshell/storage/object/lifecycle.go | 6 ++++++ iqshell/storage/object/operations/lifecycle.go | 17 +++++++++++------ iqshell/storage/object/operations/status.go | 3 ++- iqshell/storage/object/status.go | 4 ++++ 22 files changed, 74 insertions(+), 47 deletions(-) diff --git a/cmd/asyncfetch.go b/cmd/asyncfetch.go index b5b579e5..52205341 100644 --- a/cmd/asyncfetch.go +++ b/cmd/asyncfetch.go @@ -1,10 +1,11 @@ package cmd import ( + "github.com/spf13/cobra" + "github.com/qiniu/qshell/v2/docs" "github.com/qiniu/qshell/v2/iqshell" "github.com/qiniu/qshell/v2/iqshell/storage/object/operations" - "github.com/spf13/cobra" ) func asyncFetchCmdBuilder(cfg *iqshell.Config) *cobra.Command { @@ -32,7 +33,7 @@ func asyncFetchCmdBuilder(cfg *iqshell.Config) *cobra.Command { cmd.Flags().StringVarP(&info.CallbackBody, "callback-body", "b", "", "callback body") cmd.Flags().StringVarP(&info.CallbackHost, "callback-host", "T", "", "callback HOST") - cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "storage type, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage") + cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "storage type, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage") cmd.Flags().IntVarP(&info.FileType, "storage-type", "g", 0, "storage type, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage-type", "use --file-type instead") // 废弃 storage-type diff --git a/cmd/bucket.go b/cmd/bucket.go index dbedb435..9d985895 100644 --- a/cmd/bucket.go +++ b/cmd/bucket.go @@ -1,11 +1,12 @@ package cmd import ( + "github.com/spf13/cobra" + "github.com/qiniu/qshell/v2/docs" "github.com/qiniu/qshell/v2/iqshell" "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/storage/bucket/operations" - "github.com/spf13/cobra" ) var domainsCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { @@ -136,7 +137,7 @@ var listBucketCmd2Builder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().StringVarP(&info.StartDate, "start", "s", "", "start date with format yyyy-mm-dd-hh-MM-ss, all files will be listed according to the prefix and then filtered.") cmd.Flags().StringVarP(&info.EndDate, "end", "e", "", "end date with format yyyy-mm-dd-hh-MM-ss, all files will be listed according to the prefix and then filtered.") - cmd.Flags().StringVarP(&info.FileTypes, "file-types", "", "", "Specify storage type, separated by comma, all files will be listed according to the prefix and then filtered. 0:STANDARD storage, 1:IA storage, 2 means ARCHIVE storage. 3:DEEP_ARCHIVE storage") + cmd.Flags().StringVarP(&info.FileTypes, "file-types", "", "", "Specify storage type, separated by comma, all files will be listed according to the prefix and then filtered. 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage 4:ARCHIVE_IR storage") cmd.Flags().StringVarP(&info.FileTypes, "storages", "", "", "Specify storage type, same to --file-types") _ = cmd.Flags().MarkDeprecated("storages", "use --file-types instead") // 废弃 storages diff --git a/cmd/rs.go b/cmd/rs.go index 4da46295..fcd9205d 100644 --- a/cmd/rs.go +++ b/cmd/rs.go @@ -1,10 +1,11 @@ package cmd import ( + "github.com/spf13/cobra" + "github.com/qiniu/qshell/v2/docs" "github.com/qiniu/qshell/v2/iqshell" "github.com/qiniu/qshell/v2/iqshell/storage/object/operations" - "github.com/spf13/cobra" ) var statCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { @@ -88,8 +89,9 @@ var changeLifecycleCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { }, } cmd.Flags().IntVarP(&info.ToIAAfterDays, "to-ia-after-days", "", 0, "to IA storage after some days. the range is -1 or bigger than 0. -1 means cancel to IA storage") - cmd.Flags().IntVarP(&info.ToArchiveAfterDays, "to-archive-after-days", "", 0, "to archive storage after some days. the range is -1 or bigger than 0. -1 means cancel to archive storage") - cmd.Flags().IntVarP(&info.ToDeepArchiveAfterDays, "to-deep-archive-after-days", "", 0, "to deep archive storage after some days. the range is -1 or bigger than 0. -1 means cancel to deep archive storage") + cmd.Flags().IntVarP(&info.ToArchiveIRAfterDays, "to-archive-ir-after-days", "", 0, "to ARCHIVE_IR storage after some days. the range is -1 or bigger than 0. -1 means cancel to ARCHIVE_IR storage") + cmd.Flags().IntVarP(&info.ToArchiveAfterDays, "to-archive-after-days", "", 0, "to ARCHIVE storage after some days. the range is -1 or bigger than 0. -1 means cancel to ARCHIVE storage") + cmd.Flags().IntVarP(&info.ToDeepArchiveAfterDays, "to-deep-archive-after-days", "", 0, "to DEEP_ARCHIVE storage after some days. the range is -1 or bigger than 0. -1 means cancel to DEEP_ARCHIVE storage") cmd.Flags().IntVarP(&info.DeleteAfterDays, "delete-after-days", "", 0, "delete after some days. the range is -1 or bigger than 0. -1 means cancel to delete") return cmd } @@ -242,7 +244,8 @@ var changeTypeCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { And 0 means STANDARD storage, while 1 means IA storage, while 2 means ARCHIVE storage. -while 3 means DEEP_ARCHIVE storage.`, +while 3 means DEEP_ARCHIVE storage. +while 4 means ARCHIVE_IR storage.`, Example: `change storage type of A.png(bucket:bucketA key:A.png) to ARCHIVE storage qshell chtype bucketA A.png 2 and you can check result by command: diff --git a/cmd/rsbatch.go b/cmd/rsbatch.go index 684be3b5..56a9cd28 100644 --- a/cmd/rsbatch.go +++ b/cmd/rsbatch.go @@ -176,8 +176,9 @@ Lifecycle value must great than or equal to -1, unit: day. }, } cmd.Flags().IntVarP(&info.ToIAAfterDays, "to-ia-after-days", "", 0, "to IA storage after some days. the range is -1 or bigger than 0. -1 means cancel to IA storage") - cmd.Flags().IntVarP(&info.ToArchiveAfterDays, "to-archive-after-days", "", 0, "to archive storage after some days. the range is -1 or bigger than 0. -1 means cancel to archive storage") - cmd.Flags().IntVarP(&info.ToDeepArchiveAfterDays, "to-deep-archive-after-days", "", 0, "to deep archive storage after some days. the range is -1 or bigger than 0. -1 means cancel to deep archive storage") + cmd.Flags().IntVarP(&info.ToArchiveIRAfterDays, "to-archive-ir-after-days", "", 0, "to ARCHIVE_IR storage after some days. the range is -1 or bigger than 0. -1 means cancel to ARCHIVE_IR storage") + cmd.Flags().IntVarP(&info.ToArchiveAfterDays, "to-archive-after-days", "", 0, "to ARCHIVE storage after some days. the range is -1 or bigger than 0. -1 means cancel to ARCHIVE storage") + cmd.Flags().IntVarP(&info.ToDeepArchiveAfterDays, "to-deep-archive-after-days", "", 0, "to DEEP_ARCHIVE storage after some days. the range is -1 or bigger than 0. -1 means cancel to DEEP_ARCHIVE storage") cmd.Flags().IntVarP(&info.DeleteAfterDays, "delete-after-days", "", 0, "delete after some days. the range is -1 or bigger than 0. -1 means cancel to delete") setBatchCmdDefaultFlags(cmd, &info.BatchInfo) return cmd diff --git a/cmd/upload.go b/cmd/upload.go index 749e021b..aa5f68db 100644 --- a/cmd/upload.go +++ b/cmd/upload.go @@ -98,7 +98,7 @@ var upload2CmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().StringVar(&LogLevel, "log-level", "debug", "log level") cmd.Flags().IntVar(&LogRotate, "log-rotate", 7, "log rotate days") - cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage") + cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage") cmd.Flags().IntVarP(&info.FileType, "storage", "", 0, "set storage type of file, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage @@ -135,7 +135,7 @@ var syncCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().Int64VarP(&info.ChunkSize, "resumable-api-v2-part-size", "", data.BLOCK_SIZE, "the part size when use resumable upload v2 APIs to upload, default 4M") cmd.Flags().StringVarP(&info.UpHost, "up-host", "u", "", "upload host") - cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage") + cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage") cmd.Flags().IntVarP(&info.FileType, "storage", "s", 0, "set storage type of file, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage @@ -169,7 +169,7 @@ var formUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().BoolVarP(&info.Overwrite, "overwrite", "", false, "overwrite the file of same key in bucket") cmd.Flags().StringVarP(&info.MimeType, "mimetype", "t", "", "file mime type") - cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage") + cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage") cmd.Flags().IntVarP(&info.FileType, "storage", "s", 0, "set storage type of file, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage @@ -211,7 +211,7 @@ var resumeUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().Int64VarP(&info.ChunkSize, "v2-part-size", "", data.BLOCK_SIZE, "the part size when use resumable upload v2 APIs to upload, same to --resumable-api-v2-part-size") _ = cmd.Flags().MarkDeprecated("v2-part-size", "use --resumable-api-v2-part-size instead") - cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage") + cmd.Flags().IntVarP(&info.FileType, "file-type", "", 0, "set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage") cmd.Flags().IntVarP(&info.FileType, "storage", "s", 0, "set storage type of file, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage diff --git a/docs/abfetch.md b/docs/abfetch.md index 0890119b..35570f03 100644 --- a/docs/abfetch.md +++ b/docs/abfetch.md @@ -37,7 +37,7 @@ $ qshell abfetch --doc - -b/--callback-body:回调的 http Body。 【可选】 - -T/--callback-host:回调时的 HOST 头。 【可选】 - -a/--callback-url:回调的请求地址。 【可选】 -- --file-type:抓取的资源存储在七牛存储空间的类型,0:普通存储 1:低频存储 2:归档存储 3:深度归档, 默认为: 0。 【可选】 +- --file-type:抓取的资源存储在七牛存储空间的类型,0:普通存储 1:低频存储 2:归档存储 3:深度归档 4:归档直读存储, 默认为: 0。 【可选】 - -c/--thread-count:指定抓取时使用的线程数目,默认:20。 【可选】 - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 - -s/--success-list:指定一个文件的路径,如果资源抓取成功,则将资源信息写入此文件;默认不导出。 【可选】 diff --git a/docs/batchchlifecycle.md b/docs/batchchlifecycle.md index fadb5e98..2038774e 100644 --- a/docs/batchchlifecycle.md +++ b/docs/batchchlifecycle.md @@ -9,13 +9,13 @@ * 大于 0: 设置相关的生命周期 2. 生命周期时间大小规则如下(在相关生命周期时间值大于 0 时需满足): ``` -转低频存储时间 < 转归档存储时间 < 转深度归档存储时间 +转低频存储时间 < 转归档直读存储时间 < 转归档存储时间 < 转深度归档存储时间 ``` -3. 转低频存储时间、转归档存储时间、转深度归档存储时间 和 过期删除时间 至少配置一个 +3. 转低频存储时间、转归档直读存储时间、转归档存储时间、转深度归档存储时间 和 过期删除时间 至少配置一个 # 格式 ``` -qshell batchchlifecycle [--force] [--success-list ] [--failure-list ] [--sep ] [--worker ] [--to-ia-after-days ] [--to-archive-after-days ] [--to-deep-archive-after-days ] [--delete-after-days ] <-i KeysFile> +qshell batchchlifecycle [--force] [--success-list ] [--failure-list ] [--sep ] [--worker ] [--to-ia-after-days ] [--to-archive-ir-after-days ] [--to-archive-after-days ] [--to-deep-archive-after-days ] [--delete-after-days ] <-i KeysFile> ``` # 帮助文档 @@ -44,9 +44,10 @@ $ qshell batchchlifecycle --doc // :文件名,:分割符, 其他无效内容 ``` - --to-ia-after-days:指定文件上传后并在设置的时间后转换到 `低频存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `低频存储` 的生命周期规则,单位:天【可选】 +- --to-archive-ir-after-days:指定文件上传后并在设置的时间后转换到 `归档直读存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `归档直读存储` 的生命周期规则,单位:天【可选】 - --to-archive-after-days:指定文件上传后并在设置的时间后转换到 `归档存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `归档存储` 的生命周期规则,单位:天【可选】 - --to-deep-archive-after-days:指定文件上传后并在设置的时间后转换到 `深度归档存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `深度归档存储` 的生命周期规则,单位:天【可选】 - - --delete-after-days:指定文件上传后并在设置的时间后进行 `过期删除`,删除后不可恢复;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的 `过期删除` 的生命周期规则,单位:天【可选】 +- --delete-after-days:指定文件上传后并在设置的时间后进行 `过期删除`,删除后不可恢复;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的 `过期删除` 的生命周期规则,单位:天【可选】 - -y/--force:该选项控制工具的默认行为。默认情况下,对于批量操作,工具会要求使用者输入一个验证码,确认下要进行批量文件操作了,避免操作失误的发生。如果不需要这个验证码的提示过程可以使用此选项。【可选】 - -s/--success-list:该选项指定一个文件,程序会把操作成功的资源信息导入到该文件;默认不导出。【可选】 - -e/--failure-list:该选项指定一个文件,程序会把操作失败的资源信息加上错误信息导入该文件;默认不导出。【可选】 @@ -58,7 +59,7 @@ $ qshell batchchlifecycle --doc - --record-redo-while-error:依赖于 --enable-record;命令重新执行时,命令中所有任务会从头到尾重新执行;每个任务执行前会根据记录先查看当前任务是否已经执行,如果任务已执行且失败,则再执行一次;默认为 false,当任务执行失败则跳过不再重新执行。 【可选】 # 示例 -1 比如我们要将空间 `if-pbl` 里面一些文件的生命周期改为 30 天后转低频存储,60 天后转归档存储,180 天后转深度归档存储,365 天后过期删除;我们可以指定如下的 `KeysFile` 的内容: +1 比如我们要将空间 `if-pbl` 里面一些文件的生命周期改为 30 天后转低频存储,60 天后转归档直读存储,120 天后转归档存储,180 天后转深度归档存储,365 天后过期删除;我们可以指定如下的 `KeysFile` 的内容: ``` 2015/03/22/qiniu.png 2015/photo.jpg @@ -68,7 +69,8 @@ $ qshell batchchlifecycle --doc ``` $ qshell batchchlifecycle if-pbl -i lifecycle.txt \ --to-ia-after-days 30 \ - --to-archive-after-days 60 \ + --to-archive-ir-after-days 60 \ + --to-archive-after-days 120 \ --to-deep-archive-after-days 180 \ --delete-after-days 365 ``` diff --git a/docs/batchchtype.md b/docs/batchchtype.md index 31ae1745..1bd509a5 100644 --- a/docs/batchchtype.md +++ b/docs/batchchtype.md @@ -1,5 +1,5 @@ # 简介 -`batchchtype` 命令用来为空间中的文件设置存储类型。该操作发生在同一个空间中。(将文件设置为 **深度归档存储** 或者 **归档存储** 或者 **低频存储** 或者 **普通存储**,默认:文件为 **普通存储**) +`batchchtype` 命令用来为空间中的文件设置存储类型。该操作发生在同一个空间中。(将文件设置为 **深度归档存储** 或者 **归档存储** 或者 **归档直读存储** 或者 **低频存储** 或者 **普通存储**,默认:文件为 **普通存储**) # 格式 ``` @@ -23,7 +23,7 @@ $ qshell batchchtype --doc - Bucket:空间名,可以为公开空间或私有空间。【必选】 # 选项 -- -i/--input-file:接受一个文件, 文件内容每行包含 `原文件名` 和 `存储类型`,存储类型用数字表示,0 为普通存储,1 为低频存储,2 为归档存储,3 为深度归档存储。每行多个元素名之间用分割符分隔(默认 tab 制表符); 如果需要自定义分割符,可以使用 `-F` 或 `--sep` 选项指定自定义的分隔符。如果没有通过该选项指定该文件参数, 从标准输入读取内容。每行包含 `文件名` 和 `存储类型`;具体格式如下:(【可选】) +- -i/--input-file:接受一个文件, 文件内容每行包含 `原文件名` 和 `存储类型`,存储类型用数字表示,0 为普通存储,1 为低频存储,2 为归档存储,3 为深度归档存储,4 为归档直读存储。每行多个元素名之间用分割符分隔(默认 tab 制表符); 如果需要自定义分割符,可以使用 `-F` 或 `--sep` 选项指定自定义的分隔符。如果没有通过该选项指定该文件参数, 从标准输入读取内容。每行包含 `文件名` 和 `存储类型`;具体格式如下:(【可选】) ``` 1 // :文件名,:分割符,1:低频存储。 ``` diff --git a/docs/chlifecycle.md b/docs/chlifecycle.md index a300d473..a53a5234 100644 --- a/docs/chlifecycle.md +++ b/docs/chlifecycle.md @@ -9,13 +9,13 @@ * 大于 0: 设置相关的生命周期 2. 生命周期时间大小规则如下(在相关生命周期时间值大于 0 时需满足): ``` -转低频存储时间 < 转归档存储时间 < 转深度归档存储时间 +转低频存储时间 < 转归档直读存储时间 < 转归档存储时间 < 转深度归档存储时间 ``` -3. 转低频存储时间、转归档存储时间、转深度归档存储时间 和 过期删除时间 至少配置一个 +3. 转低频存储时间、转归档直读存储时间、转归档存储时间、转深度归档存储时间 和 过期删除时间 至少配置一个 # 格式 ``` -qshell chlifecycle [--to-ia-after-days ] [--to-archive-after-days ] [--to-deep-archive-after-days ] [--delete-after-days ] +qshell chlifecycle [--to-ia-after-days ] [--to-archive-ir-after-days ] [--to-archive-after-days ] [--to-deep-archive-after-days ] [--delete-after-days ] ``` # 帮助文档 @@ -36,17 +36,19 @@ $ qshell chlifecycle --doc # 选项 - --to-ia-after-days:指定文件上传后并在设置的时间后转换到 `低频存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `低频存储` 的生命周期规则,单位:天【可选】 +- --to-archive-ir-after-days:指定文件上传后并在设置的时间后转换到 `归档直读存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `归档直读存储` 的生命周期规则,单位:天【可选】 - --to-archive-after-days:指定文件上传后并在设置的时间后转换到 `归档存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `归档存储` 的生命周期规则,单位:天【可选】 - --to-deep-archive-after-days:指定文件上传后并在设置的时间后转换到 `深度归档存储类型`;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的转 `深度归档存储` 的生命周期规则,单位:天【可选】 - --delete-after-days:指定文件上传后并在设置的时间后进行 `过期删除`,删除后不可恢复;值范围为 -1 或者大于 0,设置为 -1 表示取消已设置的 `过期删除` 的生命周期规则,单位:天【可选】 # 示例 -1 比如我们要将空间 `if-pbl` 里面 `qiniu.png` 文件的生命周期改为 30 天后转低频存储,60 天后转归档存储,180 天后转深度归档存储,365 天后过期删除: +1 比如我们要将空间 `if-pbl` 里面 `qiniu.png` 文件的生命周期改为 30 天后转低频存储,60 天后转归档直读存储,120 天后转归档存储,180 天后转深度归档存储,365 天后过期删除: ``` $ qshell chlifecycle if-pbl qiniu.png \ --to-ia-after-days 30 \ - --to-archive-after-days 60 \ + --to-archive-ir-after-days 60 \ + --to-archive-after-days 120 \ --to-deep-archive-after-days 180 \ --delete-after-days 365 ``` @@ -64,7 +66,8 @@ PutTime: 16768889367943931 -> 2023-02-20 18:28:56.7943931 +0800 MimeType: text/plain Expiration: 1710518400 -> 2024-03-16 00:00:00 +0800 CST TransitionToIA: 1681574400 -> 2023-04-16 00:00:00 +0800 CST -TransitionToArchive: 1684166400 -> 2023-05-16 00:00:00 +0800 CST +TransitionToArchiveIR: 1684166400 -> 2023-05-16 00:00:00 +0800 CST +TransitionToArchive: 1684166400 -> 2023-06-16 00:00:00 +0800 CST TransitionToDeepArchive: 1694534400 -> 2023-09-13 00:00:00 +0800 CST FileType: 0 -> 标准存储 ``` diff --git a/docs/chtype.md b/docs/chtype.md index 07491717..170c098e 100644 --- a/docs/chtype.md +++ b/docs/chtype.md @@ -22,10 +22,10 @@ $ qshell chtype --doc # 参数 - Bucket:空间名,可以为公开空间或私有空间。【必选】 - Key:空间中的文件名。【必选】 -- FileType:给文件指定的新的存储类型,其中可选值为 `0` 代表 `普通存储`,`1` 代表 `低频存储`,`2` 代表 `归档存储`,`3` 代表 `深度归档存储`。【必选】 +- FileType:给文件指定的新的存储类型,其中可选值为 `0` 代表 `普通存储`,`1` 代表 `低频存储`,`2` 代表 `归档存储`,`3` 代表 `深度归档存储`,`4` 代表 `归档直读存储`。【必选】 注: -`归档存储` 直接转 `普通存储` 或 `低频存储` 会失败,需要先通过 restorear 命令恢复后再转。 +`归档存储` 或 `深度归档存储` 直接转 `普通存储` 或 `低频存储` 会失败,需要先通过 restorear 命令恢复后再转。 # 示例 修改 `if-pbl` 空间中 `qiniu.png` 图片的存储类型为 `低频存储` diff --git a/docs/fput.md b/docs/fput.md index 4e8ccdfd..502be0c7 100644 --- a/docs/fput.md +++ b/docs/fput.md @@ -31,7 +31,7 @@ $ qshell fput --doc # 选项 - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 - -t/--mimetype:指定文件的 MimeType。 【可选】 -- --file-type:文件存储类型,默认为 `0`(标准存储),`1` 为低频存储,`2` 为归档存储,`3` 为深度归档存储。 【可选】 +- --file-type:文件存储类型,默认为 `0`(标准存储),`1` 为低频存储,`2` 为归档存储,`3` 为深度归档存储,`4` 为归档直读存储。 【可选】 - -u/--up-host: 指定上传域名。 【可选】 - -l/--callback-urls:上传回调地址, 可以指定多个地址,以逗号分隔。 【可选】 - -T/--callback-host:上传回调的 HOST, 必须和 CallbackUrls 一起指定。 【可选】 diff --git a/docs/listbucket2.md b/docs/listbucket2.md index 5a8b7047..ba5e6f61 100644 --- a/docs/listbucket2.md +++ b/docs/listbucket2.md @@ -40,7 +40,7 @@ $ qshell listbucket2 --doc - --output-file-max-size:每个输出文件的最大尺寸,大于此值会自动创建新的文件(新文件的文件名规律示例,源文件:/x/x/a.txt,新建文件为:/x/x/a-${index}.txt,index 为创建文件的序列号,从 0 开始),0:不限制单个输出文件的尺寸,单位:B,默认:0。 【可选】 - --start:根据列举前缀列举整个空间,然后从中筛选出文件上传日期在 之后的文件;格式:yyyy-mm-dd-hh-MM-ss eg:2022-01-10-08-30-20 。【可选】 - --end:根据列举前缀列举整个空间, 然后从中筛选出文件上传日期在之前的文件;格式:yyyy-mm-dd-hh-MM-ss eg:2022-01-10-08-30-20 。【可选】 -- --file-types:根据列举前缀列举整个空间,然后从中筛选出满足七牛存储类型的文件;配置多个存储类型时中间用逗号隔开(eg: 1,2,3);`0`:`普通存储`,`1`:`低频存储`,`2`:`归档存储`,`3`:`深度归档存储`。 +- --file-types:根据列举前缀列举整个空间,然后从中筛选出满足七牛存储类型的文件;配置多个存储类型时中间用逗号隔开(eg: 1,2,3);`0`:`普通存储`,`1`:`低频存储`,`2`:`归档存储`,`3`:`深度归档存储`,`4`:`归档直读存储`。 - --mimetypes:根据列举前缀列举整个空间,然后从中筛选出满足 MimeType 的文件;配置多个 MimeType 时中间用逗号隔开(eg: image/*,video/)。 - --min-file-size:根据列举前缀列举整个空间,然后从中筛选出文件大小大于该值的文件;单位:B 。 - --max-file-size:根据列举前缀列举整个空间,然后从中筛选出文件大小小于该值的文件;单位:B 。 diff --git a/docs/qupload.md b/docs/qupload.md index 3888b439..98e959da 100644 --- a/docs/qupload.md +++ b/docs/qupload.md @@ -79,7 +79,7 @@ $ qshell qupload --doc - log_file:上传日志的输出文件,默认为输出到 `record_root` 指定的文件中,具体文件路径可以在终端输出看到。 【可选】 - log_rotate:上传日志文件的切换周期,单位为天,默认为 7 天即切换到新的上传日志文件。 【可选】 - log_stdout:上传日志是否同时输出一份到标准终端,默认为 `true`。 【可选】 -- file_type:文件存储类型;`0`:标准存储,`1`:低频存储,`2`:归档存储,`3`:深度归档存储;默认为 `0`(标准存储)。 【可选】 +- file_type:文件存储类型;`0`:标准存储,`1`:低频存储,`2`:归档存储,`3`:深度归档存储,`4`:归档直读存储;默认为 `0`(标准存储)。 【可选】 - delete_on_success:上传成功的文件,同时删除本地文件,以达到节约磁盘的目的,比如日志归档的场景,默认为 `false`,如果需要开启功能,设置为 `true` 即可。【可选】 - resumable_api_v2:使用分片 V2 进行上传,默认为 `false` 使用分片 V1 。【可选】 - resumable_api_v2_part_size:使用分片 V2 进行上传时定制分片大小,默认 4194304(4M) 。【可选】 diff --git a/docs/qupload2.md b/docs/qupload2.md index 585d0d3e..1553da37 100644 --- a/docs/qupload2.md +++ b/docs/qupload2.md @@ -28,7 +28,7 @@ Flags: --check-size check file size -e, --failure-list string upload failure file list --file-list string file list to upload - --file-type int set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage + --file-type int set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage -h, --help help for qupload2 --ignore-dir ignore the dir in the dest file key --key-prefix string key prefix prepended to dest file key diff --git a/docs/rput.md b/docs/rput.md index b158b25f..7c2a8dcc 100644 --- a/docs/rput.md +++ b/docs/rput.md @@ -10,7 +10,7 @@ qshell rput [--overwrite] [--v2] [--mimetype ] [--callback-urls ] [--callback-host ] [--file-type ] ``` -其中 `Overwrite`,`MimeType`,`FileType` (0: 标准存储, 1: 低频存储, 2: 归档存储, 3: 深度归档存储)参数可根据需要指定一个或者多个,参数顺序随意,程序会自动识别。 +其中 `Overwrite`,`MimeType`,`FileType` (0: 标准存储, 1: 低频存储, 2: 归档存储, 3: 深度归档存储, 4: 归档直读存储)参数可根据需要指定一个或者多个,参数顺序随意,程序会自动识别。 # 帮助文档 可以在命令行输入如下命令获取帮助文档: @@ -33,7 +33,7 @@ $ qshell rput --doc # 选项 - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 - -t/--mimetype:指定文件的 MimeType 。【可选】 -- --file-type:文件存储类型;0: 标准存储, 1: 低频存储, 2: 归档存储, 3: 深度归档存储;默认为`0`(标准存储)。 【可选】 +- --file-type:文件存储类型;0: 标准存储, 1: 低频存储, 2: 归档存储, 3: 深度归档存储, 4: 归档直读存储;默认为`0`(标准存储)。 【可选】 - -l/--callback-urls:上传回调地址,可以指定多个地址,以逗号分开。【可选】 - -T/--callbackHost:上传回调HOST, 必须和 CallbackUrls 一起指定。 【可选】 - --resumable-api-v2:使用分片上传 API V2 进行上传,默认为 `false`, 使用 V1 上传。【可选】 diff --git a/docs/sync.md b/docs/sync.md index 363061e0..2646f034 100644 --- a/docs/sync.md +++ b/docs/sync.md @@ -32,7 +32,7 @@ $ qshell sync --doc # 选项 - -k/--key:该资源保存在空间中的 key,不配置时使用资源 Url 中文件名作为存储的 key。 【可选】 - -u/--uphost:上传入口的 IP 地址,一般在大文件的情况下,可以指定上传入口的 IP 来减少 DNS 环节,提升同步速度。 【可选】 -- --file-type:文件存储类型,默认为 `0` (标准存储),`1` 为低频存储,`2` 为归档存储,`3` 为深度归档存储,【可选】 +- --file-type:文件存储类型,默认为 `0` (标准存储),`1` 为低频存储,`2` 为归档存储,`3` 为深度归档存储,`4` 为归档直读存储【可选】 - --resumable-api-v2:使用分片 v2 进行上传;默认使用 v1。 【可选】 - --resumable-api-v2-part-size:使用分片上传 API V2 进行上传时的分片大小,默认为 4M 。【可选】 - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 diff --git a/iqshell/storage/bucket/list.go b/iqshell/storage/bucket/list.go index 1c30c1fe..cf25f894 100644 --- a/iqshell/storage/bucket/list.go +++ b/iqshell/storage/bucket/list.go @@ -26,7 +26,7 @@ type ListApiInfo struct { StartTime time.Time // list item 的 put time 区间的开始时间 【闭区间】 EndTime time.Time // list item 的 put time 区间的终止时间 【闭区间】 Suffixes []string // list item 必须包含后缀 - FileTypes []int // list item 存储类型,多个使用逗号隔开, 0:普通存储 1:低频存储 2:归档存储 3:深度归档存储 + FileTypes []int // list item 存储类型,多个使用逗号隔开, 0:普通存储 1:低频存储 2:归档存储 3:深度归档存储 4:归档直读存储 MimeTypes []string // list item Mimetype类型,多个使用逗号隔开 MinFileSize int64 // 文件最小值,单位: B MaxFileSize int64 // 文件最大值,单位: B diff --git a/iqshell/storage/bucket/operations/list.go b/iqshell/storage/bucket/operations/list.go index a31e910f..50e1c891 100644 --- a/iqshell/storage/bucket/operations/list.go +++ b/iqshell/storage/bucket/operations/list.go @@ -25,7 +25,7 @@ type ListInfo struct { StartDate string // list item 的 put time 区间的开始时间 【闭区间】 【可选】 EndDate string // list item 的 put time 区间的终止时间 【闭区间】 【可选】 Suffixes string // list item 必须包含后缀 【可选】 - FileTypes string // list item 存储类型,多个使用逗号隔开, 0:普通存储 1:低频存储 2:归档存储 3:深度归档存储 【可选】 + FileTypes string // list item 存储类型,多个使用逗号隔开, 0:普通存储 1:低频存储 2:归档存储 3:深度归档存储 4:归档直读存储【可选】 MimeTypes string // list item Mimetype类型,多个使用逗号隔开 【可选】 MinFileSize string // 文件最小值,单位: B 【可选】 MaxFileSize string // 文件最大值,单位: B 【可选】 diff --git a/iqshell/storage/object/lifecycle.go b/iqshell/storage/object/lifecycle.go index 8dc98a2d..4aa52101 100644 --- a/iqshell/storage/object/lifecycle.go +++ b/iqshell/storage/object/lifecycle.go @@ -2,7 +2,9 @@ package object import ( "fmt" + "github.com/qiniu/go-sdk/v7/storage" + "github.com/qiniu/qshell/v2/iqshell/common/alert" "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/storage/object/batch" @@ -14,6 +16,7 @@ type ChangeLifecycleApiInfo struct { Bucket string `json:"bucket"` Key string `json:"key"` ToIAAfterDays int `json:"to_ia_after_days"` // 转换到 低频存储类型,设置为 -1 表示取消 + ToArchiveIRAfterDays int `json:"to_archive_ir_after_days"` // 转换到 归档直读存储类型, 设置为 -1 表示取消 ToArchiveAfterDays int `json:"to_archive_after_days"` // 转换到 归档存储类型, 设置为 -1 表示取消 ToDeepArchiveAfterDays int `json:"to_deep_archive_after_days"` // 转换到 深度归档存储类型, 设置为 -1 表示取消 DeleteAfterDays int `json:"delete_after_days"` // 过期删除,删除后不可恢复,设置为 -1 表示取消 @@ -32,6 +35,9 @@ func (l *ChangeLifecycleApiInfo) ToOperation() (string, *data.CodeError) { if l.ToIAAfterDays != 0 { lifecycleSetting += fmt.Sprintf("/toIAAfterDays/%d", l.ToIAAfterDays) } + if l.ToArchiveIRAfterDays != 0 { + lifecycleSetting += fmt.Sprintf("/toArchiveIRAfterDays/%d", l.ToArchiveIRAfterDays) + } if l.ToArchiveAfterDays != 0 { lifecycleSetting += fmt.Sprintf("/toArchiveAfterDays/%d", l.ToArchiveAfterDays) } diff --git a/iqshell/storage/object/operations/lifecycle.go b/iqshell/storage/object/operations/lifecycle.go index 319a8780..fc48ddc8 100644 --- a/iqshell/storage/object/operations/lifecycle.go +++ b/iqshell/storage/object/operations/lifecycle.go @@ -2,6 +2,8 @@ package operations import ( "fmt" + "path/filepath" + "github.com/qiniu/qshell/v2/iqshell" "github.com/qiniu/qshell/v2/iqshell/common/alert" "github.com/qiniu/qshell/v2/iqshell/common/data" @@ -12,7 +14,6 @@ import ( "github.com/qiniu/qshell/v2/iqshell/storage/bucket" "github.com/qiniu/qshell/v2/iqshell/storage/object" "github.com/qiniu/qshell/v2/iqshell/storage/object/batch" - "path/filepath" ) type ChangeLifecycleInfo object.ChangeLifecycleApiInfo @@ -26,6 +27,7 @@ func (info *ChangeLifecycleInfo) Check() *data.CodeError { } if info.ToIAAfterDays == 0 && + info.ToArchiveIRAfterDays == 0 && info.ToArchiveAfterDays == 0 && info.ToDeepArchiveAfterDays == 0 && info.DeleteAfterDays == 0 { @@ -51,10 +53,10 @@ func ChangeLifecycle(cfg *iqshell.Config, info *ChangeLifecycleInfo) { } if result.IsSuccess() { - lifecycleValues := []int{info.ToIAAfterDays, info.ToArchiveAfterDays, + lifecycleValues := []int{info.ToIAAfterDays, info.ToArchiveIRAfterDays, info.ToArchiveAfterDays, info.ToDeepArchiveAfterDays, info.DeleteAfterDays} - lifecycleDescs := []string{"to IA storage", "to archive storage", - "to deep archive storage", "delete"} + lifecycleDescs := []string{"to IA storage", "to ARCHIVE_IR storage", "to ARCHIVE storage", + "to DEEP_ARCHIVE storage", "delete"} log.InfoF("Change lifecycle Success, [%s:%s]", info.Bucket, info.Key) for i := 0; i < len(lifecycleValues); i++ { lifecycleValue := lifecycleValues[i] @@ -79,6 +81,7 @@ type BatchChangeLifecycleInfo struct { BatchInfo batch.Info // Bucket string // ToIAAfterDays int // 转换到 低频存储类型,设置为 -1 表示取消 + ToArchiveIRAfterDays int // 转换到 归档直读存储类型, 设置为 -1 表示取消 ToArchiveAfterDays int // 转换到 归档存储类型, 设置为 -1 表示取消 ToDeepArchiveAfterDays int // 转换到 深度归档存储类型, 设置为 -1 表示取消 DeleteAfterDays int // 过期删除,删除后不可恢复,设置为 -1 表示取消 @@ -94,6 +97,7 @@ func (info *BatchChangeLifecycleInfo) Check() *data.CodeError { } if info.ToIAAfterDays == 0 && + info.ToArchiveIRAfterDays == 0 && info.ToArchiveAfterDays == 0 && info.ToDeepArchiveAfterDays == 0 && info.DeleteAfterDays == 0 { @@ -141,6 +145,7 @@ func BatchChangeLifecycle(cfg *iqshell.Config, info BatchChangeLifecycleInfo) { Bucket: info.Bucket, Key: listObject.Key, ToIAAfterDays: info.ToIAAfterDays, + ToArchiveIRAfterDays: info.ToArchiveIRAfterDays, ToArchiveAfterDays: info.ToArchiveAfterDays, ToDeepArchiveAfterDays: info.ToDeepArchiveAfterDays, DeleteAfterDays: info.DeleteAfterDays, @@ -155,8 +160,8 @@ func BatchChangeLifecycle(cfg *iqshell.Config, info BatchChangeLifecycleInfo) { } in := (*ChangeLifecycleInfo)(apiInfo) if result.IsSuccess() { - log.InfoF("Change lifecycle Success, [%s:%s] => '%d:%d:%d:%d'", in.Bucket, in.Key, - in.ToIAAfterDays, in.ToArchiveAfterDays, in.ToDeepArchiveAfterDays, in.DeleteAfterDays) + log.InfoF("Change lifecycle Success, [%s:%s] => '%d:%d:%d:%d:%d'", in.Bucket, in.Key, + in.ToIAAfterDays, in.ToArchiveIRAfterDays, in.ToArchiveAfterDays, in.ToDeepArchiveAfterDays, in.DeleteAfterDays) } else { data.SetCmdStatusError() log.ErrorF("Change lifecycle Failed, [%s:%s], Code: %d, Error: %s", in.Bucket, in.Key, result.Code, result.Error) diff --git a/iqshell/storage/object/operations/status.go b/iqshell/storage/object/operations/status.go index 53cdd8ae..fb2f13d4 100644 --- a/iqshell/storage/object/operations/status.go +++ b/iqshell/storage/object/operations/status.go @@ -166,6 +166,7 @@ func getResultInfo(bucket, key string, status object.StatusResult) string { } lifecycleFieldAdder("Expiration", status.Expiration) lifecycleFieldAdder("TransitionToIA", status.TransitionToIA) + lifecycleFieldAdder("TransitionToArchiveIR", status.TransitionToArchiveIR) lifecycleFieldAdder("TransitionToArchive", status.TransitionToARCHIVE) lifecycleFieldAdder("TransitionToDeepArchive", status.TransitionToDeepArchive) @@ -174,7 +175,7 @@ func getResultInfo(bucket, key string, status object.StatusResult) string { return statInfo } -var objectTypes = []string{"标准存储", "低频存储", "归档存储", "深度归档存储"} +var objectTypes = []string{"标准存储", "低频存储", "归档存储", "深度归档存储", "归档直读存储"} func getFileTypeDescription(fileTypes int) string { typeString := "未知类型" diff --git a/iqshell/storage/object/status.go b/iqshell/storage/object/status.go index afb45ddc..7e579583 100644 --- a/iqshell/storage/object/status.go +++ b/iqshell/storage/object/status.go @@ -3,8 +3,10 @@ package object import ( "context" "fmt" + "github.com/qiniu/go-sdk/v7/auth" "github.com/qiniu/go-sdk/v7/storage" + "github.com/qiniu/qshell/v2/iqshell/common/alert" "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/storage/bucket" @@ -47,6 +49,8 @@ type StatusResult struct { TransitionToIA int64 `json:"transitionToIA"` // 文件生命周期中转为归档存储的日期,int64 类型,Unix 时间戳格式 TransitionToARCHIVE int64 `json:"transitionToARCHIVE"` + // 文件生命周期中转为归档存储的日期,int64 类型,Unix 时间戳格式 + TransitionToArchiveIR int64 `json:"transitionToArchiveIR"` // 文件生命周期中转为深度归档存储的日期,int64 类型,Unix 时间戳格式 TransitionToDeepArchive int64 `json:"transitionToDeepArchive"` } From 74c586cab77f7fbfef58e49bf8f174b623803c5e Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 29 Nov 2023 16:01:35 +0800 Subject: [PATCH 06/13] changelifecycle --- docs/chlifecycle.md | 10 +++++----- go.mod | 2 +- go.sum | 4 ++-- iqshell/storage/object/operations/type_change.go | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/chlifecycle.md b/docs/chlifecycle.md index a53a5234..6446f7b8 100644 --- a/docs/chlifecycle.md +++ b/docs/chlifecycle.md @@ -64,10 +64,10 @@ FileHash: lozgLP_MAdAKZkPCXGvfd0LIDSUI Fsize: 5444314 -> 5.19MB PutTime: 16768889367943931 -> 2023-02-20 18:28:56.7943931 +0800 CST MimeType: text/plain -Expiration: 1710518400 -> 2024-03-16 00:00:00 +0800 CST -TransitionToIA: 1681574400 -> 2023-04-16 00:00:00 +0800 CST -TransitionToArchiveIR: 1684166400 -> 2023-05-16 00:00:00 +0800 CST -TransitionToArchive: 1684166400 -> 2023-06-16 00:00:00 +0800 CST -TransitionToDeepArchive: 1694534400 -> 2023-09-13 00:00:00 +0800 CST +Expiration: 1732809600 -> 2024-11-29 00:00:00 +0800 CST +TransitionToIA: 1703865600 -> 2023-12-30 00:00:00 +0800 CST +TransitionToArchiveIR: 1706457600 -> 2024-01-29 00:00:00 +0800 CST +TransitionToArchive: 1711641600 -> 2024-03-29 00:00:00 +0800 CST +TransitionToDeepArchive: 1716825600 -> 2024-05-28 00:00:00 +0800 CST FileType: 0 -> 标准存储 ``` diff --git a/go.mod b/go.mod index e9a78f97..b2c57c56 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ require ( github.com/astaxie/beego v1.12.3 github.com/aws/aws-sdk-go v1.37.31 github.com/mitchellh/go-homedir v1.1.0 - github.com/qiniu/go-sdk/v7 v7.15.0 + github.com/qiniu/go-sdk/v7 v7.18.1 github.com/schollz/progressbar/v3 v3.8.6 github.com/spf13/cast v1.3.1 github.com/spf13/cobra v1.1.3 diff --git a/go.sum b/go.sum index 3845951a..c04e2e75 100644 --- a/go.sum +++ b/go.sum @@ -263,8 +263,8 @@ github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsT github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= github.com/qiniu/dyn v1.3.0/go.mod h1:E8oERcm8TtwJiZvkQPbcAh0RL8jO1G0VXJMW3FAWdkk= -github.com/qiniu/go-sdk/v7 v7.15.0 h1:vkxZZHM2Ed0qHeIx7NF3unXav+guaVIXlEsCCkpQAww= -github.com/qiniu/go-sdk/v7 v7.15.0/go.mod h1:nqoYCNo53ZlGA521RvRethvxUDvXKt4gtYXOwye868w= +github.com/qiniu/go-sdk/v7 v7.18.1 h1:DHWqWwVy1m4zMNDS4OzEo2tKPCyJCt0RbsqEFk9dzZ4= +github.com/qiniu/go-sdk/v7 v7.18.1/go.mod h1:nqoYCNo53ZlGA521RvRethvxUDvXKt4gtYXOwye868w= github.com/qiniu/x v1.10.5/go.mod h1:03Ni9tj+N2h2aKnAz+6N0Xfl8FwMEDRC2PAlxekASDs= github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= diff --git a/iqshell/storage/object/operations/type_change.go b/iqshell/storage/object/operations/type_change.go index a1de5166..3b03bee3 100644 --- a/iqshell/storage/object/operations/type_change.go +++ b/iqshell/storage/object/operations/type_change.go @@ -45,8 +45,8 @@ func (info *ChangeTypeInfo) getTypeOfInt() (int, *data.CodeError) { return -1, data.NewEmptyError().AppendDesc("Parse type error:" + err.Error()) } - if ret < 0 || ret > 3 { - return -1, data.NewEmptyError().AppendDesc("type must be one of 0, 1, 2, 3") + if ret < 0 || ret > 4 { + return -1, data.NewEmptyError().AppendDesc("type must be one of 0, 1, 2, 3, 4") } return ret, nil } From 75b049f802631b95b573fc3e6453752cc695e57d Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 27 Dec 2023 10:42:38 +0800 Subject: [PATCH 07/13] do not check file storage type --- cmd_test/version_test.go | 3 ++- iqshell/storage/object/operations/type_change.go | 3 --- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/cmd_test/version_test.go b/cmd_test/version_test.go index dbccd228..ef0ac37a 100644 --- a/cmd_test/version_test.go +++ b/cmd_test/version_test.go @@ -3,9 +3,10 @@ package cmd import ( - "github.com/qiniu/qshell/v2/cmd_test/test" "strings" "testing" + + "github.com/qiniu/qshell/v2/cmd_test/test" ) func TestVersion(t *testing.T) { diff --git a/iqshell/storage/object/operations/type_change.go b/iqshell/storage/object/operations/type_change.go index 3b03bee3..e61679ad 100644 --- a/iqshell/storage/object/operations/type_change.go +++ b/iqshell/storage/object/operations/type_change.go @@ -45,9 +45,6 @@ func (info *ChangeTypeInfo) getTypeOfInt() (int, *data.CodeError) { return -1, data.NewEmptyError().AppendDesc("Parse type error:" + err.Error()) } - if ret < 0 || ret > 4 { - return -1, data.NewEmptyError().AppendDesc("type must be one of 0, 1, 2, 3, 4") - } return ret, nil } From dd608c03471722c5407845053710c782476656bf Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 27 Dec 2023 11:02:19 +0800 Subject: [PATCH 08/13] handle buckets list: shared --- iqshell/storage/servers/all_buckets.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/iqshell/storage/servers/all_buckets.go b/iqshell/storage/servers/all_buckets.go index c422bdcc..23f6692d 100644 --- a/iqshell/storage/servers/all_buckets.go +++ b/iqshell/storage/servers/all_buckets.go @@ -2,7 +2,9 @@ package servers import ( "fmt" + "github.com/qiniu/go-sdk/v7/auth" + "github.com/qiniu/qshell/v2/iqshell/common/config" "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/common/utils" @@ -48,7 +50,6 @@ func BucketInfoDetailDescriptionStringFormat() string { } type ListApiInfo struct { - Shared bool Region string } @@ -65,9 +66,9 @@ func allBuckets(cfg *config.Config, info ListApiInfo) ([]BucketInfo, *data.CodeE } ucHost := cfg.Hosts.GetOneUc() - reqURL := fmt.Sprintf("%s/v3/buckets?shared=%v", utils.Endpoint(cfg.UseHttps.Value(), ucHost), info.Shared) + reqURL := fmt.Sprintf("%s/v3/buckets", utils.Endpoint(cfg.UseHttps.Value(), ucHost)) if len(info.Region) > 0 { - reqURL = fmt.Sprintf("%s®ion=%s", reqURL, info.Region) + reqURL = fmt.Sprintf("%s?region=%s", reqURL, info.Region) } var buckets []BucketInfo e := bucketManager.Client.CredentialedCall(workspace.GetContext(), bucketManager.Mac, auth.TokenQiniu, &buckets, "POST", reqURL, nil) From cc60f01757fb9438399bad859ed06be296353071 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Wed, 27 Dec 2023 17:44:26 +0800 Subject: [PATCH 09/13] more upload policy --- cmd/upload.go | 93 ++++++++++++++++--- docs/fput.md | 18 ++++ docs/qupload.md | 19 ++++ docs/qupload2.md | 25 +++++ docs/rput.md | 21 ++++- docs/sync.md | 20 ++++ .../storage/object/upload/operations/batch.go | 39 ++++++-- .../object/upload/operations/batch_data.go | 86 +++++++++++++---- .../storage/object/upload/operations/sync.go | 7 +- .../object/upload/operations/upload.go | 27 +++++- 10 files changed, 305 insertions(+), 50 deletions(-) diff --git a/cmd/upload.go b/cmd/upload.go index aa5f68db..11eed348 100644 --- a/cmd/upload.go +++ b/cmd/upload.go @@ -1,7 +1,6 @@ package cmd import ( - "github.com/qiniu/go-sdk/v7/storage" "github.com/spf13/cobra" "github.com/qiniu/qshell/v2/docs" @@ -48,9 +47,7 @@ var upload2CmdBuilder = func(cfg *iqshell.Config) *cobra.Command { LogRotate = 7 ) info := operations.BatchUpload2Info{ - UploadConfig: operations.UploadConfig{ - Policy: &storage.PutPolicy{}, - }, + UploadConfig: operations.UploadConfig{}, } cmd := &cobra.Command{ Use: "qupload2", @@ -102,19 +99,34 @@ var upload2CmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().IntVarP(&info.FileType, "storage", "", 0, "set storage type of file, same to --file-type") _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage - cmd.Flags().StringVarP(&info.Policy.CallbackURL, "callback-urls", "l", "", "upload callback urls, separated by comma") - cmd.Flags().StringVarP(&info.Policy.CallbackHost, "callback-host", "T", "", "upload callback host") //cmd.Flags().StringVar(&cfg.CmdCfg.Up.BindUpIp, "bind-up-ip", "", "upload host ip to bind") //cmd.Flags().StringVar(&cfg.CmdCfg.Up.BindRsIp, "bind-rs-ip", "", "rs host ip to bind") //cmd.Flags().StringVar(&cfg.CmdCfg.Up.BindNicIp, "bind-nic-ip", "", "local network interface card to bind") + cmd.Flags().StringVarP(&info.EndUser, "end-user", "", "", "Owner identification") + cmd.Flags().StringVarP(&info.CallbackURL, "callback-urls", "l", "", "upload callback urls, separated by comma") + cmd.Flags().StringVarP(&info.CallbackHost, "callback-host", "T", "", "upload callback host") + cmd.Flags().StringVarP(&info.CallbackBody, "callback-body", "", "", "upload callback body") + cmd.Flags().StringVarP(&info.CallbackBodyType, "callback-body-type", "", "", "upload callback body type") + cmd.Flags().StringVarP(&info.PersistentOps, "persistent-ops", "", "", "List of pre-transfer persistence processing instructions that are triggered after successful resource upload. This parameter is not supported when fileType=2 or 3 (upload archive storage or deep archive storage files). Supports magic variables and custom variables. Each directive is an API specification string, and multiple directives are separated by ;.") + cmd.Flags().StringVarP(&info.PersistentNotifyURL, "persistent-notify-url", "", "", "URL to receive notification of persistence processing results. It must be a valid URL that can make POST requests normally on the public Internet and respond successfully. The content obtained by this URL is consistent with the processing result of the persistence processing status query. To send a POST request whose body format is application/json, you need to read the body of the request in the form of a read stream to obtain it.") + cmd.Flags().StringVarP(&info.PersistentPipeline, "persistent-pipeline", "", "", "Transcoding queue name. After the resource is successfully uploaded, an independent queue is designated for transcoding when transcoding is triggered. If it is empty, it means that the public queue is used, and the processing speed is slower. It is recommended to use a dedicated queue.") + cmd.Flags().IntVarP(&info.DetectMime, "detect-mime", "", 0, `Turn on the MimeType detection function and perform detection according to the following rules; if the correct value cannot be detected, application/octet-stream will be used by default. +If set to a value of 1, the file MimeType information passed by the uploader will be ignored, and the MimeType value will be detected in the following order: + 1. Detection content; + 2. Check the file extension; + 3. Check the Key extension. +The default value is set to 0.If the uploader specifies MimeType (except application/octet-stream), this value will be used directly. Otherwise, the MimeType value will be detected in the following order: + 1. Check the file extension; + 2. Check the Key extension; + 3. Detect content. +Set to a value of -1 and use this value regardless of what value is specified on the uploader.`) + cmd.Flags().Uint64VarP(&info.TrafficLimit, "traffic-limit", "", 0, "Upload request single link speed limit to control client bandwidth usage. The speed limit value range is 819200 ~ 838860800, and the unit is bit/s.") return cmd } var syncCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { - info := operations.SyncInfo{ - Policy: &storage.PutPolicy{}, - } + info := operations.SyncInfo{} cmd := &cobra.Command{ Use: "sync [-k ]", Short: "Sync big file to qiniu bucket", @@ -141,13 +153,30 @@ var syncCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().BoolVarP(&info.Overwrite, "overwrite", "", false, "overwrite the file of same key in bucket") + cmd.Flags().StringVarP(&info.Policy.EndUser, "end-user", "", "", "Owner identification") + cmd.Flags().StringVarP(&info.Policy.CallbackURL, "callback-urls", "l", "", "upload callback urls, separated by comma") + cmd.Flags().StringVarP(&info.Policy.CallbackHost, "callback-host", "T", "", "upload callback host") + cmd.Flags().StringVarP(&info.Policy.CallbackBody, "callback-body", "", "", "upload callback body") + cmd.Flags().StringVarP(&info.Policy.CallbackBodyType, "callback-body-type", "", "", "upload callback body type") + cmd.Flags().StringVarP(&info.Policy.PersistentOps, "persistent-ops", "", "", "List of pre-transfer persistence processing instructions that are triggered after successful resource upload. This parameter is not supported when fileType=2 or 3 (upload archive storage or deep archive storage files). Supports magic variables and custom variables. Each directive is an API specification string, and multiple directives are separated by ;.") + cmd.Flags().StringVarP(&info.Policy.PersistentNotifyURL, "persistent-notify-url", "", "", "URL to receive notification of persistence processing results. It must be a valid URL that can make POST requests normally on the public Internet and respond successfully. The content obtained by this URL is consistent with the processing result of the persistence processing status query. To send a POST request whose body format is application/json, you need to read the body of the request in the form of a read stream to obtain it.") + cmd.Flags().StringVarP(&info.Policy.PersistentPipeline, "persistent-pipeline", "", "", "Transcoding queue name. After the resource is successfully uploaded, an independent queue is designated for transcoding when transcoding is triggered. If it is empty, it means that the public queue is used, and the processing speed is slower. It is recommended to use a dedicated queue.") + cmd.Flags().IntVarP(&info.Policy.DetectMime, "detect-mime", "", 0, `Turn on the MimeType detection function and perform detection according to the following rules; if the correct value cannot be detected, application/octet-stream will be used by default. +If set to a value of 1, the file MimeType information passed by the uploader will be ignored, and the MimeType value will be detected in the following order: + 1. Detection content; + 2. Check the file extension; + 3. Check the Key extension. +The default value is set to 0.If the uploader specifies MimeType (except application/octet-stream), this value will be used directly. Otherwise, the MimeType value will be detected in the following order: + 1. Check the file extension; + 2. Check the Key extension; + 3. Detect content. +Set to a value of -1 and use this value regardless of what value is specified on the uploader.`) + cmd.Flags().Uint64VarP(&info.Policy.TrafficLimit, "traffic-limit", "", 0, "Upload request single link speed limit to control client bandwidth usage. The speed limit value range is 819200 ~ 838860800, and the unit is bit/s.") return cmd } var formUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { - info := operations.UploadInfo{ - Policy: &storage.PutPolicy{}, - } + info := operations.UploadInfo{} cmd := &cobra.Command{ Use: "fput ", Short: "Form upload a local file", @@ -174,15 +203,31 @@ var formUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { _ = cmd.Flags().MarkDeprecated("storage", "use --file-type instead") // 废弃 storage cmd.Flags().StringVarP(&info.UpHost, "up-host", "u", "", "uphost") + + cmd.Flags().StringVarP(&info.Policy.EndUser, "end-user", "", "", "Owner identification") cmd.Flags().StringVarP(&info.Policy.CallbackURL, "callback-urls", "l", "", "upload callback urls, separated by comma") cmd.Flags().StringVarP(&info.Policy.CallbackHost, "callback-host", "T", "", "upload callback host") + cmd.Flags().StringVarP(&info.Policy.CallbackBody, "callback-body", "", "", "upload callback body") + cmd.Flags().StringVarP(&info.Policy.CallbackBodyType, "callback-body-type", "", "", "upload callback body type") + cmd.Flags().StringVarP(&info.Policy.PersistentOps, "persistent-ops", "", "", "List of pre-transfer persistence processing instructions that are triggered after successful resource upload. This parameter is not supported when fileType=2 or 3 (upload archive storage or deep archive storage files). Supports magic variables and custom variables. Each directive is an API specification string, and multiple directives are separated by ;.") + cmd.Flags().StringVarP(&info.Policy.PersistentNotifyURL, "persistent-notify-url", "", "", "URL to receive notification of persistence processing results. It must be a valid URL that can make POST requests normally on the public Internet and respond successfully. The content obtained by this URL is consistent with the processing result of the persistence processing status query. To send a POST request whose body format is application/json, you need to read the body of the request in the form of a read stream to obtain it.") + cmd.Flags().StringVarP(&info.Policy.PersistentPipeline, "persistent-pipeline", "", "", "Transcoding queue name. After the resource is successfully uploaded, an independent queue is designated for transcoding when transcoding is triggered. If it is empty, it means that the public queue is used, and the processing speed is slower. It is recommended to use a dedicated queue.") + cmd.Flags().IntVarP(&info.Policy.DetectMime, "detect-mime", "", 0, `Turn on the MimeType detection function and perform detection according to the following rules; if the correct value cannot be detected, application/octet-stream will be used by default. +If set to a value of 1, the file MimeType information passed by the uploader will be ignored, and the MimeType value will be detected in the following order: + 1. Detection content; + 2. Check the file extension; + 3. Check the Key extension. +The default value is set to 0.If the uploader specifies MimeType (except application/octet-stream), this value will be used directly. Otherwise, the MimeType value will be detected in the following order: + 1. Check the file extension; + 2. Check the Key extension; + 3. Detect content. +Set to a value of -1 and use this value regardless of what value is specified on the uploader.`) + cmd.Flags().Uint64VarP(&info.Policy.TrafficLimit, "traffic-limit", "", 0, "Upload request single link speed limit to control client bandwidth usage. The speed limit value range is 819200 ~ 838860800, and the unit is bit/s.") return cmd } var resumeUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { - info := operations.UploadInfo{ - Policy: &storage.PutPolicy{}, - } + info := operations.UploadInfo{} cmd := &cobra.Command{ Use: "rput ", Short: "Resumable upload a local file", @@ -217,8 +262,26 @@ var resumeUploadCmdBuilder = func(cfg *iqshell.Config) *cobra.Command { cmd.Flags().IntVarP(&info.ResumeWorkerCount, "worker", "c", 3, "worker count") cmd.Flags().StringVarP(&info.UpHost, "up-host", "u", "", "uphost") + + cmd.Flags().StringVarP(&info.Policy.EndUser, "end-user", "", "", "Owner identification") cmd.Flags().StringVarP(&info.Policy.CallbackURL, "callback-urls", "l", "", "upload callback urls, separated by comma") cmd.Flags().StringVarP(&info.Policy.CallbackHost, "callback-host", "T", "", "upload callback host") + cmd.Flags().StringVarP(&info.Policy.CallbackBody, "callback-body", "", "", "upload callback body") + cmd.Flags().StringVarP(&info.Policy.CallbackBodyType, "callback-body-type", "", "", "upload callback body type") + cmd.Flags().StringVarP(&info.Policy.PersistentOps, "persistent-ops", "", "", "List of pre-transfer persistence processing instructions that are triggered after successful resource upload. This parameter is not supported when fileType=2 or 3 (upload archive storage or deep archive storage files). Supports magic variables and custom variables. Each directive is an API specification string, and multiple directives are separated by ;.") + cmd.Flags().StringVarP(&info.Policy.PersistentNotifyURL, "persistent-notify-url", "", "", "URL to receive notification of persistence processing results. It must be a valid URL that can make POST requests normally on the public Internet and respond successfully. The content obtained by this URL is consistent with the processing result of the persistence processing status query. To send a POST request whose body format is application/json, you need to read the body of the request in the form of a read stream to obtain it.") + cmd.Flags().StringVarP(&info.Policy.PersistentPipeline, "persistent-pipeline", "", "", "Transcoding queue name. After the resource is successfully uploaded, an independent queue is designated for transcoding when transcoding is triggered. If it is empty, it means that the public queue is used, and the processing speed is slower. It is recommended to use a dedicated queue.") + cmd.Flags().IntVarP(&info.Policy.DetectMime, "detect-mime", "", 0, `Turn on the MimeType detection function and perform detection according to the following rules; if the correct value cannot be detected, application/octet-stream will be used by default. +If set to a value of 1, the file MimeType information passed by the uploader will be ignored, and the MimeType value will be detected in the following order: + 1. Detection content; + 2. Check the file extension; + 3. Check the Key extension. +The default value is set to 0. If the uploader specifies MimeType (except application/octet-stream), this value will be used directly. Otherwise, the MimeType value will be detected in the following order: + 1. Check the file extension; + 2. Check the Key extension; + 3. Detect content. +Set to a value of -1 and use this value regardless of what value is specified on the uploader.`) + cmd.Flags().Uint64VarP(&info.Policy.TrafficLimit, "traffic-limit", "", 0, "Upload request single link speed limit to control client bandwidth usage. The speed limit value range is 819200 ~ 838860800, and the unit is bit/s.") return cmd } diff --git a/docs/fput.md b/docs/fput.md index 502be0c7..43701142 100644 --- a/docs/fput.md +++ b/docs/fput.md @@ -35,6 +35,24 @@ $ qshell fput --doc - -u/--up-host: 指定上传域名。 【可选】 - -l/--callback-urls:上传回调地址, 可以指定多个地址,以逗号分隔。 【可选】 - -T/--callback-host:上传回调的 HOST, 必须和 CallbackUrls 一起指定。 【可选】 +- --callback-body:上传成功后,七牛云向业务服务器发送 Content-Type: application/x-www-form-urlencoded 的 POST 请求。业务服务器可以通过直接读取请求的 query 来获得该字段,支持魔法变量和自定义变量。callbackBody 要求是合法的 url query string。例如key=$(key)&hash=$(etag)&w=$(imageInfo.width)&h=$(imageInfo.height)。如果callbackBodyType指定为application/json,则callbackBody应为json格式,例如:{“key”:"$(key)",“hash”:"$(etag)",“w”:"$(imageInfo.width)",“h”:"$(imageInfo.height)"}。【可选】 +- --callback-body-type:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --end-user:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 +- --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 +- --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 + # 示例 1 上传本地文件 `/Users/jemy/Documents/qiniu.jpg` 到空间 `if-pbl` 里面。 diff --git a/docs/qupload.md b/docs/qupload.md index 98e959da..359502df 100644 --- a/docs/qupload.md +++ b/docs/qupload.md @@ -90,6 +90,25 @@ $ qshell qupload --doc - 未通过 `-L` 指定工作目录时为 `用户目录/.qshell/users/$CurrentUserName/qdownload/$jobId` - 注意 `jobId` 是根据上传任务动态生成;具体方式为 MD5("$SrcDir:$Bucket:$FileList"); `CurrentUserName` 当前用户的名称 - worker_count:分片上传中单个文件并发上传的分片数;默认为 3。【可选】 +- callback_urls:上传回调地址,可以指定多个地址,以逗号分开。【可选】 +- callback_host:上传回调HOST, 必须和 CallbackUrls 一起指定。 【可选】 +- callback_body:上传成功后,七牛云向业务服务器发送 Content-Type: application/x-www-form-urlencoded 的 POST 请求。业务服务器可以通过直接读取请求的 query 来获得该字段,支持魔法变量和自定义变量。callbackBody 要求是合法的 url query string。例如key=$(key)&hash=$(etag)&w=$(imageInfo.width)&h=$(imageInfo.height)。如果callbackBodyType指定为application/json,则callbackBody应为json格式,例如:{“key”:"$(key)",“hash”:"$(etag)",“w”:"$(imageInfo.width)",“h”:"$(imageInfo.height)"}。【可选】 +- callback_body_type:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- end_user:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- persistent_ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 +- persistent_notify_url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 +- persistent_pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 +- detect_mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- traffic_limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 对于那么多的参数,我们可以分为几类来解释: diff --git a/docs/qupload2.md b/docs/qupload2.md index 1553da37..8a953914 100644 --- a/docs/qupload2.md +++ b/docs/qupload2.md @@ -21,11 +21,23 @@ Usage: Flags: --bucket string bucket + --callback-body string upload callback body -T, --callback-host string upload callback host -l, --callback-urls string upload callback urls, separated by comma --check-exists check file key whether in bucket before upload --check-hash check hash --check-size check file size + --detect-mime int Turn on the MimeType detection function and perform detection according to the following rules; if the correct value cannot be detected, application/octet-stream will be used by default. + If set to a value of 1, the file MimeType information passed by the uploader will be ignored, and the MimeType value will be detected in the following order: + 1. Detection content; + 2. Check the file extension; + 3. Check the Key extension. + The default value is set to 0. If the uploader specifies MimeType (except application/octet-stream), this value will be used directly. Otherwise, the MimeType value will be detected in the following order: + 1. Check the file extension; + 2. Check the Key extension; + 3. Detect content. + Set to a value of -1 and use this value regardless of what value is specified on the uploader. + --end-user string Owner identification -e, --failure-list string upload failure file list --file-list string file list to upload --file-type int set storage type of file, 0:STANDARD storage, 1:IA storage, 2:ARCHIVE storage, 3:DEEP_ARCHIVE storage, 4:ARCHIVE_IR storage @@ -37,6 +49,9 @@ Flags: --log-rotate int log rotate days (default 7) --overwrite overwrite the file of same key in bucket -w, --overwrite-list string upload success (overwrite) file list + --persistent-notify-url string URL to receive notification of persistence processing results. It must be a valid URL that can make POST requests normally on the public Internet and respond successfully. The content obtained by this URL is consistent with the processing result of the persistence processing status query. To send a POST request whose body format is application/json, you need to read the body of the request in the form of a read stream to obtain it. + --persistent-ops string List of pre-transfer persistence processing instructions that are triggered after successful resource upload. This parameter is not supported when fileType=2 or 3 (upload archive storage or deep archive storage files). Supports magic variables and custom variables. Each directive is an API specification string, and multiple directives are separated by ;. + --persistent-pipeline string Transcoding queue name. After the resource is successfully uploaded, an independent queue is designated for transcoding when transcoding is triggered. If it is empty, it means that the public queue is used, and the processing speed is slower. It is recommended to use a dedicated queue. --put-threshold int chunk upload threshold, unit: B (default 8388608) --record-root string record root dir, and will save record info to the dir(db and log), default /.qshell --rescan-local rescan local dir to upload newly add files @@ -50,6 +65,16 @@ Flags: --src-dir string src dir to upload -s, --success-list string upload success file list --thread-count int multiple thread count (default 1) + --traffic-limit uint Upload request single link speed limit to control client bandwidth usage. The speed limit value range is 819200 ~ 838860800, and the unit is bit/s. --up-host string upload host --worker-count int the number of concurrently uploaded parts of a single file in resumable upload (default 3) + +Global Flags: + --colorful console colorful mode + -C, --config string set config file (default is $HOME/.qshell.json) + -D, --ddebug deep debug mode + -d, --debug debug mode + --doc document of command + -L, --local use current directory qshell workspace (default is $HOME/.qshell) + --silence silence mode, The console only outputs warnings、errors and some important information ``` diff --git a/docs/rput.md b/docs/rput.md index 7c2a8dcc..1a1513e9 100644 --- a/docs/rput.md +++ b/docs/rput.md @@ -34,11 +34,28 @@ $ qshell rput --doc - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 - -t/--mimetype:指定文件的 MimeType 。【可选】 - --file-type:文件存储类型;0: 标准存储, 1: 低频存储, 2: 归档存储, 3: 深度归档存储, 4: 归档直读存储;默认为`0`(标准存储)。 【可选】 -- -l/--callback-urls:上传回调地址,可以指定多个地址,以逗号分开。【可选】 -- -T/--callbackHost:上传回调HOST, 必须和 CallbackUrls 一起指定。 【可选】 - --resumable-api-v2:使用分片上传 API V2 进行上传,默认为 `false`, 使用 V1 上传。【可选】 - --resumable-api-v2-part-size:使用分片上传 API V2 进行上传时的分片大小,默认为 4M 。【可选】 - --sequential-read-file: 文件读为顺序读,不涉及跳读;开启后,上传中的分片数据会被加载至内存。此选项可能会增加挂载网络文件系统的文件上传速度。默认是:false。【可选】 +- -l/--callback-urls:上传回调地址,可以指定多个地址,以逗号分开。【可选】 +- -T/--callback-host:上传回调HOST, 必须和 CallbackUrls 一起指定。 【可选】 +- --callback-body:上传成功后,七牛云向业务服务器发送 Content-Type: application/x-www-form-urlencoded 的 POST 请求。业务服务器可以通过直接读取请求的 query 来获得该字段,支持魔法变量和自定义变量。callbackBody 要求是合法的 url query string。例如key=$(key)&hash=$(etag)&w=$(imageInfo.width)&h=$(imageInfo.height)。如果callbackBodyType指定为application/json,则callbackBody应为json格式,例如:{“key”:"$(key)",“hash”:"$(etag)",“w”:"$(imageInfo.width)",“h”:"$(imageInfo.height)"}。【可选】 +- --callback-body-type:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --end-user:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 +- --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 +- --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 # 示例 1 上传本地文件 `/Users/jemy/Documents/qiniu.mp4` 到空间 `if-pbl` 里面。 diff --git a/docs/sync.md b/docs/sync.md index 2646f034..8c82fe17 100644 --- a/docs/sync.md +++ b/docs/sync.md @@ -36,6 +36,26 @@ $ qshell sync --doc - --resumable-api-v2:使用分片 v2 进行上传;默认使用 v1。 【可选】 - --resumable-api-v2-part-size:使用分片上传 API V2 进行上传时的分片大小,默认为 4M 。【可选】 - --overwrite:是否覆盖空间已有文件,默认为 `false`。 【可选】 +- -l/--callback-urls:上传回调地址,可以指定多个地址,以逗号分开。【可选】 +- -T/--callback-host:上传回调HOST, 必须和 CallbackUrls 一起指定。 【可选】 +- --callback-body:上传成功后,七牛云向业务服务器发送 Content-Type: application/x-www-form-urlencoded 的 POST 请求。业务服务器可以通过直接读取请求的 query 来获得该字段,支持魔法变量和自定义变量。callbackBody 要求是合法的 url query string。例如key=$(key)&hash=$(etag)&w=$(imageInfo.width)&h=$(imageInfo.height)。如果callbackBodyType指定为application/json,则callbackBody应为json格式,例如:{“key”:"$(key)",“hash”:"$(etag)",“w”:"$(imageInfo.width)",“h”:"$(imageInfo.height)"}。【可选】 +- --callback-body-type:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --end-user:上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置为 application/json。【可选】 +- --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 +- --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 +- --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 + ##### 备注: 上传入口的域名对应的 IP 地址一般情况下是不变的,减少 DNS 的查询环节,可以提升同步速度和稳定性。 diff --git a/iqshell/storage/object/upload/operations/batch.go b/iqshell/storage/object/upload/operations/batch.go index 6f4e08b8..0779a418 100644 --- a/iqshell/storage/object/upload/operations/batch.go +++ b/iqshell/storage/object/upload/operations/batch.go @@ -77,10 +77,8 @@ func BatchUpload(cfg *iqshell.Config, info BatchUploadInfo) { EnableStdin: info.EnableStdin, UploadConfig: DefaultUploadConfig(), } - upload2Info.UploadConfig.Policy = &storage.PutPolicy{ - CallbackURL: info.CallbackUrl, - CallbackHost: info.CallbackHost, - } + upload2Info.UploadConfig.CallbackHost = info.CallbackHost + upload2Info.UploadConfig.CallbackURL = info.CallbackUrl if err := utils.UnMarshalFromFile(info.UploadConfigFile, &upload2Info.UploadConfig); err != nil { data.SetCmdStatusError() @@ -123,9 +121,6 @@ func (info *BatchUpload2Info) Check() *data.CodeError { if err := info.UploadConfig.Check(); err != nil { return err } - if info.UploadConfig.Policy == nil { - info.UploadConfig.Policy = &storage.PutPolicy{} - } if len(info.ItemSeparate) == 0 { info.ItemSeparate = data.DefaultLineSeparate } @@ -168,6 +163,7 @@ func BatchUpload2(cfg *iqshell.Config, info BatchUpload2Info) { func batchUpload(info BatchUpload2Info) { + log.DebugF("upload config:%+v", info) dbPath := filepath.Join(workspace.GetJobDir(), ".ldb") log.InfoF("upload status db file path:%s", dbPath) @@ -288,8 +284,33 @@ func batchUploadFlow(info BatchUpload2Info, uploadConfig UploadConfig, dbPath st Progress: nil, }, RelativePathToSrcPath: fileRelativePath, - Policy: uploadConfig.Policy, - DeleteOnSuccess: uploadConfig.DeleteOnSuccess, + Policy: storage.PutPolicy{ + Scope: "", + IsPrefixalScope: 0, + Expires: 0, + InsertOnly: 0, + EndUser: uploadConfig.EndUser, + ReturnURL: "", + ReturnBody: "", + CallbackURL: uploadConfig.CallbackURL, + CallbackHost: uploadConfig.CallbackHost, + CallbackBody: uploadConfig.CallbackBody, + CallbackBodyType: uploadConfig.CallbackBodyType, + PersistentOps: uploadConfig.PersistentOps, + PersistentNotifyURL: uploadConfig.PersistentNotifyURL, + PersistentPipeline: uploadConfig.PersistentPipeline, + ForceSaveKey: false, + SaveKey: "", + FsizeMin: 0, + FsizeLimit: 0, + DetectMime: uploadConfig.DetectMime, + MimeLimit: "", + FileType: uploadConfig.FileType, + CallbackFetchKey: uploadConfig.CallbackFetchKey, + DeleteAfterDays: uploadConfig.DeleteAfterDays, + TrafficLimit: uploadConfig.TrafficLimit, + }, + DeleteOnSuccess: uploadConfig.DeleteOnSuccess, } uploadInfo.TokenProvider = createTokenProviderWithMac(mac, uploadInfo) return uploadInfo, nil diff --git a/iqshell/storage/object/upload/operations/batch_data.go b/iqshell/storage/object/upload/operations/batch_data.go index ea8649ba..a89a7f6c 100644 --- a/iqshell/storage/object/upload/operations/batch_data.go +++ b/iqshell/storage/object/upload/operations/batch_data.go @@ -6,8 +6,6 @@ import ( "path/filepath" "strings" - "github.com/qiniu/go-sdk/v7/storage" - "github.com/qiniu/qshell/v2/iqshell/common/alert" "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/common/utils" @@ -45,7 +43,65 @@ type UploadConfig struct { RecordRoot string `json:"record_root,omitempty"` SequentialReadFile bool `json:"sequential_read_file"` // 文件顺序读 - Policy *storage.PutPolicy `json:"policy"` + // 唯一属主标识。特殊场景下非常有用,例如根据 App-Client 标识给图片或视频打水印。 + EndUser string `json:"end_user,omitempty"` + + // 上传成功后,七牛云向业务服务器发送 POST 请求的 URL。必须是公网上可以正常进行 POST 请求并能响应 HTTP/1.1 200 OK 的有效 URL。 + // 另外,为了给客户端有一致的体验,我们要求 callbackUrl 返回包 Content-Type 为 “application/json”,即返回的内容必须是合法的 + // JSON 文本。出于高可用的考虑,本字段允许设置多个 callbackUrl(用英文符号 ; 分隔),在前一个 callbackUrl 请求失败的时候会依次 + // 重试下一个 callbackUrl。一个典型例子是:http:///callback;http:///callback,并同时指定下面的 callbackHost 字段。 + // 在 callbackUrl 中使用 ip 的好处是减少对 dns 解析的依赖,可改善回调的性能和稳定性。指定 callbackUrl,必须指定 callbackbody, + // 且值不能为空。 + CallbackURL string `json:"callback_url,omitempty"` + + // 上传成功后,七牛云向业务服务器发送回调通知时的 Host 值。与 callbackUrl 配合使用,仅当设置了 callbackUrl 时才有效。 + CallbackHost string `json:"callback_host,omitempty"` + + // 上传成功后,七牛云向业务服务器发送 Content-Type: application/x-www-form-urlencoded 的 POST 请求。业务服务器可以通过直接读取 + // 请求的 query 来获得该字段,支持魔法变量和自定义变量。callbackBody 要求是合法的 url query string。 + // 例如key=$(key)&hash=$(etag)&w=$(imageInfo.width)&h=$(imageInfo.height)。如果callbackBodyType指定为application/json, + // 则callbackBody应为json格式,例如:{“key”:"$(key)",“hash”:"$(etag)",“w”:"$(imageInfo.width)",“h”:"$(imageInfo.height)"}。 + CallbackBody string `json:"callback_body,omitempty"` + + // 上传成功后,七牛云向业务服务器发送回调通知 callbackBody 的 Content-Type。默认为 application/x-www-form-urlencoded,也可设置 + // 为 application/json。 + CallbackBodyType string `json:"callback_body_type,omitempty"` + + // 资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自 + // 定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。请参阅persistenOps详解与示例。同时添加 persistentPipeline 字段,使用专 + // 用队列处理,请参阅persistentPipeline。 + PersistentOps string `json:"persistent_ops,omitempty"` + + // 接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能响应 HTTP/1.1 200 OK 的有效 URL。该 URL 获取的内容和持久化处 + // 理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body + // 才能获取。 + PersistentNotifyURL string `json:"persistent_notify_url,omitempty"` + + // 转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。 + PersistentPipeline string `json:"persistent_pipeline,omitempty"` + + // saveKey 的优先级设置。为 true 时,saveKey不能为空,会忽略客户端指定的key,强制使用saveKey进行文件命名。参数不设置时, + // 默认值为false + ForceSaveKey bool `json:"force_save_key,omitempty"` + + // 开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。 + // 默认设为 0 时:如上传端指定了 MimeType 则直接使用该值,否则按如下顺序侦测 MimeType 值: + // 1. 检查文件扩展名; + // 2. 检查 Key 扩展名; + // 3. 侦测内容。 + // 设为 1 时:则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + // 1. 侦测内容; + // 2. 检查文件扩展名; + // 3. 检查 Key 扩展名。 + // 设为 -1 时:无论上传端指定了何值直接使用该值。 + DetectMime int `json:"detect_mime,omitempty"` + + CallbackFetchKey uint8 `json:"callback_fetch_key,omitempty"` + + DeleteAfterDays int `json:"delete_after_days,omitempty"` + + // 上传单链接限速,单位:bit/s;范围:819200 - 838860800(即800Kb/s - 800Mb/s),如果超出该范围将返回 400 错误 + TrafficLimit uint64 `json:"traffic_limit,omitempty"` } func DefaultUploadConfig() UploadConfig { @@ -78,7 +134,6 @@ func DefaultUploadConfig() UploadConfig { DisableForm: false, WorkerCount: 3, RecordRoot: "", - Policy: nil, } } @@ -168,21 +223,14 @@ func (up *UploadConfig) Check() *data.CodeError { return data.NewEmptyError().AppendDesc("wrong Filetype, It should be one of 0, 1, 2, 3") } - if up.Policy != nil { - //if (up.Policy.CallbackURL == "" && up.Policy.CallbackHost != "") || - // (up.Policy.CallbackURL != "" && up.Policy.CallbackHost == "") { - // return data.NewEmptyError().AppendDesc("callbackUrls and callback must exist at the same time") - //} - - if up.Policy.CallbackURL != "" { - callbackUrls := strings.Replace(up.Policy.CallbackURL, ",", ";", -1) - up.Policy.CallbackURL = callbackUrls - if len(up.Policy.CallbackBody) == 0 { - up.Policy.CallbackBody = "key=$(key)&hash=$(etag)" - } - if len(up.Policy.CallbackBodyType) == 0 { - up.Policy.CallbackBodyType = "application/x-www-form-urlencoded" - } + if up.CallbackURL != "" { + callbackUrls := strings.Replace(up.CallbackURL, ",", ";", -1) + up.CallbackURL = callbackUrls + if len(up.CallbackBody) == 0 { + up.CallbackBody = "key=$(key)&hash=$(etag)" + } + if len(up.CallbackBodyType) == 0 { + up.CallbackBodyType = "application/x-www-form-urlencoded" } } diff --git a/iqshell/storage/object/upload/operations/sync.go b/iqshell/storage/object/upload/operations/sync.go index 71d4dd9c..398d1ab7 100644 --- a/iqshell/storage/object/upload/operations/sync.go +++ b/iqshell/storage/object/upload/operations/sync.go @@ -1,6 +1,8 @@ package operations import ( + "path/filepath" + "github.com/qiniu/qshell/v2/iqshell" "github.com/qiniu/qshell/v2/iqshell/common/alert" "github.com/qiniu/qshell/v2/iqshell/common/data" @@ -8,7 +10,6 @@ import ( "github.com/qiniu/qshell/v2/iqshell/common/progress" "github.com/qiniu/qshell/v2/iqshell/common/utils" "github.com/qiniu/qshell/v2/iqshell/common/workspace" - "path/filepath" ) type SyncInfo UploadInfo @@ -26,7 +27,7 @@ func (info *SyncInfo) Check() *data.CodeError { if info.Overwrite && len(info.SaveKey) == 0 { return alert.CannotEmptyError("Overwrite mode and Key", "") } - return nil + return checkPolicy(&info.Policy) } func SyncFile(cfg *iqshell.Config, info SyncInfo) { @@ -44,6 +45,8 @@ func SyncFile(cfg *iqshell.Config, info SyncInfo) { return } + log.DebugF("upload config:%+v", info) + info.CacheDir = workspace.GetJobDir() info.Progress = progress.NewPrintProgress(" 进度") ret, err := uploadFile((*UploadInfo)(&info)) diff --git a/iqshell/storage/object/upload/operations/upload.go b/iqshell/storage/object/upload/operations/upload.go index 006318c1..dea5fe0e 100644 --- a/iqshell/storage/object/upload/operations/upload.go +++ b/iqshell/storage/object/upload/operations/upload.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "path/filepath" + "strings" "time" "github.com/qiniu/go-sdk/v7/auth/qbox" @@ -21,8 +22,9 @@ import ( type UploadInfo struct { upload.ApiInfo + RelativePathToSrcPath string // 相对与上传文件夹的路径信息 - Policy *storage.PutPolicy + Policy storage.PutPolicy DeleteOnSuccess bool } @@ -39,13 +41,30 @@ func (info *UploadInfo) Check() *data.CodeError { if utils.IsNetworkSource(info.FilePath) { return alert.Error("file can't be network source", "") } - return nil + + return checkPolicy(&info.Policy) } func (info *UploadInfo) WorkId() string { return fmt.Sprintf("%s:%s:%s", info.FilePath, info.ToBucket, info.SaveKey) } +func checkPolicy(policy *storage.PutPolicy) *data.CodeError { + if policy.CallbackURL == "" { + return nil + } + + callbackUrls := strings.Replace(policy.CallbackURL, ",", ";", -1) + policy.CallbackURL = callbackUrls + if len(policy.CallbackBody) == 0 { + policy.CallbackBody = "key=$(key)&hash=$(etag)" + } + if len(policy.CallbackBodyType) == 0 { + policy.CallbackBodyType = "application/x-www-form-urlencoded" + } + return nil +} + func UploadFile(cfg *iqshell.Config, info UploadInfo) { cfg.JobPathBuilder = func(cmdPath string) string { resumeVersion := "v1" @@ -61,6 +80,8 @@ func UploadFile(cfg *iqshell.Config, info UploadInfo) { return } + log.DebugF("upload config:%+v", info) + info.CacheDir = workspace.GetJobDir() info.Progress = progress.NewPrintProgress(" 进度") ret, err := uploadFile(&info) @@ -126,7 +147,7 @@ func createTokenProvider(info *UploadInfo) (provider func() string, err *data.Co } func createTokenProviderWithMac(mac *qbox.Mac, info *UploadInfo) func() string { - policy := *info.Policy + policy := info.Policy policy.Scope = info.ToBucket policy.InsertOnly = 1 // 仅新增不覆盖 if info.Overwrite { From 946bce85fa49c0ef9bd7e87c23e8d04cae7dc201 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Fri, 29 Dec 2023 14:53:05 +0800 Subject: [PATCH 10/13] delete file type check --- iqshell/storage/object/upload/operations/batch_data.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/iqshell/storage/object/upload/operations/batch_data.go b/iqshell/storage/object/upload/operations/batch_data.go index ea8649ba..c88fa3fc 100644 --- a/iqshell/storage/object/upload/operations/batch_data.go +++ b/iqshell/storage/object/upload/operations/batch_data.go @@ -164,10 +164,6 @@ func (up *UploadConfig) Check() *data.CodeError { } } - if up.FileType < 0 || up.FileType > 3 { - return data.NewEmptyError().AppendDesc("wrong Filetype, It should be one of 0, 1, 2, 3") - } - if up.Policy != nil { //if (up.Policy.CallbackURL == "" && up.Policy.CallbackHost != "") || // (up.Policy.CallbackURL != "" && up.Policy.CallbackHost == "") { From 8d84b902c7e2cd0e3ad9aacb85c78c593ca4f3a1 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Thu, 11 Jan 2024 10:12:04 +0800 Subject: [PATCH 11/13] change doc --- docs/fput.md | 22 ++++++++++++---------- docs/qupload.md | 20 +++++++++++--------- docs/rput.md | 22 ++++++++++++---------- docs/sync.md | 22 ++++++++++++---------- 4 files changed, 47 insertions(+), 39 deletions(-) diff --git a/docs/fput.md b/docs/fput.md index 43701142..b43507c4 100644 --- a/docs/fput.md +++ b/docs/fput.md @@ -41,16 +41,18 @@ $ qshell fput --doc - --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 - --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 - --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 -- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 - 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: - 1) 侦测内容; - 2) 检查文件扩展名; - 3) 检查 Key 扩展名。 - 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: - 1) 检查文件扩展名; - 2) 检查 Key 扩展名; - 3) 侦测内容。 - 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 +``` + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +``` - --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 diff --git a/docs/qupload.md b/docs/qupload.md index 359502df..6990ca10 100644 --- a/docs/qupload.md +++ b/docs/qupload.md @@ -99,15 +99,17 @@ $ qshell qupload --doc - persistent_notify_url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 - persistent_pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 - detect_mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 - 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: - 1) 侦测内容; - 2) 检查文件扩展名; - 3) 检查 Key 扩展名。 - 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: - 1) 检查文件扩展名; - 2) 检查 Key 扩展名; - 3) 侦测内容。 - 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +``` + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +``` - traffic_limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 diff --git a/docs/rput.md b/docs/rput.md index 1a1513e9..edfbf28b 100644 --- a/docs/rput.md +++ b/docs/rput.md @@ -45,16 +45,18 @@ $ qshell rput --doc - --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 - --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 - --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 -- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 - 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: - 1) 侦测内容; - 2) 检查文件扩展名; - 3) 检查 Key 扩展名。 - 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: - 1) 检查文件扩展名; - 2) 检查 Key 扩展名; - 3) 侦测内容。 - 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 +``` + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +``` - --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 # 示例 diff --git a/docs/sync.md b/docs/sync.md index 8c82fe17..fcf82cb0 100644 --- a/docs/sync.md +++ b/docs/sync.md @@ -44,16 +44,18 @@ $ qshell sync --doc - --persistent-ops:资源上传成功后触发执行的预转持久化处理指令列表。fileType=2或3(上传归档存储或深度归档存储文件)时,不支持使用该参数。支持魔法变量和自定义变量。每个指令是一个 API 规格字符串,多个指令用;分隔。【可选】 - --persistent-notify-url:接收持久化处理结果通知的 URL。必须是公网上可以正常进行 POST 请求并能成功响应的有效 URL。该 URL 获取的内容和持久化处理状态查询的处理结果一致。发送 body 格式是 Content-Type 为 application/json 的 POST 请求,需要按照读取流的形式读取请求的 body 才能获取。【可选】 - --persistent-pipeline:转码队列名。资源上传成功后,触发转码时指定独立的队列进行转码。为空则表示使用公用队列,处理速度比较慢。建议使用专用队列。【可选】 -- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 - 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: - 1) 侦测内容; - 2) 检查文件扩展名; - 3) 检查 Key 扩展名。 - 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: - 1) 检查文件扩展名; - 2) 检查 Key 扩展名; - 3) 侦测内容。 - 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +- --detect-mime:开启 MimeType 侦测功能,并按照下述规则进行侦测;如不能侦测出正确的值,会默认使用 application/octet-stream 。【可选】 +``` + 1. 设为 1 值,则忽略上传端传递的文件 MimeType 信息,并按如下顺序侦测 MimeType 值: + 1) 侦测内容; + 2) 检查文件扩展名; + 3) 检查 Key 扩展名。 + 2. 默认设为 0 值,如上传端指定了 MimeType(application/octet-stream 除外)则直接使用该值,否则按如下顺序侦测 MimeType 值: + 1) 检查文件扩展名; + 2) 检查 Key 扩展名; + 3) 侦测内容。 + 3. 设为 -1 值,无论上传端指定了何值直接使用该值。 +``` - --traffic-limit:上传请求单链接速度限制,控制客户端带宽占用。限速值取值范围为 819200 ~ 838860800,单位为 bit/s。【可选】 From 2b970baf1692748bba31263b89018f14e3ac86db Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Thu, 11 Jan 2024 10:46:48 +0800 Subject: [PATCH 12/13] change test case --- cmd_test/upload_form_test.go | 6 ++++-- cmd_test/upload_test.go | 6 +++--- iqshell/storage/object/batch/data.go | 2 ++ iqshell/storage/object/operations/status.go | 1 + 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/cmd_test/upload_form_test.go b/cmd_test/upload_form_test.go index a0552256..950f307b 100644 --- a/cmd_test/upload_form_test.go +++ b/cmd_test/upload_form_test.go @@ -3,9 +3,10 @@ package cmd import ( - "github.com/qiniu/qshell/v2/cmd_test/test" "strings" "testing" + + "github.com/qiniu/qshell/v2/cmd_test/test" ) func TestFormUpload(t *testing.T) { @@ -19,7 +20,8 @@ func TestFormUpload(t *testing.T) { result, errs := test.RunCmdWithError("fput", test.Bucket, "qshell_fput_1M", path, "--mimetype", "image/jpg", "--storage", "0", - "--file-type", "1") + "--file-type", "1", + "--end-user", "10") if len(errs) > 0 { t.Fail() } diff --git a/cmd_test/upload_test.go b/cmd_test/upload_test.go index 3f8d5b28..31ede9c8 100644 --- a/cmd_test/upload_test.go +++ b/cmd_test/upload_test.go @@ -4,11 +4,12 @@ package cmd import ( "encoding/json" - "github.com/qiniu/qshell/v2/cmd_test/test" - "github.com/qiniu/qshell/v2/iqshell/storage/object/upload/operations" "path/filepath" "strings" "testing" + + "github.com/qiniu/qshell/v2/cmd_test/test" + "github.com/qiniu/qshell/v2/iqshell/storage/object/upload/operations" ) func TestQUpload(t *testing.T) { @@ -71,7 +72,6 @@ func TestQUpload(t *testing.T) { DisableForm: false, WorkerCount: 4, RecordRoot: "", - Policy: nil, }, LogFile: logPath, RecordRoot: recordPath, diff --git a/iqshell/storage/object/batch/data.go b/iqshell/storage/object/batch/data.go index d02bed53..cd43dc45 100644 --- a/iqshell/storage/object/batch/data.go +++ b/iqshell/storage/object/batch/data.go @@ -3,6 +3,7 @@ package batch import ( "encoding/base64" "fmt" + "github.com/qiniu/qshell/v2/iqshell/common/data" "github.com/qiniu/qshell/v2/iqshell/common/flow" ) @@ -58,6 +59,7 @@ type OperationResult struct { PutTime int64 `json:"putTime"` MimeType string `json:"mimeType"` Type int `json:"type"` + EndUser string `json:"endUser"` Error string `json:"error"` Parts []int64 `json:"parts"` } diff --git a/iqshell/storage/object/operations/status.go b/iqshell/storage/object/operations/status.go index fb2f13d4..b08d59f0 100644 --- a/iqshell/storage/object/operations/status.go +++ b/iqshell/storage/object/operations/status.go @@ -147,6 +147,7 @@ func getResultInfo(bucket, key string, status object.StatusResult) string { fieldAdder("Fsize", status.FSize, utils.FormatFileSize(status.FSize)) fieldAdder("PutTime", status.PutTime, time.Unix(0, status.PutTime*100).String()) fieldAdder("MimeType", status.MimeType, "") + fieldAdder("EndUser", status.EndUser, "") fieldAdderWithValueDescs("Status", status.Status, map[interface{}]string{1: "禁用"}, From e2262c52800f27bf2d5e759a24e2a2bb2a51a103 Mon Sep 17 00:00:00 2001 From: YangSen-qn Date: Tue, 30 Jan 2024 16:28:48 +0800 Subject: [PATCH 13/13] buckets cmd --- iqshell/storage/servers/all_buckets.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/iqshell/storage/servers/all_buckets.go b/iqshell/storage/servers/all_buckets.go index 23f6692d..bff5cd9e 100644 --- a/iqshell/storage/servers/all_buckets.go +++ b/iqshell/storage/servers/all_buckets.go @@ -66,9 +66,9 @@ func allBuckets(cfg *config.Config, info ListApiInfo) ([]BucketInfo, *data.CodeE } ucHost := cfg.Hosts.GetOneUc() - reqURL := fmt.Sprintf("%s/v3/buckets", utils.Endpoint(cfg.UseHttps.Value(), ucHost)) + reqURL := fmt.Sprintf("%s/v3/buckets?shared=rd", utils.Endpoint(cfg.UseHttps.Value(), ucHost)) if len(info.Region) > 0 { - reqURL = fmt.Sprintf("%s?region=%s", reqURL, info.Region) + reqURL = fmt.Sprintf("%s®ion=%s", reqURL, info.Region) } var buckets []BucketInfo e := bucketManager.Client.CredentialedCall(workspace.GetContext(), bucketManager.Mac, auth.TokenQiniu, &buckets, "POST", reqURL, nil)