0.介绍

Rclone 是一个用于管理云存储上的文件的命令行程序。它是云供应商 Web 存储接口的功能丰富的替代方案。超过 70 种云存储产品支持 rclone，包括 S3 对象存储、企业和消费者文件存储服务以及标准传输协议。

Rclone 具有与 unix 命令 rsync、cp、mv、mount、ls、ncdu、tree、rm 和 cat 等价的强大云功能。Rclone 熟悉的语法包括 shell 管道支持和–dry-run保护。它可以在命令行、脚本中或通过其API使用。
用户称rclone为“云存储的瑞士军刀”、 “技术与魔法无异”。

Rclone 真正照顾您的数据。它始终保留时间戳并验证校验和。通过有限的带宽传输；间歇性连接，或受配额限制，可以从上次传输的好文件开始重新启动。您可以 检查文件的完整性。在可能的情况下，rclone 采用服务器端传输来最大程度地减少本地带宽使用，并在不使用本地磁盘的情况下从一个提供商传输到另一个提供商。

虚拟后端包装本地和云文件系统以应用 加密、 压缩、 分块、 散列和 连接。

Rclone将任何本地、云或虚拟文件系统挂载为 Windows、macOS、linux 和 FreeBSD 上的磁盘，并且还通过 SFTP、 HTTP、 WebDAV、 FTP和 DLNA为这些文件系统提供服务。

Rclone 是成熟的开源软件，最初受到 rsync 的启发，用Go编写。友好的支持社区熟悉各种用例。官方 Ubuntu、Debian、Fedora、Brew 和 Chocolatey 存储库。包括 rclone。对于最新版本，建议从 rclone.org 下载。

Rclone 广泛应用于 Linux、Windows 和 Mac 上。第三方开发人员使用 rclone 命令行或 API 创建创新的备份、恢复、GUI 和业务流程解决方案。
Rclone 负责与云存储通信的繁重工作。

Rclone 可以帮助您：

• 将文件备份（并加密）到云存储
• 从云存储恢复（和解密）文件
• 将云数据镜像到其他云服务或本地
• 将数据迁移到云端，或在云存储供应商之间迁移
• 将多个加密、缓存或多样化的云存储挂载为磁盘
• 使用lsf、ljson、size、ncdu分析和解释云存储中保存的数据
• 将文件系统合并在一起，将多个本地和/或云文件系统呈现为一个整体

1.下载备份工具

版本参考：https://rclone.org/downloads/

2.解压备份工具

unzip rclone-v1.63.1-linux-amd64.zip

3.配置

# 获取minio地址与端口号
kubectl get svc -o wide |grep minio

# 开始配置rclone
./rclone config

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.

# 新建配置
n/s/q> n

# 设置配置名称
name> minio 
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
 2 / Akamai NetStorage
   \ (netstorage)
 3 / Alias for an existing remote
   \ (alias)
 4 / Amazon Drive
   \ (amazon cloud drive)
 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
   \ (s3)
 6 / Backblaze B2
   \ (b2)
 7 / Better checksums for other remotes
   \ (hasher)
 8 / Box
   \ (box)
 9 / Cache a remote
   \ (cache)
10 / Citrix Sharefile
   \ (sharefile)
11 / Combine several remotes into one
   \ (combine)
12 / Compress a remote
   \ (compress)
13 / Dropbox
   \ (dropbox)
14 / Encrypt/Decrypt a remote
   \ (crypt)
15 / Enterprise File Fabric
   \ (filefabric)
16 / FTP
   \ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
   \ (google cloud storage)
18 / Google Drive
   \ (drive)
19 / Google Photos
   \ (google photos)
20 / HTTP
   \ (http)
21 / Hadoop distributed file system
   \ (hdfs)
22 / HiDrive
   \ (hidrive)
23 / In memory object storage system.
   \ (memory)
24 / Internet Archive
   \ (internetarchive)
25 / Jottacloud
   \ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
   \ (koofr)
27 / Local Disk
   \ (local)
28 / Mail.ru Cloud
   \ (mailru)
29 / Mega
   \ (mega)
30 / Microsoft Azure Blob Storage
   \ (azureblob)
31 / Microsoft OneDrive
   \ (onedrive)
32 / OpenDrive
   \ (opendrive)
33 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
   \ (swift)
34 / Oracle Cloud Infrastructure Object Storage
   \ (oracleobjectstorage)
35 / Pcloud
   \ (pcloud)
36 / PikPak
   \ (pikpak)
37 / Put.io
   \ (putio)
38 / QingCloud Object Storage
   \ (qingstor)
39 / SMB / CIFS
   \ (smb)
40 / SSH/SFTP
   \ (sftp)
41 / Sia Decentralized Cloud
   \ (sia)
42 / Storj Decentralized Cloud Storage
   \ (storj)
43 / Sugarsync
   \ (sugarsync)
44 / Transparently chunk/split large files
   \ (chunker)
45 / Union merges the contents of several upstream fs
   \ (union)
46 / Uptobox
   \ (uptobox)
47 / WebDAV
   \ (webdav)
48 / Yandex Disk
   \ (yandex)
49 / Zoho
   \ (zoho)
50 / premiumize.me
   \ (premiumizeme)
51 / seafile
   \ (seafile)

# 选择链接类型
Storage> 5

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Amazon Web Services (AWS) S3
   \ (AWS)
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ (Alibaba)
 3 / Arvan Cloud Object Storage (AOS)
   \ (ArvanCloud)
 4 / Ceph Object Storage
   \ (Ceph)
 5 / China Mobile Ecloud Elastic Object Storage (EOS)
   \ (ChinaMobile)
 6 / Cloudflare R2 Storage
   \ (Cloudflare)
 7 / DigitalOcean Spaces
   \ (DigitalOcean)
 8 / Dreamhost DreamObjects
   \ (Dreamhost)
 9 / Google Cloud Storage
   \ (GCS)
10 / Huawei Object Storage Service
   \ (HuaweiOBS)
11 / IBM COS S3
   \ (IBMCOS)
12 / IDrive e2
   \ (IDrive)
13 / IONOS Cloud
   \ (IONOS)
14 / Seagate Lyve Cloud
   \ (LyveCloud)
15 / Liara Object Storage
   \ (Liara)
16 / Minio Object Storage
   \ (Minio)
17 / Netease Object Storage (NOS)
   \ (Netease)
18 / Petabox Object Storage
   \ (Petabox)
19 / RackCorp Object Storage
   \ (RackCorp)
20 / Scaleway Object Storage
   \ (Scaleway)
21 / SeaweedFS S3
   \ (SeaweedFS)
22 / StackPath Object Storage
   \ (StackPath)
23 / Storj (S3 Compatible Gateway)
   \ (Storj)
24 / Tencent Cloud Object Storage (COS)
   \ (TencentCOS)
25 / Wasabi Object Storage
   \ (Wasabi)
26 / Qiniu Object Storage (Kodo)
   \ (Qiniu)
27 / Any other S3 compatible provider
   \ (Other)

# 选择小类别
provider> 16

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value.
 1 / Enter AWS credentials in the next step.
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ "true"

# 选择凭据类型
env_auth> 1

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").

# 输入minio AK 
access_key_id> minio

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").

# 输入minio SK 
secret_access_key> <your passwd>

Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
   / Use this if unsure.
 1 | Will use v4 signatures and an empty region.
   \ ""
   / Use this only if v4 signatures don't work.
 2 | E.g. pre Jewel/v10 CEPH.
   \ "other-v2-signature"

# 输入链接区域
region> us-east-1

Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a string value. Press Enter for the default ("").

# 输入minio地址
endpoint> http://localhost:9000

Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").

# 回车略过
location_constraint> 

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ "private"
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ "public-read"
   / Owner gets FULL_CONTROL.
 3 | The AllUsers group gets READ and WRITE access.
   | Granting this on a bucket is generally not recommended.
   \ "public-read-write"
   / Owner gets FULL_CONTROL.
 4 | The AuthenticatedUsers group gets READ access.
   \ "authenticated-read"
   / Object owner gets FULL_CONTROL.
 5 | Bucket owner gets READ access.
   | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-read"
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-full-control"

# 配置访问权限
acl> 1

Option server_side_encryption.
The server-side encryption algorithm used when storing this object in S3.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
 3 / aws:kms
   \ "aws:kms"

# 使用默认配置
server_side_encryption> 

Option sse_kms_key_id.
If using KMS ID you must provide the ARN of Key.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value.
 1 / None
   \ ""
 2 / arn:aws:kms:*
   \ "arn:aws:kms:us-east-1:*"

# 使用默认配置
sse_kms_key_id> 

Edit advanced config?
y) Yes
n) No (default)

# 不使用高级配置
y/n> n

--------------------
[minio]
type = s3
provider = Minio
access_key_id = minio
secret_access_key = <yourpasswd>
region = us-east-1
endpoint = http://localhost:9000
acl = private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote

# 保存当前配置
y/e/d> y

Current remotes:

Name                 Type
====                 ====
minio                s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

# 退出配置页面
e/n/d/r/c/s/q> q

4.测试配置是否正确

./rclone ls minio:
./rclone lsd minio:

5.备份

5.1 备份至本地文件夹

# 新建一个用来备份的文件夹
mkdir minio_data

# 进行备份，<local-host>代表目录，这里可以用第二个直接备份到minio_data中
./rclone copy minio: <local-host>
./rclone copy minio: minio_data/

5.2 从minioA 到 minioB

# 进行全局复制
./rclone copy minio: new_minio:

# bucket间复制
./rclone copy minio:bie/ new_minio:/bie/

# 文件间复制
./rclone copy minio:bie/package new_minio:/bie/package

6. 数据校验

6.1 数据容量校验

# 数据校验
./rclone size minio:

6.2 数据双向diff

# 双向比对,这里以bie的桶为示例

## 文件夹与minio服务比对
./rclone check test new_minio:bie/package

## minio 与 minio比对
./rclone check minio: new_minio:
./rclone check minio:bie new_minio:bie
./rclone check minio:bie/package new_minio:bie/package

正常情况下，differences应该为0，若有异常，可以查看ERROR的实际报错

7.恢复

7.1 恢复配置

# 获取minio地址与端口号
kubectl get svc -o wide |grep minio

# 开始配置rclone
./rclone config

minio n/s/q> n 
name> new_minio 
Storage> 5
provider> 16 
env_auth> 1 
access_key_id> minio 
secret_access_key> <passwd> 
region> us-east-1 
# 注意IP地址及端口，现有环境minio的svc地址及端口 
endpoint> http://localhost:9000 
location_constraint> 
acl> 1 
server_side_encryption> 
sse_kms_key_id> 
y/n> n 
y/e/d> y
e/n/d/r/c/s/q> q

7.2 从本地文件夹恢复

# minio_data 为之前备份的数据 上述图片状态为恢复数据正常结束,可登录minio的web界面查看之前的数据是否恢复。
./rclone -P copy minio_data news:/ 

8. 数据同步

# 使源和目标相同，仅修改目标
./rclone sync test new_minio:
./rclone sync minio: new_minio:
./rclone sync minio:bie new_minio:bie
./rclone sync minio:bie/package new_minio:bie/package

# 双向同步，仅适用于在线，多个minio服务
# 1.63.1版本，双向同步为实验功能，数据很多的情况下，不建议使用
./rclone bisync minio: new_minio:
./rclone bisync minio:bie new_minio:bie
./rclone bisync minio:bie/package new_minio:bie/package

9.常用指令

• rclone config : 配置会话；
• rclone copy : 将文件从原文件夹复制至目标文件夹，跳过已复制的文件；
• rclone sync : 将文件从原文件夹同步至目标文件夹，只修改目的地；
• rclone move : 将文件从原文件夹移动至目标文件夹；
• rclone delete : 删除路径中的内容；
• rclone purge : 清空指定路径下的所有文件数据；
• rclone mkdir : 创建一个新目录；
• rclone rmdir : 删除一个空目录；
• rclone rmdirs : 删除路径下的所有空目录；
• rclone check : 检查源和目标的数据是否匹配；
• rclone ls : 列出指定路径下所有的文件包含文件的大小及路径；
• rclone lsd : 列出路径中的所有目录containers和buckets；
• rclone lsl : 列出具有大小、修改时间和路径中的所有对象；
• rclone md5sum : 为路径中的所有对象生成一个md5sum文件；
• rclone sha1sum : 为路径中的所有对象生成一个sha1sum文件；
• rclone size : 返回远程路径中对象的总大小和数量；
• rclone version : 显示版本号；
• rclone cleanup : 如果可能的话，清理remote；
• rclone dedupe : 交互式查找重复文件并删除重命名它们；
• rclone authorize : 远程认证；
• rclone cat : 连接文件并将它们发送到stdout；
• rclone copyto : 将文件从源复制到目标，跳过已复制的文件；
• rclone genautocomplete : rclone输出完成脚本；
• rclone gendocs : rclone将markdown文档输出到所提供的目录；
• rclone listremotes : 列出所有远程的配置文件；
• rclone mount : 将远程目标挂载至本地；
• rclone moveto : 将文件或目录从源移动至目标；
• rclone obscure : 在rclone.conf文件中使用模糊密码；
• rclone cryptcheck : 检验远程认证；
• rclone about : 获取配额信息；