CHANGELOG.md

This file describes changes relevant to all users that are made in each released version of restic from the perspective of the user.

Important Changes in 0.X.Y

• We've disabled handling SIGPIPE again. Turns out, writing to broken TCP connections also raised SIGPIPE, so restic exits on the first write to a broken connection. Instead, restic should retry the request. restic#1459 restic#1457 restic#1466

• The command diff was added, it allows comparing two snapshots and listing all differences. restic#11 restic#1460 restic#1462

Small changes

• We've added code to detect old cache directories of repositories that haven't been used in a long time, restic now prints a note when it detects that such dirs exist. Also, the option --cleanup-cache was added to automatically remove such directories. That's not a problem because the cache will be rebuild once a repo is accessed again. restic#1436

• The cache directory on Windows and Darwin was not correct, instead the directory .cache was used. restic#1454

• By default, the access time for files and dirs is not saved any more. It is not possible to reliably disable updating the access time during a backup, so for the next backup the access time is different again. This means a lot of metadata is saved. If you want to save the access time anyway, pass --with-atime to the backup command. restic#1452

• We've improved the s3 backend to work with DigitalOcean Spaces. restic#1459 restic#1457

• The cancellation logic was improved, restic can now shut down cleanly when requested to do so (e.g. via ctrl+c). restic#1439

Important Changes in 0.8.0

• A vulnerability was found in the restic restorer, which allowed attackers in special circumstances to restore files to a location outside of the target directory. Due to the circumstances we estimate this to be a low-risk vulnerability, but urge all users to upgrade to the latest version of restic.

  Exploiting the vulnerability requires a Linux/Unix system which saves backups via restic and a Windows systems which restores files from the repo. In addition, the attackers need to be able to create create files with arbitrary names which are then saved to the restic repo. For example, by creating a file named "..\test.txt" (which is a perfectly legal filename on Linux) and restoring a snapshot containing this file on Windows, it would be written to the parent of the target directory.

  We'd like to thank Tyler Spivey for reporting this responsibly!

  restic#1445

• The s3 backend used the subdir restic within a bucket if no explicit path after the bucket name was specified. Since this version, restic does not use this default path any more. If you created a repo on s3 in a bucket without specifying a path within the bucket, you need to add /restic at the end of the repository specification to access your repo: s3:s3.amazonaws.com/bucket/restic restic#1292 restic#1437

• We've added a local cache for metadata so that restic doesn't need to load all metadata (snapshots, indexes, ...) from the repo each time it starts. By default the cache is active, but there's a new global option --no-cache that can be used to disable the cache. By deafult, the cache a standard cache folder for the OS, which can be overridden with --cache-dir. The cache will automatically populate, indexes and snapshots are saved as they are loaded. Cache directories for repos that haven't been used recently can automatically be removed by restic with the --cleanup-cache option. restic#1040 restic#29 restic#738 restic#282 restic#1287 restic#1436

• A related change was to by default create pack files in the repo that contain either data or metadata, not both mixed together. This allows easy caching of only the metadata files. The next run of restic prune will untangle mixed files automatically. restic#1265

• The Google Cloud Storage backend no longer requires the service account to have the storage.buckets.get permission ("Storage Admin" role) in restic init if the bucket already exists. restic#1281

• Added support for rate limiting through --limit-upload and --limit-download flags. restic#1216 restic#1336 restic#1358

• Failed backend requests are now automatically retried. restic#1353

• We've added the dump command which prints a file from a snapshot to stdout. This can e.g. be used to restore files read with backup --stdin. restic#510 restic#1346

Small changes

• The directory structure in the fuse mount now exposes a symlink latest which points to the latest snapshot in that particular directory. restic#1249

• The option --compact was added to the forget command to provide the same compact view as the snapshots command. restic#1269

• We've re-enabled a workaround for minio-go (the library we're using to access s3 backends), this reduces memory usage. restic#1256 restic#1267

• The sftp backend now prompts for the password if a password is necessary for login. restic#448 restic#1270

• The generate command has been added, which replaces the now removed commands manpage and autocomplete. This release of restic contains the most recent manpages in doc/man and the auto-completion files for bash and zsh in doc/bash-completion.sh and doc/zsh-completion.zsh restic#1274 restic#1282

• A bug was discovered in the library we're using to access Backblaze, it now reuses already established TCP connections which should be a lot faster and not cause network failures any more. restic#1291 restic#1301

• Another bug in the forget command caused prune not to be run when --prune was specified without a policy, e.g. when only snapshot IDs that should be forgotten are listed manually. This is corrected now. restic#1317

• The check command now explicetly prints No errors were found when no errors could be found. restic#1319 restic#1303

• The fuse mount now has an ids subdirectory which contains the snapshots below their (short) IDs. restic#1102 restic#1299 restic#1320

• The backup command was improved, it now caches the result of excludes for a directory. restic#1271 restic#1326

• We've added the --cacert option which can be used to pass one (or more) CA certificates to restic. These are used in addition to the system CA certificates to verify HTTPS certificates (e.g. for the REST backend). restic#1114 restic#1276

• When the list of files/dirs to be saved is read from a file with --files-from, comment lines (starting with #) are now ignored. restic#1367 restic#1368

Important Changes in 0.7.3

• For large backups stored in Google Cloud Storage, the prune command fails because listing only returns the first 1000 files. This has been corrected, no data is lost in the process. In addition, a plausibility check was added to prune. restic#1246 restic#1247

Important Changes in 0.7.2

• We've added an official docker image and a Dockerfile to build this image in docker/. restic#1061

• The git repository layout was changed to resemble the layout typically used in Go projects, we're not using gb for building restic any more and vendoring the dependencies is now taken care of by dep. restic#1126

• We now support saving backups on Google Cloud Storage. restic#1134 restic#1052 restic#211

• We've added support for Microsoft Azure Blob Storage as a restic backend. restic#1149 restic#1059 restic#609

• In the course of supporting Microsoft Azure Blobe Storage Go 1.8 is now a requirement to build restic.

• The restore command has been improved: When dirs are excluded (or not included) in a restore, they are not loaded from the repo any more. restic#1044

• Name collisions are now resolved by appending a counter. restic#1179 restic#1209

Small changes

• The key command now prompts for a password even if the original password to access a repo has been specified via the RESTIC_PASSWORD environment variable or a password file. restic#1132 restic#1133

• Properly report errors when reading files with exclude patterns. restic#1144

• We now automatically generate man pages for all restic commands, see the subdir doc/man. restic#697 restic#1147

• The key remove command was corrected and now works as documented. restic#1164

• When a restic command other than init is used with a local repository and the repository directory does not exist, restic creates the directory structure. That's an error, only the init command should create the dir. restic#1167 restic#1182

• Restic now prints stats on all BSD systems (not only on darwin) when SIGINFO is received (usually when ctrl+t is pressed). restic#1203 restic#1082 (comment)

• Since a few releases restic had the ability to write profiling files for memory and CPU usage when debug is enabled. It was discovered that when restic is interrupted (ctrl+c is pressed), the proper shutdown hook is not run. This is now corrected. restic#1191

• A new option --exclude-caches was added that allows excluding cache directories (that are tagged as such). This is a special case of a more generic option --exclude-if-present which excludes a directory if a file with a specific name (and contents) is present. restic#317 restic#1170 restic#1224

• The forget command now has an option --group-by that allows flexible grouping policies. restic#1196

• The date and time restic records for a new backup can now be specified externally by passing --time to the backup command. restic#1205

• The option --compact was added to the snapshots command to get a better overview of the snapshots in a repo. It limits each snapshot to a single line. restic#1218 restic#1223

Important Changes in 0.7.1

• The migrate command for chaning the s3legacy layout to the default layout for s3 backends has been improved: It can now be restarted with restic migrate --force s3_layout and automatically retries operations on error. restic#1073 restic#1075

Small changes

• The local and sftp backends now create the subdirs below data/ on open/init. This way, restic makes sure that they always exist. This is connected to an issue for the sftp server: restic/rest-server#11 (comment) restic#1055 restic#1077 restic#1105

• When no S3 credentials are specified in the environment variables, restic now tries to load credentials from an IAM instance profile when the s3 backend is used. restic#1067 restic#1086

• On Darwin and FreeBSD, restic now prints stats when SIGINFO is received (usually when ctrl+t is pressed). restic#1082

• The dependencies have been updated. restic#1108 restic#1124

• A bug was found (and corrected) in the index rebuilding after prune, which led to indexes which include blobs that were not present in the repo any more. There were already checks in place which detected this situation and aborted with an error message. A new run of either prune or rebuild-index corrected the index files. This is now fixed and a test has been added to detect this. restic#1115

• Errors for chmod() on Unix for filesystems which do not support it (e.g. smb mounted via gvfs) are now ignored. restic#1080 restic#1112

• The semantic for the --tags option to forget and snapshots was clarified: restic#1081 restic#1090

Important Changes in 0.7.0

• New "swift" backend: A new backend for the OpenStack Swift cloud storage protocol has been added, https://wiki.openstack.org/wiki/Swift restic#975 restic#648

• New "b2" backend: A new backend for Backblaze B2 cloud storage service has been added, https://www.backblaze.com restic#512 restic#978

• Improved performance for the find command: Restic recognizes paths it has already checked for the files in question, so the number of backend requests is reduced a lot. restic#989 restic#993

• Improved performance for the fuse mount: Listing directories which contain large files now is significantly faster. restic#998

• The default layout for the s3 backend is now default (instead of s3legacy). Also, there's a new migrate command to convert an existing repo, it can be run like this: restic migrate s3_layout restic#965 restic#1004

• The fuse mount now has two more directories: tags contains a subdir for each tag, which in turn contains only the snapshots that have this tag. The subdir hosts contains a subdir for each host that has a snapshot, and the subdir contains the snapshots for that host. restic#636 restic#1050

Small changes

• For the s3 backend we're back to using the high-level API the s3 client library for uploading data, a few users reported dropped connections (which the library will automatically retry now). restic#1013 restic#1023 restic#1025

• The prune command has been improved and will now remove invalid pack files, for example files that have not been uploaded completely because a backup was interrupted. restic#1029 restic#1036

• restic now tries to detect when an invalid/unknown backend is used and returns an error message. restic#1021 restic#1070

Important Changes in 0.6.1

This is mostly a bugfix release and only contains small changes:

• We've fixed a bug where rebuild-index would corrupt the index when used with the s3 backend together with the default layout. This is not the default setting.

• Backends based on HTTP now allow several idle connections in parallel. This is especially important for the REST backend, which (when used with a local server) may create a lot connections and exhaust available ports quickly. restic#985 restic#986

• Regular status report: We've removed the status report that was printed every 10 seconds when restic is run non-interactively. You can still force reporting the current status by sending a USR1 signal to the process. restic#974

• The build.go now strips the temporary directory used for compilation from the binary. This is the first step in enabling reproducible builds. restic#981

Important Changes in 0.6.0

Consistent forget policy

The forget command was corrected to be more consistent in which snapshots are to be forgotten. It is possible that the new code removes more snapshots than before, so please review what would be deleted by using the --dry-run option.

restic#957 restic#953

Unified repository layout

Up to now the s3 backend used a special repository layout. We've decided to unify the repository layout and implemented the default layout also for the s3 backend. For creating a new repository on s3 with the default layout, use restic -o s3.layout=default init. For further commands the option is not necessary any more, restic will automatically detect the correct layout to use. A future version will switch to the default layout for new repositories.

restic#966 restic#965

Memory and time improvements for the s3 backend

We've updated the library used for accessing s3, switched to using a lower level API and added caching for some requests. This lead to a decrease in memory usage and a great speedup. In addition, we added benchmark functions for all backends, so we can track improvements over time. The Continuous Integration test service we're using (Travis) now runs the s3 backend tests not only against a Minio server, but also against the Amazon s3 live service, so we should be notified of any regressions much sooner.

restic#962 restic#960 restic#946 restic#938 restic#883