Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

better way to enable/disable ANSI escape codes in messages #332

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
30dba04
Makefile: install: force symlinks. README: logcheck filters. profil…
natmaka Jun 9, 2022
5b7018e
psd-resync.service: no ANSI escapes when logging
natmaka Jun 10, 2022
9662c1b
README formatting
natmaka Jun 10, 2022
6be9cd1
README formatting
natmaka Jun 10, 2022
135e1e0
README minor fixes
natmaka Jun 10, 2022
b0747cb
no ANSI escapes when logging
natmaka Jun 9, 2022
138ae49
Merge branch 'graysky2:master' into master
natmaka Jun 12, 2022
2d4fb41
testing
natmaka Jun 17, 2022
de729f6
Merge branch 'master' of github.com:natmaka/profile-sync-daemon
natmaka Jun 17, 2022
0ecb8aa
minor
natmaka Jun 17, 2022
cadcc57
testing
natmaka Jun 21, 2022
e371293
README minor fixes
natmaka Jun 10, 2022
93feed9
testing
natmaka Jun 17, 2022
6aefd89
no ANSI escapes when logging
natmaka Jun 9, 2022
cc2b51a
minor
natmaka Jun 17, 2022
1af88be
testing
natmaka Jun 21, 2022
11d0897
merge
natmaka Jun 21, 2022
2edabab
testing
natmaka Jun 25, 2022
4b787e4
Merge branch 'graysky2:master' into master
natmaka Jan 2, 2023
d748b6f
psd-overlay-helper deletes workdir on umount
graysky2 Dec 11, 2020
e3b8945
README formatting
natmaka Jun 10, 2022
f87e9e9
README formatting
natmaka Jun 10, 2022
8089853
README minor fixes
natmaka Jun 10, 2022
142f56b
testing
natmaka Jun 17, 2022
e9b4b09
minor
natmaka Jun 17, 2022
b9053e1
testing
natmaka Jun 25, 2022
dd0fe21
Move $VOLATILE to dedicated dir in $XDG_RUNTIME_DIR
ranisalt Aug 26, 2022
c6a488c
VOLATILE at the end of the sync must match new value
graysky2 Aug 28, 2022
2a3f577
psd.in: fix grep statement and VERSION
graysky2 Sep 10, 2022
1dfebd2
Per XDG Base Directory Specification, create XDG_RUNTIME_DIR in /tmp …
morgant Jan 1, 2023
f2ca01c
README and MIT: update
graysky2 Jan 1, 2023
3dd2fe1
Fix `surf` support (rm wrong '/')
al-mission-2016 May 5, 2022
9cbc9f6
Add vscode dir
Apr 29, 2022
c576a55
Merge branch 'master' of github.com:natmaka/profile-sync-daemon
natmaka Jan 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Profile-sync-daemon (psd) is a tiny pseudo-daemon designed to manage your browser's profile in tmpfs and to periodically sync it back to your physical disc (HDD/SSD). This is accomplished via a symlinking step and an innovative use of rsync to maintain back-up and synchronization between the two. One of the major design goals of psd is a completely transparent user experience.

## Good Advice
Always backup your browser profile(s) before using psd for the first time.
Before using psd for the first time **stop (quit, exit) your browser then backup its profile(s)**.

## Users of eCryptFS
User of eCryptFS are encouraged not to use psd unless willing to help troubleshoot suspected browser corruption. See [#158](https://github.com/graysky2/profile-sync-daemon/issues/158).
Expand Down Expand Up @@ -55,9 +55,13 @@ If you are interested in packaging psd for your favorite distro, please contact
## logcheck
Using logcheck? Here are some ways to filter out log lines:
```regexp
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ profile-sync-daemon\[[0-9]+]\]: .*(google-chrome|firefox) (re|un)sync successful
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ profile-sync-daemon\[[0-9]+\]: psd startup check successful$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ profile-sync-daemon\[[0-9]+\]:[[:space:]]+[^[:space:]]+ (re|un)sync successful
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[[0-9]+\]: Started Timer for profile-sync-daemon
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ profile-sync-daemon\[[0-9]+\]: psd startup check successful$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ profile-sync-daemon\[[0-9]+\]:[[:space:]]+[^[:space:]]+ (re|un)sync successful
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[[0-9]+\]: psd(-resync)?\.service: Consumed [0-9\.]+s CPU time\.$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[[0-9]+\]: psd(-resync)\.timer: Succeeded\.$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ systemd\[[0-9]+\]: run-user-[0-9]+-[^[:space:]]+: Succeeded\.$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sudo:[[:space:]]+guest : PWD=/home/guest ; USER=root ; COMMAND=/usr/bin/psd-overlay-helper
```
8 changes: 3 additions & 5 deletions common/profile-sync-daemon.in
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,8 @@

# needed for debian >=8.x
PATH=$PATH:/sbin

if [[ -n "$LAUNCHED_BY_SYSTEMD" ]] ; then
# This script was invoked by systemd
# Thx to notes-jj: https://serverfault.com/questions/926349/systemd-tell-if-script-was-run-by-systemd-or-by-user
if [[ ! -t 1 ]] ; then
# Thx D. Clayton: https://stackoverflow.com/questions/911168/how-can-i-detect-if-my-shell-script-is-running-through-a-pipe
# No 'echo' ANSI escape codes
BLD=''
RED=''
Expand Down Expand Up @@ -515,7 +513,7 @@ do_sync_for() {
ln -s /dev/null "$DIR"
fi

# sync the tmpfs targets to the disc
# sync the tmpfs target to the non-tmpfs filesystem
if [[ -e "$DIR"/.flagged ]]; then
REPORT="resync"
if [[ $OLFS -eq 1 ]]; then
Expand Down
10 changes: 5 additions & 5 deletions doc/psd-overlay-helper.1
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
.TH psd-overlay-helper 1 "13 June 2016" "" ""
.SH NAME
\fBpsd-overlay-helper \fP- Script to use overlay file system for profile-sync-daemon.
\fBpsd-overlay-helper \fP- Script enabling the "profile-sync-daemon" to use an 'overlay' file system in order to reduce the amount of writing operations.
\fB
.SH DESCRIPTION
Profile-sync-daemon (psd) is a tiny pseudo-daemon designed to manage browser profile/profiles in tmpfs and to periodically sync back to the physical disc (HDD/SSD). This is accomplished by an innovative use of rsync to maintain synchronization between a tmpfs copy and media-bound backup of the browser profile/profiles. Additionally, psd features several crash recovery features.
Profile-sync-daemon (psd) is a tiny pseudo-daemon designed to manage browser profile(s) in a 'tmpfs' file system and to periodically sync back to the main file system (HDD/SSD). This is accomplished by an innovative use of rsync to maintain synchronization between a copy on a tmpfs and media-bound backup of the browser profile(s). Additionally, psd hass several crash-recovery features.

The psd-overlay-helper script is used internally to enable the usage of the overlay file system. It should not be called by the user directly.
The psd-overlay-helper script is used internally to enable the usage of the 'overlay' file system. It should not be called by the user directly.
.SH CONTRIBUTE
Users wishing to contribute to this project should fork it and send a pull request. Source is freely available on the project page linked below.
Users wishing to contribute to this project should fork it and send a pull request. The source code is freely available on the project page linked below.
.SH ONLINE
.IP \(bu 3
Project page: https://github.com/graysky2/profile-sync-daemon
.IP \(bu 3
Wiki page: https://wiki.archlinux.org/index.php/Profile-sync-daemon
Wiki page: https://wiki.archlinux.org/title/Profile-sync-daemon
.SH AUTHOR
graysky (graysky AT archlinux DOT us)
40 changes: 21 additions & 19 deletions doc/psd.1
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
.\" Text automatically generated by txt2man
.TH profile-sync-daemon "22 September 2020" "" ""
.SH NAME
\fBprofile-sync-daemon \fP- Symlinks and syncs browser profiles to RAM (tmpfs) thus reducing HDD/SSD calls and speeding up browsers.
\fBprofile-sync-daemon \fP- Symlinks and syncs browser profile(s) to RAM (thanks to tmpfs), thus reducing HDD/SSD activity and speeding-up browsers.
\fB
.SH DESCRIPTION
Profile-sync-daemon (psd) is a tiny pseudo-daemon designed to manage browser profile/profiles in tmpfs and to periodically sync back to the physical disc (HDD/SSD). This is accomplished by an innovative use of rsync to maintain synchronization between a tmpfs copy and media-bound backup of the browser profile/profiles. Additionally, psd features several crash-recovery features.
Profile-sync-daemon (psd) is a tiny pseudo-daemon (a systemd service) designed to manage browser profile(s) in a 'tmpfs' file system and to periodically sync back to the main file system (HDD/SSD). This is accomplished by an innovative use of rsync to maintain synchronization between a copy on a tmpfs and media-bound backup of the browser profile(s). Additionally, psd hass several crash-recovery features.
.PP
Design goals of psd:
.RS
Expand All @@ -16,20 +16,22 @@ Reduced wear to physical discs (particularly SSDs).
Speed.
.RE
.PP
Since the profile/profiles, browser cache*, etc. are relocated into tmpfs (RAM disk), the corresponding I/O associated with using the browser is also redirected from the physical disc to the RAM, thus reducing wear to the physical disc and improving browser responsiveness.
Since the profile(s), browser cache*, etc. are relocated into tmpfs (sort-of RAM disk), the I/O corresponding to the browser activity is redirected from the physical disc to the RAM, thus reducing wear to the physical disc and improving responsiveness.
.PP
*Note that some browsers such as Chrome/Chromium, Firefox (since v21), Midori, and Rekonq actually keep their cache directories separate from their browser profile directory. It is not within the scope of profile-sync-daemon to modify this behavior; users wishing to relocate this directory, may refer to the following url for several work-arounds: https://wiki.archlinux.org/index.php/Chromium_Tips_and_Tweaks#Cache_in_tmpfs
For the best overall performance using tmpfs along with swap space is recommended.
.PP
*Note that some browsers such as Chrome/Chromium, Firefox (since v21), Midori, and Rekonq actually keep their cache directories separate from their profile directory. Modifying this behavior is outside the scope of profile-sync-daemon ; users wishing to relocate this directory may refer to the following URL for several work-arounds: https://wiki.archlinux.org/title/Chromium#Cache_in_tmpfs
.SH SETUP
$XDG_CONFIG_HOME/psd/psd.conf (referred to hereafter as "the config file") contains all user managed settings.
$XDG_CONFIG_HOME/psd/psd.conf (referred to hereafter as "the config file") contains all user-managed settings.
.PP
NOTE: edits made to the config file while psd is active will be applied only after the service has been restarted.
.RS
.IP \(bu 3
Optionally enable the use of overlayfs to improve sync speed and to use a smaller memory footprint. Do this in the USE_OVERLAYFS variable. The user will require no password sudo rights to /usr/bin/psd-overlay-helper to use this option and the kernel must support overlayfs version 22 or higher. See the FAQ below for additional details.
Optionally enable the use of overlayfs to improve sync speed and to use a smaller memory footprint. Do this through the USE_OVERLAYFS variable. The 'sudo' utility must let the user account running the browser invoke /usr/bin/psd-overlay-helper as root without prompting for the password and the kernel must support overlayfs version 22 or higher. See the FAQ below for additional details.
.IP \(bu 3
Optionally have psd resync to the filesystem prior to a sleep call. This can help in the event that the system does not properly wake up. Do this in the USE_SUSPSYNC variable.
Optionally have psd resync to the filesystem prior to a power-management transition (suspend, hibernate...). This can help in the event that the system does not properly wake up. Do this in the USE_SUSPSYNC variable.
.IP \(bu 3
Optionally define which browsers are to be managed in the BROWSERS array. If none are defined, the default is all detected browsers.
Optionally define in the BROWSERS array which browser(s) is/are to be managed. If none is defined, the default is all detected browsers.
.IP \(bu 3
Optionally disable the use of crash-recovery snapshots (not recommended). Do this in the USE_BACKUPS variable.
.IP \(bu 3
Expand All @@ -39,17 +41,17 @@ Optionally define the number of crash-recovery snapshots to keep. Do this in the
NOTE: occasionally, updates/changes are made to the default config file (/usr/share/psd/psd.conf) upstream. The user copy ($XDG_CONFIG_HOME/psd/psd.conf) will need to be diffed against it.
.SH RUNNING PSD
.SS PREVIEW MODE
The preview option can be called to show users exactly what psd will do/is doing based on the entries in the config file. It will also provide useful information such as profile size, paths, and if any recovery snapshots have been created.
The 'preview' option can be used to show exactly what psd will do/is doing based on the entries in the config file. It will also provide useful information such as profile size, paths, and (if any) existing recovery snapshot(s).
.PP
.nf
.fam C
$ psd p

Profile-sync-daemon v6.39
Profile-sync-daemon v6.44

systemd service: active
resync-timer: active
sync on sleep: enabled
sync on sleep: disabled
use overlayfs: enabled

Psd will manage the following per /home/facade/.config/psd/.psd.conf settings:
Expand Down Expand Up @@ -78,7 +80,7 @@ The preview option can be called to show users exactly what psd will do/is doing
.fam T
.fi
.SS START AND STOP PSD
Psd ships with a systemd user service to start or stop it (psd.service). Additionally, a provided resync-timer will run an hourly resync from tmpfs back to the disk. The resync-timer is started automatically with psd.service so there is no need to start the timer; only start psd.service.
Psd ships with a systemd user service to start or stop it (psd.service). Additionally, a provided resync-timer will run an hourly resync from tmpfs back to the main file system (HDD/SSD). The resync-timer is started automatically with psd.service so there is no need to start the timer: starting psd.service is sufficient.
.PP
.nf
.fam C
Expand All @@ -98,7 +100,7 @@ The clean mode will delete ALL recovery snapshots that have accumulated. Only ru
.fam C
$ psd c

Profile-sync-daemon v6.39
Profile-sync-daemon v6.44

Deleting 2 crashrecovery dirs for profile /home/facade/.config/chromium
/home/facade/.config/chromium-backup-crashrecovery-20200423_171359
Expand Down Expand Up @@ -193,15 +195,15 @@ Since psd is just a bash script with a systemd service, it should run on any fla
.SH FAQ
Q1: What is overlayfs mode?
.PP
A1: Overlayfs is a simple union filesystem mainlined in the Linux kernel version 3.18.0. When used with psd, a reduced memory footprint and faster sync operations can be realized. The magic is in how the overlay mount only writes out data that has changed rather than the entire profile. The same recovery features psd uses in its default mode are also active when running in overlayfs mode.
A1: Overlayfs is a simple union filesystem type mainlined in the Linux kernel version 3.18.0. When used with psd, a reduced memory footprint and faster sync operations can be realized. The magic is in how the overlay mount only writes out data that has changed rather than the entire profile. The same recovery features psd uses in its default mode are also active when running in overlayfs mode.
.PP
See the example in the PREVIEW MODE section above which shows a system using overlayfs to illustrate the typical memory savings. Note the "overlayfs size" report compared to the total "profile size" report for each profile. Be aware that these numbers will change depending on just how much new data is written to the profile, but in common use cases, the overlayfs size will always be less than the profile size.
.PP
Q2: How do I enable overlayfs mode?
.PP
A2: First, be sure psd is not active or else any changes to the config file will be ignored until it is restarted. Overlayfs mode is enabled with the USE_OVERLAYFS= variable which should be set to "yes" in the config file. Psd will automatically detect the overlayfs version available to the kernel if it is configured to use one of them. It is recommended to run psd in preview mode to verify that the system can in fact use overlayfs.
.PP
Users wanting to use overlayfs mode MUST have sudo rights without password prompt to /usr/bin/psd-overlay-helper or global sudo rights without password prompt. If the user does not have global rights, add the following line to /etc/sudoers after any other lines defining sudo access. It is recommended to use /usr/bin/visudo as root to set this up:
The user account using the browser MUST have sudo rights without password prompt to /usr/bin/psd-overlay-helper or global sudo rights without password prompt. If this account (hereby named 'foo') does not have global rights, add the following line to /etc/sudoers after any other lines defining sudo access. It is recommended to use /usr/bin/visudo as root to set this up:
.PP
.nf
.fam C
Expand All @@ -211,15 +213,15 @@ Users wanting to use overlayfs mode MUST have sudo rights without password promp
.fi
Q3: Why do I have another browser profile directory "foo-back-ovfs" when I enable overlayfs?
.PP
A3: The way overlayfs works is to mount a read-only base copy (so-called lower dir) of the profile, and manage the new data on top of that. In order to avoid resyncing to the read-only filesystem, a copy is used instead. So using overlayfs is a trade-off: faster initial sync times and less memory usage vs. disk space in the home dir.
A3: The overlayfs works by mounting a read-only base copy (so-called "lower dir") of the profile, and manage the new data on top of that. In order to avoid resyncing to the read-only filesystem, a copy is used instead. So using overlayfs is a trade-off: faster initial sync times and less memory usage vs. disk space in the home dir.
.PP
Q4: I need more memory to accommodate my profile/profiles in /run/user/xxxx. How can I allocate more?
Q4: I need more memory to accommodate my profile(s) in /run/user/xxxx. How can I allocate more?
.PP
A4: The standard way of controlling the size of /run/user/ is the RuntimeDirectorySize directive in logind.conf (see the man page for logind.conf for more). By default, 10% of physical memory is used but one can increase it safely. Remember that tmpfs only consumes what is actually used; the number specified here is just a maximum allowed.
A4: The standard way of controlling the size of /run/user/ is the RuntimeDirectorySize directive in logind.conf (see the man page for logind.conf). By default, 10% of physical memory is used but one can increase it (beware: if you oversize your tmpfs instances the machine will deadlock, read about this at https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html ). Remember that tmpfs only consumes what is actually used; the number specified here is just a maximum allowed.
.PP
Q5: My system crashed for some reason and psd didn't sync back. What do I do?
.PP
A5: The "last good" backup of the browser profile/profiles should be the filesystem. Upon restarting psd (on a reboot for example), a check is performed to see if the symlink to the tmpfs copy of the profile is invalid. If it is invalid, psd will snapshot the "last good" backup before it rotates it back into place. This is more for a sanity check that psd did no harm and that any data loss was a function of something else.
A5: The "last good" backup of the browser profile/profiles should be on the main file system (HDD/SSD). Upon restarting psd (on a reboot for example), a check is performed to see if the symlink to the tmpfs copy of the profile is invalid. If it is invalid, psd will snapshot the "last good" backup before rotating it back into place. This is more for a sanity check that psd did no harm and that any data loss was a done by something else.
.PP
Q6: Where can I find this snapshot?
.PP
Expand Down
1 change: 0 additions & 1 deletion init/psd-resync.service
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ BindsTo=psd.service
[Service]
Type=oneshot
ExecStart=/usr/bin/profile-sync-daemon resync
Environment=LAUNCHED_BY_SYSTEMD=1

[Install]
WantedBy=default.target
1 change: 0 additions & 1 deletion init/psd.service
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/profile-sync-daemon startup
ExecStop=/usr/bin/profile-sync-daemon unsync
Environment=LAUNCHED_BY_SYSTEMD=1

[Install]
WantedBy=default.target