#136 Duply 2.4.x is not compatible with duplicity 2.x

[tengel]

Duply 2.4.x is not compatible with duplicity 2.x - Arch just upgraded their duplicity package to 2.0.2 which results in invalid usage (commandline options) between the two tools. Ignoring all the extraneous output, the "backup" error presents as:

duplicity: error: argument {cleanup,cl,collection-status,st,full,fb,incremental,inc,ib,list-current-files,ls,remove-all-but-n-full,ra,remove-all-inc-of-but-n-full,ri,remove-older-than,ro,restore,rb,verify,vb}: invalid choice: '/home/tengel' (choose from 'cleanup', 'cl', 'collection-status', 'st', 'full', 'fb', 'incremental', 'inc', 'ib', 'list-current-files', 'ls', 'remove-all-but-n-full', 'ra', 'remove-all-inc-of-but-n-full', 'ri', 'remove-older-than', 'ro', 'restore', 'rb', 'verify', 'vb')

...and the "status" error presents as:

ComCommandLineError: Source should be url, not directory.  Got '.duplicity-ignore' instead.

Duplicity 2.x changelog about the params: https://gitlab.com/duplicity/duplicity/-/issues/152

[tengel]

There are apparently (I've downgraded duplicity so cannot confirm) other issues in the autodetection of full or incrementals with 2.x, a few links:

• https://bugs.archlinux.org/task/79448?project=1&string=duplicity
• https://gitlab.com/duplicity/duplicity/-/issues/733

[ede]

yeah duplicity 2.x had a ruff start. this duplicity issue is planned to be fixed with next duplicity v2.1 release. please check then and report back!

Thanks!.. ede

[ede]

duplicity v2.1 is released. would you mind to doublecheck? ..ede

[tengel]

Will do once Arch team generates (testing) packages for it, they're not available just yet (I just submitted an out-of-date report to the packagers to notify),

[ede]

the status error looks fishy. can you please provide your profile config file? obfuscate sensitive strings in it before uploading it! ..ede

[tengel]

Sure thing, this is a well-tenured profile (running for years?) so no recent changes on my side, it's not very complex as I hand it off to rclone instead of using one of the duplicity internal backends: (where "okin" is the name of the laptop/configs)

GPG_KEY='XXXXXXXX'
GPG_PW='xxxxxxxx'
GPG_PW_SIGN="${GPG_PW}"
GPG_OPTS='--pinentry-mode loopback --no-tty'
TARGET='rclone://okin:'
SOURCE='/home/tengel'
FILENAME='.duplicity-ignore'
DUPL_PARAMS="$DUPL_PARAMS --exclude-if-present '$FILENAME'"
MAX_AGE=2M
MAX_FULL_BACKUPS=2
MAX_FULLS_WITH_INCRS=1

The corresponding exclude file is just a generic set of patterns which look like:

- /home/*/Downloads
- /home/*/VirtualBox**
...

...and the script that runs it is very basic outside the bashisms:

...
echo "" >> ${LOGFILE}
duply okin backup 1>>${LOGFILE} 2>&1
duply okin status 1>>${LOGFILE} 2>&1
echo "" >> ${LOGFILE}
...

[tengel]

(duplicity 2.1.0)

While it now works, there's an error in the output for "backup" which indicates to me duplicity is not receiving the expected input now; duply okin backup has this embedded (new output not there before with duplicity 1.2.3):

--- Start running command BKP at 2023-09-02 08:28:21.835 ---
No valid action command found. Will imply 'backup' because a path source was given and target is a url location.
...

When I run bash -x duply okin backup I do see that the commandline is simply missing any action, such as the word "backup" as expected by the new duplicity. If I take the commandline and simply change duplicity --name duply_okin ... to duplicity backup --name duply_okin ... then it's happy.

---

The status call is still broken; it appears that the option --exclude-if-present is no longer available or possible when used with the "duplicity collection-status" command based on some quick debugging. Running bash -x duply oking status we tease out:

duplicity collection-status \
  --name duply_okin --encrypt-key XXXXXXXX \
   --sign-key XXXXXXXX --verbosity 4 \
   --gpg-options '--pinentry-mode loopback --no-tty' \
   --exclude-if-present .duplicity-ignore rclone://xxxxxxxx:

Running duplicity collection-status --help does not show "exclude-if-present" a valid option; taking the above command and removing it works as expected to give me the status of the duply profile backups.

---

Fixing the "backup" command looks easy I think, as all it took was this:

$ diff -uN /usr/bin/duply ./duplytest 
--- /usr/bin/duply  2023-09-02 08:12:04.000000000 -0500
+++ ./duplytest 2023-09-02 08:43:05.149764244 -0500
@@ -2605,7 +2605,7 @@
     ( run_script "$script" )
     ;;
   'bkp')
-    duplify -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
+    duplify backup -- "${dupl_opts[@]}" $EXCLUDE_PARAM "$EXCLUDE" \
           "$SOURCE" "$BACKEND_URL"
     ;;
   'incr')

Fixing the "status" problem though is harder as those are all shared DUPL_PARAMS from the config applied to all operations. It's possible this manifests in other subcommands (purge, etc.) which I haven't tried yet as I don't want to risk my backup data until we sort things out.

$  for subc in backup incr full \
   verify list-current-files cleanup \
   remove-older-than remove-all-but-n-full \
   remove-all-inc-of-but-n-full \
   collection-status; do echo -n "${subc}: "; \
   duplicity ${subc} --help | grep -m1 -c \
   exclude-if-present; done;

backup: 1
incr: 1
full: 1
verify: 1
list-current-files: 0
cleanup: 0
remove-older-than: 0
remove-all-but-n-full: 0
remove-all-inc-of-but-n-full: 0
collection-status: 0

If we're to believe the help output as a source of truth, this option is now invalid for a number of duplicity subcommands so cannot be passed in generically...

[ede]

thanks for the detailed checkup :)

quick side notes:
- to check generated command lines use the duply parameter --preview
- --pinentry-mode loopback --no-tty are added by duplicity already since quite a while and can be ommitted

1. wrt. command backup
   we will need to stay back compatible for some while still. so just adding it w/o checking which duplicity we are on is no option. fortunately version parsing code is already in duplicity so it'll be a quick fix. until then the notice is not blocking anything.
   btw. running duplicity w/o command is still valid, it just prints that log notice now.

2. wrt. --exclude-if-present
   you are totally right. the command line parser was reimplemented in duplicity 2.x because python retired the optparse module used originally. the new argparse module does create the help as well, so it's output can be taken as "source of truth".
   i will have to filter the option accordingly which is already done for other exclusion options.

thanks for reporting/checking. i will have a look next week. please be patient. Thanks!

[tengel]

thanks for reporting/checking. i will have a look next week. please be patient. Thanks!

Thanks, no problem for me right away - this specific exclude feature was/is a part of the defaults, I searched my source data and don't actually use .duplicity-ignore anywhere so it was simple enough for me to comment out those config lines and be working OK along with understanding the other items above. My personal usage is very straightforward and simple (not using any other commandline options, e.g.) so no stress from me. :)

[ede]

good to hear. quick suggestion though. if you feel backups are crucial you should do a restore or verify now to make sure these still work as expected too ;)

[lds]

ouhh big problems in 2.x duplicity

duply local fetch etc/passwd /tmp/file
CommandLineError: Invalid or missing action command and cannot be implied from the given arguments. ['--file-to-restore', 'etc/passwd', 'boto3+s3://xxxxxxxx',   '/tmp/file']

--file-to-restore is deprecated in 2.1.0 and 'action' is mandatory (restore)

this duplicity command works:

duplicity restore --name test --no-encryption --verbosity 4 --s3-region-name=xxxx --s3-endpoint-url https://xxxxxxxxxx --restore-time now --path-to-restore etc/passwd boto3+s3://xxxxxx /tmp/file

[ede]

hey lds,

i am aware and working on it. actually the action command is not really mandatory but '--file-to-restore' is no valid option anymore. changed to --path-to-restore hence it is mistaken as the command as the error states.

anyway, please stick to duplicity 1.2.3 for now until i fixed up all the new inconsistencies.

Thanks! ..ede

[ede]

hello tengel, lds,

there is a new devel version https://duply.net/tmp/duply.sh . please try that one and come back with results. please test against duplicity 2.1 or later.
a list with changes can be found in the top of the file or in the online source code https://duply.net/Code#Latest_Development_Snapshot .

Thanks!.. ede

[lds]

Hello ede

It seems correct to me despite cosmetic errors on fetch

duply local fetch etc/hosts /tmp/hosts
Start duply v2.5.0dev, time is 2023-09-21 16:55:46.
Using profile '/root/.duply/local'.
Using installed duplicity version 2.1.1, gpg 2.2.19 (Home: /root/.gnupg), awk 'mawk 1.3.4 20200120', grep 'grep (GNU grep) 3.4', bash '5.0.17(1)-release (x86_64-pc-linux-gnu)'.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - En/Decryption skipped. (GPG='disabled')

--- Start running command FETCH at 2023-09-21 16:55:47.302 ---
No valid action command found. Will imply 'restore' because url source was given and target is a local path.
--- Finished state OK at 2023-09-21 16:55:53.247 - Runtime 00:00:05.944 ---

backup verify purge purgeFull purge-full work very well

great job :)

[lds]

it seems that the fulls are no longer triggered
I had to force the full backup with duply local full

my conf

MAX_FULL_BACKUPS=2
MAX_AGE=4W
MAX_FULLBKP_AGE=2W
DUPL_PARAMS="$DUPL_PARAMS --s3-endpoint-url http://xxxxxx --s3-region-name xxxx --full-if-older-than $MAX_FULLBKP_AGE"

[lds]

it may be because the rotation was not done correctly due to problems linked to the duplicity version

by forcing two fulls I was able to return to a normal state

[ede]

triggered how? if you run backup on an empty target it creates a full.

wrt.

cosmetic errors on fetch & restore

please try this snapshot https://duply.net/tmp/duply.sh

[lds]

ok cosmetic errors have disappeared :)
thanks

[Rhomeo]

With the above snapshot, backup works without the "no valid command" warning. status throws an error, though:

# duply mail status
Start duply v2.5.0dev, time is 2023-09-22 08:12:29.
Using profile '/etc/duply/mail'.
Using installed duplicity version 2.1.1, python 3.9.2 (/opt/ddtbackup-py3/venv3/bin/python3) 'PYTHONPATH=:/usr/lib/python39.zip:/usr/lib/python3.9:/usr/lib/python3.9/lib-dynload:/opt/ddtbackup-py3/venv3/lib/python3.9/site-packages:/usr/local/lib/python3.9/dist-packages:/usr/lib/python3/dist-packages', gpg 2.2.27 (Home: /root/.gnupg), awk 'GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)', grep 'grep (GNU grep) 3.6', bash '5.1.4(1)-release (x86_64-pc-linux-gnu)'.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - En/Decryption skipped. (GPG='disabled')

--- Start running command STATUS at 2023-09-22 08:12:29.435 ---
CommandLineError: Source should be url, not directory.  Got '100' instead.
Enter 'duplicity --help' for help screen.
2023-09-22 08:12:29.606 Task 'STATUS' failed with exit code '23'.
--- Finished state FAILED 'code 23' at 2023-09-22 08:12:29.606 - Runtime 00:00:00.170 ---

The culprit seems to be that I set VOLSIZE=100 in duply's config:

# duply mail status --preview
[...]
--- Start running command STATUS at 2023-09-22 08:12:42.979 ---
TMPDIR='/tmp'      duplicity  collection-status --name duply_mail --no-encryption  --verbosity '4'    --full-if-older-than 1M --volsize 100     'file:///var/tmp/backup.mnt/Mail'

And indeed, removing --volsize 100 from the command shown by --preview is enough to make it work.

[ede]

hey Rhomeo,

can you please recheck with the new snapshot https://duply.net/tmp/duply.sh ?

Thanks! ..ede

[Rhomeo]

status works fine for me with the new snapshot.
Thanks for the quick fix!

[tengel]

@ede, testing the latest dev snapshot (Sat Sep 23 02:09:07 PM UTC 2023) looks great except one thing - I've tested: verify, list, full, purgeIncr, purgeFull, backup, status, restore, fetch, cleanup. (verify found a corrupt incremental the day before this bug report, like @lds had).

The verifyPath seems to not be happy with including the exclude file. Using something like

duply okin verifyPath Pictures/ ./Pictures/

Results in the below (where /home/*/Downloads is a literal line item in the exclude file):

Reading globbing filelist /home/tengel/.duply/okin/exclude
Fatal Error: The file specification
    /home/*/Downloads
cannot match any files in the base directory
    Pictures

The --preview output shows --exclude-filelist /home/tengel/.duply/okin/exclude being passed to duplicity verify, so if I just edit the commandline to remove that option then the verify works as expected. Redacted for GPG info:

TMPDIR='/tmp'  PASSPHRASE=xxx SIGN_PASSPHRASE=xxx duplicity \
   verify --name duply_okin --encrypt-key xxx --sign-key xxx \
   --verbosity '4' --path-to-restore Pictures/ \
   'rclone://zzzz:' ./Pictures/

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Sep 23 08:10:01 2023
Verify complete: 1 file(s) compared, 0 difference(s) found.

[ede]

@tengel, looks like a duplicity issue. verify as such accepts in/excludes. would you mind creating an issue on https://gitlab.com/duplicity/duplicity/-/issues so we might dig in and find out what the problem is.

stabbing i the dark i'd guess the combination of relative source path and absolute exclusion confuses duplicity somehow.

[tengel]

@ede understood thank you - I personally do not use this feature so it may have always been a problem, my intent is/was to help test the new duply dev work for regressions which looks great. I don't think this duplicity verify problem should stop us from releasing a new duply version for packagers to distribute etc. thanks! :)

[ede]

agreed. will do so when time permits. thanks All of you!