Download script and more test cases

- Added download script.
- Document more test cases, how to test and expected behavior.

Signed-off-by: Fon E. Noel NFEBE <[email protected]>

Author: nfebe

.gitignore

@@ -1,4 +1,3 @@
-test-tree/challenging-names
-test-tree/apod
+test-tree/
 __pycache__
 venv

README.md

@@ -10,24 +10,6 @@ protocol.
 For more context, see [this
 discussion](https://chat.opentechstrategies.com/#narrow/stream/73-Permanent/topic/QA/near/155527).
 
-## Testing scope
-
-The scope of testing here verifies the possibility of correctly uploading and downloading
- a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
- which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
-
-### What file types and scenarios are covered?
-
-- Text and png images with obscure names generated via [generate-tree.py](generate-tree.py)
-- Images in `.jpg` and `.png` format downloaded from [APOD](https://apod.nasa.gov/apod) via [apod-downloader.py](apod-downloader.py)
-- Compressed files in `.zip` and `.tar`
-- Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
-- Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
-
-### What file types and scenarios are left out?
-
-Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
-
 ## Usage
 
 You would have to install the python requirements used in this repo.
@@ -50,7 +32,23 @@ Run `./upload-test.py test-tree/apod --archive-path "/archives/rclone QA 1 (0a0j
 
 *That said, the archive path used in the sample command would have to be updated to match some archive created on Permanent.org*
 
-### Challenging Names
+## Testing scope
+
+The scope of testing here verifies the possibility of correctly uploading and downloading
+ a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
+ which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
+
+### What file types are tested?
+
+- Text and png images with obscure names generated via [generate-tree.py](generate-tree.py)
+- Images in `.jpg` and `.png` format downloaded from [APOD](https://apod.nasa.gov/apod) via [apod-downloader.py](apod-downloader.py)
+- Compressed files in `.zip` and `.tar`
+- Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
+- Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
+
+### What test cases are covered?
+
+#### Challenging Names
 
 Run `./generate-tree.py` to generate test data, which will be placed
 in a new subdirectory named `test-tree/challenging-names`.
@@ -62,6 +60,61 @@ first, of course).  See the long comment at the top of
 [upload-test.sh](upload-test.sh) for information about what it's
 trying to do and what problems we know about so far.
 
+#### Duplicates
+
+A duplicate is a file/folder with exactly the same name. Of course this is not possible on regular file systems but Permanent does support it.
+There is a deduplication algorithm from Permanent that the `sftp-service` relies to ensure that files with identical names on Permanent won't be 
+ be considered as the same on regular file systems.
+
+##### How test duplicate
+
+- Create a folder in the test archive of the remote (permanent.org or permanent.dev depending on your test target) e.g 'duplicates'.
+- Upload at least two copies of multiple identical files into the folder `duplicates` for example (`file.txt`, `file.txt`, `file.txt` and `photo.png`, `photo.png` ...)
+- Run the download test script against the duplicate folder. In this case:
+
+```
+`./test-download.py --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/" --remote-dir "duplicates"`
+```
+##### Expected results
+
+- Check download folder and ensure that results looks like:
+
+*Result from `tree` program*
+```
+├── file (1).txt
+├── file (2).txt
+├── file.txt
+├── Photo (1).png
+└── Photo.png
+
+0 directories, 5 files
+```
+##### Multiple Identical Uploads
+
+This test case captures what happens if you sync the same path with unchanged content multiples times.
+
+##### How test identical uploads
+
+- Generate challenging names if not generated earlier, see [Challenging Names](#challenging-names)
+
+Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-414 --log-file=duplicate-upload-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+*Notice the use of the `--only` flag which specifies only files containing the number `414` should be uploaded, you can change this number to follow a string pattern in the generated challenging names but the provide example works just fine.*
+
+##### Expected results
+
+- `rclone` should report `Sizes identical` and `Unchanged skipping`
+
+```
+2023/03/29 14:54:00 DEBUG : 002-dupe-test.txt: Sizes identical
+2023/03/29 14:54:00 DEBUG : 002-dupe-test.txt: Unchanged skipping
+```
+- No duplicates should be be seen on Permanent UI.
+
+### What file types and scenarios are left out?
+
+Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
+
 ## Web Interface
 
 For prod, just go to the site as per usual.  For dev, go to

generate-tree.py

@@ -27,7 +27,6 @@
 """
 
 import os
-import sys
 import shutil
 
 test_tree_top = "test-tree/challenging-names"

test-download.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+
+from utils import log, parse_cli, rclone_download
+
+DOWNLOAD_DIR = "test-tree/downloads"
+
+import os
+
+def main():
+    # Do some initial setup, parse cli, etc
+    cli = parse_cli()
+    rclone_download(os.path.join(DOWNLOAD_DIR, cli.remote_dir), cli.remote_dir)
+
+if __name__ == "__main__":
+    main()

upload-test.py

@@ -1,14 +1,13 @@
 #!/usr/bin/env python3
 
 from utils import log, parse_cli, rclone_upload
+import os
+import sys
 
 CHALLENGING_NAMES_DIR = "test-tree/challenging-names"
 APOD_DIR = "test-tree/apod"
 MISC_DIR = "test-tree/misc"
 
-import os
-import sys
-
 gentree = __import__("generate-tree")
 
 def omit_p(fname, omit_list):
@@ -38,7 +37,6 @@ def skip_p(fname, cli):
 
     return False
 
-
 def main():
     # Do some initial setup, parse cli, etc
     cli = parse_cli()

utils.py

@@ -1,50 +1,42 @@
 #!/usr/bin/env python3
+import os
+import argparse
+import datetime
+import subprocess
+from pathlib import Path
+import __main__
+
 RCLONE_REMOTE = "permanent"
 ARCHIVE_PATH = ""
 TIMEOUT = 5 * 60
 LOG_FILE = "log.txt"
 
-import argparse
-import datetime
-import subprocess
-
 RCLONE = subprocess.check_output("which rclone", shell=True).strip().decode("utf-8")
 
+
 def which(cmd):
     """Return path to cmd"""
     return subprocess.check_output(f"which {cmd}", shell=True).strip().decode("utf-8")
 
+
 def log(msg, echo=True):
     """Print message to log file (and screen if echo is True)"""
     if echo:
         print(msg)
-    with open(LOG_FILE, "a") as fh:
+    with open(LOG_FILE, "a", encoding="utf-8") as fh:
         fh.write(msg)
         fh.write("\n")
 
+
 def slurp_if_e(fname):
     if os.path.exists(fname):
-        with open(fname) as fh:
+        with open(fname, encoding="utf-8") as fh:
             return fh.read()
     return ""
 
-def rclone_upload(fname, remote_dir, timeout: int = TIMEOUT):
-    args = []
-    if timeout > 0:
-        args.extend(["timeout", str(timeout)])
-
-    args.extend(
-        [
-            RCLONE,
-            "copy",
-            "-vv",
-            "--size-only",  # server doesn't do mtime
-            "--sftp-set-modtime=false",  # server doesn't do mtime
-            fname,
-            f"{RCLONE_REMOTE}:{ARCHIVE_PATH}{remote_dir}",
-        ]
-    )
 
+def run(fname, args):
+    """Execute rclone command pass in args on path fname"""
     start_time = datetime.datetime.now()
     try:
         process = subprocess.Popen(
@@ -66,24 +58,70 @@ def rclone_upload(fname, remote_dir, timeout: int = TIMEOUT):
 
     return process
 
+
+def rclone_upload(fname, remote_dir, timeout: int = TIMEOUT):
+    """Upload to rlcone"""
+    args = []
+    if timeout > 0:
+        args.extend(["timeout", str(timeout)])
+
+    args.extend(
+        [
+            RCLONE,
+            "copy",
+            "-vv",
+            "--size-only",  # server doesn't do mtime
+            "--sftp-set-modtime=false",  # server doesn't do mtime
+            fname,
+            f"{RCLONE_REMOTE}:{ARCHIVE_PATH}{remote_dir}",
+        ]
+    )
+    return run(fname, args)
+
+
+def rclone_download(fname, remote_dir, timeout: int = TIMEOUT):
+    """Download from rclone"""
+    args = []
+    if timeout > 0:
+        args.extend(["timeout", str(timeout)])
+
+    args.extend(
+        [
+            RCLONE,
+            "copy",
+            "-vv",
+            "--size-only",  # server doesn't do mtime
+            "--sftp-set-modtime=false",  # server doesn't do mtime
+            f"{RCLONE_REMOTE}:{ARCHIVE_PATH}{remote_dir}",
+            fname,
+        ]
+    )
+    return run(fname, args)
+
+
 def parse_cli():
     global LOG_FILE
     global RCLONE_REMOTE
     global ARCHIVE_PATH
 
+    program = Path(__main__.__file__).stem
     parser = argparse.ArgumentParser(
-        prog="upload-test",
+        prog=program,
         description="QA test Permanent rclone",
         epilog="For challenging-names, id is a 3-digit number.  For apod, it is a date in %Y-%m-%d format.",
     )
-    parser.add_argument("directory")
+    if program == "upload-test":
+        parser.add_argument("directory")
     parser.add_argument("--log-file", help=f"path to log file (defaults to {LOG_FILE})")
     parser.add_argument(
         "--omit",
         help="specify file of ids to omit (misc and challenging-names)",
     )
     parser.add_argument("--only", help="only test one file id")
-    parser.add_argument("--remote", help="Name of configured rclone remote such as permanent-prod or permanent-dev")
+    parser.add_argument(
+        "--remote",
+        help="Name of configured rclone remote such as permanent-prod or permanent-dev",
+    )
     parser.add_argument("--archive-path", help="Archive path in Permanent.")
     parser.add_argument(
         "--remote-dir",
@@ -112,6 +150,9 @@ def parse_cli():
         RCLONE_REMOTE = args.remote
     else:
         log("No rclone remote set. Attempting with default remote `permanent`...", True)
-        log("If the default remote `permanent` is not configured uploads would fail.", True)
+        log(
+            "If the default remote `permanent` is not configured uploads would fail.",
+            True,
+        )
 
-    return args
+    return args