Merge pull request #1 from OpenTechStrategies/update-usage

Update usage

Author: nfebe

.gitignore

@@ -1,3 +1,4 @@
 test-tree/challenging-names
 test-tree/apod
 __pycache__
+venv

README.md

@@ -10,22 +10,52 @@ protocol.
 For more context, see [this
 discussion](https://chat.opentechstrategies.com/#narrow/stream/73-Permanent/topic/QA/near/155527).
 
+## Testing scope
+
+The scope of testing here verifies the possibility of correctly uploading and downloading
+ a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
+ which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
+
+### What file types and scenarios are covered?
+
+- Text and png images with obscure names generated via [generate-tree.py](generate-tree.py)
+- Images in `.jpg` and `.png` format downloaded from [APOD](https://apod.nasa.gov/apod) via [apod-downloader.py](apod-downloader.py)
+- Compressed files in `.zip` and `.tar`
+- Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
+- Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
+
+### What file types and scenarios are left out?
+
+Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
+
 ## Usage
 
+You would have to install the python requirements used in this repo.
+
+1. `python -m venv venv`
+2. `source venv/bin/activate`
+3. `pip install -r requirements.txt`
+
+*It's possible to just install the requirements on your workspace however steps 1 and  2 would create and activate a virtual environment for this project alone! Recommended!*
+
 ### APOD
 
 Run `./apod-downloader.py` to download a set of Astronomy Of The Day
 photos, in a new directory called `test-tree/apod`.
 
-Run `./upload-test.sh test-tree/apod` to try uploading the APOD photos
+Run `./upload-test.py test-tree/apod --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/"` to try uploading the APOD photos
 (and some html) all at once to a directory on the server.
 
+**NB: The `--archive-path` argument specifies the route to the the specific permanent archive to which uploads would be made. So how do you get the archive path? See [Constructing archive path](https://github.com/permanentOrg/sftp-service/#downloading-from-permanent)**
+
+*That said, the archive path used in the sample command would have to be updated to match some archive created on Permanent.org*
+
 ### Challenging Names
 
 Run `./generate-tree.py` to generate test data, which will be placed
 in a new subdirectory named `test-tree/challenging-names`.
 
-Then run `./upload-test.sh test-tree/challenging-names` to try uploading the data (you'll need to
+Then run `./upload-test.py test-tree/challenging-names --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/"` to try uploading the data (you'll need to
 [configure
 rclone](https://github.com/PermanentOrg/sftp-service#running-rclone-against-permanentorg-instances)
 first, of course).  See the long comment at the top of

requirements.txt

@@ -0,0 +1,3 @@
+beautifulsoup4==4.12.0
+python_dateutil==2.8.2
+requests==2.25.1

upload-test.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
-RCLONE_REMOTE = "permanent-"
-ARCHIVE_PATH = "/archives/rclone QA 1 (0a0j-0000)/My Files/"
+RCLONE_REMOTE = "permanent"
+ARCHIVE_PATH = ""
 CHALLENGING_NAMES_DIR = "test-tree/challenging-names"
 APOD_DIR = "test-tree/apod"
 MISC_DIR = "test-tree/misc"
@@ -52,6 +52,7 @@ def omit_p(fname, omit_list):
 def parse_cli():
     global LOG_FILE
     global RCLONE_REMOTE
+    global ARCHIVE_PATH
 
     parser = argparse.ArgumentParser(
         prog="upload-test",
@@ -65,7 +66,8 @@ def parse_cli():
         help="specify file of ids to omit (misc and challenging-names)",
     )
     parser.add_argument("--only", help="only test one file id")
-    parser.add_argument("--remote", help="prod or dev", default="dev")
+    parser.add_argument("--remote", help="Name of configured rclone remote such as permanent-prod or permanent-dev")
+    parser.add_argument("--archive-path", help="Archive path in Permanent.")
     parser.add_argument(
         "--remote-dir",
         help="remote subdirectory (defaults to 'test-tree')",
@@ -87,7 +89,13 @@ def parse_cli():
         LOG_FILE = args.log_file
     if args.omit:
         args.omit_files = slurp_if_e(args.omit).strip().split("\n")
-    RCLONE_REMOTE += args.remote
+    if args.archive_path:
+        ARCHIVE_PATH = args.archive_path
+    if args.remote:
+        RCLONE_REMOTE = args.remote
+    else:
+        log("No rclone remote set. Attempting with default remote `permanent`...", True)
+        log("If the default remote `permanent` is not configured uploads would fail.", True)
 
     return args