Merge pull request #5 from OpenTechStrategies/3-nested-folders

Document tests for nested folders

Author: nfebe

.gitignore

@@ -2,3 +2,4 @@ test-tree/*
 !test-tree/misc/
 __pycache__
 venv
+log*.txt

README.md

@@ -34,8 +34,8 @@ Run `./upload-test.py test-tree/apod --archive-path "/archives/rclone QA 1 (0a0j
 
 ## Testing scope
 
-The scope of testing here verifies the possibility of correctly uploading and downloading
- a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
+The scope of testing via the test cases documented below verifies the possibility of correctly uploading and downloading
+ a defined set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
  which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
 
 ### What file types are tested?
@@ -46,9 +46,9 @@ The scope of testing here verifies the possibility of correctly uploading and do
 - Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
 - Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
 
-### What test cases are covered?
+## Test Cases?
 
-#### Challenging Names
+### Challenging Names
 
 Run `./generate-tree.py` to generate test data, which will be placed
 in a new subdirectory named `test-tree/challenging-names`.
@@ -60,24 +60,24 @@ first, of course).  See the long comment at the top of
 [upload-test.sh](upload-test.sh) for information about what it's
 trying to do and what problems we know about so far.
 
-#### Duplicates
+### Duplicates
 
 A duplicate is a file/folder with exactly the same name. Of course this is not possible on regular file systems but Permanent does support it.
 There is a deduplication algorithm from Permanent that the `sftp-service` relies to ensure that files with identical names on Permanent won't be 
  be considered as the same on regular file systems.
 
-##### How test duplicate
+#### How test duplicate
 
 - Create a folder in the test archive of the remote (permanent.org or permanent.dev depending on your test target) e.g 'duplicates'.
 - Upload at least two copies of multiple identical files into the folder `duplicates` for example (`file.txt`, `file.txt`, `file.txt` and `photo.png`, `photo.png` ...)
 - Run the download test script against the duplicate folder. In this case:
 
 ```
-`./test-download.py --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/" --remote-dir "duplicates"`
+`./test-download.py --remote=prod --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/" --remote-dir=duplicates`
 ```
-##### Expected results
+#### Expected results
 
-- Check download folder and ensure that results looks like:
+- Check downloads folder in `test-tree/downloads` and ensure that results looks like:
 
 *Result from `tree` program*
 ```
@@ -89,19 +89,19 @@ There is a deduplication algorithm from Permanent that the `sftp-service` relies
 
 0 directories, 5 files
 ```
-##### Multiple Identical Uploads
+#### Multiple Identical Uploads
 
 This test case captures what happens if you sync the same path with unchanged content multiples times.
 
-##### How test identical uploads
+#### How test identical uploads
 
 - Generate challenging names if not generated earlier, see [Challenging Names](#challenging-names)
 
-Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-414 --log-file=duplicate-upload-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-414 --log-file=log-duplicate-upload.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
 
 *Notice the use of the `--only` flag which specifies only files containing the number `414` should be uploaded, you can change this number to follow a string pattern in the generated challenging names but the provide example works just fine.*
 
-##### Expected results
+#### Expected results
 
 - `rclone` should report `Sizes identical` and `Unchanged skipping`
 
@@ -111,8 +111,8 @@ Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-4
 ```
 - No duplicates should be be seen on Permanent UI.
 
-##### Large uploads
-###### Uploads
+#### Large uploads
+##### Uploads
 
 To test large file (`400MB` +) uploads, a couple of large files are required. Some ready-made test files can be downloaded via:
 
@@ -136,7 +136,43 @@ and then run `./special-files-downloader.py --my-source my_files.txt`
 
 Once the files are on disk:
 
-Run `./upload-test.py test-tree/special-files/large --remote-dir=large-files --log-file=large-files-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+Run `./upload-test.py test-tree/special-files/large --remote-dir=large-files --log-file=log-large-files.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+#### Nested folders/files
+
+##### Uploading
+
+We have a default nest of folders that goes down 4 levels.
+
+Run `./upload-test.py test-tree/misc/nested/ --remote-dir=nested --log-file=log-nested.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+Verify in the Permanent UI that the folder set to remote dir `--remote-dir` in this case `nested` contains the nested folder with the following structure.
+
+*Result from `tree` program*
+
+```
+test-tree/misc/nested/
+├── nested-level-1
+│   ├── nested-level-2
+│   │   ├── nested-level-3
+│   │   │   └── record-level-3.txt
+│   │   └── record-level-2.txt
+│   └── record-level-1.txt
+└── record-level-0.txt
+```
+
+To test a nest with more levels, simply paste a nested folder structure inside `test-tree/misc/nested` or manually create more folder levels in the existing nest.
+
+##### Downloading
+
+*The steps in the upload section above must be completed before this step*
+
+Run
+
+`./test-download.py --remote=prod --archive-path="/archives/rclone QA 1 (0a21-0000)/My Files/" --remote-dir=nested`
+
+Check downloads folder in `test-tree/downloads` and ensure `downloads/nested` that results looks like the structured previous uploaded in the nested folder upload tests above.
+
 
 ### What file types and scenarios are left out?
 

test-tree/misc/nested/nested-level-1/nested-level-2/nested-level-3/record-level-3.txt

@@ -0,0 +1 @@
+Nested record level 3.

test-tree/misc/nested/nested-level-1/nested-level-2/record-level-2.txt

@@ -0,0 +1 @@
+Nested record level 2.

test-tree/misc/nested/nested-level-1/record-level-1.txt

@@ -0,0 +1 @@
+Nested record level 0.

test-tree/misc/nested/record-level-0.txt

@@ -0,0 +1 @@
+Nested record level 0.

upload-test.py

@@ -7,6 +7,7 @@
 CHALLENGING_NAMES_DIR = "test-tree/challenging-names"
 APOD_DIR = "test-tree/apod"
 MISC_DIR = "test-tree/misc"
+NESTED_DIR = "test-tree/misc/nested"
 
 gentree = __import__("generate-tree")
 
@@ -49,21 +50,23 @@ def main():
 
             # Try to rclone this file
             log(f"Trying {fname}...")
-            out = rclone_upload(os.path.join(CHALLENGING_NAMES_DIR, fname), cli.remote_dir)
+            rclone_upload(os.path.join(CHALLENGING_NAMES_DIR, fname), cli.remote_dir)
     elif os.path.abspath(cli.directory) == os.path.abspath(APOD_DIR):
-        out = rclone_upload(APOD_DIR, cli.remote_dir, timeout=0)
+        rclone_upload(APOD_DIR, cli.remote_dir, timeout=0)
     elif os.path.abspath(cli.directory) == os.path.abspath(MISC_DIR):
         for fname in os.listdir(MISC_DIR):
             if skip_p(fname, cli):
                 continue
 
             # Try to rclone this file
             log(f"Trying {fname}...")
-            out = rclone_upload(os.path.join(MISC_DIR, fname), cli.remote_dir)
+            rclone_upload(os.path.join(MISC_DIR, fname), cli.remote_dir)
 
             # Upload file 2 twice
             if fname[0:3] == "002":
-                out = rclone_upload(os.path.join(MISC_DIR, fname), cli.remote_dir)
+                rclone_upload(os.path.join(MISC_DIR, fname), cli.remote_dir)
+    elif os.path.abspath(cli.directory) == os.path.abspath(NESTED_DIR):
+        rclone_upload(NESTED_DIR, cli.remote_dir, timeout=0)
     else:
         sys.exit("Not sure what to do with that directory.")