Merge pull request #2 from OpenTechStrategies/download-script

nfebe · web-flow · commit 4e9f30831aaa · 2023-04-10T11:00:42.000+01:00
Download script and more test cases
diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,3 @@
-test-tree/challenging-names
-test-tree/apod
+test-tree/
 __pycache__
 venv
diff --git a/README.md b/README.md
@@ -10,24 +10,6 @@ protocol.
 For more context, see [this
 discussion](https://chat.opentechstrategies.com/#narrow/stream/73-Permanent/topic/QA/near/155527).
 
-## Testing scope
-
-The scope of testing here verifies the possibility of correctly uploading and downloading
- a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
- which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
-
-### What file types and scenarios are covered?
-
-- Text and png images with obscure names generated via [generate-tree.py](generate-tree.py)
-- Images in `.jpg` and `.png` format downloaded from [APOD](https://apod.nasa.gov/apod) via [apod-downloader.py](apod-downloader.py)
-- Compressed files in `.zip` and `.tar`
-- Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
-- Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
-
-### What file types and scenarios are left out?
-
-Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
-
 ## Usage
 
 You would have to install the python requirements used in this repo.
@@ -50,7 +32,23 @@ Run `./upload-test.py test-tree/apod --archive-path "/archives/rclone QA 1 (0a0j
 
 *That said, the archive path used in the sample command would have to be updated to match some archive created on Permanent.org*
 
-### Challenging Names
+## Testing scope
+
+The scope of testing here verifies the possibility of correctly uploading and downloading
+ a finite set of file types in a particular size range to [Permanent.org](Permanent.org) using [rclone](https://rclone.org/)
+ which talks to permanent using the [SFTP service](https://github.com/PermanentOrg/sftp-service)
+
+### What file types are tested?
+
+- Text and png images with obscure names generated via [generate-tree.py](generate-tree.py)
+- Images in `.jpg` and `.png` format downloaded from [APOD](https://apod.nasa.gov/apod) via [apod-downloader.py](apod-downloader.py)
+- Compressed files in `.zip` and `.tar`
+- Videos in `.mp4`, `.webm`, `.gifs` and `.3gp` common in mobile devices.
+- Executable files in `.exe`, `.run`, `.sh`, `.dep` and extension-less bin executables.
+
+### What test cases are covered?
+
+#### Challenging Names
 
 Run `./generate-tree.py` to generate test data, which will be placed
 in a new subdirectory named `test-tree/challenging-names`.
@@ -62,6 +60,93 @@ first, of course).  See the long comment at the top of
 [upload-test.sh](upload-test.sh) for information about what it's
 trying to do and what problems we know about so far.
 
+#### Duplicates
+
+A duplicate is a file/folder with exactly the same name. Of course this is not possible on regular file systems but Permanent does support it.
+There is a deduplication algorithm from Permanent that the `sftp-service` relies to ensure that files with identical names on Permanent won't be 
+ be considered as the same on regular file systems.
+
+##### How test duplicate
+
+- Create a folder in the test archive of the remote (permanent.org or permanent.dev depending on your test target) e.g 'duplicates'.
+- Upload at least two copies of multiple identical files into the folder `duplicates` for example (`file.txt`, `file.txt`, `file.txt` and `photo.png`, `photo.png` ...)
+- Run the download test script against the duplicate folder. In this case:
+
+```
+`./test-download.py --archive-path "/archives/rclone QA 1 (0a0j-0000)/My Files/" --remote-dir "duplicates"`
+```
+##### Expected results
+
+- Check download folder and ensure that results looks like:
+
+*Result from `tree` program*
+```
+├── file (1).txt
+├── file (2).txt
+├── file.txt
+├── Photo (1).png
+└── Photo.png
+
+0 directories, 5 files
+```
+##### Multiple Identical Uploads
+
+This test case captures what happens if you sync the same path with unchanged content multiples times.
+
+##### How test identical uploads
+
+- Generate challenging names if not generated earlier, see [Challenging Names](#challenging-names)
+
+Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-414 --log-file=duplicate-upload-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+*Notice the use of the `--only` flag which specifies only files containing the number `414` should be uploaded, you can change this number to follow a string pattern in the generated challenging names but the provide example works just fine.*
+
+##### Expected results
+
+- `rclone` should report `Sizes identical` and `Unchanged skipping`
+
+```
+2023/03/29 14:54:00 DEBUG : 002-dupe-test.txt: Sizes identical
+2023/03/29 14:54:00 DEBUG : 002-dupe-test.txt: Unchanged skipping
+```
+- No duplicates should be be seen on Permanent UI.
+
+##### Large uploads
+###### Uploads
+
+To test large file (`400MB` +) uploads, a couple of large files are required. Some ready-made test files can be downloaded via:
+
+`./special-files-downloader.py --large`
+
+If you have your own large files or other kinds of files you would like to run tests with, you can list the links to those files in a text file like so:
+
+'my_files.txt'
+```
+https://link.com/to/file_1.extension
+https://link.com/to/file_2.extension
+https://link.com/to/file_3.extension
+```
+
+and then run `./special-files-downloader.py --my-source my_files.txt`
+
+- *You can specify as many paths as you want inside the file*
+- *You can name the the source text file anything you want but pass the right name and path to `--my-source`*
+
+**You don't need to download any files if you already have some special files on your computer, simply copy such files into one of these directories `test-tree/special-files/`, `test-tree/special-files/large`, `test-tree/special-files/zips`,  or `test-tree/special-files/custom`**
+
+Once the files are on disk:
+
+Run `./upload-test.py test-tree/special-files/large --remote-dir=large-files --log-file=large-files-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+### What file types and scenarios are left out?
+
+Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
+
+## Troubleshooting
+
+- Remember that the commands are examples and some of the arguments may not apply to your specific environment.
+    - *For example ensure that arguments such as `--remote`, `--archive-path` are updated and correct*
+
 ## Web Interface
 
 For prod, just go to the site as per usual.  For dev, go to
diff --git a/generate-tree.py b/generate-tree.py
@@ -27,7 +27,6 @@
 """
 
 import os
-import sys
 import shutil
 
 test_tree_top = "test-tree/challenging-names"
diff --git a/source_large_files.txt b/source_large_files.txt
@@ -0,0 +1 @@
+https://www.quintic.com/software/sample_videos/Cricket%20Bowling%20150fps%201200.avi
diff --git a/source_zip_files.txt b/source_zip_files.txt
diff --git a/special-files-downloader.py b/special-files-downloader.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+import os
+import sys
+import argparse
+from urllib.parse import urlparse
+import requests
+
+SPECIAL_FILES_ROOT = "test-tree/special-files/"
+LARGE_FILES_PATH = SPECIAL_FILES_ROOT + "large/"
+ZIP_FILES_PATH = SPECIAL_FILES_ROOT + "zips/"
+CUSTOM_FILES_PATH = SPECIAL_FILES_ROOT + "custom/"
+CHUNK_SIZE = 1024 * 1024
+LARGE_FILE_URLS = []
+ZIP_FILE_URLS = []
+
+
+def parse_cli():
+    """Prepare parser"""
+    parser = argparse.ArgumentParser(
+        prog="special-files-downloader", description="Download special test files"
+    )
+
+    parser.add_argument(
+        "--large", help="Download large files for testing", action="store_true"
+    )
+    parser.add_argument(
+        "--zip", help="Download zip files for testing", action="store_true"
+    )
+    parser.add_argument(
+        "--my-source", help="Download files from links listed in text file path"
+    )
+    parser.add_argument(
+        "--all", help="Download all earmarked special files.", action="store_true"
+    )
+
+    return parser
+
+
+def check_paths():
+    """Ensure special-file folders required in test-tree are present"""
+    if not os.path.exists(SPECIAL_FILES_ROOT):
+        os.makedirs(SPECIAL_FILES_ROOT)
+    if not os.path.exists(LARGE_FILES_PATH):
+        os.makedirs(LARGE_FILES_PATH)
+    if not os.path.exists(ZIP_FILES_PATH):
+        os.makedirs(ZIP_FILES_PATH)
+    if not os.path.exists(CUSTOM_FILES_PATH):
+        os.makedirs(CUSTOM_FILES_PATH)
+
+
+def get_file_urls():
+    """Get links to default special-files required for testing"""
+    global LARGE_FILE_URLS
+    global ZIP_FILE_URLS
+    large_files_handle = open("source_large_files.txt", "r", encoding="utf-8")
+    large_files = large_files_handle.readlines()
+    large_files_handle.close()
+    LARGE_FILE_URLS = map(lambda x: x.strip(), large_files)
+    zip_files_handle = open("source_zip_files.txt", "r", encoding="utf-8")
+    zip_files = zip_files_handle.readlines()
+    zip_files_handle.close()
+    ZIP_FILE_URLS = map(lambda x: x.strip(), zip_files)
+
+
+def download_file_from_url(url, path):
+    """Download file in url to path"""
+    fname = os.path.basename(urlparse(url).path)
+    print(f"\nDownloading {fname}\n")
+    underline = "=" * len(fname)
+    underline = underline + "============"
+    print(underline + "\n")
+    size = 0
+    with requests.get(url, stream=True) as response:
+        if response.status_code == 200:
+            with open(path + fname, "wb") as file:
+                for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
+                    file.write(chunk)
+                    size = size + (CHUNK_SIZE)
+                    print(f"Downloaded {size} bytes of {fname} ...")
+
+
+def main():
+    "Script entry point"
+    check_paths()
+    get_file_urls()
+    parser = parse_cli()
+    if len(sys.argv) == 1:
+        parser.print_help()
+        print(
+            "\n========================\n| No downloads done... |\n========================\n"
+        )
+    args = parser.parse_args()
+
+    if args.large:
+        for url in LARGE_FILE_URLS:
+            download_file_from_url(url, LARGE_FILES_PATH)
+    if args.zip:
+        for url in ZIP_FILE_URLS:
+            download_file_from_url(url, ZIP_FILE_URLS)
+    if args.all:
+        ALL_URLS = ZIP_FILE_URLS + LARGE_FILE_URLS
+        for url in ALL_URLS:
+            download_file_from_url(url, SPECIAL_FILES_ROOT)
+    if args.my_source:
+        source = open(args.my_source, "r")
+        source = source.readlines()
+        for url in source:
+            download_file_from_url(url.strip(), CUSTOM_FILES_PATH)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/test-download.py b/test-download.py
@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+
+from utils import log, parse_cli, rclone_download
+
+DOWNLOAD_DIR = "test-tree/downloads"
+
+import os
+
+def main():
+    # Do some initial setup, parse cli, etc
+    cli = parse_cli()
+    rclone_download(os.path.join(DOWNLOAD_DIR, cli.remote_dir), cli.remote_dir)
+
+if __name__ == "__main__":
+    main()
diff --git a/upload-test.py b/upload-test.py
diff --git a/utils.py b/utils.py

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+https://www.quintic.com/software/sample_videos/Cricket%20Bowling%20150fps%201200.avi`