Add test notes for special files

- How to prepare large files for upload tests
- Document commands to use for large file upload tests

Signed-off-by: Fon E. Noel NFEBE <[email protected]>

Author: nfebe

README.md

@@ -111,10 +111,42 @@ Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-4
 ```
 - No duplicates should be be seen on Permanent UI.
 
+##### Large uploads
+###### Uploads
+
+To test for large file uploads. A couple of large files are needed and can be downloaded via:
+
+`./special-files-downloader --large`
+
+If you have you own large files you would like to test with, you can list the links to those files in a text file like so:
+
+'my_files.txt'
+```
+https://link.com/to/file_1.extension
+https://link.com/to/file_2.extension
+https://link.com/to/file_3.extension
+```
+
+and then run `./special-files-downloader --my-sources my_files.txt` 
+
+- *You can specify as many paths as you want*
+- *You can name the the source text file anything you want but pass the right name and path to `--my-sources`*
+
+**You don't need to download any files if you already have some special files on your computer, simply copy such files into one of these directories `test-tree/special-files/large`, `test-tree/special-files/zips`, `test-tree/special-files/mixed` or `test-tree/special-files/custom`**
+
+Once the files are on disk:
+
+Run `./upload-test.py test-tree/special-files --remote-dir=special-files --log-file=special-files-log.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
 ### What file types and scenarios are left out?
 
 Anything not included in the section above describing what is currently covered is by implication excluded from these tests.
 
+## Troubleshooting
+
+- Remember that the commands are examples and some of the arguments may not apply to your specific environment.
+    - *For example ensure that arguments such as `--remote`, `--archive-path` are updated and correct*
+
 ## Web Interface
 
 For prod, just go to the site as per usual.  For dev, go to

special-files-downloader.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+import os
+import sys
+import argparse
+import requests
+from urllib.parse import urlparse
+
+SPECIAL_FILES_PATH = "test-tree/special-files/"
+LARGE_FILE_URIS = [
+    "https://www.quintic.com/software/sample_videos/Cricket%20Bowling%20150fps%201200.avi"
+]
+ZIP_FILE_URLS = []
+
+
+def parse_cli():
+    parser = argparse.ArgumentParser(
+        prog="special-files-downloader", description="Download special test files"
+    )
+
+    parser.add_argument("--large", help=f"Download large files for testing")
+    parser.add_argument("--zip", help=f"Download zip files for testing")
+    parser.add_argument(
+        "--my-sources", help=f"Download files from links listed in text file path"
+    )
+    parser.add_argument(
+        "--all",
+        help="Download all earmarked special files.",
+    )
+
+    return parser
+
+
+def download_file_from_url(uri, path):
+    response = requests.get(uri, allow_redirects=False)
+    fname = os.path.basename(urlparse(uri).path)
+    if response.status_code == 200:
+        with open(path + fname, "wb") as file:
+            file.write(response.content)
+
+
+def main():
+    parser = parse_cli()
+    if len(sys.argv) == 1:
+        parser.print_help()
+        print("No downloads done....")
+    args = parser.parse_args()
+
+    if args.large:
+        for uri in LARGE_FILE_URIS:
+            download_file_from_url(uri, SPECIAL_FILES_PATH + "large/")
+    if args.zip:
+        for uri in ZIP_FILE_URLS:
+            download_file_from_url(uri, SPECIAL_FILES_PATH + "zips/")
+    if args.all:
+        ALL = ZIP_FILE_URLS + LARGE_FILE_URIS
+        for uri in ALL:
+            download_file_from_url(uri, SPECIAL_FILES_PATH + "mixed/")
+    if args.my_sources:
+        sources = open(args.my_sources, "r")
+        sources = sources.readlines()
+        for source in sources:
+            download_file_from_url(source, SPECIAL_FILES_PATH + "custom/")
+
+
+if __name__ == "__main__":
+    main()