Merge pull request PermanentOrg#16 from OpenTechStrategies/wider-range-test-data

Add instructions for variety testing

Author: nfebe

.gitignore

@@ -1,5 +1,7 @@
 test-tree/*
 !test-tree/misc/
+test-tree/misc/variety/*
+!test-tree/misc/variety/files.zip
 __pycache__
 venv
 log*.txt

README.md

@@ -111,8 +111,8 @@ Run `./upload-test.py test-tree/challenging-names --only=414 --remote-dir=test-4
 ```
 - No duplicates should be be seen on Permanent UI.
 
+### Uploads
 #### Large uploads
-##### Uploads
 
 To test large file (`400MB` +) uploads, a couple of large files are required. Some ready-made test files can be downloaded via:
 
@@ -192,14 +192,14 @@ Run
 Check the downloads folder in `test-tree/downloads` and ensure that the `downloads/nested` directory has a structure like the nested directory uploaded in the [nested uploads test](#nested-uploads).
 
 
-#### Quantity Tests
+### Quantity Tests
 
 To test uploads/downloads with a large number of files, we definitely need "a large number" of files on either side (local/remote) of the process.
 
 To generate a number of files with a specific size in `test-tree/special-files`, run `./create-files.py --quantity 10 --size 10000 --root-name "10-10B"` (*In this case, the command would generate 10, 10 bytes files.*)
 
 *Take note that in the command `--quantity`, `--size` and `--root-name` are arguments whose values you can change. Quanity for number of files, size for file zie and root name for the name of the parent folder that would hold the files*
-##### Large number of uploads
+#### Large number of uploads
 
 For 1000, 1B Files:
 - Run `./create-files.py`
@@ -223,8 +223,42 @@ For 1000, 5MB Files:
 
 *Of course, by looking at the pattern above, other number-size arrangements can be generated for further testing.*
 
+#### Variety of file types/sizes
 
-##### Large number of downloads
+*Prepare data variety*
+
+To test uploads with a variety of file types/sizes, unzip the files archive in `/test-tree/misc/variety/files.zip` into same directory `/test-tree/misc/variety/`
+
+- Unzip : `unzip ./test-tree/misc/variety/files.zip -d ./test-tree/misc/variety` 
+
+*Otherwise* 
+
+You can add your own files in the range of a few kilo bytes to about 50 mega bytes in the same location (`/test-tree/misc/variety/files`).
+
+Ideally, a few images (`.png`, `.jpg`), document files (`.docx`, `.ppt`, `.xlxs`), video and sound files, and archive files.
+
+
+*Duplicate data* 
+
+Once you have the files in place, you duplicate the files to achieve a desire volume using the duplication script described in the next section.
+
+The `./duplicate-files` script is designed to create multiple copies of files in a specified source directory. It allows you to make 'n' copies of each file found in the source directory and save them in either the same directory or a different destination directory.
+
+Usage: `./duplicate-files <source_path> <n> <destination_path (optional)>`
+
+- Create a duplication by 10 `duplicate-files ./test-tree/misc/variety/files 10`
+
+*You can change the number to reduce or increase the number of files and consequently the resulting size.
+
+
+##### *Variety upload test*
+
+Now you can test uploads with the variety of files set up in `./test-tree/variety`
+
+- Run `./upload-test.py test-tree/misc/variety/files --remote-dir=variety --log-file=variety.txt --remote=prod --archive-path="/archives/QA (0a21-0000)/My Files/"`
+
+
+#### Large number of downloads
 
 
 

duplicate-files

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+if [ $# -lt 2 ]; then
+  echo "Usage: $0 <source_path> <number_of_duplicates> [destination_path]"
+  exit 1
+fi
+
+source_path="$1"
+n="$2"
+destination_path="$3"
+
+if [ ! -d "$source_path" ]; then
+  echo "Error: The specified source path does not exist."
+  exit 1
+fi
+
+if [ "$n" -le 0 ]; then
+  echo "Error: 'n' should be a positive integer."
+  exit 1
+fi
+
+if [ -z "$destination_path" ]; then
+  destination_path="$source_path"
+fi
+
+# Check if the destination path already exists
+if [ ! -d "$destination_path" ]; then
+  echo "Notice: The specified destination path does not exist. Creating it..."
+  mkdir -p "$destination_path"
+fi
+
+# Loop through each file in the specified source path
+for file in "$source_path"/*; do
+  if [ -f "$file" ]; then
+    filename=$(basename "$file")
+    
+    # Check if there's an extension
+    if [[ "$filename" == *.* ]]; then
+      file_extension="${filename##*.}"
+      filename_without_extension="${filename%.*}"
+    else
+      file_extension=""
+      filename_without_extension="$filename"
+    fi
+    
+    # Create 'n' copies of the file in the destination path
+    for ((i = 1; i <= n; i++)); do
+      if [ -z "$file_extension" ]; then
+        new_filename="${filename} ($i)"
+      else
+        new_filename="${filename_without_extension} ($i).$file_extension"
+      fi
+      
+      cp "$file" "$destination_path/$new_filename"
+      echo "Created copy: $destination_path/$new_filename"
+    done
+  fi
+done
+
+echo "Done."