[Performance] Investigate performance of GET /cloud-api/folders

[DenizAltunkapan]

GET /cloud-api/folders has frequent long response times, which makes Cloud Page loading and folder navigation feel slow.
Please profile this endpoint and identify optimization opportunities.

Focus:

• Measure latency across proxy, backend, DB, and filesystem.
• Check for expensive recursion, N+1 queries, and missing indexes (user_id, parent_id).
• Propose concrete improvements with expected impact.

Acceptance Criteria

• Baseline latency metrics documented (including p95).
• Primary bottleneck(s) identified with evidence.
• Clear optimization plan (or implementation) delivered.

Labels
performance, backend, cloud-page, api, investigation

[SuganthiThomas]

Hi @DenizAltunkapan ,

Can i work on this issue?

Thank You,
Suganthi T

[DenizAltunkapan]

@SuganthiThomas i assigned you

[SuganthiThomas]

Hi @DenizAltunkapan ,

I have set up the workspace setup and started to work on performance analysis part .

Thank You

[SuganthiThomas]

Hi @DenizAltunkapan,

I implemented several optimizations to improve the performance of the folder traversal API:

🔹 Key Improvements

1. Reduced filesystem I/O by eliminating redundant directory scans and expensive operations.
2. Replaced costly MIME type detection (Files.probeContentType) with a lightweight extension-based approach.
3. Minimized repeated metadata reads to improve scalability for large directory structures.
   Simplified path resolution by replacing toRealPath() with toAbsolutePath() where safe.

🔹 Expensive Operations Identified

Paths.get(rootPath).toRealPath().normalize() (disk I/O heavy)
Files.probeContentType() (slow MIME detection)
Multiple Files.list(path) calls (redundant directory scans)

🔹 Code-Level Optimizations

1. Used a single directory scan with grouping:
   try (var stream = Files.list(path)) {
   result = stream.collect(Collectors.groupingBy(p -> {
   if (Files.isSymbolicLink(p)) return "symlinks";
   if (Files.isDirectory(p, LinkOption.NOFOLLOW_LINKS)) return "folders";
   if (Files.isRegularFile(p, LinkOption.NOFOLLOW_LINKS)) return "files";
   return "others";
   }));
   }
   Avoided unnecessary validatePath() calls for symlinks.

2. Replaced:
   filePath.toRealPath().normalize()

with:

filePath.toAbsolutePath().normalize()

🔹 Performance Impact
Before: ~198 ms to build folder tree
After: ~45 ms

➡️ ~4x improvement in response time

🔹 Notes

1. toAbsolutePath() was used instead of toRealPath() to avoid disk I/O, while still maintaining correct relative path computation.
   2, LinkOption.NOFOLLOW_LINKS ensures symlinks are not traversed, improving both performance and safety.

I have pasted the code that i have implamented
private FolderDto readFolder(String rootPath,Path rootReal,Path path) throws IOException {
// Resolve root path once for relative path calculation
Map<String, List> result;

  try (var stream = Files.list(path)) {
      result = stream.collect(Collectors.groupingBy(p -> {
          if (Files.isSymbolicLink(p)) return "symlinks";
          if (Files.isDirectory(p, LinkOption.NOFOLLOW_LINKS)) return "folders";
          if (Files.isRegularFile(p, LinkOption.NOFOLLOW_LINKS)) return "files";
          return "others";
      }));
  }

List<FolderDto> subfolders;
List<Path> entries = result.getOrDefault("folders", List.of());
subfolders = entries.parallelStream()
        .map(subPath -> {
            try {
               // validatePath(rootPath,rootReal, subPath);
                return readFolder(rootPath, rootReal, subPath);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        })
        .toList();


List<FileDto> files;
files = result.getOrDefault("files", List.of()).parallelStream()
        .map(
              filePath -> {
                try {
                  // Validate each child path to prevent symlink escapes
                 // validatePath(rootPath, filePath);
                  
                  // Calculate relative path from root to file
                  String relativePath;
                  
                    relativePath =
                        Paths.get(rootPath).relativize(filePath.toAbsolutePath()).toString();
                    relativePath = relativePath.replace('\\', '/');
                  

                  BasicFileAttributes attrs =
                      Files.readAttributes(filePath, BasicFileAttributes.class);
                  return new FileDto(
                      filePath.getFileName().toString(),
                      relativePath,
                      attrs.size(),
                      getMimeType(filePath),
                      attrs.lastModifiedTime().toMillis());
                } catch (IOException e) {
                  throw new FileAccessException(
                      "Failed to read file attributes: "
                          + filePath
                          + " with exception: "
                          + e.getMessage());
                }
              })
          .collect(Collectors.toList());


// Calculate relative path for the current folder
String folderRelativePath;
folderRelativePath = Paths.get(rootPath).relativize(path.toAbsolutePath()).toString();
folderRelativePath = folderRelativePath.replace('\\', '/');
if (folderRelativePath.isEmpty()) {
  folderRelativePath = ".";}

try {
  BasicFileAttributes folderAttrs = Files.readAttributes(path, BasicFileAttributes.class);
  return new FolderDto(
      path.getFileName().toString(),
      folderRelativePath,
      subfolders,
      files,
      folderAttrs.lastModifiedTime().toMillis());
} catch (IOException e) {
  throw new FileAccessException(
      "Failed to read folder attributes: " + path + " with exception: " + e.getMessage());
}

}
/***/
Creaed getMimeType(Path filePath) to avoid Files.probeContentType
//
public static String getMimeType(Path filePath) {

  String fileName=filePath.getFileName().toString();
 
  if (fileName == null || !fileName.contains(".")) {
      return "application/octet-stream";
  }
  String ext = fileName.substring(fileName.lastIndexOf('.') + 1).toLowerCase();

  String mimeType = switch (ext) {

      // 📄 Text / Docs
      case "txt" -> "text/plain";
      case "log" -> "text/plain";
      case "csv" -> "text/csv";
      case "json" -> "application/json";
      case "xml" -> "application/xml";
      case "md" -> "text/markdown";

      // 🧾 Documents
      case "pdf" -> "application/pdf";
      case "doc" -> "application/msword";
      case "docx" -> "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
      case "xls" -> "application/vnd.ms-excel";
      case "xlsx" -> "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
      case "ppt" -> "application/vnd.ms-powerpoint";
      case "pptx" -> "application/vnd.openxmlformats-officedocument.presentationml.presentation";

      // 🖼 Images
      case "jpg", "jpeg" -> "image/jpeg";
      case "png" -> "image/png";
      case "gif" -> "image/gif";
      case "bmp" -> "image/bmp";
      case "webp" -> "image/webp";
      case "svg" -> "image/svg+xml";

      // 🎥 Video
      case "mp4" -> "video/mp4";
      case "mkv" -> "video/x-matroska";
      case "avi" -> "video/x-msvideo";
      case "mov" -> "video/quicktime";
      case "wmv" -> "video/x-ms-wmv";

      // 🎵 Audio
      case "mp3" -> "audio/mpeg";
      case "wav" -> "audio/wav";
      case "aac" -> "audio/aac";
      case "flac" -> "audio/flac";

      // 📦 Archives
      case "zip" -> "application/zip";
      case "rar" -> "application/vnd.rar";
      case "7z" -> "application/x-7z-compressed";
      case "tar" -> "application/x-tar";
      case "gz" -> "application/gzip";

      // 💻 Code
      case "java" -> "text/x-java-source";
      case "py" -> "text/x-python";
      case "js" -> "text/javascript";
      case "ts" -> "text/typescript";
      case "html" -> "text/html";
      case "css" -> "text/css";
      case "c", "cpp" -> "text/x-c++src";

      // 🔐 Default
      default -> "application/octet-stream";
  };
  if (mimeType == null) {
      try {
          mimeType = Files.probeContentType(filePath);
      } catch (Exception e) {
          mimeType = "application/octet-stream";
      }
  }

  return mimeType != null ? mimeType : "application/octet-stream";

}

Please let me know your suggestions.

Thank You,

[DenizAltunkapan]

@SuganthiThomas Thanks for the improvements, the reduction in filesystem I/O and removal of expensive operations definitely helps performance.

However, the core issue described here is not fully addressed yet:
I think the main issue is that the API still recursively traverses the full folder tree, which leads to over-fetching. The intended solution was to only fetch the current directory level (depth = 1) and load subfolders on demand.

Additionally, removing validatePath introduces potential security risks (e.g. path traversal / symlink escape), especially since toRealPath was replaced with toAbsolutePath.

Also, using parallelStream for filesystem operations may not provide benefits and could introduce overhead in I/O-bound scenarios.

I would suggest:

• aligning the implementation with the lazy-loading approach (no recursion)
• reintroducing proper path validation
• reconsidering parallel streams for I/O operations

The current changes improve performance, but do not yet solve the underlying architectural issue.