Skip to content

XWIKI-22845: Introduce a S3 storage support for attachment content, history and deleted attachments and documents #4642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from
Draft

XWIKI-22845: Introduce a S3 storage support for attachment content, history and deleted attachments and documents #4642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
47d4c9e
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Sep 4, 2025
62f259d
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Sep 10, 2025
4b3d733
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Sep 11, 2025
2fe51c0
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Sep 15, 2025
c6c8403
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Oct 7, 2025
39a5ac5
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Oct 8, 2025
a407973
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Oct 8, 2025
f50f343
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Oct 8, 2025
0498428
XWIKI-22845: Introduce a S3 storage support for attachment content, h…
michitux Oct 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion xwiki-platform-core/xwiki-platform-store/xwiki-platform-store-filesystem-oldcore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<name>XWiki Platform - Store - Filesystem - Old Core</name>
<description>Implement various oldcore store APIs based on filesystem.</description>
<properties>
<xwiki.jacoco.instructionRatio>0.41</xwiki.jacoco.instructionRatio>
<xwiki.jacoco.instructionRatio>0.39</xwiki.jacoco.instructionRatio>
<!-- Old names of this module used for retro compatibility when resolving dependencies of old extensions -->
<xwiki.extension.features>org.xwiki.platform:xwiki-platform-store-filesystem-attachments</xwiki.extension.features>
</properties>
Expand All @@ -45,6 +45,11 @@
<artifactId>xwiki-commons-component-api</artifactId>
<version>${commons.version}</version>
</dependency>
<dependency>
<groupId>org.xwiki.commons</groupId>
<artifactId>xwiki-commons-store-blob-filesystem</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this module supposed to only use the API ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory, yes, but I didn't feel like rewriting the migrations on top of the public API due to the high risk of introducing regressions in particular as some migrations move directories with the attachment data several times, which wouldn't be a good idea with S3, anyway. For this reason, I introduced a way in the file system store to get the actual filesystem path, and I'm using this in the migrations to keep their code mostly unchanged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's only for the migrations, I guess we could move those to some dedicated module.

<version>${commons.version}</version>
</dependency>
<!-- Needed for DocumentReferenceSerializer implementation. -->
<dependency>
<groupId>org.xwiki.platform</groupId>
Expand Down
19 changes: 10 additions & 9 deletions ...tem-oldcore/src/main/java/org/xwiki/store/filesystem/internal/AttachmentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobStoreException;

/**
* A means of getting files for storing information about a given attachment.
Expand All @@ -30,23 +31,23 @@
public interface AttachmentFileProvider
{
/**
* @return the File for storing the latest version of the attachment's content.
* @return the Blob for storing the latest version of the attachment's content.
*/
File getAttachmentContentFile();
Blob getAttachmentContentFile() throws BlobStoreException;

/**
* Get the meta file for the attachment. The meta file contains information about each version of the attachment
* Get the meta blob for the attachment. The meta file contains information about each version of the attachment
* such as who saved it.
*
* @return the File for storing meta data for each version of an attachment.
* @return the Blob for storing meta data for each version of an attachment.
*/
File getAttachmentVersioningMetaFile();
Blob getAttachmentVersioningMetaFile() throws BlobStoreException;

/**
* Get a uniquely named file for storing a particular version of the attachment.
* Get a uniquely named Blob for storing a particular version of the attachment.
*
* @param versionName the name of the version of the attachment eg: "1.1" or "1.2"
* @return the File for storing the content of a particular version of the attachment.
* @return the Blob for storing the content of a particular version of the attachment.
*/
File getAttachmentVersionContentFile(String versionName);
Blob getAttachmentVersionContentFile(String versionName) throws BlobStoreException;
}
32 changes: 22 additions & 10 deletions ...core/src/main/java/org/xwiki/store/filesystem/internal/DefaultAttachmentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobPath;
import org.xwiki.store.blob.BlobStore;
import org.xwiki.store.blob.BlobStoreException;

/**
* A means of getting files for storing information about a given attachment.
Expand All @@ -36,10 +39,15 @@ public class DefaultAttachmentFileProvider implements AttachmentFileProvider
*/
private static final String ATTACH_ARCHIVE_META_FILENAME = "~METADATA.xml";

/**
* The blob store where the attachment information is stored.
*/
protected final BlobStore store;

/**
* The directory where all information about this attachment resides.
*/
private final File attachmentDir;
private final BlobPath attachmentDir;

/**
* The name of the attached file.
Expand All @@ -49,19 +57,21 @@ public class DefaultAttachmentFileProvider implements AttachmentFileProvider
/**
* The Constructor.
*
* @param store the blob store where the attachment information is stored.
* @param attachmentDir a directory where all information about this attachment is stored.
* @param fileName the name of the attachment file.
*/
public DefaultAttachmentFileProvider(final File attachmentDir, final String fileName)
public DefaultAttachmentFileProvider(BlobStore store, final BlobPath attachmentDir, final String fileName)
{
this.store = store;
this.attachmentDir = attachmentDir;
this.attachmentFileName = fileName;
}

/**
* @return the directory where information about this attachment is stored.
*/
protected File getAttachmentDir()
protected BlobPath getAttachmentDir()
{
return this.attachmentDir;
}
Expand All @@ -75,20 +85,22 @@ protected String getAttachmentFileName()
}

@Override
public File getAttachmentContentFile()
public Blob getAttachmentContentFile() throws BlobStoreException
{
return new File(this.attachmentDir, StoreFileUtils.getStoredFilename(this.attachmentFileName, null));
return this.store.getBlob(
this.attachmentDir.resolve(StoreFileUtils.getStoredFilename(this.attachmentFileName, null)));
}

@Override
public File getAttachmentVersioningMetaFile()
public Blob getAttachmentVersioningMetaFile() throws BlobStoreException
{
return new File(this.attachmentDir, ATTACH_ARCHIVE_META_FILENAME);
return this.store.getBlob(this.attachmentDir.resolve(ATTACH_ARCHIVE_META_FILENAME));
}

@Override
public File getAttachmentVersionContentFile(final String versionName)
public Blob getAttachmentVersionContentFile(final String versionName) throws BlobStoreException
{
return new File(this.attachmentDir, StoreFileUtils.getStoredFilename(this.attachmentFileName, versionName));
return this.store.getBlob(
this.attachmentDir.resolve(StoreFileUtils.getStoredFilename(this.attachmentFileName, versionName)));
}
}
16 changes: 11 additions & 5 deletions ...c/main/java/org/xwiki/store/filesystem/internal/DefaultDeletedAttachmentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,12 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobPath;
import org.xwiki.store.blob.BlobStore;
import org.xwiki.store.blob.BlobStoreException;

import com.xpn.xwiki.doc.XWikiAttachment;

/**
* A means of getting files for storing information about a given deleted attachment.
Expand All @@ -40,17 +45,18 @@ public class DefaultDeletedAttachmentFileProvider extends DefaultAttachmentFileP
/**
* The Constructor.
*
* @param store the blob store where the attachment information is stored.
* @param attachmentDir the location where the information about the deleted attachment will be stored.
* @param fileName the name of the attachment file.
*/
public DefaultDeletedAttachmentFileProvider(final File attachmentDir, final String fileName)
public DefaultDeletedAttachmentFileProvider(BlobStore store, final BlobPath attachmentDir, final String fileName)
{
super(attachmentDir, fileName);
super(store, attachmentDir, fileName);
}

@Override
public File getDeletedAttachmentMetaFile()
public Blob getDeletedAttachmentMetaFile() throws BlobStoreException
{
return new File(this.getAttachmentDir(), DELETED_ATTACH_META_FILENAME);
return this.store.getBlob(this.getAttachmentDir().resolve(DELETED_ATTACH_META_FILENAME));
}
}
20 changes: 15 additions & 5 deletions ...n/java/org/xwiki/store/filesystem/internal/DefaultDeletedDocumentContentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobPath;
import org.xwiki.store.blob.BlobStore;
import org.xwiki.store.blob.BlobStoreException;

/**
* A means of getting files for storing information about a given deleted document.
Expand All @@ -34,22 +37,29 @@ public class DefaultDeletedDocumentContentFileProvider implements DeletedDocumen
*/
private static final String DELETED_DOCUMENT_FILE_NAME = "content.xml";

/**
* The blob store where the document information is stored.
*/
protected final BlobStore store;

/**
* The directory where all information about this deleted document resides.
*/
private final File deletedDocumentDir;
private final BlobPath deletedDocumentDir;

/**
* @param store the blob store where the document information is stored.
* @param deletedDocumentDir the location where the information about the deleted document will be stored.
*/
public DefaultDeletedDocumentContentFileProvider(final File deletedDocumentDir)
public DefaultDeletedDocumentContentFileProvider(BlobStore store, final BlobPath deletedDocumentDir)
{
this.store = store;
this.deletedDocumentDir = deletedDocumentDir;
}

@Override
public File getDeletedDocumentContentFile()
public Blob getDeletedDocumentContentFile() throws BlobStoreException
{
return new File(this.deletedDocumentDir, DELETED_DOCUMENT_FILE_NAME);
return this.store.getBlob(this.deletedDocumentDir.resolve(DELETED_DOCUMENT_FILE_NAME));
}
}
7 changes: 4 additions & 3 deletions ...core/src/main/java/org/xwiki/store/filesystem/internal/DeletedAttachmentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobStoreException;

/**
* A means of getting files for storing information about a given attachment.
Expand All @@ -30,7 +31,7 @@
public interface DeletedAttachmentFileProvider extends AttachmentFileProvider
{
/**
* @return the File for storing the information about the deleted attachment such as who deleted it.
* @return the Blob for storing the information about the deleted attachment such as who deleted it.
*/
File getDeletedAttachmentMetaFile();
Blob getDeletedAttachmentMetaFile() throws BlobStoreException;
}
7 changes: 4 additions & 3 deletions ...src/main/java/org/xwiki/store/filesystem/internal/DeletedDocumentContentFileProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
*/
package org.xwiki.store.filesystem.internal;

import java.io.File;
import org.xwiki.store.blob.Blob;
import org.xwiki.store.blob.BlobStoreException;

/**
* A means of getting files for storing information about a given deleted document.
Expand All @@ -30,7 +31,7 @@
public interface DeletedDocumentContentFileProvider
{
/**
* @return the File for storing the content of the deleted document.
* @return the Blob for storing the content of the deleted document.
*/
File getDeletedDocumentContentFile();
Blob getDeletedDocumentContentFile() throws BlobStoreException;
}
Loading