Skip to content

Streaming bed file in analyzer #259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
khoroshevskyi wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Streaming bed file in analyzer #259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
931bc60
Streaming bed file in analyzer
khoroshevskyi Feb 11, 2026
2294830
Merge branch 'master' into streaming_wasm
khoroshevskyi Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ui/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"generate-types-local": "npx openapi-typescript http://localhost:8000/openapi.json -o bedbase-types.d.ts"
},
"dependencies": {
"@databio/gtars": "^0.5.3",
"@databio/gtars": "/home/bnt4me/virginia/repos/gtars/gtars-wasm/pkg",
"@tanstack/react-query": "^5.28.0",
"@tanstack/react-query-devtools": "^5.28.0",
"@tanstack/react-table": "^8.15.3",
Expand Down
2 changes: 1 addition & 1 deletion ui/src/components/bed-analytics-components/chromosome-stats-panel.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,4 +119,4 @@ const ChromosomeStatsPanel: React.FC<Props> = ({ rs, selectedFile }) => {
);
};

export default ChromosomeStatsPanel;
export default ChromosomeStatsPanel;
198 changes: 100 additions & 98 deletions ui/src/pages/bed-analytics.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import { useState, useRef, useEffect } from 'react';
import { useState, useRef, useEffect, useCallback } from 'react';
import { useSearchParams, useNavigate } from 'react-router-dom';
import { Layout } from '../components/layout.tsx';
import { RegionSet, ChromosomeStatistics } from '@databio/gtars';
import { handleBedFileInput } from '../utils.ts';
import { bytesToSize } from '../utils.ts';
import ChromosomeStatsPanel from '../components/bed-analytics-components/chromosome-stats-panel.tsx';
import RegionDistributionPlot from '../components/bed-analytics-components/bed-plots.tsx';
Expand All @@ -15,6 +14,7 @@ type BedGenomeStats = components['schemas']['RefGenValidReturnModel'];
export const BEDAnalytics = () => {
const [rs, setRs] = useState<RegionSet | null>(null);
const [loadingRS, setLoadingRS] = useState(false);
const [progress, setProgress] = useState<{ percent: number; status: string } | null>(null);
const [totalProcessingTime, setTotalProcessingTime] = useState<number | null>(null);
const [selectedFile, setSelectedFile] = useState<File | null>(null);
const [inputMode, setInputMode] = useState<'file' | 'url'>('file');
Expand All @@ -28,6 +28,43 @@ export const BEDAnalytics = () => {
const analyzeGenomeMutation = useAnalyzeGenome();

const fileInputRef = useRef<HTMLInputElement>(null);
const workerRef = useRef<Worker | null>(null);
const startTimeRef = useRef<number>(0);

// Initialize worker
useEffect(() => {
workerRef.current = new Worker(new URL('../workers/bedAnalyzerWorker.ts', import.meta.url), {
type: 'module',
});

workerRef.current.onmessage = (e) => {
const { type } = e.data;

if (type === 'status') {
setProgress((prev) => ({ percent: prev?.percent ?? 0, status: e.data.message }));
} else if (type === 'progress') {
setProgress({
percent: e.data.percent,
status: 'Processing file...',
});
} else if (type === 'result') {
const endTime = performance.now();
const regionSet = new RegionSet(e.data.entries);
setRs(regionSet);
setTotalProcessingTime(endTime - startTimeRef.current);
setLoadingRS(false);
setProgress(null);
} else if (type === 'error') {
console.error('Worker error:', e.data.message);
setLoadingRS(false);
setProgress(null);
}
};

return () => {
workerRef.current?.terminate();
};
}, []);

useEffect(() => {
const urlParam = searchParams.get('bedUrl');
Expand All @@ -38,73 +75,35 @@ export const BEDAnalytics = () => {
}, [searchParams]);

useEffect(() => {
initializeRegionSet();
initializeAnalysis();
}, [triggerSearch]);
Comment on lines 77 to 79
Copy link

Copilot AI Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This effect calls initializeAnalysis but doesn’t include it in the dependency array. With plugin:react-hooks/recommended enabled and --max-warnings 0, this will fail lint due to react-hooks/exhaustive-deps. Refactor so the effect satisfies exhaustive-deps without changing the intended “only run when triggerSearch changes” behavior (e.g., make initializeAnalysis stable via refs, or restructure the effect logic accordingly).

Copilot uses AI. Check for mistakes.

const fetchBedFromUrl = async (url: string): Promise<File> => {
// console.log(`${url[0]}, ${url[1]}, ${url}`);
const fetchUrl =
url.length === 32 && !url.startsWith('http')
? `https://api.bedbase.org/v1/files/files/${url[0]}/${url[1]}/${url}.bed.gz`
: url;
// console.log(`${fetchUrl}`);
const response = await fetch(fetchUrl);
if (!response.ok) {
throw new Error(`Failed to fetch BED file: ${response.statusText}`);
}
const blob = await response.blob();
const fileName = fetchUrl.split('/').pop() || 'remote-bed-file.bed';
return new File([blob], fileName, { type: 'text/plain' });
};

const initializeRegionSet = async () => {
let fileToProcess: File | null = null;
const initializeAnalysis = useCallback(() => {
if (!workerRef.current) return;

if (inputMode === 'file' && selectedFile) {
fileToProcess = selectedFile;
setLoadingRS(true);
setRs(null);
setTotalProcessingTime(null);
setProgress({ percent: 0, status: 'Starting...' });
startTimeRef.current = performance.now();
workerRef.current.postMessage({ file: selectedFile });
} else if (inputMode === 'url' && bedUrl.trim()) {
try {
fileToProcess = await fetchBedFromUrl(bedUrl.trim());
} catch (error) {
console.error('Error fetching URL:', error);
return;
}
}

if (fileToProcess) {
setLoadingRS(true);
setRs(null);
setTotalProcessingTime(null);

try {
const startTime = performance.now();

const syntheticEvent = {
target: { files: [fileToProcess] },
} as unknown as Event;

await handleBedFileInput(syntheticEvent, (entries) => {
setTimeout(() => {
const rs = new RegionSet(entries);
const endTime = performance.now();
const totalTimeMs = endTime - startTime;

setRs(rs);
setTotalProcessingTime(totalTimeMs);
setLoadingRS(false);
}, 10);
});
} catch (error) {
setLoadingRS(false);
console.error('Error loading file:', error);
}
setProgress({ percent: 0, status: 'Starting...' });
startTimeRef.current = performance.now();
workerRef.current.postMessage({ url: bedUrl.trim() });
}
};
}, [inputMode, selectedFile, bedUrl]);

const unloadFile = () => {
setRs(null);
setTotalProcessingTime(null);
setSelectedFile(null);
setBedUrl('');
setProgress(null);
if (fileInputRef.current) {
fileInputRef.current.value = '';
}
Expand Down Expand Up @@ -303,27 +302,34 @@ export const BEDAnalytics = () => {
</div>
</div>


<div className="mt-4">
{loadingRS && (
<div
className="d-inline-flex align-items-center gap-2 px-3 py-2 bg-success bg-opacity-10 border border-success border-opacity-25 rounded-pill">
<div className="spinner-border spinner-border-sm text-success">
<span className="visually-hidden">Loading...</span>
{loadingRS && progress && (
<div className="mb-3">
<div className="d-inline-flex align-items-center gap-2 px-3 py-2 bg-success bg-opacity-10 border border-success border-opacity-25 rounded-pill mb-2">
<div className="spinner-border spinner-border-sm text-success">
<span className="visually-hidden">Loading...</span>
</div>
<span className="small text-success fw-medium">{progress.status}</span>
</div>
<span className="small text-success fw-medium">
Loading and analyzing...
</span>
{progress.percent >= 0 && (
<div className="progress" style={{ height: '6px' }}>
<div
className="progress-bar bg-primary"
role="progressbar"
style={{ width: `${progress.percent}%`, transition: 'width 0.1s ease' }}
aria-valuenow={progress.percent}
aria-valuemin={0}
aria-valuemax={100}
/>
</div>
)}
</div>
)}

{rs && !loadingRS && (
<div
className="d-inline-flex align-items-center gap-2 px-3 py-2 bg-primary bg-opacity-10 border border-primary border-opacity-25 rounded-pill">
<div className="d-inline-flex align-items-center gap-2 px-3 py-2 bg-primary bg-opacity-10 border border-primary border-opacity-25 rounded-pill">
<div className="bg-primary rounded-circle p-1" />
<span className="small text-primary fw-medium">
Results ready
</span>
<span className="small text-primary fw-medium">Results ready</span>
</div>
)}

Expand All @@ -333,34 +339,30 @@ export const BEDAnalytics = () => {
<div className="mt-3 p-3 border rounded shadow-sm bg-white">
<table className="table table-sm mb-0">
<tbody>
<tr>
<th scope="row">Identifier</th>
<td>{rs.identifier}</td>
</tr>
<tr>
<th scope="row">Mean region width</th>
<td>{rs.meanRegionWidth}</td>
</tr>
<tr>
<th scope="row">Total number of regions</th>
<td>{rs.numberOfRegions}</td>
</tr>
<tr>
<th scope="row">Total number of nucleotides</th>
<td>{rs.nucleotidesLength}</td>
</tr>
{/*<tr>*/}
{/* <th scope="row">First row</th>*/}
{/* <td>{rs.first_region}</td>*/}
{/*</tr>*/}
<tr>
<th scope="row">Data Format</th>
<td>{classify?.data_format}</td>
</tr>
<tr>
<th scope="row">BED compliance</th>
<td>{classify?.bed_compliance}</td>
</tr>
<tr>
<th scope="row">Identifier</th>
<td>{rs.identifier}</td>
</tr>
<tr>
<th scope="row">Mean region width</th>
<td>{rs.meanRegionWidth}</td>
</tr>
<tr>
<th scope="row">Total number of regions</th>
<td>{rs.numberOfRegions}</td>
</tr>
<tr>
<th scope="row">Total number of nucleotides</th>
<td>{rs.nucleotidesLength}</td>
</tr>
<tr>
<th scope="row">Data Format</th>
<td>{classify?.data_format}</td>
</tr>
<tr>
<th scope="row">BED compliance</th>
<td>{classify?.bed_compliance}</td>
</tr>
</tbody>
</table>
<div className="mt-3">
Expand Down Expand Up @@ -413,4 +415,4 @@ export const BEDAnalytics = () => {
)}
</Layout>
);
};
};
112 changes: 112 additions & 0 deletions ui/src/workers/bedAnalyzerWorker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
import init, {
bedParserNew,
bedParserUpdate,
bedParserFinish,
bedParserFree,
} from '@databio/gtars';

let wasmReady = false;

async function ensureWasm() {
if (wasmReady) return;
await init();
wasmReady = true;
}

async function processStream(
reader: ReadableStreamDefaultReader<Uint8Array>,
totalSize: number,
parser: number,
) {
let bytesProcessed = 0;

while (true) {
const { done, value } = await reader.read();
if (done) break;

bedParserUpdate(parser, value);

bytesProcessed += value.length;
self.postMessage({
type: 'progress',
bytesProcessed,
totalSize,
percent: totalSize > 0 ? Math.round((100 * bytesProcessed) / totalSize) : -1,
});
Comment on lines +29 to +35
Copy link

Copilot AI Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Progress updates are posted to the main thread for every streamed chunk. For large files this can generate a very high message rate and noticeably slow parsing/UI updates. Consider throttling these messages (e.g., post at most every N ms or every M bytes, always sending a final 100% update at the end).

Copilot uses AI. Check for mistakes.
}
}

async function processFile(file: File) {
self.postMessage({ type: 'status', message: 'Loading WASM module...' });
await ensureWasm();

const parser = bedParserNew();

try {
self.postMessage({ type: 'status', message: 'Processing file...' });

const stream = file.stream();
const reader = stream.getReader();
await processStream(reader, file.size, parser);

self.postMessage({ type: 'status', message: 'Building RegionSet...' });
const entries = bedParserFinish(parser);

self.postMessage({ type: 'result', entries });
} catch (err) {
bedParserFree(parser);
throw err;
}
}

async function processUrl(url: string) {
self.postMessage({ type: 'status', message: 'Loading WASM module...' });
await ensureWasm();

self.postMessage({ type: 'status', message: 'Fetching file...' });

const fetchUrl =
url.length === 32 && !url.startsWith('http')
? `https://api.bedbase.org/v1/files/files/${url[0]}/${url[1]}/${url}.bed.gz`
: url;

const response = await fetch(fetchUrl);
if (!response.ok) {
throw new Error(`Failed to fetch BED file: ${response.statusText}`);
}

const parser = bedParserNew();

try {
self.postMessage({ type: 'status', message: 'Processing file...' });

const contentLength = response.headers.get('content-length');
const totalSize = contentLength ? parseInt(contentLength, 10) : 0;
const reader = response.body!.getReader();
Copy link

Copilot AI Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

response.body can be null (e.g., for certain response types or environments), but this code uses a non-null assertion and will throw at runtime if it is. Handle the null case explicitly and emit a worker error message (or throw a clearer error) before calling getReader().

Suggested change
const reader = response.body!.getReader();
const body = response.body;
if (!body) {
throw new Error('Failed to process BED file: response body is not available as a readable stream.');
}
const reader = body.getReader();

Copilot uses AI. Check for mistakes.
await processStream(reader, totalSize, parser);

self.postMessage({ type: 'status', message: 'Building RegionSet...' });
const entries = bedParserFinish(parser);

self.postMessage({ type: 'result', entries });
} catch (err) {
bedParserFree(parser);
throw err;
}
}

self.onmessage = async (e: MessageEvent) => {
const { file, url } = e.data;

try {
if (file) {
await processFile(file);
} else if (url) {
await processUrl(url);
} else {
throw new Error('No file or URL provided');
}
} catch (error) {
self.postMessage({ type: 'error', message: (error as Error).message });
}
};