Waste water data submission #448

leoraba · 2024-12-05T16:59:49Z

Summary

To be able to prepare, validate and submit environmental Data.

Details

Updated Next.js dynamic routes to load environmental or clinical submission pages.
Created 3 new environment variables to configure Environmental Submission Service.
Environmental submission React components are in a folder components/pages/submission/environmental and Clinical are in components/pages/submission/clinical
Using available endpoints from submission-service to upload, commit and keep track of status and systemIds of records within the Submission
Using available endpoint from submission-service to get Historial of previous submissions

leoraba · 2024-12-05T17:00:31Z

.env.schema

+NEXT_PUBLIC_ENVIRONMENTAL_SUBMISSION_API_URL=
+NEXT_PUBLIC_ENVIRONMENTAL_SUBMISSION_CATEGORY_ID=


environment variables to set up Submission service

what's a category id in this context?

Lyric data category ID. Lyric is built to manage data of multiple data-dictionaries and multiple submitting projects, so when submitting we need to identify both which data category (data-dictionary) we are submitting for, and what our project ID is. Not sure if we have a separate env var for the project ID or not...

components/pages/submission/Clinical/index.tsx

justincorrigible · 2024-12-11T19:41:16Z

components/pages/submission/Environmental/NewSubmissions/DropZone.tsx

@@ -48,7 +48,7 @@ const DropZone = ({
 		isDragActive,
 		// isFileTooLarge,
 	} = useDropzone({
-		accept: '.fa,.gz,.fasta,.tsv,text/tab-separated-values',
+		accept: '.csv',


would it be valid/desirable to also accept tsv?

Do you mean to convert it to .csv for them? Or is this expecting that they may submit .tsv in the future and we should accept that?

components/pages/submission/Environmental/NewSubmissions/index.tsx

joneubank · 2025-01-03T15:04:55Z

components/pages/submission/Clinical/index.tsx


-import NewSubmissions from './NewSubmissions';
-import PreviousSubmissions from './PreviousSubmissions';
+import PageContent from './PageContent';


Did we name this component? It is an extremely vague component name.

joneubank

Great stuff, lots of good work here. No major problems found so I am approving, but I left a handful of comments mostly around type management.

joneubank · 2025-01-03T15:06:41Z

components/pages/submission/Environmental/NewSubmissions/index.tsx

+				// TODO: Verify if the user's session permissions allow them
+				// to upload environmental data to this organization.


We have userHasWriteScopes flag in the surrounding if statement... is this an additional check for the specific organization the data is submitted for?

joneubank · 2025-01-03T15:08:53Z

components/pages/submission/Environmental/NewSubmissions/index.tsx

+					case 'INVALID_FILE_EXTENSION':
+					case 'FILE_READ_ERROR':
+					case 'UNRECOGNIZED_HEADER':
+					case 'MISSING_REQUIRED_HEADER': {


Should we communicate the specific error?

joneubank · 2025-01-03T15:10:04Z

components/pages/submission/Environmental/NewSubmissions/index.tsx

+				Waste water metadata is submitted as a <span className="code">.csv</span> file .The file
+				name must match the Study name for the Submission. Multiple files are accepted.


Good opportunity for an example valid file name?

joneubank · 2025-01-03T15:12:51Z

components/pages/submission/Environmental/PreviousSubmissions/index.tsx

+				// TODO: Fetch list of previous submisions
+				// setPreviousSubmissions();


Use tickets not TODOs?

joneubank · 2025-01-03T15:15:41Z

components/pages/submission/clinical/PageContent.tsx

@@ -0,0 +1,81 @@
+/*
+ *
+ * Copyright (c) 2021 The Ontario Institute for Cancer Research. All rights reserved


Copyright date

joneubank · 2025-01-03T16:16:48Z

global/hooks/useEnvironmentalData/index.ts

+							return parsedStream && JSON.parse(parsedStream);
+						})
+						.catch((error: Error | string) => {
+							console.log(`Offending stream at ${origin}`, stream);


This error log message is not very helpful. If I try to upload something and the log says "Offending stream" I will have more questions than answers. At least the text is unique enough that we can find our way back to the source code, but an error message with more context would be better.

joneubank · 2025-01-03T16:51:07Z

global/hooks/useEnvironmentalData/index.ts

+	const getErrorDetailsMessage = (
+		errors: ErrorDetails[],
+		index: number,
+	): { status: 'ERROR' | 'PROCESSING' | 'COMPLETE' | 'QUEUED'; message: string } => {


Return type is a bit weird... This function could be returning { status: 'PROCESSING' } | { status: 'ERROR'; message: string }. We never return status 'COMPLETE' or 'QUEUED' and we return an empty message sometimes...

If we do intend to have the status return any of the statuses, we should type this like status: UploadStatusType

joneubank · 2025-01-03T16:51:56Z

global/hooks/useEnvironmentalData/index.ts

+	 * @param param0
+	 * @returns
+	 */
+	const submitData = async ({ body }: { body: FormData }) => {


This is returning Promise<any>. Can we clean that typing up at all?

joneubank · 2025-01-03T17:09:28Z

global/hooks/useEnvironmentalData/types.ts

As a general rule, we don't want to use the suffix Type in our types. Same as we don't call our variables things like messageString or countInt. Aim for descriptive names so that when a function says it requires an UploadData parameter I know to pass the actual object with UploadData and not an UploadDataType type definition.

joneubank · 2025-01-03T17:10:51Z

global/hooks/useEnvironmentalData/types.ts

+export type DataRecordValue =
+	| string
+	| string[]
+	| number
+	| number[]
+	| boolean
+	| boolean[]
+	| undefined;
+export type DataRecord = Record<string, DataRecordValue>;


Looks like we are replicating some TSV parsing types that are used in lectern/lyric. Might be a sign we want a separate package dedicated to tabular data file work.

data submission

b8660aa

leoraba commented Dec 5, 2024

View reviewed changes

leoraba requested review from justincorrigible and joneubank December 5, 2024 17:07

leoraba added 3 commits December 5, 2024 15:14

organization name

4b8a029

Update submission instructions

07080e0

get organization name from file name

b320bb9

leoraba force-pushed the feat/environmental-submission branch from 59b5097 to b320bb9 Compare December 5, 2024 21:28

justincorrigible reviewed Dec 11, 2024

View reviewed changes

components/pages/submission/Clinical/index.tsx Outdated Show resolved Hide resolved

justincorrigible reviewed Dec 11, 2024

View reviewed changes

components/pages/submission/Environmental/NewSubmissions/index.tsx Show resolved Hide resolved

Capitalize first letter filename

ff5f64e

leoraba force-pushed the feat/environmental-submission branch from 6cba35c to d050c70 Compare December 11, 2024 21:34

revert copyright date

ba39caa

leoraba force-pushed the feat/environmental-submission branch from d050c70 to ba39caa Compare December 11, 2024 21:39

joneubank reviewed Jan 3, 2025

View reviewed changes

joneubank approved these changes Jan 3, 2025

View reviewed changes

This was referenced Jan 9, 2025

WW - Submission Summary Page virusseq/roadmap#54

Closed

WW - Build Submission Dashboard UI virusseq/roadmap#51

Closed

leoraba linked an issue Jan 9, 2025 that may be closed by this pull request

WW - Build Submission Dashboard UI virusseq/roadmap#51

Closed

leoraba added 5 commits January 9, 2025 21:18

fix types

4a624fa

Upload error message

cc290aa

fix error types

2e9ff03

new env variable sample ID

37a73ff

Abort asynchronous calls using AbortController

9120d6e

leoraba linked an issue Jan 13, 2025 that may be closed by this pull request

WW - Submission Summary Page virusseq/roadmap#54

Closed

leoraba merged commit 96a7e80 into wastewater Jan 13, 2025
2 checks passed

leoraba deleted the feat/environmental-submission branch January 13, 2025 14:17

leoraba mentioned this pull request Jan 13, 2025

WW - Built authentication on lyric virusseq/roadmap#53

Closed

leoraba linked an issue Jan 13, 2025 that may be closed by this pull request

WW - Built authentication on lyric virusseq/roadmap#53

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Waste water data submission #448

Waste water data submission #448

leoraba commented Dec 5, 2024 •

edited

Loading

leoraba Dec 5, 2024

justincorrigible Dec 5, 2024

joneubank Jan 3, 2025 •

edited

Loading

justincorrigible Dec 11, 2024

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank left a comment

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

joneubank Jan 3, 2025

		NEXT_PUBLIC_ENVIRONMENTAL_SUBMISSION_API_URL=
		NEXT_PUBLIC_ENVIRONMENTAL_SUBMISSION_CATEGORY_ID=

		// TODO: Verify if the user's session permissions allow them
		// to upload environmental data to this organization.

		Waste water metadata is submitted as a <span className="code">.csv</span> file .The file
		name must match the Study name for the Submission. Multiple files are accepted.

		// TODO: Fetch list of previous submisions
		// setPreviousSubmissions();

Waste water data submission #448

Waste water data submission #448

Conversation

leoraba commented Dec 5, 2024 • edited Loading

Summary

Details

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneubank Jan 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneubank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leoraba commented Dec 5, 2024 •

edited

Loading

joneubank Jan 3, 2025 •

edited

Loading