ETL Unfinished Features List

[gtarcea]

• For samples show all measurements for each sample attribute
• The samples/processes pages under experiment is different from the samples/processes pages under projects
• FILE: Keyword not processed
• Check existence of files on server during the checking stage
• Experiments that are not finished being built can still be accessed from the sidebar
• Mark Warnings and Errors differently so user can tell them apart
• Sort list of sample attributes alphabetically
• Support new ETL on project experiment page (like V1 does now)
• NOTE: Keyword not processed
• Delete Experiment leaves a bunch of cruft in the database (its only doing enough to remove experiment, processes and samples from the UI)
• Cannot specify multiple spreadsheets in the UI
• Spreadsheet is not associated with samples/processes from the spreadsheet
• ETL Doesn't give feedback if it encounters problems while performing the actual load (as opposed to the initial parse)
• Multiple processes per worksheet is not supported
• Spreadsheet load status still requires the user to press the sync button
• If experiment exists at time of load (not time of parse) then ETL will silently fail
• There is a limit on the number of errors that can be sent back (not sure where limit is) and the system will silently fail when that limit is reached
• If servers are shutdown during spreadsheet processing that ETL job will be left in limbo
• If a sample appears multiple times in a worksheet with a parent column then the created experiment workflow is not correct (and no error or warning is generated)
• There is no way to specify custom keywords
• If a process attribute is specified multiple times then it results in multiple entries for that process attribute (don't know what the correct behavior should be)
• No support for supporting Branden's case of sample states rather than processes when constructing workflow
• The UI shows sample states as their property set id rather than something meaningful
• There isn't any way to specify a measurement as a best measure in the spreadsheet
• Samples/Processes lists aren't paged, resulting in long load times if there are large numbers of samples or processes
• All measurements and process attributes are attributed as "OBJECT" type meaning there is no way to edit them in the UI
• The workflow editor doesn't show most measurements in the samples Details pages

[stvdwtt]

I'm not sure that "Create Experiment from File" is the most clear way to communicate that a user can ETL the file. I think having the word spreadsheet in the button communicates that this is an option only for spreadsheets (you can't create an experiment from an image). I think using the word parse would clarify what's going to happen, but maybe that word would trip some people up.

Possible alternates:
Parse File to Create Experiment
Parse Spreadsheet to Create Experiment
Create Experiment from this Spreadsheet
Create Experiment from Spreadsheet

Some of these are a bit longer than the current wording, which is not great.

I'm happy to be over-ruled if everyone else thinks the current wording is clear.

[stvdwtt]

Comments on the ETL screen:

• The text is great, I think it gives the right level of detail. Eventually, it should link to an ETL FAQ page (which we'd have to make)
• I think it'd be worth showing a no-parent example, right under the parent example
• It isn't clear that the "Name" field at the bottom is for the experiment name. At first I thought I was supposed to put the parent name in there after checking the parent box, until I thought about it for a couple seconds and realized that didn't make sense. I recommend changing that initial grayed out text to "Name for the new experiment"

[stvdwtt]

Comments on the ETL validation screen:

I got what I think are a bunch of warnings on the ETL validation screen (see below). I would be helpful to communicate how important these are. Is this just a log of normal events? Am I being warned of non-fatal abnormalities? Are these fatal errors? The text at the top tells be that ETL validation succeeded, so I'm guessing that these aren't fatal errors.

[stvdwtt]

Comment on the "ETL is being run in the background" message:

The message says to hit "Sync" to check if ETL is completed, but doesn't tell me how to know if it has completed.

Suggestion:
Have something along the lines of:
"The new experiment will appear on the Experiments page (accessible from the left sidebar) when ETL is complete. Click "Sync" on the sidebar to refresh that page."