[CAT-12] Workflow Metrics in the SDK (#114)

* items submitted

* workflow metrics changes

* updated sphinx docs

Author: goatrocks

docs/_sources/auto-review.rst.txt

@@ -13,7 +13,7 @@ Using Auto Review
 Auto review is currently only available via api call. To use it, the following steps must be performed:
 
 - Submit a document through your workflow
-- Retreive the submission
+- Retrieve the submission
 - Add a flag on each prediction result you wish to Auto Review in the submission to designate if you would like to accept or reject it
 - Submit the changes to Review
 
@@ -31,7 +31,7 @@ Example usage of Auto Review::
             workflow_id=workflow_id, files=["my_file.pdf"]
         )
     ) 
-    # Retreive the submission results
+    # Retrieve the submission results
     submissions = client.call(WaitForSubmissions(submission_ids))
     submission = submissions[0]
     raw_result = client.call(RetrieveStorageObject(submission.result_file))
@@ -79,7 +79,7 @@ When submitting changes for a given document using SubmitReview, you can reject
 
 Accepting No Value
 ==================
-Even when all predictions are accepted, a reviewer will still need to manually accept any labels that are empty. However, you can automatically accept empty values for specific lables using the "_no_value" key. Within the model results, set the dictionary key "_no_value" equal to a dictionary containing the name of your model keying to a list of labels. This list of labels should contain all the labels you would like to automatically accept empty values for. Accepted empty labels will appear in review as accepted labels. Please note that you can only auto-accept empty labels that do not contain any predictions. Labels that contain predictions should not be inlcuded in the list of no value labels. In the example below empty labels are only accepted if the specified label is not present in the predictions::
+Even when all predictions are accepted, a reviewer will still need to manually accept any labels that are empty. However, you can automatically accept empty values for specific labels using the "_no_value" key. Within the model results, set the dictionary key "_no_value" equal to a dictionary containing the name of your model keying to a list of labels. This list of labels should contain all the labels you would like to automatically accept empty values for. Accepted empty labels will appear in review as accepted labels. Please note that you can only auto-accept empty labels that do not contain any predictions. Labels that contain predictions should not be inlcuded in the list of no value labels. In the example below empty labels are only accepted if the specified label is not present in the predictions::
 
     from indico.queries import (
         WorkflowSubmission,
@@ -93,7 +93,7 @@ Even when all predictions are accepted, a reviewer will still need to manually a
             workflow_id=workflow_id, files=["my_file.pdf"]
         )
     ) 
-    # Retreive the submission results
+    # Retrieve the submission results
     submissions = client.call(WaitForSubmissions(submission_ids))
     submission = submissions[0]
     raw_result = client.call(RetrieveStorageObject(submission.result_file))

docs/_sources/examples.rst.txt

@@ -15,3 +15,4 @@ Examples
     image-predictions-ex
     submission-ex
     user-metrics-ex
+    workflow-metrics-ex

docs/_sources/index.rst.txt

@@ -41,6 +41,7 @@ Reference
     filters
     graphql_queries
     user_metrics
+    workflow-metrics
 
 **Types**
 
@@ -52,3 +53,4 @@ Reference
     model-group-type
     model-type
     user-metrics-types
+    workflow-metrics-types

docs/_sources/workflow-metrics-ex.rst.txt

@@ -0,0 +1,6 @@
+Fetching Workflow Metrics
+***********************************************************
+
+An example that illustrates how to fetch worklow metric data.
+
+.. literalinclude:: ../examples/get-workflow-metrics.py

docs/_sources/workflow-metrics-types.rst.txt

@@ -0,0 +1,23 @@
+**********************
+Workflow Metrics Types
+**********************
+
+.. automodule:: indico.types.workflow_metrics
+.. autoclass:: indico.types.workflow_metrics.WorkflowMetricsOptions
+.. autoclass:: indico.types.workflow_metrics.TimeOnTaskMetric
+.. autoclass:: indico.types.workflow_metrics.TimeOnTaskMetrics
+.. autoclass:: indico.types.workflow_metrics.DailyTimeOnTaskMetric
+.. autoclass:: indico.types.workflow_metrics.SubmissionMetric
+.. autoclass:: indico.types.workflow_metrics.DailySubmissionMetric
+.. autoclass:: indico.types.workflow_metrics.DailyQueueMetric
+.. autoclass:: indico.types.workflow_metrics.QueueMetrics
+.. autoclass:: indico.types.workflow_metrics.PredictionMetric
+.. autoclass:: indico.types.workflow_metrics.PredictionMetrics
+.. autoclass:: indico.types.workflow_metrics.DailyPredictionMetric
+.. autoclass:: indico.types.workflow_metrics.StpMetric
+.. autoclass:: indico.types.workflow_metrics.DailyStpMetric
+.. autoclass:: indico.types.workflow_metrics.ClassStpMetrics
+.. autoclass:: indico.types.workflow_metrics.ModelStpMetrics
+.. autoclass:: indico.types.workflow_metrics.WorkflowStpMetrics
+.. autoclass:: indico.types.workflow_metrics.StraightThroughProcessing
+.. autoclass:: indico.types.workflow_metrics.WorkflowMetrics

docs/_sources/workflow-metrics.rst.txt

@@ -0,0 +1,7 @@
+*****************
+Workflow Metrics
+*****************
+
+.. autoclass:: indico.queries.workflow_metrics.GetWorkflowMetrics
+
+

docs/auto-review.html

@@ -124,12 +124,14 @@
 <li class="toctree-l1"><a class="reference internal" href="exports.html">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 
@@ -210,7 +212,7 @@ <h2>Using Auto Review<a class="headerlink" href="#using-auto-review" title="Perm
 <p>Auto review is currently only available via api call. To use it, the following steps must be performed:</p>
 <ul class="simple">
 <li><p>Submit a document through your workflow</p></li>
-<li><p>Retreive the submission</p></li>
+<li><p>Retrieve the submission</p></li>
 <li><p>Add a flag on each prediction result you wish to Auto Review in the submission to designate if you would like to accept or reject it</p></li>
 <li><p>Submit the changes to Review</p></li>
 </ul>
@@ -227,7 +229,7 @@ <h2>Using Auto Review<a class="headerlink" href="#using-auto-review" title="Perm
         <span class="n">workflow_id</span><span class="o">=</span><span class="n">workflow_id</span><span class="p">,</span> <span class="n">files</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;my_file.pdf&quot;</span><span class="p">]</span>
     <span class="p">)</span>
 <span class="p">)</span>
-<span class="c1"># Retreive the submission results</span>
+<span class="c1"># Retrieve the submission results</span>
 <span class="n">submissions</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">WaitForSubmissions</span><span class="p">(</span><span class="n">submission_ids</span><span class="p">))</span>
 <span class="n">submission</span> <span class="o">=</span> <span class="n">submissions</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
 <span class="n">raw_result</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">RetrieveStorageObject</span><span class="p">(</span><span class="n">submission</span><span class="o">.</span><span class="n">result_file</span><span class="p">))</span>
@@ -269,7 +271,7 @@ <h2>Auto Reject a Document<a class="headerlink" href="#auto-reject-a-document" t
 </div>
 <div class="section" id="accepting-no-value">
 <h2>Accepting No Value<a class="headerlink" href="#accepting-no-value" title="Permalink to this headline">¶</a></h2>
-<p>Even when all predictions are accepted, a reviewer will still need to manually accept any labels that are empty. However, you can automatically accept empty values for specific lables using the “_no_value” key. Within the model results, set the dictionary key “_no_value” equal to a dictionary containing the name of your model keying to a list of labels. This list of labels should contain all the labels you would like to automatically accept empty values for. Accepted empty labels will appear in review as accepted labels. Please note that you can only auto-accept empty labels that do not contain any predictions. Labels that contain predictions should not be inlcuded in the list of no value labels. In the example below empty labels are only accepted if the specified label is not present in the predictions:</p>
+<p>Even when all predictions are accepted, a reviewer will still need to manually accept any labels that are empty. However, you can automatically accept empty values for specific labels using the “_no_value” key. Within the model results, set the dictionary key “_no_value” equal to a dictionary containing the name of your model keying to a list of labels. This list of labels should contain all the labels you would like to automatically accept empty values for. Accepted empty labels will appear in review as accepted labels. Please note that you can only auto-accept empty labels that do not contain any predictions. Labels that contain predictions should not be inlcuded in the list of no value labels. In the example below empty labels are only accepted if the specified label is not present in the predictions:</p>
 <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">indico.queries</span> <span class="kn">import</span> <span class="p">(</span>
     <span class="n">WorkflowSubmission</span><span class="p">,</span>
     <span class="n">WaitForSubmissions</span><span class="p">,</span>
@@ -282,7 +284,7 @@ <h2>Accepting No Value<a class="headerlink" href="#accepting-no-value" title="Pe
         <span class="n">workflow_id</span><span class="o">=</span><span class="n">workflow_id</span><span class="p">,</span> <span class="n">files</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;my_file.pdf&quot;</span><span class="p">]</span>
     <span class="p">)</span>
 <span class="p">)</span>
-<span class="c1"># Retreive the submission results</span>
+<span class="c1"># Retrieve the submission results</span>
 <span class="n">submissions</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">WaitForSubmissions</span><span class="p">(</span><span class="n">submission_ids</span><span class="p">))</span>
 <span class="n">submission</span> <span class="o">=</span> <span class="n">submissions</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
 <span class="n">raw_result</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">RetrieveStorageObject</span><span class="p">(</span><span class="n">submission</span><span class="o">.</span><span class="n">result_file</span><span class="p">))</span>

docs/client.html

@@ -118,12 +118,14 @@
 <li class="toctree-l1"><a class="reference internal" href="exports.html">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 

docs/config.html

@@ -44,7 +44,7 @@
     <link rel="index" title="Index" href="genindex.html" />
     <link rel="search" title="Search" href="search.html" />
     <link rel="next" title="IndicoClient" href="client.html" />
-    <link rel="prev" title="Sending New Samples to a Workflow (aka Workflow Submission)" href="submission-ex.html" /> 
+    <link rel="prev" title="Fetching User Metrics" href="user-metrics-ex.html" /> 
 </head>
 
 <body class="wy-body-for-nav">
@@ -118,12 +118,14 @@
 <li class="toctree-l1"><a class="reference internal" href="exports.html">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 
@@ -234,7 +236,7 @@
           <footer>
     <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
         <a href="client.html" class="btn btn-neutral float-right" title="IndicoClient" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
-        <a href="submission-ex.html" class="btn btn-neutral float-left" title="Sending New Samples to a Workflow (aka Workflow Submission)" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
+        <a href="user-metrics-ex.html" class="btn btn-neutral float-left" title="Fetching User Metrics" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
     </div>
 
   <hr/>

docs/datasets.html

@@ -118,12 +118,14 @@
 <li class="toctree-l1"><a class="reference internal" href="exports.html">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 

docs/docextract.html

@@ -118,12 +118,14 @@
 <li class="toctree-l1"><a class="reference internal" href="exports.html">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 

docs/examples.html

@@ -114,6 +114,7 @@
 <li class="toctree-l2"><a class="reference internal" href="image-predictions-ex.html">Get Predictions from Image Model</a></li>
 <li class="toctree-l2"><a class="reference internal" href="submission-ex.html">Sending New Samples to a Workflow (aka Workflow Submission)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="user-metrics-ex.html">Fetching User Metrics</a></li>
+<li class="toctree-l2"><a class="reference internal" href="workflow-metrics-ex.html">Fetching Workflow Metrics</a></li>
 </ul>
 </li>
 </ul>
@@ -131,12 +132,15 @@
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
 <li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
+<li class="toctree-l1"><a class="reference internal" href="workflow-metrics.html">Workflow Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="workflow-metrics-types.html">User Metrics Types</a></li>
 </ul>
 
 
@@ -218,6 +222,7 @@ <h1>Examples<a class="headerlink" href="#examples" title="Permalink to this head
 <li class="toctree-l1"><a class="reference internal" href="image-predictions-ex.html">Get Predictions from Image Model</a></li>
 <li class="toctree-l1"><a class="reference internal" href="submission-ex.html">Sending New Samples to a Workflow (aka Workflow Submission)</a></li>
 <li class="toctree-l1"><a class="reference internal" href="user-metrics-ex.html">Fetching User Metrics</a></li>
+<li class="toctree-l1"><a class="reference internal" href="workflow-metrics-ex.html">Fetching Workflow Metrics</a></li>
 </ul>
 </div>
 </div>

docs/exports.html

@@ -118,12 +118,14 @@
 <li class="toctree-l1 current"><a class="current reference internal" href="#">Exports</a></li>
 <li class="toctree-l1"><a class="reference internal" href="filters.html">Filters</a></li>
 <li class="toctree-l1"><a class="reference internal" href="graphql_queries.html">Common GraphQL Queries</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user_metrics.html">User Metrics</a></li>
 </ul>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="dataset-type.html">Dataset Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="jobs-type.html">Jobs Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-group-type.html">Model Group Types</a></li>
 <li class="toctree-l1"><a class="reference internal" href="model-type.html">Model Types</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-metrics-types.html">User Metrics Types</a></li>
 </ul>