Merge pull request #513 from Police-Data-Accessibility-Project/mc_ds_868_syncs

mc_ds_868_syncs

Author: maxachis

ENV.md

@@ -57,19 +57,28 @@ Note that some tasks/subtasks are themselves enabled by other tasks.
 
 ### Scheduled Task Flags
 
-| Flag                                | Description                                                                   |
-|-------------------------------------|-------------------------------------------------------------------------------|
-| `SCHEDULED_TASKS_FLAG`              | All scheduled tasks. Disabling disables all other scheduled tasks.            |
-| `PUSH_TO_HUGGING_FACE_TASK_FLAG`    | Pushes data to HuggingFace.                                                   |
-| `POPULATE_BACKLOG_SNAPSHOT_TASK_FLAG` | Populates the backlog snapshot.                                               |
-| `DELETE_OLD_LOGS_TASK_FLAG`         | Deletes old logs.                                                             |
-| `RUN_URL_TASKS_TASK_FLAG`           | Runs URL tasks.                                                               |
-| `IA_PROBE_TASK_FLAG`            | Extracts and links Internet Archives metadata to URLs.                        |
-| `IA_SAVE_TASK_FLAG`             | Saves URLs to Internet Archives.                                              |
-| `MARK_TASK_NEVER_COMPLETED_TASK_FLAG` | Marks tasks that were started but never completed (usually due to a restart). |
-| `DELETE_STALE_SCREENSHOTS_TASK_FLAG` | Deletes stale screenshots for URLs already validated.                         |
-| `TASK_CLEANUP_TASK_FLAG` | Cleans up tasks that are no longer needed. |
+| Flag                                   | Description                                                                   |
+|----------------------------------------|-------------------------------------------------------------------------------|
+| `SCHEDULED_TASKS_FLAG`                 | All scheduled tasks. Disabling disables all other scheduled tasks.            |
+| `PUSH_TO_HUGGING_FACE_TASK_FLAG`       | Pushes data to HuggingFace.                                                   |
+| `POPULATE_BACKLOG_SNAPSHOT_TASK_FLAG`  | Populates the backlog snapshot.                                               |
+| `DELETE_OLD_LOGS_TASK_FLAG`            | Deletes old logs.                                                             |
+| `RUN_URL_TASKS_TASK_FLAG`              | Runs URL tasks.                                                               |
+| `IA_PROBE_TASK_FLAG`                   | Extracts and links Internet Archives metadata to URLs.                        |
+| `IA_SAVE_TASK_FLAG`                    | Saves URLs to Internet Archives.                                              |
+| `MARK_TASK_NEVER_COMPLETED_TASK_FLAG`  | Marks tasks that were started but never completed (usually due to a restart). |
+| `DELETE_STALE_SCREENSHOTS_TASK_FLAG`   | Deletes stale screenshots for URLs already validated.                         |
+| `TASK_CLEANUP_TASK_FLAG`               | Cleans up tasks that are no longer needed. |
 | `REFRESH_MATERIALIZED_VIEWS_TASK_FLAG` | Refreshes materialized views. |
+| `DS_APP_SYNC_AGENCY_ADD_TASK_FLAG`     | Adds new agencies to the Data Sources App|
+| `DS_APP_SYNC_AGENCY_UPDATE_TASK_FLAG`       | Updates existing agencies in the Data Sources App|
+| `DS_APP_SYNC_AGENCY_DELETE_TASK_FLAG`       | Deletes agencies in the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_ADD_TASK_FLAG`     | Adds new data sources to the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_UPDATE_TASK_FLAG`  | Updates existing data sources in the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_DELETE_TASK_FLAG`  | Deletes data sources in the Data Sources App|
+| `DS_APP_SYNC_META_URL_ADD_TASK_FLAG`        | Adds new meta URLs to the Data Sources App|
+| `DS_APP_SYNC_META_URL_UPDATE_TASK_FLAG`     | Updates existing meta URLs in the Data Sources App|
+| `DS_APP_SYNC_META_URL_DELETE_TASK_FLAG`     | Deletes meta URLs in the Data Sources App|
 
 ### URL Task Flags
 
@@ -81,7 +90,6 @@ URL Task Flags are collectively controlled by the `RUN_URL_TASKS_TASK_FLAG` flag
 | `URL_HTML_TASK_FLAG`                | URL HTML scraping task.                               |
 | `URL_RECORD_TYPE_TASK_FLAG`         | Automatically assigns Record Types to URLs.           |
 | `URL_AGENCY_IDENTIFICATION_TASK_FLAG` | Automatically assigns and suggests Agencies for URLs. |
-| `URL_SUBMIT_APPROVED_TASK_FLAG`     | Submits approved URLs to the Data Sources App.        |
 | `URL_MISC_METADATA_TASK_FLAG`       | Adds misc metadata to URLs.                           |
 | `URL_AUTO_RELEVANCE_TASK_FLAG`      | Automatically assigns Relevances to URLs.             |
 | `URL_PROBE_TASK_FLAG`               | Probes URLs for web metadata.                         |
@@ -90,7 +98,6 @@ URL Task Flags are collectively controlled by the `RUN_URL_TASKS_TASK_FLAG` flag
 | `URL_AUTO_VALIDATE_TASK_FLAG`       | Automatically validates URLs.                         |
 | `URL_AUTO_NAME_TASK_FLAG`           | Automatically names URLs.                             |
 | `URL_SUSPEND_TASK_FLAG`             | Suspends URLs meeting suspension criteria.            |
-| `URL_SUBMIT_META_URLS_TASK_FLAG`    | Submits meta URLs to the Data Sources App.            |
 
 ### Agency ID Subtasks