Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Templating tool for simple pipelines #31

Open
2 tasks done
adlersantos opened this issue May 20, 2021 · 0 comments
Open
2 tasks done

Templating tool for simple pipelines #31

adlersantos opened this issue May 20, 2021 · 0 comments
Labels
feature request New feature or request

Comments

@adlersantos
Copy link
Member

adlersantos commented May 20, 2021

Description

Instead of going through the pipeline development instructions for simple pipelines (such as a simple CSV to BigQuery dump), we can use a templating tool that creates all the necessary files and folders, with the pipeline config being auto-populated based on some set of inputs.

For example,

$ create_pipeline_template \
    --pipeline-group "yadda" \
    --pipeline-name "my_simple_pipeline" \
    --source="https://some/data.csv" \
    --destination-type "bigquery" \
    --destination-name "dataset.some_bigquery_table" \
    --rename-column "Number of Cases:cases" \
    --rename-column "Date:date" \
    --remove-null "column=date"

Checklist

  • I created this issue in accordance with the Code of Conduct.
  • This issue is appropriately labeled.
@adlersantos adlersantos added the feature request New feature or request label May 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant