Skip to content

Commit f5947f5

Browse files
jim-barlowjim-barlow
committed
updated bqtools docs
1 parent 2450311 commit f5947f5

File tree

7 files changed

+421
-7
lines changed

7 files changed

+421
-7
lines changed

‎docs/reference/bqmanager/index.md renamed to ‎docs/.archive/bqmanager/index.md

File renamed without changes.

‎docs/reference/bqtools/resources/create.md renamed to ‎docs/reference/bqtools/functions/create.md

+11-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,15 @@
1-
These functions are used to create resources from SQL definitions and options.
1+
These functions are used to create resources from SQL definitions and deployment-specific options.
22

3-
# **`create_function`**
3+
4+
5+
6+
7+
8+
9+
10+
11+
12+
# **create_function**
413
_**Attribute**_ | Value
514
--- | ---
615
_**Name**_ | `create_function`

‎docs/reference/bqtools/functions/metadata.md

+205
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
These functions are used to query resource metadata to support efficient, automated data workflows.
2+
3+
# **GET Date Partitions**
4+
_**Attribute**_ | Value
5+
--- | ---
6+
_**Function Name**_ | `get_date_partitions`
7+
_**ID**_ | `bqtools.[region].get_date_partitions`
8+
_**Latest Version**_ | `bqtools:v1.0.0`
9+
_**Description**_ | Returns an array of dates corresponding to all existing partitions in a single partitioned table, sorted by descending date.
10+
_**Type**_ | `PROCEDURE`
11+
_**Arguments**_ | `partitioned_table_id STRING, OUT partitions ARRAY<DATE>`
12+
_**Returns**_ | `OUT partitions ARRAY<DATE>`
13+
_**Dependencies**_ | `bqtools-qb.[region].get_date_partitions`
14+
15+
!!! info "Function Usage"
16+
=== "EU"
17+
```sql
18+
DECLARE partitioned_table_id STRING;
19+
DECLARE partitions ARRAY<DATE>;
20+
21+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
22+
23+
CALL bqtools.eu.get_date_partitions(partitioned_table_id, partitions);
24+
```
25+
26+
=== "US"
27+
```sql
28+
DECLARE partitioned_table_id STRING;
29+
DECLARE partitions ARRAY<DATE>;
30+
31+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
32+
33+
CALL bqtools.us.get_date_partitions(partitioned_table_id, partitions);
34+
```
35+
36+
# **GET First Date Partition**
37+
_**Attribute**_ | Value
38+
--- | ---
39+
_**Function Name**_ | `get_first_date_partition`
40+
_**ID**_ | `bqtools.[region].get_first_date_partition`
41+
_**Latest Version**_ | `bqtools:v1.0.0`
42+
_**Description**_ | Returns the first partition date from a single date-partitioned table.
43+
_**Type**_ | `PROCEDURE`
44+
_**Arguments**_ | `partitioned_table_id STRING, OUT first_partition DATE`
45+
_**Returns**_ | `OUT first_partition DATE`
46+
_**Dependencies**_ | `bqtools.[region].get_date_partitions`
47+
48+
!!! info "Function Usage"
49+
=== "EU"
50+
```sql
51+
DECLARE partitioned_table_id STRING;
52+
DECLARE first_partition DATE;
53+
54+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
55+
56+
CALL bqtools.eu.get_first_date_partition(partitioned_table_id, first_partition);
57+
```
58+
59+
=== "US"
60+
```sql
61+
DECLARE partitioned_table_id STRING;
62+
DECLARE first_partition DATE;
63+
64+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
65+
66+
CALL bqtools.us.get_first_date_partition(partitioned_table_id, first_partition);
67+
```
68+
69+
# **GET Last Date Partition**
70+
_**Attribute**_ | Value
71+
--- | ---
72+
_**Function Name**_ | `get_last_date_partition`
73+
_**ID**_ | `bqtools.[region].get_last_date_partition`
74+
_**Latest Version**_ | `bqtools:v1.0.0`
75+
_**Description**_ | Returns the last partition date from a single date-partitioned table.
76+
_**Type**_ | `PROCEDURE`
77+
_**Arguments**_ | `partitioned_table_id STRING, OUT last_partition DATE`
78+
_**Returns**_ | `OUT last_partition DATE`
79+
_**Dependencies**_ | `bqtools.[region].get_date_partitions`
80+
81+
!!! info "Function Usage"
82+
=== "EU"
83+
```sql
84+
DECLARE partitioned_table_id STRING;
85+
DECLARE last_partition DATE;
86+
87+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
88+
89+
CALL bqtools.eu.get_first_date_partition(partitioned_table_id, last_partition);
90+
```
91+
92+
=== "US"
93+
```sql
94+
DECLARE partitioned_table_id STRING;
95+
DECLARE last_partition DATE;
96+
97+
SET partitioned_table_id = 'project_id.dataset_name.partitioned_table_name';
98+
99+
CALL bqtools.us.get_first_date_partition(partitioned_table_id, last_partition);
100+
```
101+
102+
# **GET Date Shards**
103+
_**Attribute**_ | Value
104+
--- | ---
105+
_**Function Name**_ | `get_date_shards`
106+
_**ID**_ | `bqtools.[region].get_date_shards`
107+
_**Latest Version**_ | `bqtools:v1.0.0`
108+
_**Description**_ | Returns an array of `shard_dates` corresponding to all existing date shards in a single date-sharded table, sorted by descending date.
109+
_**Type**_ | `PROCEDURE`
110+
_**Arguments**_ | `sharded_table_dataset_id STRING, sharded_table_prefix STRING, OUT shard_dates ARRAY<DATE>`
111+
_**Returns**_ | `OUT shard_dates ARRAY<DATE>`
112+
_**Dependencies**_ | `bqtools-qb.[region].get_date_shards`
113+
114+
!!! info "Function Usage"
115+
=== "EU"
116+
```sql
117+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
118+
DECLARE shard_dates ARRAY<DATE>;
119+
120+
SET sharded_table_dataset_id = 'project_id.dataset_name';
121+
SET sharded_table_prefix = 'my_table_prefix_';
122+
123+
CALL bqtools.eu.get_date_shards(sharded_table_dataset_id, sharded_table_prefix, shard_dates);
124+
```
125+
126+
=== "US"
127+
```sql
128+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
129+
DECLARE shard_dates ARRAY<DATE>;
130+
131+
SET sharded_table_dataset_id = 'project_id.dataset_name';
132+
SET sharded_table_prefix = 'my_table_prefix_';
133+
134+
CALL bqtools.us.get_date_shards(sharded_table_dataset_id, sharded_table_prefix, shard_dates);
135+
```
136+
137+
# **GET First Date Shard**
138+
_**Attribute**_ | Value
139+
--- | ---
140+
_**Function Name**_ | `get_first_date_shard`
141+
_**ID**_ | `bqtools.[region].get_first_date_shard`
142+
_**Latest Version**_ | `bqtools:v1.0.0`
143+
_**Description**_ | Returns the first `shard_date` from a single sharded table.
144+
_**Type**_ | `PROCEDURE`
145+
_**Arguments**_ | `sharded_table_dataset_id STRING, sharded_table_prefix STRING, OUT first_shard DATE`
146+
_**Returns**_ | `OUT first_shard DATE`
147+
_**Dependencies**_ | `bqtools.[region].get_date_shards`
148+
149+
!!! info "Function Usage"
150+
=== "EU"
151+
```sql
152+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
153+
DECLARE first_shard DATE;
154+
155+
SET sharded_table_dataset_id = 'project_id.dataset_name';
156+
SET sharded_table_prefix = 'my_table_prefix_';
157+
158+
CALL bqtools.eu.get_first_date_shard(sharded_table_dataset_id, sharded_table_prefix, first_shard);
159+
```
160+
161+
=== "US"
162+
```sql
163+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
164+
DECLARE first_shard DATE;
165+
166+
SET sharded_table_dataset_id = 'project_id.dataset_name';
167+
SET sharded_table_prefix = 'my_table_prefix_';
168+
169+
CALL bqtools.us.get_first_date_shard(sharded_table_dataset_id, sharded_table_prefix, first_shard);
170+
```
171+
172+
# **GET Last Date Shard**
173+
_**Attribute**_ | Value
174+
--- | ---
175+
_**Function Name**_ | `get_last_date_shard`
176+
_**ID**_ | `bqtools.[region].get_last_date_shard`
177+
_**Latest Version**_ | `bqtools:v1.0.0`
178+
_**Description**_ | Returns the last `shard_date` from a single date-sharded table.
179+
_**Type**_ | `PROCEDURE`
180+
_**Arguments**_ | `sharded_table_dataset_id STRING, sharded_table_prefix STRING, OUT last_shard DATE`
181+
_**Returns**_ | `OUT last_shard DATE`
182+
_**Dependencies**_ | `bqtools.[region].get_date_shards`
183+
184+
!!! info "Function Usage"
185+
=== "EU"
186+
```sql
187+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
188+
DECLARE last_shard DATE;
189+
190+
SET sharded_table_dataset_id = 'project_id.dataset_name';
191+
SET sharded_table_prefix = 'my_table_prefix_';
192+
193+
CALL bqtools.eu.get_first_date_shard(sharded_table_dataset_id, sharded_table_prefix, last_shard);
194+
```
195+
196+
=== "US"
197+
```sql
198+
DECLARE sharded_table_dataset_id, sharded_table_prefix STRING;
199+
DECLARE last_shard DATE;
200+
201+
SET sharded_table_dataset_id = 'project_id.dataset_name';
202+
SET sharded_table_prefix = 'my_table_prefix_';
203+
204+
CALL bqtools.us.get_first_date_shard(sharded_table_dataset_id, sharded_table_prefix, last_shard);
205+
```

‎docs/reference/bqtools/functions/partitions.md

+124
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
These functions support creation, insertion and deletion of data partitions to support composition of idempotent data operations.
2+
3+
# **DELETE Date Partitions**
4+
_**Attribute**_ | Value
5+
--- | ---
6+
_**Function Name**_ | `delete_date_partitions`
7+
_**ID**_ | `bqtools.[region].delete_date_partitions`
8+
_**Latest Version**_ | `bqtools:v1.0.0`
9+
_**Description**_ | Deletes a range of date-partitions from a date-partitioned destination table.
10+
_**Type**_ | `PROCEDURE`
11+
_**Arguments**_ | `table_id STRING, date_column_name STRING, start_date DATE, end_date DATE`
12+
_**Returns**_ | `None`
13+
_**Dependencies**_ | `bqtools-qb.[region].delete_date_partitions`
14+
15+
!!! info "Function Usage"
16+
=== "EU"
17+
```sql
18+
DECLARE destination_table_id, date_column_name STRING;
19+
DECLARE start_date, end_date DATE;
20+
21+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
22+
SET date_column_name = 'event_date';
23+
SET start_date = CURRENT_DATE - 7;
24+
SET end_date = CURRENT_DATE;
25+
26+
CALL bqtools.eu.delete_date_partitions(destination_table_id, event_date, date_column_name, start_date, end_date);
27+
```
28+
29+
=== "US"
30+
```sql
31+
DECLARE destination_table_id, date_column_name STRING;
32+
DECLARE start_date, end_date DATE;
33+
34+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
35+
SET date_column_name = 'event_date';
36+
SET start_date = CURRENT_DATE - 7;
37+
SET end_date = CURRENT_DATE;
38+
39+
CALL bqtools.us.delete_date_partitions(destination_table_id, event_date, date_column_name, start_date, end_date);
40+
```
41+
42+
# **INSERT Date Partitions**
43+
_**Attribute**_ | Value
44+
--- | ---
45+
_**Function Name**_ | `insert_date_partitions_from_date_bounded_table_function`
46+
_**ID**_ | `bqtools.[region].insert_date_partitions_from_date_bounded_table_function`
47+
_**Latest Version**_ | `bqtools:v1.0.0`
48+
_**Description**_ | Inserts a range of date partitions from a date-bounded table function into a date-partitioned destination table.
49+
_**Type**_ | `PROCEDURE`
50+
_**Arguments**_ | `destination_table_id STRING, source_table_function_id STRING, start_date DATE, end_date DATE`
51+
_**Returns**_ | `None`
52+
_**Dependencies**_ | `bqtools-qb.[region].insert_date_partitions_from_date_bounded_table_function`
53+
54+
!!! info "Function Usage"
55+
=== "EU"
56+
```sql
57+
DECLARE destination_table_id, source_table_function_id STRING;
58+
DECLARE start_date, end_date DATE;
59+
60+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
61+
SET source_table_function_id = 'project_id.dataset_name.table_function_name';
62+
63+
SET start_date = CURRENT_DATE - 7;
64+
SET end_date = CURRENT_DATE;
65+
66+
CALL bqtools.eu.insert_date_partitions_from_date_bounded_table_function(destination_table_id, source_table_function_id, start_date, end_date);
67+
```
68+
69+
=== "US"
70+
```sql
71+
DECLARE destination_table_id, source_table_function_id STRING;
72+
DECLARE start_date, end_date DATE;
73+
74+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
75+
SET source_table_function_id = 'project_id.dataset_name.table_function_name';
76+
77+
SET start_date = CURRENT_DATE - 7;
78+
SET end_date = CURRENT_DATE;
79+
80+
CALL bqtools.us.insert_date_partitions_from_date_bounded_table_function(destination_table_id, source_table_function_id, start_date, end_date);
81+
```
82+
83+
# **REPLACE Date Partitions**
84+
_**Attribute**_ | Value
85+
--- | ---
86+
_**Function Name**_ | `replace_date_partitions_from_date_bounded_table_function`
87+
_**ID**_ | `bqtools.[region].replace_date_partitions_from_date_bounded_table_function`
88+
_**Latest Version**_ | `bqtools:v1.0.0`
89+
_**Description**_ | Deletes a range of date partitions from a date-partitioned destination table and inserts the same range of date partitions from a date-bounded table function.
90+
_**Type**_ | `PROCEDURE`
91+
_**Arguments**_ | `destination_table_id STRING, date_column_name STRING, source_table_function_id STRING, start_date DATE, end_date DATE`
92+
_**Returns**_ | `None`
93+
_**Dependencies**_ | `bqtools.[region].delete_date_partitions`, `bqtools.[region].insert_date_partitions_from_date_bounded_table_function`
94+
95+
!!! info "Function Usage"
96+
=== "EU"
97+
```sql
98+
DECLARE destination_table_id, date_column_name, source_table_function_id STRING;
99+
DECLARE start_date, end_date DATE;
100+
101+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
102+
SET date_column_name = 'event_date';
103+
SET source_table_function_id = 'project_id.dataset_name.table_function_name';
104+
105+
SET start_date = CURRENT_DATE - 7;
106+
SET end_date = CURRENT_DATE;
107+
108+
CALL bqtools.eu.replace_date_partitions_from_date_bounded_table_function(destination_table_id, date_column_name, source_table_function_id, start_date, end_date);
109+
```
110+
111+
=== "US"
112+
```sql
113+
DECLARE destination_table_id, date_column_name, source_table_function_id STRING;
114+
DECLARE start_date, end_date DATE;
115+
116+
SET destination_table_id = 'project_id.dataset_name.partitioned_table_name';
117+
SET date_column_name = 'event_date';
118+
SET source_table_function_id = 'project_id.dataset_name.table_function_name';
119+
120+
SET start_date = CURRENT_DATE - 7;
121+
SET end_date = CURRENT_DATE;
122+
123+
CALL bqtools.us.replace_date_partitions_from_date_bounded_table_function(destination_table_id, date_column_name, source_table_function_id, start_date, end_date);
124+
```

0 commit comments

Comments
 (0)