Skip to content

Commit baff7dc

Browse files
jaceklaskowskijaceklaskowski
committed
TemporaryTableCredentialsService and S3 Storage Location
1 parent 7edcfcd commit baff7dc

File tree

1 file changed

+125
-0
lines changed

1 file changed

+125
-0
lines changed

‎docs/server/TemporaryTableCredentialsService.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,27 @@ Method | URL | Handler | Params
66
-|-|-|-
77
POST | | [generateTemporaryTableCredential](#generateTemporaryTableCredential) | [GenerateTemporaryTableCredential](GenerateTemporaryTableCredential.md)
88

9+
## TableRepository { #TABLE_REPOSITORY }
10+
11+
`TemporaryTableCredentialsService` gets at the system-wide [TableRepository](../persistent-storage/TableRepository.md#getInstance) instance when created.
12+
13+
## Looking Up Table Credentials { #generateTemporaryTableCredential }
14+
15+
```java
16+
HttpResponse generateTemporaryTableCredential(
17+
GenerateTemporaryTableCredential generateTemporaryTableCredential)
18+
```
19+
20+
`generateTemporaryTableCredential` [looks up the table metadata](../persistent-storage/TableRepository.md#getTableById) (by the `tableId` of the given `GenerateTemporaryTableCredential`) in the [TableRepository](#TABLE_REPOSITORY).
21+
22+
Unless the storage location is on S3 (starts with `s3://` prefix), `generateTemporaryTableCredential` assumes local file system and returns empty credentials.
23+
24+
For a S3 storage location, `generateTemporaryTableCredential` [finds the S3 storage location credentials](TemporaryCredentialUtils.md#findS3BucketConfig).
25+
26+
## Demo
27+
28+
### Local File System
29+
930
```console
1031
$ http http://localhost:8081/api/2.1/unity-catalog/temporary-table-credentials \
1132
table_id=b5d6db68-5eca-485c-be5f-5f53d4f27f60 \
@@ -21,3 +42,107 @@ server: Armeria/1.28.4
2142
"expiration_time": null
2243
}
2344
```
45+
46+
### S3 Storage Location
47+
48+
Register a table on S3.
49+
50+
```console
51+
$ ./bin/uc table create \
52+
--full_name unity.default.s3_creds_demo \
53+
--columns "id INT, name STRING" \
54+
--storage_location "s3://japila.pl/my_demo_bucket/s3_creds_demo"
55+
┌────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
56+
│ KEY │ VALUE │
57+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
58+
│NAME │s3_creds_demo │
59+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
60+
│CATALOG_NAME │unity │
61+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
62+
│SCHEMA_NAME │default │
63+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
64+
│TABLE_TYPE │EXTERNAL │
65+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
66+
│DATA_SOURCE_FORMAT │DELTA │
67+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
68+
│COLUMNS │{"name":"id","type_text":"int","type_json":"{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,│
69+
│ │\"metadata\":{}}","type_name":"INT","type_precision":0,"type_scale":0,"type_interval_type":null,"pos│
70+
│ │ition":0,"comment":null,"nullable":true,"partition_index":null} │
71+
│ │{"name":"name","type_text":"string","type_json":"{\"name\":\"name\",\"type\":\"string\",\"nullable\"│
72+
│ │:true,\"metadata\":{}}","type_name":"STRING","type_precision":0,"type_scale":0,"type_interval_type":│
73+
│ │null,"position":1,"comment":null,"nullable":true,"partition_index":null} │
74+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
75+
│STORAGE_LOCATION │s3://japila.pl/my_demo_bucket/s3_creds_demo │
76+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
77+
│COMMENT │null │
78+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
79+
│PROPERTIES │{} │
80+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
81+
│CREATED_AT │1720539043045 │
82+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
83+
│UPDATED_AT │null │
84+
├────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
85+
│TABLE_ID │020270b2-621e-4157-a73d-6a63c3b86bdd │
86+
└────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
87+
```
88+
89+
```console
90+
$ http http://localhost:8081/api/2.1/unity-catalog/temporary-table-credentials \
91+
table_id=020270b2-621e-4157-a73d-6a63c3b86bdd \
92+
operation=READ
93+
94+
HTTP/1.1 400 Bad Request
95+
content-length: 192
96+
content-type: application/json
97+
date: Tue, 9 Jul 2024 14:54:48 GMT
98+
server: Armeria/1.28.4
99+
100+
{
101+
"details": [
102+
{
103+
"@type": "google.rpc.ErrorInfo",
104+
"metadata": {},
105+
"reason": "FAILED_PRECONDITION"
106+
}
107+
],
108+
"error_code": "FAILED_PRECONDITION",
109+
"message": "S3 bucket configuration not found.",
110+
"stack_trace": null
111+
}
112+
```
113+
114+
Add a S3 bucket configuration to `etc/conf/server.properties` for the table location and bounce the UC server.
115+
116+
```console
117+
$ cat etc/conf/server.properties
118+
server.env=dev
119+
## temp credential config for s3 (Multiple s3 config can be added by incrementing the index)
120+
s3.bucketPath.0=s3://japila.pl
121+
s3.accessKey.0=FIXME_accessKey
122+
s3.secretKey.0=FIXME_secretKey
123+
s3.sessionToken.0=FIXME_sessionToken
124+
```
125+
126+
```console
127+
./bin/start-uc-server
128+
```
129+
130+
```console
131+
$ http http://localhost:8081/api/2.1/unity-catalog/temporary-table-credentials \
132+
table_id=020270b2-621e-4157-a73d-6a63c3b86bdd \
133+
operation=READ
134+
HTTP/1.1 200 OK
135+
content-length: 158
136+
content-type: application/json
137+
date: Tue, 9 Jul 2024 15:32:04 GMT
138+
server: Armeria/1.28.4
139+
140+
{
141+
"aws_temp_credentials": {
142+
"access_key_id": "FIXME_accessKey",
143+
"secret_access_key": "FIXME_secretKey",
144+
"session_token": "FIXME_sessionToken"
145+
},
146+
"expiration_time": null
147+
}
148+
```

0 commit comments

Comments
 (0)