Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
369 changes: 4 additions & 365 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@

* [Java on Google Cloud][cloud-java]

## Documentation

See the [official guide](https://cloud.google.com/java/docs/setup) to get setup and started with development.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not directly related to this PR. I noticed that we use BigQuery as an example in the guide, do we want to use a pure GAPIC instead so it is more generic?


## Supported APIs

Libraries are available on GitHub and Maven Central for developing Java applications that interact with individual Google Cloud services:
Expand Down Expand Up @@ -236,371 +240,6 @@ If the service is not listed, [google-api-java-client][google-api-java-client-se

*When building Java applications, preference should be given to the libraries listed in the table.*



## Specifying a Project ID

Most `google-cloud` libraries require a project ID. There are multiple ways to specify this project ID.

1. When using `google-cloud` libraries from within Compute/App Engine, there's no need to specify a project ID. It is automatically inferred from the production environment.
2. When using `google-cloud` elsewhere, you can do one of the following:
* Supply the project ID when building the service options. For example, to use Datastore from a project with ID "PROJECT_ID", you can write:

```java
Datastore datastore = DatastoreOptions.newBuilder().setProjectId("PROJECT_ID").build().getService();
```
* Specify the environment variable `GOOGLE_CLOUD_PROJECT` to be your desired project ID.
* Set the project ID using the [Google Cloud SDK](https://cloud.google.com/sdk/?hl=en). To use the SDK, [download the SDK](https://cloud.google.com/sdk/?hl=en) if you haven't already, and set the project ID from the command line. For example:

```
gcloud config set project PROJECT_ID
```

`google-cloud` determines the project ID from the following sources in the listed order, stopping once it finds a value:

1. The project ID supplied when building the service options
2. Project ID specified by the environment variable `GOOGLE_CLOUD_PROJECT`
3. The App Engine / Compute Engine project ID
4. The project ID specified in the JSON credentials file pointed by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
5. The Google Cloud SDK project ID

In cases where the library may expect a project ID explicitly, we provide a helper that can provide the inferred project ID:
```java
import com.google.cloud.ServiceOptions;
...
String projectId = ServiceOptions.getDefaultProjectId();
```

## Authentication

`google-cloud-java` uses
[https://github.com/googleapis/google-auth-library-java](https://github.com/googleapis/google-auth-library-java)
to authenticate requests. `google-auth-library-java` supports a wide range of authentication types;
see the project's [README](https://github.com/google/google-auth-library-java/blob/main/README.md)
and [javadoc](https://cloud.google.com/java/docs/reference/google-auth-library/latest/overview) for more
details.

### Google Cloud Platform environment

When using Google Cloud libraries from a Google Cloud Platform environment such as Compute Engine,
Kubernetes Engine, or App Engine, no additional authentication steps are necessary.

For example:

```java
Storage storage = StorageOptions.getDefaultInstance().getService();
```

or:

```java
CloudTasksClient cloudTasksClient = CloudTasksClient.create();
```

### Other environments

#### Using a service account (recommended)

1. [Generate a JSON service account key](https://cloud.google.com/storage/docs/authentication?hl=en#service_accounts).

2. After downloading that key, you must do one of the following:
* Define the environment variable GOOGLE_APPLICATION_CREDENTIALS to be the location of the key.
For example:
```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json
```
* Supply the JSON credentials file when building the service options. For example, this Storage
object has the necessary permissions to interact with your Google Cloud Storage data:
```java
Storage storage = StorageOptions.newBuilder()
.setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream("/path/to/my/key.json")))
.build()
.getService();
```

#### Local development/testing

If running locally for development/testing, you can use the [Google Cloud SDK](https://cloud.google.com/sdk/).
Create Application Default Credentials with `gcloud auth application-default login`, and then
`google-cloud` will automatically detect such credentials.

#### Existing OAuth2 access token

If you already have an OAuth2 access token, you can use it to authenticate (notice that in this case, the
access token will not be automatically refreshed):

```java
Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime));
Storage storage = StorageOptions.newBuilder()
.setCredentials(credentials)
.build()
.getService();
```

or:

```java
Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime));
CloudTasksSettings cloudTasksSettings = CloudTasksSettings.newBuilder()
.setCredentialProvider(FixedCredentialsProvider.create(credentials))
.build();
CloudTasksClient cloudTasksClient = CloudTasksClient.create(cloudTasksSettings);
```

### Application Default Credentials

If no credentials are provided, `google-cloud` will attempt to detect them from the environment
using `GoogleCredentials.getApplicationDefault()` which will search for Application Default
Credentials in the following locations (in order):

1. The credentials file pointed to by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
2. Credentials provided by the Google Cloud SDK `gcloud auth application-default login` command
3. Google App Engine built-in credentials
4. Google Cloud Shell built-in credentials
5. Google Compute Engine built-in credentials

### Authenticating with an API Key

[Authenticating with API Keys](https://cloud.google.com/docs/authentication/api-keys) is supported by a handful of Google Cloud APIs.

We are actively exploring ways to improve the API Key experience.
Currently, to use an API Key with a Java client library, you need to set the header for the relevant service Client manually.

For example, to set the API Key with the [Language service](https://cloud.google.com/java/docs/reference/google-cloud-language/latest/overview):

```java
public LanguageServiceClient createGrpcClientWithApiKey(String apiKey) throws Exception {
// Manually set the api key via the header
Map<String, String> header = new HashMap<String, String>() { {put("x-goog-api-key", apiKey);}};
FixedHeaderProvider headerProvider = FixedHeaderProvider.create(header);

// Create the client
TransportChannelProvider transportChannelProvider = InstantiatingGrpcChannelProvider.newBuilder().setHeaderProvider(headerProvider).build();
LanguageServiceSettings settings = LanguageServiceSettings.newBuilder().setTransportChannelProvider(transportChannelProvider).build();
LanguageServiceClient client = LanguageServiceClient.create(settings);
return client;
}
```

An example instantiation with the Language Client using rest:
```java
public LanguageServiceClient createRestClientWithApiKey(String apiKey) throws Exception {
// Manually set the api key header
Map<String, String> header = new HashMap<String, String>() { {put("x-goog-api-key", apiKey);}};
FixedHeaderProvider headerProvider = FixedHeaderProvider.create(header);

// Create the client
TransportChannelProvider transportChannelProvider = InstantiatingHttpJsonChannelProvider.newBuilder().setHeaderProvider(headerProvider).build();
LanguageServiceSettings settings = LanguageServiceSettings.newBuilder().setTransportChannelProvider(transportChannelProvider).build();
LanguageServiceClient client = LanguageServiceClient.create(settings);
return client;
}
```

## Troubleshooting

To get help, follow the instructions in the [Troubleshooting document](https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md).

## Configuring a Proxy

Google Cloud client libraries use HTTPS and gRPC in underlying communication
with the services.
In both protocols, you can configure a proxy using `https.proxyHost`
and (optional) `https.proxyPort` properties.

### gRPC Custom Proxy Configuration

For a more custom proxy with gRPC, you will need supply a `ProxyDetector` to
the `ManagedChannelBuilder`:

```java
import com.google.api.core.ApiFunction;
import com.google.api.gax.rpc.TransportChannelProvider;
import com.google.cloud.tasks.v2.CloudTasksClient;
import com.google.cloud.tasks.v2.CloudTasksSettings;
import com.google.cloud.tasks.v2.stub.CloudTasksStubSettings;
import io.grpc.HttpConnectProxiedSocketAddress;
import io.grpc.ManagedChannelBuilder;
import io.grpc.ProxiedSocketAddress;
import io.grpc.ProxyDetector;

import javax.annotation.Nullable;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.SocketAddress;

public CloudTasksClient getService() throws IOException {
TransportChannelProvider transportChannelProvider =
CloudTasksStubSettings.defaultGrpcTransportProviderBuilder()
.setChannelConfigurator(
new ApiFunction<ManagedChannelBuilder, ManagedChannelBuilder>() {
@Override
public ManagedChannelBuilder apply(ManagedChannelBuilder managedChannelBuilder) {
return managedChannelBuilder.proxyDetector(
new ProxyDetector() {
@Nullable
@Override
public ProxiedSocketAddress proxyFor(SocketAddress socketAddress)
throws IOException {
return HttpConnectProxiedSocketAddress.newBuilder()
.setUsername(PROXY_USERNAME)
.setPassword(PROXY_PASSWORD)
.setProxyAddress(new InetSocketAddress(PROXY_HOST, PROXY_PORT))
.setTargetAddress((InetSocketAddress) socketAddress)
.build();
}
});
}
})
.build();
CloudTasksSettings cloudTasksSettings =
CloudTasksSettings.newBuilder()
.setTransportChannelProvider(transportChannelProvider)
.build();
return CloudTasksClient.create(cloudTasksSettings);
}
```

## Long Running Operations

Long running operations (LROs) are often used for API calls that are expected to
take a long time to complete (i.e. provisioning a GCE instance or a Dataflow pipeline).
The initial API call creates an "operation" on the server and returns an Operation ID
to track its progress. LRO RPCs have the suffix `Async` appended to the call name
(i.e. `clusterControllerClient.createClusterAsync()`)

Our generated clients provide a nice interface for starting the operation and
then waiting for the operation to complete. This is accomplished by returning an
[`OperationFuture`](https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.longrunning.OperationFuture).
When calling `get()` on the `OperationFuture`, the client library will poll the operation to
check the operation's status.

For example, take a sample `createCluster` Operation in google-cloud-dataproc v4.20.0:
```java
try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
CreateClusterRequest request =
CreateClusterRequest.newBuilder()
.setProjectId("{PROJECT_ID}")
.setRegion("{REGION}")
.setCluster(Cluster.newBuilder().build())
.setRequestId("{REQUEST_ID}")
.setActionOnFailedPrimaryWorkers(FailureAction.forNumber(0))
.build();
OperationFuture<Cluster, ClusterOperationMetadata> future =
clusterControllerClient.createClusterOperationCallable().futureCall(request);
// Do something.
Cluster response = future.get();
} catch (CancellationException e) {
// Exceeded the default RPC timeout without the Operation completing.
// Library is no longer polling for the Operation status. Consider
// increasing the timeout.
}
```

### LRO Timeouts
The polling operations have a default timeout that varies from service to service.
The library will throw a `java.util.concurrent.CancellationException` with the message:
`Task was cancelled.` if the timeout exceeds the operation. A `CancellationException`
does not mean that the backend GCP Operation was cancelled. This exception is thrown from the
client library when it has exceeded the total timeout without receiving a successful status from the operation.
Our client libraries respect the configured values set in the OperationTimedPollAlgorithm for each RPC.

Note: The client library handles the Operation's polling mechanism for you. By default, there is no need
to manually poll the status yourself.

### Default LRO Values
Each LRO RPC has a set of pre-configured default values. You can find these values by
searching in each Client's `StubSettings`'s class. The default LRO settings are initialized
inside the `initDefaults()` method in the nested Builder class.

For example, in google-cloud-aiplatform v3.24.0, the default [OperationTimedPollAlgorithm](https://github.com/googleapis/google-cloud-java/blob/9ae786d1acdc7354adf86b78691570668caa293d/java-aiplatform/google-cloud-aiplatform/src/main/java/com/google/cloud/aiplatform/v1/stub/EndpointServiceStubSettings.java#L755-L765)
has these default values:
```java
OperationTimedPollAlgorithm.create(
RetrySettings.newBuilder()
.setInitialRetryDelay(Duration.ofMillis(5000L))
.setRetryDelayMultiplier(1.5)
.setMaxRetryDelay(Duration.ofMillis(45000L))
.setInitialRpcTimeout(Duration.ZERO)
.setRpcTimeoutMultiplier(1.0)
.setMaxRpcTimeout(Duration.ZERO)
.setTotalTimeout(Duration.ofMillis(300000L))
.build())
```
Both retries and LROs share the same RetrySettings class. Note the corresponding link:
- Total Timeout (Max Time allowed for polling): 5 minutes
- Initial Retry Delay (Initial delay before first poll): 5 seconds
- Max Retry Delay (Maximum delay between each poll): 45 seconds
- Retry Delay Multiplier (Multiplier value to increase the poll delay): 1.5

The RPC Timeout values have no use in LROs and can be omitted or set to the default values
(`Duration.ZERO` for Timeouts or `1.0` for the multiplier).

### Configuring LRO Timeouts
To configure the LRO values, create an OperationTimedPollAlgorithm object and update the
RPC's polling algorithm. For example:
```java
ClusterControllerSettings.Builder settingsBuilder = ClusterControllerSettings.newBuilder();
TimedRetryAlgorithm timedRetryAlgorithm = OperationTimedPollAlgorithm.create(
RetrySettings.newBuilder()
.setInitialRetryDelay(Duration.ofMillis(500L))
.setRetryDelayMultiplier(1.5)
.setMaxRetryDelay(Duration.ofMillis(5000L))
.setInitialRpcTimeout(Duration.ZERO) // ignored
.setRpcTimeoutMultiplier(1.0) // ignored
.setMaxRpcTimeout(Duration.ZERO) // ignored
.setTotalTimeout(Duration.ofHours(24L)) // set polling timeout to 24 hours
.build());
settingsBuilder.createClusterOperationSettings()
.setPollingAlgorithm(timedRetryAlgorithm);
ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settingsBuilder.build());
```

Note: The configuration above *only* modifies the LRO values for the `createClusterOperation` RPC.
The other RPCs in the Client will still use each RPC's pre-configured LRO values.

## Managing Dependencies

If you are using more than one Google Cloud client library, we recommend you use one of
our Bill of Material (BOM) artifacts to help manage dependency versions. For more information,
see [Using the Cloud Client Libraries](https://cloud.google.com/java/docs/bom).

## Java Versions

Java 8 or above is required for using the clients in this repository.

## Supported Platforms

Clients in this repository use either HTTP or gRPC for the transport layer. All
HTTP-based clients should work in all environments.

For clients that use gRPC, the supported platforms are constrained by the platforms
that [Forked Tomcat Native](http://netty.io/wiki/forked-tomcat-native.html) supports,
which for architectures means only x86_64, and for operating systems means Mac OS X,
Windows, and Linux. Additionally, gRPC constrains the use of platforms with
threading restrictions.

Thus, the following are not supported:

- Android
- Consider [Firebase](https://firebase.google.com), which includes many of these APIs.
- It is possible to use these libraries in many cases, although it is unsupported.
You can find examples, such as [this one](https://github.com/GoogleCloudPlatform/android-docs-samples/tree/master/speech/SpeechRecognitionClient),
in this [example repository](https://github.com/GoogleCloudPlatform/android-docs-samples) but consider the risks carefully before using these libraries in an application.
- Raspberry Pi (since it runs on the ARM architecture)
- Google App Engine Standard Java 7

The following environments should work (among others):

- standalone Windows on x86_64
- standalone Mac OS X on x86_64
- standalone Linux on x86_64
- Google Compute Engine (GCE)
- Google Container Engine (GKE)
- Google App Engine Standard Java 8 (GAE Std J8)
- Google App Engine Flex (GAE Flex)
- Alpine Linux (Java 11+)

## Testing

This library provides tools to help write tests for code that uses google-cloud services.
Expand Down
Loading