Merge matrix feature branch into master.

Features included in this merge:
- Added script to build and upload docker image for matrix test.
- Added script to create test cases and created go__master testcases based on it.
- Created dictionary for runtimes and gRPC releases for supported languages.
- Added go 1.7 and 1.8 Dockerfile/templates.

See tools/interop_matrix/README.md for details.

Author: yongni

templates/tools/dockerfile/go_build_interop.sh.include

@@ -0,0 +1,47 @@
+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)

templates/tools/dockerfile/interoptest/grpc_interop_go1.7/build_interop.sh.template

@@ -0,0 +1,3 @@
+%YAML 1.2
+--- |
+  <%include file="../../go_build_interop.sh.include"/>  

templates/tools/dockerfile/interoptest/grpc_interop_go1.8/build_interop.sh.template

@@ -0,0 +1,3 @@
+%YAML 1.2
+--- |
+  <%include file="../../go_build_interop.sh.include"/>  

third_party/protobuf

@@ -13,6 +13,8 @@ container engine, BigQuery etc)
 
 internal_ci: Support for running tests on an internal CI platform.
 
+interop_matrix: Scripts to build, upload, and run gRPC clients in docker with various language/runtimes. 
+
 jenkins: Support for running tests on Jenkins.
 
 run_tests: Scripts to run gRPC tests in parallel.

tools/README.md

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)
+  

tools/dockerfile/interoptest/grpc_interop_go1.7/build_interop.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)
+  

tools/dockerfile/interoptest/grpc_interop_go1.8/build_interop.sh

@@ -0,0 +1,40 @@
+# Overview
+
+This directory contains scripts that facilitate building and running gRPC tests for combinations of language/runtimes (known as matrix).
+
+The setup builds gRPC docker images for each language/runtime and upload it to Google Container Registry (GCR). These images, encapsulating gRPC stack
+from specific releases/tag, are used to test version compatiblity between gRPC release versions.
+
+## Instructions for creating GCR images
+- Edit  `./client_matrix.py` to include desired gRPC release.
+- Run `tools/interop_matrix/create_matrix_images.py`.  Useful options:
+  - `--git_checkout` enables git checkout grpc release branch/tag.
+  - `--release` specifies a git release tag.  Make sure it is a valid tag in the grpc github rep.
+  - `--language` specifies a language.
+  For examle, To build all languages for all gRPC releases across all runtimes, do `tools/interop_matrix/create_matrix_images.py --git_checkout --release=all`.
+- Verify the newly created docker images are uploaded to GCR.  For example:
+  - `gcloud beta container images list --repository gcr.io/grpc-testing` shows image repos.
+  - `gcloud beta container images list-tags gcr.io/grpc-testing/grpc_interop_go1.7` show tags for a image repo.
+
+## Instructions for adding new language/runtimes*
+- Create new `Dockerfile.template`, `build_interop.sh.template` for the language/runtime under `template/tools/dockerfile/`.
+- Run `tools/buildgen/generate_projects.sh` to create corresponding files under `tools/dockerfile/`.
+- Add language/runtimes to `client_matrix.py` following existing language/runtimes examples.
+- Run `tools/interop_matrix/create_matrix_images.py` which will build and upload images to GCR.  Unless you are also building images for a gRPC release, make sure not to set `--gcr_tag` (the default tag 'master' is used for testing).
+
+*: Please delete your docker images at https://pantheon.corp.google.com/gcr/images/grpc-testing?project=grpc-testing afterwards.  Permissions to access GrpcTesting project is required for this step.
+
+## Instructions for creating new test cases
+- Create test cases by running `LANG=<lang> [RELEASE=<release>] ./create_testcases.sh`.  For example,
+  - `LANG=go ./create_testcases.sh` will generate `./testcases/go__master`, which is also a functional bash script.
+  - `LANG=go KEEP_IMAGE=1 ./create_testcases.sh` will generate `./testcases/go__master` and keep the local docker image so it can be invoked simply via `./testcases/go__master`.  Note: remove local docker images manually afterwards with `docker rmi <image_id>`.
+- Stage and commit the generated test case file `./testcases/<lang>__<release>`.
+
+## Instructions for running test cases against a GCR image
+- Run test cases by specifying `docker_image` variable inline with the test case script created above.
+For example:
+  - `docker_image=gcr.io/grpc-testing/grpc_interop_go1.7:master ./testcases/go__master` will run go__master test cases against `go1.7` with gRPC release `master` docker image in GCR.
+
+
+Note:
+- File path starting with `tools/` or `template/` are relative to the grpc repo root dir.  File path starting with `./` are relative to current directory (`tools/interop_matrix`).

tools/gcp/utils/gcr_upload.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python2.7
+# Copyright 2017, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# Dictionaries used for client matrix testing.
+
+def get_github_repo(lang):
+  return {
+      'go': '[email protected]:grpc/grpc-go.git',
+      'java': '[email protected]:grpc/grpc-java.git',
+  }.get(lang, '[email protected]:grpc/grpc.git')
+
+# Dictionary of runtimes per language
+LANG_RUNTIME_MATRIX = {
+    'go': ['go1.7', 'go1.8'],
+}
+
+# Dictionary of releases per language.  For each language, we need to provide
+# a tuple of release tag (used as the tag for the GCR image) and also github hash.
+LANG_RELEASE_MATRIX = {
+    'go': ['v1.0.1-GA', 'v1.3.0'],
+}