Skip to content

Commit 55fb48d

Browse files
authored
API Generator (#177)
* API Generator Signed-off-by: Theo Truong <[email protected]> * # Took advantage of the provided gem folder to grep for folders representing namespaces. We don't need to hardcode existing namespaces anymore! Signed-off-by: Theo Truong <[email protected]> --------- Signed-off-by: Theo Truong <[email protected]>
1 parent fd322d4 commit 55fb48d

25 files changed

+930
-0
lines changed

.rubocop.yml

+2
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ require:
77
AllCops:
88
TargetRubyVersion: 2.5
99
NewCops: enable
10+
Exclude:
11+
- 'api_generator/**/*'
1012

1113
RSpec/ImplicitExpect:
1214
Enabled: false

CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
44
## [Unreleased]
55
### Added
66
- Added `remote_store.restore` action ([#176](https://github.com/opensearch-project/opensearch-ruby/pull/176))
7+
- Added API Generator ([#139](https://github.com/opensearch-project/opensearch-ruby/issues/139))
78
### Changed
89
- Merged `opensearch-transport`, `opensearch-api`, and `opensearch-dsl` into `opensearch-ruby` ([#133](https://github.com/opensearch-project/opensearch-ruby/issues/133))
910
- Bumped `mocha` gem from 1.x.x to 2.x.x ([#178](https://github.com/opensearch-project/opensearch-ruby/pull/178))

DEVELOPER_GUIDE.md

+5
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
- [Build and Test](#build-and-test)
77
- [Integration Tests](#integration-tests)
88
- [Linter](#linter)
9+
- [Generate API Actions](#generate-api-actions)
910
- [Submitting Changes](#submitting-changes)
1011

1112
# Developer Guide
@@ -85,6 +86,10 @@ rubocop -a
8586
rubocop --auto-gen-config
8687
```
8788

89+
### Generate API Actions
90+
91+
All changes to the API actions should be done via the `api_generator`. For more information, see the [API Generator's USER_GUIDE](./api_generator/USER_GUIDE.md).
92+
8893
## Submitting Changes
8994

9095
See [CONTRIBUTING](CONTRIBUTING.md).

api_generator/.rubocop.yml

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
require: rubocop-rake
2+
AllCops:
3+
Include:
4+
- 'lib/**/*.rb'
5+
- 'Rakefile'
6+
NewCops: enable
7+
8+
Metrics/CyclomaticComplexity:
9+
Enabled: false
10+
Metrics/MethodLength:
11+
Enabled: false
12+
Metrics/AbcSize:
13+
Enabled: false
14+
Metrics/PerceivedComplexity:
15+
Enabled: false
16+
17+
Layout/EmptyLineAfterGuardClause:
18+
Enabled: false
19+
20+
Style/MultilineBlockChain:
21+
Enabled: false

api_generator/.ruby-version

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.1.0

api_generator/USER_GUIDE.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
This API Generator generates API actions for the OpenSearch Ruby client based off of [OpenSearch OpenAPI specification](https://github.com/opensearch-project/opensearch-api-specification/blob/main/OpenSearch.openapi.json). All changes to the API actions should be done via the this generator. If you find an error in the API actions, it most likely comes from the spec. So, please submit a new issue to the [OpenSearch API specification](https://github.com/opensearch-project/opensearch-api-specification/issues/new/choose) repo first.
2+
3+
---
4+
5+
### Usage
6+
This generator should be run everytime the OpenSearch API Specification is updated to propagate the changes to the Ruby client. For now, this must be done manually:
7+
- Create a new branch from `main`
8+
- Download the latest OpenSearch API Specification from [The API Spec Repo](https://github.com/opensearch-project/opensearch-api-specification/blob/main/OpenSearch.openapi.json)
9+
- Run the generator with the API Spec downloaded previously (see below)
10+
- Run Rubocop with `-a` flag to remove redundant spacing from the generated code `rubocop -a`
11+
- Commit and create a PR to merge the updated API actions into `main`.
12+
13+
### Generate API Actions
14+
Install all dependencies
15+
```bash
16+
bundle install
17+
```
18+
19+
Import the API Generator and load the OpenSearch OpenAPI specification into a generator instance
20+
```ruby
21+
require './lib/api_generator'
22+
generator = ApiGenerator.new('./OpenSearch.openapi.json')
23+
```
24+
25+
The `generate` method accepts the path to the root directory of the `opensearch-ruby` gem as a parameter. By default, it points to the parent directory of the folder containing the generator script. For example to generate all actions into the `tmp` directory:
26+
```ruby
27+
generator.generate('./tmp')
28+
```
29+
30+
You can also target a specific API version by passing in the version number as a parameter. For example to generate all actions for version `1.0` into the `tmp` directory:
31+
```ruby
32+
generator.generate(version: '1.0')
33+
```
34+
35+
The generator also support incremental generation. For example, to generate all actions of the `cat` namespace:
36+
```ruby
37+
generator.generate(namespace: 'cat')
38+
```
39+
40+
To limit it to specific actions of a namespace:
41+
```ruby
42+
generator.generate(namespace: 'cat', actions: %w[aliases allocation])
43+
```
44+
45+
Note that the root namespace is presented by an empty string `''`. For example, to generate all actions of the root namespace for OS version 2.3:
46+
```ruby
47+
generator.generate(version: '2.3', namespace: '')
48+
```

api_generator/gemfile

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# The OpenSearch Contributors require contributions made to
4+
# this file be licensed under the Apache-2.0 license or a
5+
# compatible open source license.
6+
7+
# frozen_string_literal: true
8+
9+
source 'https://rubygems.org'
10+
11+
gem 'rake'
12+
gem 'rubocop', '~> 1.44', require: false
13+
gem 'rubocop-rake', require: false
14+
gem 'openapi3_parser'
15+
gem 'mustache', '~> 1'
16+
gem 'awesome_print'
17+
gem 'activesupport', '~> 7'

api_generator/lib/action.rb

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# The OpenSearch Contributors require contributions made to
4+
# this file be licensed under the Apache-2.0 license or a
5+
# compatible open source license.
6+
7+
# frozen_string_literal: true
8+
9+
require_relative 'operation'
10+
require_relative 'version'
11+
require_relative 'parameter'
12+
13+
# A collection of operations that comprise a single API Action
14+
class Action
15+
attr_reader :group, :name, :namespace, :http_verbs, :urls, :description, :external_docs,
16+
:parameters, :path_params, :query_params,
17+
:body, :body_description, :body_required
18+
19+
# @param [Array<Operation>] operations
20+
def initialize(operations)
21+
@operations = operations
22+
@group = operations.first.group
23+
@name = operations.first.action
24+
@namespace = operations.first.namespace
25+
@http_verbs = operations.map(&:http_verb).uniq
26+
@urls = operations.map(&:url).uniq
27+
@description = operations.map(&:description).find(&:present?)
28+
@external_docs = operations.map(&:external_docs).find(&:present?)
29+
@external_docs = nil if @external_docs == 'https://opensearch.org/docs/latest'
30+
31+
dup_params = operations.flat_map(&:parameters)
32+
@path_params = dup_params.select { |p| p.in == 'path' }
33+
path_param_names = @path_params.map(&:name).to_set
34+
@query_params = dup_params.select { |p| p.in == 'query' && !path_param_names.include?(p.name) }
35+
@parameters = @path_params + @query_params
36+
@parameters.each { |p| p.spec.node_data['required'] = p.name.in?(required_components) }
37+
38+
@body = operations.map(&:request_body).find(&:present?)
39+
@body_required = 'body'.in?(required_components)
40+
@body_description = @body&.content&.[]('application/json')&.schema&.description if @body.present?
41+
end
42+
43+
# @return [Set<String>] The names of input components that are required by the action.
44+
# A component is considered required if it is required by all operations that make up the action.
45+
def required_components
46+
@required_components ||= @operations.map do |op|
47+
set = Set.new(op.parameters.select(&:required?).map(&:name))
48+
set.add('body') if op.request_body&.required?
49+
set
50+
end.reduce(&:intersection)
51+
end
52+
end

api_generator/lib/action_generator.rb

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# The OpenSearch Contributors require contributions made to
4+
# this file be licensed under the Apache-2.0 license or a
5+
# compatible open source license.
6+
7+
# frozen_string_literal: true
8+
9+
require_relative 'base_generator'
10+
require_relative 'action'
11+
12+
# Generate an API Action via Mustache
13+
class ActionGenerator < BaseGenerator
14+
self.template_file = './templates/action.mustache'
15+
attr_reader :module_name, :method_name, :valid_params_constant_name,
16+
:method_description, :argument_descriptions, :external_docs
17+
18+
# Actions that use perform_request_simple_ignore_404
19+
SIMPLE_IGNORE_404 = %w[exists
20+
indices.exists
21+
indices.exists_alias
22+
indices.exists_template
23+
indices.exists_type].to_set.freeze
24+
25+
# Actions that use perform_request_complex_ignore_404
26+
COMPLEX_IGNORE_404 = %w[delete
27+
get
28+
indices.flush_synced
29+
indices.delete_template
30+
indices.delete
31+
security.get_role
32+
security.get_user
33+
snapshot.status
34+
snapshot.get
35+
snapshot.get_repository
36+
snapshot.delete_repository
37+
snapshot.delete
38+
update
39+
watcher.delete_watch].to_set.freeze
40+
41+
# Actions that use perform_request_ping
42+
PING = %w[ping].to_set.freeze
43+
44+
# @param [Pathname] output_folder
45+
# @param [Action] action
46+
def initialize(output_folder, action)
47+
super(output_folder)
48+
@action = action
49+
@urls = action.urls.map { |u| u.split('/').select(&:present?) }.uniq
50+
@external_docs = action.external_docs
51+
@module_name = action.namespace&.camelize
52+
@method_name = action.name.underscore
53+
@valid_params_constant_name = "#{action.name.upcase}_QUERY_PARAMS"
54+
@method_description = action.description
55+
@argument_descriptions = params_desc + [body_desc].compact
56+
end
57+
58+
def url_components
59+
@urls.max_by(&:length)
60+
.map { |e| e.starts_with?('{') ? "_#{e[/{(.+)}/, 1]}" : "'#{e}'" }
61+
.join(', ')
62+
end
63+
64+
def http_verb
65+
case @action.http_verbs.sort
66+
when %w[get post]
67+
'body ? OpenSearch::API::HTTP_POST : OpenSearch::API::HTTP_GET'
68+
when %w[post put]
69+
diff_param = @urls.map(&:to_set).sort_by(&:size).reverse.reduce(&:difference).first
70+
"_#{diff_param[/{(.+)}/, 1]} ? OpenSearch::API::HTTP_PUT : OpenSearch::API::HTTP_POST"
71+
else
72+
"OpenSearch::API::HTTP_#{@action.http_verbs.first.upcase}"
73+
end
74+
end
75+
76+
def required_args
77+
@action.required_components.map { |arg| { arg: } }
78+
.tap { |args| args.last&.[]=('_blank_line', true) }
79+
end
80+
81+
def path_params
82+
@action.path_params.map { |p| { name: p.name, listify: p.is_array } }
83+
.tap { |args| args.last&.[]=('_blank_line', true) }
84+
end
85+
86+
def query_params
87+
@action.query_params.map { |p| { name: p.name } }
88+
end
89+
90+
def listify_query_params
91+
@action.query_params.select(&:is_array).map { |p| { name: p.name } }
92+
.tap { |args| args.first&.[]=('_blank_line', true) }
93+
end
94+
95+
def perform_request
96+
args = 'method, url, params, body, headers'
97+
return "perform_request_simple_ignore_404(#{args})" if SIMPLE_IGNORE_404.include?(@action.group)
98+
return "perform_request_complex_ignore_404(#{args}, arguments)" if COMPLEX_IGNORE_404.include?(@action.group)
99+
return "perform_request_ping(#{args})" if PING.include?(@action.group)
100+
"perform_request(#{args}).body"
101+
end
102+
103+
private
104+
105+
def output_file
106+
create_folder(*[@output_folder, @action.namespace].compact).join("#{@action.name}.rb")
107+
end
108+
109+
def params_desc
110+
@action.parameters.map do |p|
111+
{ data_type: p.ruby_type,
112+
name: p.name,
113+
required: p.required?,
114+
description: p.description,
115+
default: p.default,
116+
deprecated: p.deprecated? }
117+
end
118+
end
119+
120+
def body_desc
121+
return unless @action.body.present?
122+
{ data_type: :Hash,
123+
name: :body,
124+
description: @action.body_description,
125+
required: @action.body_required }
126+
end
127+
end

api_generator/lib/api_generator.rb

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# The OpenSearch Contributors require contributions made to
4+
# this file be licensed under the Apache-2.0 license or a
5+
# compatible open source license.
6+
7+
# frozen_string_literal: true
8+
9+
require 'openapi3_parser'
10+
require_relative 'action'
11+
require_relative 'action_generator'
12+
require_relative 'spec_generator'
13+
require_relative 'namespace_generator'
14+
require_relative 'index_generator'
15+
16+
# Generate API endpoints for OpenSearch Ruby client
17+
class ApiGenerator
18+
HTTP_VERBS = %w[get post put patch delete patch head].freeze
19+
20+
# @param [String] openapi_spec location of the OpenSearch API spec file [required]
21+
def initialize(openapi_spec)
22+
@spec = Openapi3Parser.load_file(openapi_spec)
23+
end
24+
25+
# @param [String] gem_folder location of the API Gem folder (default to the parent folder of the generator)
26+
# @param [String] version target OpenSearch version to generate like "2.5" or "3.0"
27+
# @param [String] namespace namespace to generate (Default to all namespaces. Use '' for root)
28+
# @param [Array<String>] actions list of actions in the specified namespace to generate (Default to all actions)
29+
def generate(gem_folder = '../', version: nil, namespace: nil, actions: nil)
30+
gem_folder = Pathname gem_folder
31+
namespaces = existing_namespaces(gem_folder)
32+
target_actions(version, namespace, actions).each do |action|
33+
ActionGenerator.new(gem_folder.join('lib/opensearch/api/actions'), action).generate
34+
SpecGenerator.new(gem_folder.join('spec/opensearch/api/actions'), action).generate
35+
NamespaceGenerator.new(gem_folder.join('lib/opensearch/api/namespace'), action.namespace).generate(namespaces)
36+
end
37+
IndexGenerator.new(gem_folder.join('lib/opensearch'), namespaces).generate
38+
end
39+
40+
private
41+
42+
def target_actions(version, namespace, actions)
43+
namespace = namespace.to_s
44+
actions = Array(actions).map(&:to_s).to_set unless actions.nil?
45+
46+
operations = @spec.paths.flat_map do |url, path|
47+
path.to_h.slice(*HTTP_VERBS).compact.map do |verb, operation_spec|
48+
operation = Operation.new operation_spec, url, verb
49+
operation.part_of?(version, namespace, actions) ? operation : nil
50+
end
51+
end.compact
52+
53+
operations.group_by(&:group).values.map { |ops| Action.new ops }
54+
end
55+
56+
def existing_namespaces(gem_folder)
57+
gem_folder.join('lib/opensearch/api/actions').children.select(&:directory?).map(&:basename).map(&:to_s).to_set
58+
end
59+
end

0 commit comments

Comments
 (0)