diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index f13c1fed9e0..4e374f67386 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -14,6 +14,7 @@ The `.md` documents in this repository are rendered into HTML pages using [Jekyll](https://jekyllrb.com/). These HTML pages are hosted on [opensearch.org](https://opensearch.org/docs/latest/). ## Starting the Jekyll server locally + You can run the Jekyll server locally to view the rendered HTML pages using the following steps: 1. Install [Ruby](https://www.ruby-lang.org/en/documentation/installation/) 3.1.0 or later for your operating system. @@ -22,6 +23,7 @@ You can run the Jekyll server locally to view the rendered HTML pages using the 4. Open your browser and navigate to `http://localhost:4000` to view the rendered HTML pages. ## Using the `spec-insert` Jekyll plugin + The `spec-insert` Jekyll plugin is used to insert API components into Markdown files. The plugin downloads the [latest OpenSearch specification](https://github.com/opensearch-project/opensearch-api-specification) and renders the API components from the spec. This aims to reduce the manual effort required to keep the documentation up to date. To use this plugin, make sure that you have installed Ruby 3.1.0 or later and the required gems by running `bundle install`. @@ -67,24 +69,28 @@ bundle exec jekyll spec-insert --refresh-spec ``` ### Ignoring files and folders + The `spec-insert` plugin ignores all files and folders listed in the [./_config.yml#exclude](./_config.yml) list, which is also the list of files and folders that Jekyll ignores. ### Configuration + You can update the configuration settings for this plugin through the [config.yml](./spec-insert/config.yml) file. -_Note that tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run._ +**Note:** The tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run. ## CI/CD The `spec-insert` plugin is run as part of the CI/CD pipeline to ensure that the API components are up to date in the documentation. This is performed through the [update-api-components.yml](.github/workflows/update-api-components.yml) GitHub Actions workflow, which creates a pull request containing the updated API components every Sunday. ## Spec insert components All spec insert components accept the following arguments: + - `api` (String; required): The name of the API to render the component from. This is equivalent to the `x-operation-group` field in the OpenSearch OpenAPI Spec. - `component` (String; required): The name of the component to render, such as `query_parameters`, `path_parameters`, or `endpoints`. - `omit_header` (Boolean; Default is `false`): If set to `true`, the markdown header of the component will not be rendered. ### Endpoints To insert endpoints for the `search` API, use the following snippet: + ```markdown ``` + This table accepts the same arguments as the query parameters table except the `include_global` argument. ### Query parameters + To insert the API query parameters table of the `cat.indices` API, use the following snippet: + ```markdown ``` + +### Request and response bodies (Beta) + +To insert the request and response body tables of the `indices.create` API, use the following snippet: + +```markdown + + +``` + +**Note:**: These components are still a work in progress and may not render correctly for all APIs. + +## Spec insert coverage report +To generate a coverage report of the API components that are being used in the documentation, run the following command: + +```shell +cd spec-insert +bundle exec rake generate_utilization_coverage +``` + +The coverage report will be generated in the `spec-insert/utilization_coverage.md` by default. + diff --git a/spec-insert/.gitignore b/spec-insert/.gitignore index c9958b86d2f..b14907f21ea 100644 --- a/spec-insert/.gitignore +++ b/spec-insert/.gitignore @@ -1,2 +1,3 @@ opensearch-openapi.yaml rspec_examples.txt +utilization_coverage.md \ No newline at end of file diff --git a/spec-insert/Rakefile b/spec-insert/Rakefile new file mode 100644 index 00000000000..c60443fecb3 --- /dev/null +++ b/spec-insert/Rakefile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: Apache-2.0 +# +# The OpenSearch Contributors require contributions made to +# this file be licensed under the Apache-2.0 license or a +# compatible open source license. + +# frozen_string_literal: true + +require 'rake' +require 'active_support/all' +require_relative 'lib/coverage/utilization_coverage' +require_relative 'lib/utils' + +desc 'Generate utilization coverage of Spec-Insert components' +task :generate_utilization_coverage do + Utils.load_spec + coverage = UtilizationCoverage.new.render + file = File.join(__dir__, 'utilization_coverage.md') + File.write(file, coverage) + puts "Utilization coverage written to #{file}" +end diff --git a/spec-insert/config.yml b/spec-insert/config.yml index a1606429652..11288e2353a 100644 --- a/spec-insert/config.yml +++ b/spec-insert/config.yml @@ -1,4 +1,6 @@ param_table: + parameter_column: + freeform_text: -- freeform field -- default_column: empty_text: N/A required_column: diff --git a/spec-insert/lib/api/action.rb b/spec-insert/lib/api/action.rb index 5ad3dded773..99c3ad24b74 100644 --- a/spec-insert/lib/api/action.rb +++ b/spec-insert/lib/api/action.rb @@ -7,62 +7,104 @@ # frozen_string_literal: true require_relative 'parameter' +require_relative 'body' require_relative 'operation' -# A collection of operations that comprise a single API Action -# AKA operation-group -class Action - # @param [SpecHash] spec Parsed OpenAPI spec - def self.actions=(spec) - operations = spec.paths.flat_map do |url, ops| - ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] } +module Api + # A collection of operations that comprise a single API Action + # AKA operation-group + class Action + SUCCESS_CODES = %w[200 201 202 203 204 205 206 207 208 226].freeze + + # @param [SpecHash] spec Parsed OpenAPI spec + def self.actions=(spec) + operations = spec.paths.flat_map do |url, ops| + ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] } + end + @actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) } end - @actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }.index_by(&:full_name) - end - # @return [Hash] API Actions indexed by operation-group - def self.actions - raise 'Actions not set' unless @actions - @actions - end + # @return [Array] API Actions + def self.all + raise 'Actions not set' unless @actions + @actions + end - # @return [Array] Operations in the action - attr_reader :operations + def self.by_full_name + @by_full_name ||= all.index_by(&:full_name).to_h + end - # @param [Array] operations - def initialize(operations) - @operations = operations - @operation = operations.first - @spec = @operation&.spec - end + def self.by_namespace + @by_namespace ||= all.group_by(&:namespace) + end - # @return [Array] Input arguments. - def arguments; @arguments ||= Parameter.from_operations(@operations.map(&:spec)); end + # @return [Array] Operations in the action + attr_reader :operations - # @return [String] Full name of the action (i.e. namespace.action) - def full_name; @operation&.group; end + # @param [Array] operations + def initialize(operations) + @operations = operations + @operation = operations.first || {} + @spec = @operation&.spec + end - # return [String] Name of the action - def name; @operation&.action; end + def query_parameters + @operations.map(&:spec).flat_map(&:parameters).filter { |param| !param['x-global'] && param.in == 'query' } + .group_by(&:name).values + .map { |params| Parameter.from_param_specs(params, @operations.size) } + end - # @return [String] Namespace of the action - def namespace; @operation&.namespace; end + def path_parameters + @operations.map(&:spec).flat_map(&:parameters).filter { |param| param.in == 'path' } + .group_by(&:name).values + .map { |params| Parameter.from_param_specs(params, @operations.size) } + end - # @return [Array] Sorted unique HTTP verbs - def http_verbs; @operations.map(&:http_verb).uniq.sort; end + # @return [Api::Body, nil] Request body + def request_body + @request_body ||= + begin + operation = @operations.find { |op| op.spec.requestBody.present? } + required = @operations.all? { |op| op.spec.requestBody.required } + operation.nil? ? nil : Body.new(operation.spec.requestBody.content, required:) + end + end + + # @return [Api::Body] Response body + def response_body + @response_body ||= + begin + spec = @operations.first.spec + code = SUCCESS_CODES.find { |c| spec.responses[c].present? } + Body.new(@operations.first.spec.responses[code].content, required: nil) + end + end + + # @return [String] Full name of the action (i.e. namespace.action) + def full_name; @operation.group; end - # @return [Array] Unique URLs - def urls; @operations.map(&:url).uniq; end + # return [String] Name of the action + def name; @operation.action; end - # @return [String] Description of the action - def description; @spec&.description; end + # @return [String] Namespace of the action + def namespace; @operation.namespace || ''; end - # @return [Boolean] Whether the action is deprecated - def deprecated; @spec&.deprecated; end + # @return [Array] Sorted unique HTTP verbs + def http_verbs; @operations.map(&:http_verb).uniq.sort; end - # @return [String] Deprecation message - def deprecation_message; @spec['x-deprecation-message']; end + # @return [Array] Unique URLs + def urls; @operations.map(&:url).uniq; end - # @return [String] API reference - def api_reference; @operation&.external_docs&.url; end + # @return [String] Description of the action + def description; @spec.description; end + + # @return [Boolean] Whether the action is deprecated + def deprecated; @spec.deprecated; end + + # @return [String] Deprecation message + def deprecation_message; @spec['x-deprecation-message']; end + + # @return [String] API reference + def api_reference; @operation.external_docs.url; end + end end diff --git a/spec-insert/lib/api/body.rb b/spec-insert/lib/api/body.rb new file mode 100644 index 00000000000..e20f588cc07 --- /dev/null +++ b/spec-insert/lib/api/body.rb @@ -0,0 +1,51 @@ +# frozen_string_literal: true + +require_relative 'parameter' +require_relative 'body_parameter' + +module Api + # Request or response body + class Body + # @return [Boolean] Whether the body is in NDJSON format + attr_reader :ndjson + + # @return [Boolean] + attr_reader :required + + # @return [Array] + attr_reader :params_group + + # @param [SpecHash] content + # @param [Boolean, nil] required + def initialize(content, required:) + @required = required + @ndjson = content['application/json'].nil? + spec = content['application/json'] || content['application/x-ndjson'] + @params_group = BodyParameterGroup.from_schema( + flatten_schema(spec.schema), + description: spec.description || spec.schema.description, + ancestors: [] + ) + end + + # @param [SpecHash] schema + # @return [SpecHash] a schema with allOf flattened + def flatten_schema(schema) + return schema if schema.type.present? && schema.type != 'object' + return schema if schema.properties.present? + return schema if schema.additionalProperties.present? + return schema.anyOf.map { |sch| flatten_schema(sch) } if schema.anyOf.present? + return schema.oneOf.map { |sch| flatten_schema(sch) } if schema.oneOf.present? + return schema if schema.allOf.blank? + + schema = schema.allOf.each_with_object({ properties: {}, required: [] }) do |sch, h| + sch = flatten_schema(sch) + h[:properties].merge!(sch.properties || {}) + h[:required] += sch.required || [] + h[:additionalProperties] ||= sch.additionalProperties + end + + SpecHash.new(schema, fully_parsed: true) + end + end +end diff --git a/spec-insert/lib/api/body_parameter.rb b/spec-insert/lib/api/body_parameter.rb new file mode 100644 index 00000000000..1aaf3b221e8 --- /dev/null +++ b/spec-insert/lib/api/body_parameter.rb @@ -0,0 +1,88 @@ +# frozen_string_literal: true + +require_relative 'parameter' +require_relative '../config' + +module Api + # Represents a group of parameters of an object within a request or response body + class BodyParameterGroup + def self.from_schema(schema, description:, ancestors:) + is_array = schema.type == 'array' || schema.items.present? + parameters = BodyParameter.from_schema(is_array ? schema.items : schema) + new(parameters:, ancestors:, description:, is_array:) + end + + attr_reader :parameters, :ancestors, :description, :is_array + + # @param [Array] parameters + # @param [Array] ancestors + # @param [String] description + # @param [Boolean] is_array + def initialize(parameters:, ancestors:, description:, is_array:) + @parameters = parameters + @ancestors = ancestors + @description = description + @is_array = is_array + parameters.each { |param| param.group = self } + end + + # @return [Array] The child groups of the group + def descendants + @parameters.map(&:child_params_group).compact.flat_map do |group| + [group] + group.descendants + end + end + end + + # TODO: Handle cyclic references + # Represents a body parameter of different levels of a request or response body + class BodyParameter < Parameter + # @param [SpecHash] schema The schema of an object + # @return [Array] The parameters of the object + def self.from_schema(schema) + properties = schema.properties || {} + parameters = properties.map do |name, prop| + BodyParameter.new(name:, schema: prop, required: schema.required&.include?(name)) + end.sort { |a, b| a.name <=> b.name } + return parameters unless schema.additionalProperties + additional_schema = schema.additionalProperties == true ? {} : schema.additionalProperties + free_form_name = CONFIG.param_table.parameter_column.freeform_text + parameters + [BodyParameter.new(name: free_form_name, schema: SpecHash.new(additional_schema))] + end + + attr_accessor :group + + # @param [String] name + # @param [SpecHash] schema + # @param [Boolean] required + def initialize(name:, schema:, required: false) + super(name:, + required:, + schema:, + description: schema.description, + default: schema['default'], + deprecated: schema.deprecated || schema['x-version-deprecated'].present?, + version_deprecated: schema['x-version-deprecated'], + deprecation_message: schema['x-deprecation-message']) + @include_object = @doc_type.include?('Object') + end + + # @return [BodyParameterGroup, nil] The parameters of the object + def child_params_group + return nil unless @include_object + return @child_params_group if defined?(@child_params_group) + @child_params_group ||= BodyParameterGroup.from_schema( + @schema, + ancestors: @group.ancestors + [@name], + description: @description + ) + end + + private + + # TODO: Turn this into a configurable setting + def parse_array(schema) + "Array of #{parse_doc_type(schema.items).pluralize}" + end + end +end diff --git a/spec-insert/lib/api/operation.rb b/spec-insert/lib/api/operation.rb index 6f9fb44cc4a..571fecea075 100644 --- a/spec-insert/lib/api/operation.rb +++ b/spec-insert/lib/api/operation.rb @@ -6,29 +6,31 @@ # frozen_string_literal: true -# An API Operation -class Operation - # @return [Openapi3Parser::Node::Operation] Operation Spec - attr_reader :spec - # @return [String] URL - attr_reader :url - # @return [String] HTTP Verb - attr_reader :http_verb - # @return [String] Operation Group - attr_reader :group - # @return [String] API Action - attr_reader :action - # @return [String] API Namespace - attr_reader :namespace +module Api + # An API Operation + class Operation + # @return [SpecHash] Operation Spec + attr_reader :spec + # @return [String] URL + attr_reader :url + # @return [String] HTTP Verb + attr_reader :http_verb + # @return [String] Operation Group + attr_reader :group + # @return [String] API Action + attr_reader :action + # @return [String] API Namespace + attr_reader :namespace - # @param [Openapi3Parser::Node::Operation] spec Operation Spec - # @param [String] url - # @param [String] http_verb - def initialize(spec, url, http_verb) - @spec = spec - @url = url - @http_verb = http_verb.upcase - @group = spec['x-operation-group'] - @action, @namespace = @group.split('.').reverse + # @param [SpecHash] spec Operation Spec + # @param [String] url + # @param [String] http_verb + def initialize(spec, url, http_verb) + @spec = spec + @url = url + @http_verb = http_verb.upcase + @group = spec['x-operation-group'] + @action, @namespace = @group.split('.').reverse + end end end diff --git a/spec-insert/lib/api/parameter.rb b/spec-insert/lib/api/parameter.rb index f946c31acbd..77143c9f71a 100644 --- a/spec-insert/lib/api/parameter.rb +++ b/spec-insert/lib/api/parameter.rb @@ -1,94 +1,92 @@ # frozen_string_literal: true -module ArgLocation - PATH = :path - QUERY = :query -end +module Api + # Represents a parameter of an API action + # Acting as base class for URL parameters and Body parameters + class Parameter + # @return [String] The name of the parameter + attr_reader :name + # @return [String] The description of the parameter + attr_reader :description + # @return [Boolean] Whether the parameter is required + attr_reader :required + # @return [SpecHash] The JSON schema of the parameter + attr_reader :schema + # @return [String] Argument type in documentation + attr_reader :doc_type + # @return [String] The default value of the parameter + attr_reader :default + # @return [Boolean] Whether the parameter is deprecated + attr_reader :deprecated + # @return [String] The deprecation message + attr_reader :deprecation_message + # @return [String] The OpenSearch version when the parameter was deprecated + attr_reader :version_deprecated -# Represents a parameter of an API action -class Parameter - # @return [String] The name of the parameter - attr_reader :name - # @return [String] The description of the parameter - attr_reader :description - # @return [Boolean] Whether the parameter is required - attr_reader :required - # @return [SpecHash] The JSON schema of the parameter - attr_reader :schema - # @return [String] Argument type in documentation - attr_reader :doc_type - # @return [String] The default value of the parameter - attr_reader :default - # @return [Boolean] Whether the parameter is deprecated - attr_reader :deprecated - # @return [String] The deprecation message - attr_reader :deprecation_message - # @return [String] The OpenSearch version when the parameter was deprecated - attr_reader :version_deprecated - # @return [ArgLocation] The location of the parameter - attr_reader :location + def initialize(name:, description:, required:, schema:, default:, deprecated:, deprecation_message:, + version_deprecated:) + @name = name + @description = description + @required = required + @schema = schema + @doc_type = parse_doc_type(schema) + @default = default + @deprecated = deprecated + @deprecation_message = deprecation_message + @version_deprecated = version_deprecated + end - def initialize(name:, description:, required:, schema:, default:, deprecated:, deprecation_message:, - version_deprecated:, location:) - @name = name - @description = description - @required = required - @schema = schema - @doc_type = get_doc_type(schema) - @default = default - @deprecated = deprecated - @deprecation_message = deprecation_message - @version_deprecated = version_deprecated - @location = location - end + # @param [SpecHash] Full OpenAPI spec + def self.global=(spec) + @global = spec.components.parameters.filter { |_, p| p['x-global'] }.map { |_, p| from_param_specs([p], nil) } + end - # @param [SpecHash | nil] schema - # @return [String | nil] Documentation type - def get_doc_type(schema) - return nil if schema.nil? - union = schema.anyOf || schema.oneOf - return union.map { |sch| get_doc_type(sch) }.sort.uniq.join(' or ') if union.present? - return 'Integer' if schema.type == 'integer' - return 'Float' if schema.type == 'number' - return 'Boolean' if schema.type == 'boolean' - return 'String' if schema.type == 'string' - return 'NULL' if schema.type == 'null' - return 'List' if schema.type == 'array' - 'Object' - end + # @return [Array] Global parameters + def self.global + raise 'Global parameters not set' unless @global + @global + end - # @param [SpecHash] Full OpenAPI spec - def self.global=(spec) - @global = spec.components.parameters.filter { |_, p| p['x-global'] }.map { |_, p| from_parameters([p], 1) } - end + # @param [Array] params List of parameters of the same name + # @param [Integer, nil] opts_count Number of operations involved + # @return [UrlParameter] Single parameter distilled from the list + def self.from_param_specs(params, opts_count) + param = params.first || SpecHash.new + schema = param.schema || SpecHash.new + required = opts_count.nil? ? param.required : params.filter(&:required).size == opts_count + Parameter.new(name: param.name, + description: param.description || schema.description, + required:, + schema:, + default: param['default'] || schema['default'], + deprecated: param.deprecated || schema.deprecated, + deprecation_message: param['x-deprecation-message'] || schema['x-deprecation-message'], + version_deprecated: param['x-version-deprecated'] || schema['x-version-deprecated']) + end - # @return [Array] Global parameters - def self.global - raise 'Global parameters not set' unless @global - @global - end + private - # @param [Array] operations List of operations of the same group - # @return [Array] List of parameters of the operation group - def self.from_operations(operations) - operations.flat_map(&:parameters).filter { |param| !param['x-global'] } - .group_by(&:name).values.map { |params| from_parameters(params, operations.size) } - end + # @param [SpecHash, nil] schema + # @return [String] Documentation type + def parse_doc_type(schema) + return nil if schema.nil? + return 'any' if schema == true + union = schema.anyOf || schema.oneOf + return union.map { |sch| parse_doc_type(sch) }.uniq.sort.join(' or ') if union.present? + return parse_doc_type(schema.allOf.first) if schema.allOf.present? + type = schema.type + return 'Integer' if type == 'integer' + return 'Float' if type == 'number' + return 'Boolean' if type == 'boolean' + return 'String' if type == 'string' + return parse_array(schema) if type == 'array' || schema.items.present? + return 'NULL' if type == 'null' + return 'Object' if type == 'object' || type.nil? + raise "Unhandled JSON Schema Type: #{type}" + end - # @param [Array] params List of parameters of the same name - # @param [Integer] opts_count Number of operations involved - # @return [Parameter] Single parameter distilled from the list - def self.from_parameters(params, opts_count) - param = params.first || SpecHash.new - schema = param.schema || SpecHash.new - Parameter.new(name: param.name, - description: param.description || schema.description, - required: params.filter(&:required).size >= opts_count, - schema:, - default: param['default'] || schema['default'], - deprecated: param.deprecated || schema.deprecated, - deprecation_message: param['x-deprecation-message'] || schema['x-deprecation-message'], - version_deprecated: param['x-version-deprecated'] || schema['x-version-deprecated'], - location: params.any? { |p| p.in == 'path' } ? ArgLocation::PATH : ArgLocation::QUERY) + def parse_array(_schema) + 'List' + end end end diff --git a/spec-insert/lib/coverage/templates/utilization_coverage.mustache b/spec-insert/lib/coverage/templates/utilization_coverage.mustache new file mode 100644 index 00000000000..fa6641a3049 --- /dev/null +++ b/spec-insert/lib/coverage/templates/utilization_coverage.mustache @@ -0,0 +1,19 @@ +## Spec Insert Utilization Coverage + +The following table is a summary of the utilization coverage of each Spec Insert component. The utilization coverage is calculated as the percentage of APIs in the spec that have had the component inserted. The coverage is further broken down by the namespace for each component. +{{#components}} +
+
+ {{{component}}}: {{percent}}% - {{utilization}} of {{total}} APIs covered +{{#namespaces}} +
+{{{namespace}}} namespace: {{utilization}}/{{total}} + +{{#actions}} +- {{#utilized}}✓{{/utilized}}{{^utilized}}✗{{/utilized}} {{{name}}} +{{/actions}} +
+{{/namespaces}} +
+
+{{/components}} diff --git a/spec-insert/lib/coverage/utilization_coverage.rb b/spec-insert/lib/coverage/utilization_coverage.rb new file mode 100644 index 00000000000..88268e31a3e --- /dev/null +++ b/spec-insert/lib/coverage/utilization_coverage.rb @@ -0,0 +1,36 @@ +# frozen_string_literal: true + +require 'mustache' +require 'logger' +require_relative '../utils' +require_relative '../api/action' + +# Renders utilization coverage of Spec-Insert components to a markdown file +class UtilizationCoverage < Mustache + self.template_file = "#{__dir__}/templates/utilization_coverage.mustache" + + def components + total = Api::Action.all.count + ::Utils::COMPONENTS.map do |id, component| + utilization = ::Utils.utilized_components.values.flatten.count { |comp| comp == id } + percent = (utilization.to_f / total * 100).round(2) + { component:, utilization:, total:, percent:, namespaces: namespace_utilization(id) } + end + end + + private + + def namespace_utilization(component) + Api::Action.by_namespace.entries.sort_by(&:first).map do |namespace, actions| + namespace = '[root]' unless namespace.present? + actions = actions.map do |action| + { name: action.full_name, + utilized: ::Utils.utilized_components[action.full_name]&.include?(component) } + end.sort_by { |action| action[:name] } + total = actions.count + utilization = actions.count { |action| action[:utilized] } + percent = (utilization.to_f / total * 100).round(2) + { namespace:, utilization:, total:, percent:, actions: } + end + end +end diff --git a/spec-insert/lib/doc_processor.rb b/spec-insert/lib/doc_processor.rb index 3cddb5939c0..5888d630d67 100644 --- a/spec-insert/lib/doc_processor.rb +++ b/spec-insert/lib/doc_processor.rb @@ -32,6 +32,11 @@ def process(write_to_file: true) rendered_content end + # @return [Array] the spec inserts targeted by this processor + def spec_inserts + find_insertions(File.readlines(@file_path)).map(&:last) + end + private # @return Array<[Integer, Integer, SpecInsert]> diff --git a/spec-insert/lib/dot_hash.rb b/spec-insert/lib/dot_hash.rb index c24e6d88299..ab736d020dc 100644 --- a/spec-insert/lib/dot_hash.rb +++ b/spec-insert/lib/dot_hash.rb @@ -11,7 +11,11 @@ def initialize(hash = {}, fully_parsed: false) end def to_s - @hash.to_s + "<#{self.class}: #{@hash}>" + end + + def inspect + "<#{self.class}: #{@hash}>" end def [](key) @@ -35,7 +39,7 @@ def method_missing(name, ...) def parse(value) return value.map { |v| parse(v) } if value.is_a?(Array) - return value if value.is_a?(self.class) + return value if value.is_a?(DotHash) return value unless value.is_a?(Hash) DotHash.new(value) end diff --git a/spec-insert/lib/insert_arguments.rb b/spec-insert/lib/insert_arguments.rb index 75d43fdda04..9d10314fb77 100644 --- a/spec-insert/lib/insert_arguments.rb +++ b/spec-insert/lib/insert_arguments.rb @@ -1,10 +1,13 @@ # frozen_string_literal: true +require_relative 'utils' +require_relative 'spec_insert_error' + # Doc Insert Arguments class InsertArguments attr_reader :raw - # @param [Array] lines the lines between + # @param [Array] lines the lines between "" def initialize(lines) end_index = lines.each_with_index.find { |line, _index| line.match?(/^\s*-->/) }&.last&.- 1 @raw = lines[1..end_index].filter { |line| line.include?(':') }.to_h do |line| @@ -20,7 +23,10 @@ def api # @return [String] def component - @raw['component'] + @raw['component'].tap do |component| + raise SpecInsertError, 'Component not specified.' if component.nil? + raise SpecInsertError, "Invalid component: #{component}" unless component.in?(Utils::COMPONENTS) + end end # @return [Array] diff --git a/spec-insert/lib/jekyll-spec-insert.rb b/spec-insert/lib/jekyll-spec-insert.rb index 6c4c1fb0788..f474fdd1235 100644 --- a/spec-insert/lib/jekyll-spec-insert.rb +++ b/spec-insert/lib/jekyll-spec-insert.rb @@ -5,6 +5,7 @@ require 'yaml' require_relative 'spec_hash' require_relative 'doc_processor' +require_relative 'utils' # Jekyll plugin to insert document components generated from the spec into the Jekyll site class JekyllSpecInsert < Jekyll::Command @@ -16,25 +17,13 @@ def self.init_with_program(prog) c.option 'refresh-spec', '--refresh-spec', '-R', 'Redownload the OpenSearch API specification' c.option 'fail-on-error', '--fail-on-error', '-F', 'Fail on error' c.action do |_args, options| - spec_file = File.join(Dir.pwd, 'spec-insert/opensearch-openapi.yaml') - excluded_paths = YAML.load_file('_config.yml')['exclude'] - download_spec(spec_file, forced: options['refresh-spec']) - SpecHash.load_file(spec_file) - run_once(excluded_paths, fail_on_error: options['fail-on-error']) - watch(excluded_paths, fail_on_error: options['fail-on-error']) if options['watch'] + Utils.load_spec(forced: options['refresh-spec'], logger: Jekyll.logger) + Utils.target_files.each { |file| process_file(file, fail_on_error: options['fail-on-error']) } + watch(fail_on_error: options['fail-on-error']) if options['watch'] end end end - def self.download_spec(spec_file, forced: false) - return if !forced && File.exist?(spec_file) && (File.mtime(spec_file) > 1.day.ago) - Jekyll.logger.info 'Downloading OpenSearch API specification...' - system 'curl -L -X GET ' \ - 'https://github.com/opensearch-project/opensearch-api-specification' \ - '/releases/download/main-latest/opensearch-openapi.yaml ' \ - "-o #{spec_file}" - end - def self.process_file(file, fail_on_error: false) DocProcessor.new(file, logger: Jekyll.logger).process rescue StandardError => e @@ -43,19 +32,11 @@ def self.process_file(file, fail_on_error: false) Jekyll.logger.error "Error processing #{relative_path}: #{e.message}" end - def self.run_once(excluded_paths, fail_on_error: false) - excluded_paths = excluded_paths.map { |path| File.join(Dir.pwd, path) } - Dir.glob(File.join(Dir.pwd, '**/*.md')) - .filter { |file| excluded_paths.none? { |excluded| file.start_with?(excluded) } } - .each { |file| process_file(file, fail_on_error: fail_on_error) } - end - - def self.watch(excluded_paths, fail_on_error: false) - Jekyll.logger.info "\nWatching for changes...\n" - excluded_paths = excluded_paths.map { |path| /\.#{path}$/ } + def self.watch(fail_on_error: false) + excluded_paths = Utils.config_exclude.map { |path| /\.#{path}$/ } Listen.to(Dir.pwd, only: /\.md$/, ignore: excluded_paths) do |modified, added, _removed| - (modified + added).each { |file| process_file(file, fail_on_error: fail_on_error) } + (modified + added).each { |file| process_file(file, fail_on_error:) } end.start trap('INT') { exit } diff --git a/spec-insert/lib/renderers/body_parameters.rb b/spec-insert/lib/renderers/body_parameters.rb new file mode 100644 index 00000000000..abc7d16887a --- /dev/null +++ b/spec-insert/lib/renderers/body_parameters.rb @@ -0,0 +1,61 @@ +# frozen_string_literal: true + +require_relative 'components/base_mustache_renderer' +require_relative 'components/parameter_table_renderer' + +# Renders request body parameters +class BodyParameters < BaseMustacheRenderer + self.template_file = "#{__dir__}/templates/body_parameters.mustache" + + def initialize(action, args, is_request:) + super(action, args) + @is_request = is_request + @body = is_request ? @action.request_body : @action.response_body + @params_group = @body.params_group + end + + def header + @header ||= "#{@is_request ? 'Request' : 'Response'} body fields" + end + + def description + name = "The #{@is_request ? 'request' : 'response'} body" + required = @body.required ? ' is __required__. It' : ' is optional. It' if @is_request + schema_desc = if @params_group.is_array + "#{name}#{required} is an __array of JSON objects__ (NDJSON). Each object has the following fields." + else + "#{name}#{required} is a JSON object with the following fields." + end + [@params_group.description, schema_desc].compact.reject(&:empty?).join("\n\n") + end + + def required + @body.required + end + + def table + ParameterTableRenderer.new(@params_group.parameters, @args, is_body: true).render + end + + def descendants + @params_group.descendants.sort_by(&:ancestors).map do |group| + { block_name: "#{@args.api}::#{@is_request ? 'request' : 'response'}_body", + summary: "#{header}: #{group.ancestors.join(' > ')}", + description: descendant_desc(group), + table: ParameterTableRenderer.new(group.parameters, @args, is_body: true).render } + end + end + + private + + # @param [Api::BodyParameterGroup] group + def descendant_desc(group) + schema_desc = + if group.is_array + "`#{group.ancestors.last}` is an __array of JSON objects__ (NDJSON). Each object has the following fields." + else + "`#{group.ancestors.last}` is a JSON object with the following fields." + end + [group.description, schema_desc].compact.reject(&:empty?).join("\n\n") + end +end diff --git a/spec-insert/lib/renderers/base_mustache_renderer.rb b/spec-insert/lib/renderers/components/base_mustache_renderer.rb similarity index 70% rename from spec-insert/lib/renderers/base_mustache_renderer.rb rename to spec-insert/lib/renderers/components/base_mustache_renderer.rb index b3d756304c9..920ba704fb3 100644 --- a/spec-insert/lib/renderers/base_mustache_renderer.rb +++ b/spec-insert/lib/renderers/components/base_mustache_renderer.rb @@ -6,7 +6,12 @@ class BaseMustacheRenderer < Mustache self.template_path = "#{__dir__}/templates" - # @param [Action] action API Action + # @param [Api::Action] + attr_reader :action + # @param [InsertArguments] + attr_reader :args + + # @param [Api::Action] action API Action # @param [InsertArguments] args def initialize(action, args) super() diff --git a/spec-insert/lib/renderers/parameter_table_renderer.rb b/spec-insert/lib/renderers/components/parameter_table_renderer.rb similarity index 66% rename from spec-insert/lib/renderers/parameter_table_renderer.rb rename to spec-insert/lib/renderers/components/parameter_table_renderer.rb index e07e820c290..4e6f1fa0b6d 100644 --- a/spec-insert/lib/renderers/parameter_table_renderer.rb +++ b/spec-insert/lib/renderers/components/parameter_table_renderer.rb @@ -1,20 +1,22 @@ # frozen_string_literal: true require_relative 'table_renderer' -require_relative '../config' +require_relative '../../config' # Renders a table of parameters of an API action class ParameterTableRenderer - COLUMNS = ['Parameter', 'Description', 'Required', 'Data type', 'Default'].freeze - DEFAULT_COLUMNS = ['Parameter', 'Data type', 'Description'].freeze + SHARED_COLUMNS = ['Description', 'Required', 'Data type', 'Default'].freeze + URL_PARAMS_COLUMNS = (['Parameter'] + SHARED_COLUMNS).freeze + BODY_PARAMS_COLUMNS = (['Property'] + SHARED_COLUMNS).freeze - # @param [Array] parameters + # @param [Array] parameters # @param [InsertArguments] args - def initialize(parameters, args) + def initialize(parameters, args, is_body: false) @config = CONFIG.param_table @parameters = filter_parameters(parameters, args) - @columns = determine_columns(args) + @is_body = is_body @pretty = args.pretty + @columns = determine_columns(args) end # @return [String] @@ -29,17 +31,18 @@ def render # @param [InsertArguments] args def determine_columns(args) if args.columns.present? - invalid = args.columns - COLUMNS + invalid = args.columns - (@is_body ? BODY_PARAMS_COLUMNS : URL_PARAMS_COLUMNS) raise ArgumentError, "Invalid column(s): #{invalid.join(', ')}." unless invalid.empty? return args.columns end required = @parameters.any?(&:required) ? 'Required' : nil default = @parameters.any? { |p| p.default.present? } ? 'Default' : nil - ['Parameter', required, 'Data type', 'Description', default].compact + name = @is_body ? 'Property' : 'Parameter' + [name, required, 'Data type', 'Description', default].compact end - # @param [Array] parameters + # @param [Array] parameters # @param [InsertArguments] args def filter_parameters(parameters, args) parameters = parameters.reject(&:deprecated) unless args.include_deprecated @@ -47,8 +50,10 @@ def filter_parameters(parameters, args) end def row(param) + parameter = "`#{param.name}`#{'
_DEPRECATED_' if param.deprecated}" { - 'Parameter' => "`#{param.name}`#{'
_DEPRECATED_' if param.deprecated}", + 'Parameter' => parameter, + 'Property' => parameter, 'Description' => description(param), 'Required' => param.required ? @config.required_column.true_text : @config.required_column.false_text, 'Data type' => param.doc_type, @@ -56,25 +61,26 @@ def row(param) } end - # @param [Parameter] param + # @param [Api::Parameter] param def description(param) deprecation = deprecation(param) - required = param.required && @columns.exclude?('Required') ? '**(Required)** ' : '' + required = param.required && @columns.exclude?('Required') ? '**(Required)**' : '' description = param.description + default = param.default.nil? || @columns.include?('Default') ? '' : "_(Default: `#{param.default}`)_" valid_values = valid_values(param) - default = param.default.nil? || @columns.include?('Default') ? '' : " _(Default: `#{param.default}`)_" - "#{deprecation}#{required}#{description}#{default}#{valid_values}" + main_line = [deprecation, required, description, default].compact.map(&:strip).reject(&:empty?).join(' ') + [main_line, valid_values].reject(&:empty?).join('
') end - # @param [Parameter] param + # @param [Api::Parameter] param def valid_values(param) enums = extract_enum_values(param.schema)&.compact return '' unless enums.present? if enums.none? { |enum| enum[:description].present? } - "
Valid values are: #{enums.map { |enum| "`#{enum[:value]}`" }.join(', ')}" + "Valid values are: #{enums.map { |enum| "`#{enum[:value]}`" }.join(', ').gsub(/, ([^,]+)$/, ', and \1')}." else - "
Valid values are:
#{enums.map { |enum| "- `#{enum[:value]}`: #{enum[:description]}" } + "Valid values are:
#{enums.map { |enum| "- `#{enum[:value]}`: #{enum[:description]}" } .join('
')}" end end @@ -93,6 +99,6 @@ def extract_enum_values(schema) def deprecation(param) message = ": #{param.deprecation_message}" if param.deprecation_message.present? since = " since #{param.version_deprecated}" if param.version_deprecated.present? - "_(Deprecated#{since}#{message})_ " if param.deprecated + "_(Deprecated#{since}#{message})_" if param.deprecated end end diff --git a/spec-insert/lib/renderers/table_renderer.rb b/spec-insert/lib/renderers/components/table_renderer.rb similarity index 100% rename from spec-insert/lib/renderers/table_renderer.rb rename to spec-insert/lib/renderers/components/table_renderer.rb diff --git a/spec-insert/lib/renderers/endpoints.rb b/spec-insert/lib/renderers/endpoints.rb index f47c9b41937..49b1155f2ab 100644 --- a/spec-insert/lib/renderers/endpoints.rb +++ b/spec-insert/lib/renderers/endpoints.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative 'base_mustache_renderer' +require_relative 'components/base_mustache_renderer' # Renders Endpoints class Endpoints < BaseMustacheRenderer diff --git a/spec-insert/lib/renderers/path_parameters.rb b/spec-insert/lib/renderers/path_parameters.rb index 715ef7028c4..0c89ec582c8 100644 --- a/spec-insert/lib/renderers/path_parameters.rb +++ b/spec-insert/lib/renderers/path_parameters.rb @@ -1,7 +1,7 @@ # frozen_string_literal: true -require_relative 'base_mustache_renderer' -require_relative 'parameter_table_renderer' +require_relative 'components/base_mustache_renderer' +require_relative 'components/parameter_table_renderer' # Renders path parameters class PathParameters < BaseMustacheRenderer @@ -18,6 +18,6 @@ def optional private def params - @params ||= @action.arguments.select { |arg| arg.location == ArgLocation::PATH } + @params ||= @action.path_parameters end end diff --git a/spec-insert/lib/renderers/query_parameters.rb b/spec-insert/lib/renderers/query_parameters.rb index f03544fce39..2073cb321c1 100644 --- a/spec-insert/lib/renderers/query_parameters.rb +++ b/spec-insert/lib/renderers/query_parameters.rb @@ -1,7 +1,7 @@ # frozen_string_literal: true -require_relative 'base_mustache_renderer' -require_relative 'parameter_table_renderer' +require_relative 'components/base_mustache_renderer' +require_relative 'components/parameter_table_renderer' # Renders query parameters class QueryParameters < BaseMustacheRenderer @@ -19,8 +19,8 @@ def optional def params return @params if defined?(@params) - @params = @action.arguments.select { |arg| arg.location == ArgLocation::QUERY } - @params += Parameter.global if @args.include_global + @params = @action.query_parameters + @params += Api::Parameter.global if @args.include_global @params end end diff --git a/spec-insert/lib/renderers/spec_insert.rb b/spec-insert/lib/renderers/spec_insert.rb index 57486a7f736..77f13dd049c 100644 --- a/spec-insert/lib/renderers/spec_insert.rb +++ b/spec-insert/lib/renderers/spec_insert.rb @@ -1,22 +1,22 @@ # frozen_string_literal: true -require_relative 'base_mustache_renderer' +require_relative 'components/base_mustache_renderer' require_relative '../insert_arguments' require_relative '../api/action' require_relative '../spec_insert_error' require_relative 'endpoints' require_relative 'path_parameters' require_relative 'query_parameters' +require_relative 'body_parameters' # Class to render spec insertions class SpecInsert < BaseMustacheRenderer - COMPONENTS = Set.new(%w[query_params path_params endpoints]).freeze self.template_file = "#{__dir__}/templates/spec_insert.mustache" - # @param [Array] arg_lines the lines between "" - def initialize(arg_lines) - args = InsertArguments.new(arg_lines) - action = Action.actions[args.api] + # @param [Array] lines the lines between "" + def initialize(lines) + args = InsertArguments.new(lines) + action = Api::Action.by_full_name[args.api] super(action, args) raise SpecInsertError, '`api` argument not specified.' unless @args.api raise SpecInsertError, "API Action '#{@args.api}' does not exist in the spec." unless @action @@ -35,6 +35,10 @@ def content PathParameters.new(@action, @args).render when :endpoints Endpoints.new(@action, @args).render + when :request_body_parameters + BodyParameters.new(@action, @args, is_request: true).render + when :response_body_parameters + BodyParameters.new(@action, @args, is_request: false).render else raise SpecInsertError, "Invalid component: #{@args.component}" end diff --git a/spec-insert/lib/renderers/templates/body_parameters.mustache b/spec-insert/lib/renderers/templates/body_parameters.mustache new file mode 100644 index 00000000000..ae6a6ddc867 --- /dev/null +++ b/spec-insert/lib/renderers/templates/body_parameters.mustache @@ -0,0 +1,15 @@ +{{^omit_header}} +## {{{header}}} + +{{{description}}} +{{/omit_header}} +{{{table}}}{{#descendants}} +
+ + {{{summary}}} + + {: .text-delta} + +{{{description}}} +{{{table}}} +
{{/descendants}} \ No newline at end of file diff --git a/spec-insert/lib/spec_hash.rb b/spec-insert/lib/spec_hash.rb index f34838bd22f..5a49fa6da77 100644 --- a/spec-insert/lib/spec_hash.rb +++ b/spec-insert/lib/spec_hash.rb @@ -2,6 +2,8 @@ require_relative 'config' require_relative 'dot_hash' +require_relative 'api/action' +require_relative 'api/parameter' # Spec class for parsing OpenAPI spec # It's basically a wrapper around a Hash that allows for accessing hash values as object attributes @@ -10,8 +12,8 @@ class SpecHash < DotHash def self.load_file(file_path) @root = YAML.load_file(file_path) parsed = SpecHash.new(@root) - Action.actions = parsed - Parameter.global = parsed + Api::Action.actions = parsed + Api::Parameter.global = parsed end # @return [Hash] Root of the raw OpenAPI Spec used to resolve $refs @@ -28,12 +30,12 @@ def description def parse(value) return value.map { |v| parse(v) } if value.is_a?(Array) - return value if value.is_a?(self.class) + return value if value.is_a?(SpecHash.class) return value unless value.is_a?(Hash) ref = value.delete('$ref') value.transform_values! { |v| parse(v) } - return SpecHash.new(value, fully_parsed: true) unless ref - SpecHash.new(parse(resolve(ref)).merge(value), fully_parsed: true) + value.merge!(resolve(ref)) if ref + SpecHash.new(value) end def resolve(ref) diff --git a/spec-insert/lib/utils.rb b/spec-insert/lib/utils.rb new file mode 100644 index 00000000000..3131eb90fe8 --- /dev/null +++ b/spec-insert/lib/utils.rb @@ -0,0 +1,63 @@ +# frozen_string_literal: true + +require 'yaml' +require_relative 'spec_hash' +require_relative 'doc_processor' + +# Utility methods for the Spec-Insert +module Utils + REPO_ROOT = File.expand_path('../..', __dir__) + SPEC_FILE = File.join(REPO_ROOT, 'spec-insert/opensearch-openapi.yaml') + COMPONENTS = { + 'endpoints' => 'Endpoints', + 'query_parameters' => 'Query Parameters', + 'path_parameters' => 'Path Parameters', + 'request_body_parameters' => 'Request Body Parameters', + 'response_body_parameters' => 'Response Body Parameters' + }.freeze + + # @return [Array] list of markdown files to insert the spec components into + def self.target_files + excluded_paths = config_exclude.map { |path| File.join(REPO_ROOT, path) } + Dir.glob(File.join(REPO_ROOT, '**/*.md')).filter do |file| + excluded_paths.none? { |exc| file.start_with?(exc) } + end + end + + # @return [Array] list of paths excluded by Jekyll + def self.config_exclude + YAML.load_file(File.join(REPO_ROOT, '_config.yml'))['exclude'] + end + + def self.load_spec(forced: false, logger: nil) + download_spec(forced:, logger:) + SpecHash.load_file(SPEC_FILE) + end + + def self.download_spec(forced: false, logger: nil) + return if !forced && File.exist?(SPEC_FILE) && (File.mtime(SPEC_FILE) > 1.day.ago) + logger&.info 'Downloading OpenSearch API specification...' + system 'curl -L -X GET ' \ + 'https://github.com/opensearch-project/opensearch-api-specification' \ + '/releases/download/main-latest/opensearch-openapi.yaml ' \ + "-o #{SPEC_FILE}" + end + + # @return [Hash] where each is an API/action name and each value is an array of generated component for that API + def self.utilized_components + @utilized_components ||= begin + logger = Logger.new(IO::NULL) + spec_inserts = target_files.flat_map { |file| DocProcessor.new(file, logger:).spec_inserts } + Set.new(spec_inserts.map { |insert| [insert.args.api, insert.args.component] }) + .to_a.group_by(&:first).transform_values { |values| values.map(&:last) } + end + end + + # @param [String] value + # @return [Boolean] + def self.parse_boolean(value) + return true if value == true || value =~ /^(true|t|yes|y|1)$/i + return false if value == false || value.nil? || value =~ /^(false|f|no|n|0)$/i + raise ArgumentError, "invalid value for Boolean: #{value}" + end +end diff --git a/spec-insert/spec/_fixtures/expected_output/body_params_tables.md b/spec-insert/spec/_fixtures/expected_output/body_params_tables.md new file mode 100644 index 00000000000..c490941b25d --- /dev/null +++ b/spec-insert/spec/_fixtures/expected_output/body_params_tables.md @@ -0,0 +1,193 @@ + +## Request body fields + +The index settings to be updated. + +The request body is __required__. It is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `lifecycle` | Object | | +| `mode` | String | | +| `routing` | Object | | +| `routing_path` | Array of Strings or String | | +| `soft_deletes` | Array of Objects | | +| `soft_deletes.retention_lease.period` | String | A duration. Units can be `nanos`, `micros`, `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours) and `d` (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. | + +
+ + Request body fields: lifecycle + + {: .text-delta} + +`lifecycle` is a JSON object with the following fields. + +| Property | Required | Data type | Description | +| :--- | :--- | :--- | :--- | +| `name` | **Required** | String | | +| `indexing_complete` | _Optional_ | Boolean or String | Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures this behavior while keeping the semantics of the field type. Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type. | +| `origination_date` | _Optional_ | Integer or String | Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures this behavior while keeping the semantics of the field type. Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type. | +| `parse_origination_date` | _Optional_ | Boolean | Set to `true` to parse the origination date from the index name. This origination date is used to calculate the index age for its phase transitions. The index name must match the pattern `^.*-{date_format}-\\d+`, where the `date_format` is `yyyy.MM.dd` and the trailing digits are optional. An index that was rolled over would normally match the full format, for example `logs-2016.10.31-000002`). If the index name doesn't match the pattern, index creation fails. | +| `rollover_alias` | _Optional_ | String | The index alias to update when the index rolls over. Specify when using a policy that contains a rollover action. When the index rolls over, the alias is updated to reflect that the index is no longer the write index. For more information about rolling indexes, see Rollover. | +| `step` | _Optional_ | Object | | + +
+
+ + Request body fields: lifecycle > step + + {: .text-delta} + +`step` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `wait_time_threshold` | String | A duration. Units can be `nanos`, `micros`, `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours) and `d` (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. | + +
+
+ + Request body fields: routing + + {: .text-delta} + +`routing` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `allocation` | Object | | +| `rebalance` | Object | | + +
+
+ + Request body fields: routing > allocation + + {: .text-delta} + +`allocation` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `disk` | Object | | +| `enable` | String | Valid values are: `all`, `new_primaries`, `none`, and `primaries`. | +| `include` | Object | | +| `initial_recovery` | Object | | +| `total_shards_per_node` | Integer or String | Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures this behavior while keeping the semantics of the field type. Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type. | + +
+
+ + Request body fields: routing > allocation > disk + + {: .text-delta} + +`disk` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `threshold_enabled` | Boolean or String | Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures this behavior while keeping the semantics of the field type. Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type. | + +
+
+ + Request body fields: routing > allocation > include + + {: .text-delta} + +`include` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `_id` | String | | +| `_tier_preference` | String | | + +
+
+ + Request body fields: routing > allocation > initial_recovery + + {: .text-delta} + +`initial_recovery` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `_id` | String | | + +
+
+ + Request body fields: routing > rebalance + + {: .text-delta} + +`rebalance` is a JSON object with the following fields. + +| Property | Required | Data type | Description | +| :--- | :--- | :--- | :--- | +| `enable` | **Required** | String | Valid values are: `all`, `none`, `primaries`, and `replicas`. | + +
+
+ + Request body fields: soft_deletes + + {: .text-delta} + +`soft_deletes` is an __array of JSON objects__ (NDJSON). Each object has the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `enabled` | Boolean or String | Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures this behavior while keeping the semantics of the field type. Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type. | +| `retention` | Object | The retention settings for soft deletes. | +| `retention_lease` | Object | | + +
+
+ + Request body fields: soft_deletes > retention + + {: .text-delta} + +The retention settings for soft deletes. + +`retention` is a JSON object with the following fields. + +| Property | Data type | Description | +| :--- | :--- | :--- | +| `operations` | Integer or String | | + +
+
+ + Request body fields: soft_deletes > retention_lease + + {: .text-delta} + +`retention_lease` is a JSON object with the following fields. + +| Property | Required | Data type | Description | +| :--- | :--- | :--- | :--- | +| `period` | **Required** | String | A duration. Units can be `nanos`, `micros`, `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours) and `d` (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. | + +
+ + + + +## Response body fields + +The response body is a JSON object with the following fields. + +| Property | Required | Data type | Description | +| :--- | :--- | :--- | :--- | +| `acknowledged` | **Required** | Boolean | For a successful response, this value is always true. On failure, an exception is returned instead. | + + diff --git a/spec-insert/spec/_fixtures/expected_output/param_tables.md b/spec-insert/spec/_fixtures/expected_output/url_params_tables.md similarity index 95% rename from spec-insert/spec/_fixtures/expected_output/param_tables.md rename to spec-insert/spec/_fixtures/expected_output/url_params_tables.md index 2a8e233fbb1..d604e9a592b 100644 --- a/spec-insert/spec/_fixtures/expected_output/param_tables.md +++ b/spec-insert/spec/_fixtures/expected_output/url_params_tables.md @@ -10,7 +10,7 @@ The following table lists the available path parameters. All path parameters are | Parameter | Data type | Description | | :--- | :--- | :--- | -| `index` | List or String | Comma-separated list of data streams, indexes, and aliases to search. Supports wildcards (`*`). To search all data streams and indexes, omit this parameter or use `*` or `_all`.
Valid values are: `_all`, `_any`, `_none` | +| `index` | List or String | Comma-separated list of data streams, indexes, and aliases to search. Supports wildcards (`*`). To search all data streams and indexes, omit this parameter or use `*` or `_all`.
Valid values are: `_all`, `_any`, and `_none`. | @@ -31,7 +31,7 @@ The following table lists the available query parameters. |:---------------|:--------------------------|:-----------------------------------------------------------------------------------------------------------------------------------|:-------------|:--------| | Boolean | `analyze_wildcard` | If true, wildcard and prefix queries are analyzed. This parameter can only be used when the q query string parameter is specified. | **Required** | `false` | | String | `analyzer` | Analyzer to use for the query string. This parameter can only be used when the q query string parameter is specified. | _Optional_ | N/A | -| List or String | `expand_wildcards` | Comma-separated list of expand wildcard options.
Valid values are: `open`, `closed`, `none`, `all` | _Optional_ | N/A | +| List or String | `expand_wildcards` | Comma-separated list of expand wildcard options.
Valid values are: `open`, `closed`, `none`, and `all`. | _Optional_ | N/A | | Boolean | `pretty` | Whether to pretty format the returned JSON response. | _Optional_ | N/A | | Boolean | `human`
_DEPRECATED_ | _(Deprecated since 3.0: Use the `format` parameter instead.)_ Whether to return human readable values for statistics. | _Optional_ | `true` | @@ -50,7 +50,7 @@ omit_header: true | :--- | :--- | | `analyze_wildcard` | **(Required)** If true, wildcard and prefix queries are analyzed. This parameter can only be used when the q query string parameter is specified. _(Default: `false`)_ | | `analyzer` | Analyzer to use for the query string. This parameter can only be used when the q query string parameter is specified. | -| `expand_wildcards` | Comma-separated list of expand wildcard options.
Valid values are: `open`, `closed`, `none`, `all` | +| `expand_wildcards` | Comma-separated list of expand wildcard options.
Valid values are: `open`, `closed`, `none`, and `all`. | diff --git a/spec-insert/spec/_fixtures/input/body_params_tables.md b/spec-insert/spec/_fixtures/input/body_params_tables.md new file mode 100644 index 00000000000..b2aa818370b --- /dev/null +++ b/spec-insert/spec/_fixtures/input/body_params_tables.md @@ -0,0 +1,12 @@ + + + + + + \ No newline at end of file diff --git a/spec-insert/spec/_fixtures/input/param_tables.md b/spec-insert/spec/_fixtures/input/url_params_tables.md similarity index 100% rename from spec-insert/spec/_fixtures/input/param_tables.md rename to spec-insert/spec/_fixtures/input/url_params_tables.md diff --git a/spec-insert/spec/_fixtures/opensearch_spec.yaml b/spec-insert/spec/_fixtures/opensearch_spec.yaml index 6428d8f57f3..d0cb3efec29 100644 --- a/spec-insert/spec/_fixtures/opensearch_spec.yaml +++ b/spec-insert/spec/_fixtures/opensearch_spec.yaml @@ -59,6 +59,30 @@ paths: - $ref: '#/components/parameters/search___query.analyze_wildcard' - $ref: '#/components/parameters/search___query.analyzer' - $ref: '#/components/parameters/search___query.expand_wildcards' + /_settings: + put: + operationId: indices.put_settings.0 + x-operation-group: indices.put_settings + x-version-added: '1.0' + description: Updates the index settings. + externalDocs: + url: https://opensearch.org/docs/latest/api-reference/index-apis/update-settings/ + requestBody: + $ref: '#/components/requestBodies/indices.put_settings' + responses: + '200': + $ref: '#/components/responses/indices.put_settings___200' + + /_bulk: + post: + operationId: bulk.0 + x-operation-group: bulk + x-version-added: '1.0' + description: Allows to perform multiple index/update/delete operations in a single request. + externalDocs: + url: https://opensearch.org/docs/latest/api-reference/document-apis/bulk/ + requestBody: + $ref: '#/components/requestBodies/bulk' components: parameters: @@ -149,6 +173,43 @@ components: items: $ref: '#/components/schemas/_common___ExpandWildcardsCompact' + requestBodies: + + indices.put_settings: + required: true + content: + application/json: + schema: + title: settings + $ref: '#/components/schemas/indices._common___IndexSettings' + + bulk: + content: + application/x-ndjson: + schema: + type: array + items: + anyOf: + - type: object + - type: object + properties: + index: + type: string + action: + type: string + enum: [index, create, delete, update] + data: + type: object + description: The operation definition and data (action-data pairs), separated by newlines + required: true + + responses: + + indices.put_settings___200: + content: + application/json: + schema: + $ref: '#/components/schemas/_common___AcknowledgedResponseBase' schemas: _common___Indices: @@ -178,3 +239,218 @@ components: - closed - none - all + + _common___AcknowledgedResponseBase: + type: object + properties: + acknowledged: + description: For a successful response, this value is always true. On failure, an exception is returned instead. + type: boolean + required: + - acknowledged + + indices._common___IndexSettings: + type: object + description: The index settings to be updated. + properties: + mode: + type: string + routing_path: + $ref: '#/components/schemas/_common___StringOrStringArray' + soft_deletes: + items: + $ref: '#/components/schemas/indices._common___SoftDeletes' + soft_deletes.retention_lease.period: + $ref: '#/components/schemas/_common___Duration' + routing: + $ref: '#/components/schemas/indices._common___IndexRouting' + lifecycle: + $ref: '#/components/schemas/indices._common___IndexSettingsLifecycle' + + _common___StringOrStringArray: + oneOf: + - type: string + - type: array + items: + type: string + + indices._common___SoftDeletes: + type: object + properties: + enabled: + description: Indicates whether soft deletes are enabled on the index. + $ref: '#/components/schemas/_common___StringifiedBoolean' + retention: + $ref: '#/components/schemas/indices._common___SoftDeletesRetention' + retention_lease: + $ref: '#/components/schemas/indices._common___RetentionLease' + + _common___Duration: + description: |- + A duration. Units can be `nanos`, `micros`, `ms` (milliseconds), `s` (seconds), `m` (minutes), `h` (hours) and + `d` (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. + pattern: ^(?:(-1)|([0-9\.]+)(?:d|h|m|s|ms|micros|nanos))$ + type: string + + indices._common___IndexRouting: + type: object + properties: + allocation: + $ref: '#/components/schemas/indices._common___IndexRoutingAllocation' + rebalance: + $ref: '#/components/schemas/indices._common___IndexRoutingRebalance' + + _common___StringifiedEpochTimeUnitMillis: + description: |- + Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures + this behavior while keeping the semantics of the field type. + + Depending on the target language, code generators can keep the union or remove it and leniently parse + strings to the target type. + oneOf: + - $ref: '#/components/schemas/_common___EpochTimeUnitMillis' + - type: string + + _common___EpochTimeUnitMillis: + allOf: + - $ref: '#/components/schemas/_common___UnitMillis' + + _common___UnitMillis: + description: The time unit for milliseconds. + type: integer + format: int64 + + indices._common___IndexSettingsLifecycle: + type: object + properties: + name: + type: string + indexing_complete: + $ref: '#/components/schemas/_common___StringifiedBoolean' + origination_date: + description: |- + If specified, this is the timestamp used to calculate the index age for its phase transitions. Use this setting + if you create a new index that contains old data and want to use the original creation date to calculate the index + age. Specified as a Unix epoch value in milliseconds. + $ref: '#/components/schemas/_common___StringifiedEpochTimeUnitMillis' + parse_origination_date: + description: |- + Set to `true` to parse the origination date from the index name. This origination date is used to calculate the index age + for its phase transitions. The index name must match the pattern `^.*-{date_format}-\\d+`, where the `date_format` is + `yyyy.MM.dd` and the trailing digits are optional. An index that was rolled over would normally match the full format, + for example `logs-2016.10.31-000002`). If the index name doesn't match the pattern, index creation fails. + type: boolean + step: + $ref: '#/components/schemas/indices._common___IndexSettingsLifecycleStep' + rollover_alias: + description: |- + The index alias to update when the index rolls over. Specify when using a policy that contains a rollover action. + When the index rolls over, the alias is updated to reflect that the index is no longer the write index. For more + information about rolling indexes, see Rollover. + type: string + required: + - name + + _common___StringifiedLong: + oneOf: + - type: integer + format: int64 + - type: string + + _common___StringifiedBoolean: + description: |- + Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures + this behavior while keeping the semantics of the field type. + + Depending on the target language, code generators can keep the union or remove it and leniently parse + strings to the target type. + oneOf: + - type: boolean + - type: string + + _common___StringifiedInteger: + description: |- + Certain APIs may return values, including numbers such as epoch timestamps, as strings. This setting captures + this behavior while keeping the semantics of the field type. + + Depending on the target language, code generators can keep the union or remove it and leniently parse + strings to the target type. + oneOf: + - type: integer + - type: string + + indices._common___SoftDeletesRetention: + type: object + description: The retention settings for soft deletes. + properties: + operations: + $ref: '#/components/schemas/_common___StringifiedLong' + + indices._common___RetentionLease: + type: object + properties: + period: + $ref: '#/components/schemas/_common___Duration' + required: + - period + + indices._common___IndexRoutingAllocation: + type: object + properties: + enable: + $ref: '#/components/schemas/indices._common___IndexRoutingAllocationOptions' + include: + $ref: '#/components/schemas/indices._common___IndexRoutingAllocationInclude' + initial_recovery: + $ref: '#/components/schemas/indices._common___IndexRoutingAllocationInitialRecovery' + disk: + $ref: '#/components/schemas/indices._common___IndexRoutingAllocationDisk' + total_shards_per_node: + $ref: '#/components/schemas/_common___StringifiedInteger' + + indices._common___IndexRoutingAllocationDisk: + type: object + properties: + threshold_enabled: + $ref: '#/components/schemas/_common___StringifiedBoolean' + indices._common___IndexRoutingAllocationInclude: + type: object + properties: + _tier_preference: + type: string + _id: + type: string + indices._common___IndexRoutingAllocationInitialRecovery: + type: object + properties: + _id: + type: string + indices._common___IndexRoutingAllocationOptions: + type: string + enum: + - all + - new_primaries + - none + - primaries + + indices._common___IndexRoutingRebalance: + type: object + properties: + enable: + $ref: '#/components/schemas/indices._common___IndexRoutingRebalanceOptions' + required: + - enable + + indices._common___IndexRoutingRebalanceOptions: + type: string + enum: + - all + - none + - primaries + - replicas + + indices._common___IndexSettingsLifecycleStep: + type: object + properties: + wait_time_threshold: + $ref: '#/components/schemas/_common___Duration' \ No newline at end of file diff --git a/spec-insert/spec/doc_processor_spec.rb b/spec-insert/spec/doc_processor_spec.rb index 76be789cf0a..4f2c46b9e7d 100644 --- a/spec-insert/spec/doc_processor_spec.rb +++ b/spec-insert/spec/doc_processor_spec.rb @@ -14,11 +14,15 @@ def test_file(file_name) expect(actual_output).to eq(expected_output) end - it 'inserts the param tables correctly' do - test_file('param_tables') + it 'inserts the endpoints correctly' do + test_file('endpoints') end - it 'inserts the Endpoints correctly' do - test_file('endpoints') + it 'inserts the url param tables correctly' do + test_file('url_params_tables') + end + + it 'inserts the body param tables correctly' do + test_file('body_params_tables') end end diff --git a/spec-insert/spec/mock_config.yml b/spec-insert/spec/mock_config.yml index 15e31f24eae..96aaeab6e19 100644 --- a/spec-insert/spec/mock_config.yml +++ b/spec-insert/spec/mock_config.yml @@ -1,4 +1,6 @@ param_table: + parameter_column: + freeform_text: -- freeform field -- default_column: empty_text: N/A required_column: