[Spec-Insert] Request and Response Body Tables [Beta] + Utilization Coverage

DEVELOPER_GUIDE.md

@@ -14,6 +14,7 @@
 The `.md` documents in this repository are rendered into HTML pages using [Jekyll](https://jekyllrb.com/). These HTML pages are hosted on [opensearch.org](https://opensearch.org/docs/latest/).
 
 ## Starting the Jekyll server locally
+
 You can run the Jekyll server locally to view the rendered HTML pages using the following steps:
 
 1. Install [Ruby](https://www.ruby-lang.org/en/documentation/installation/) 3.1.0 or later for your operating system.
@@ -22,6 +23,7 @@ You can run the Jekyll server locally to view the rendered HTML pages using the
 4. Open your browser and navigate to `http://localhost:4000` to view the rendered HTML pages.
 
 ## Using the `spec-insert` Jekyll plugin
+
 The `spec-insert` Jekyll plugin is used to insert API components into Markdown files. The plugin downloads the [latest OpenSearch specification](https://github.com/opensearch-project/opensearch-api-specification) and renders the API components from the spec. This aims to reduce the manual effort required to keep the documentation up to date.
 
 To use this plugin, make sure that you have installed Ruby 3.1.0 or later and the required gems by running `bundle install`.
@@ -67,24 +69,28 @@ bundle exec jekyll spec-insert --refresh-spec
 ```
 
 ### Ignoring files and folders
+
 The `spec-insert` plugin ignores all files and folders listed in the [./_config.yml#exclude](./_config.yml) list, which is also the list of files and folders that Jekyll ignores.
 
 ### Configuration
+
 You can update the configuration settings for this plugin through the [config.yml](./spec-insert/config.yml) file. 
 
-_Note that tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run._
+**Note:** The tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run.
 
 ## CI/CD
 The `spec-insert` plugin is run as part of the CI/CD pipeline to ensure that the API components are up to date in the documentation. This is performed through the [update-api-components.yml](.github/workflows/update-api-components.yml) GitHub Actions workflow, which creates a pull request containing the updated API components every Sunday.
 
 ## Spec insert components
 All spec insert components accept the following arguments:
+
 - `api` (String; required): The name of the API to render the component from. This is equivalent to the `x-operation-group` field in the OpenSearch OpenAPI Spec.
 - `component` (String; required): The name of the component to render, such as `query_parameters`, `path_parameters`, or `endpoints`.
 - `omit_header` (Boolean; Default is `false`): If set to `true`, the markdown header of the component will not be rendered.
 
 ### Endpoints
 To insert endpoints for the `search` API, use the following snippet:
+
 ```markdown
 <!-- spec_insert_start
 api: search
@@ -104,10 +110,13 @@ component: path_parameters
 -->
 <!-- spec_insert_end -->
 ```
+
 This table accepts the same arguments as the query parameters table except the `include_global` argument.
 
 ### Query parameters
+
 To insert the API query parameters table of the `cat.indices` API, use the following snippet:
+
 ```markdown
 <!-- spec_insert_start
 api: cat.indices
@@ -144,3 +153,28 @@ pretty: true
 -->
 <!-- spec_insert_end -->
 ```
+
+### Request and response bodies (Beta)
+
+To insert the request and response body tables of the `indices.create` API, use the following snippet:
+
+```markdown
+<!-- spec_insert_start
+api: indices.create
+component: request_body_parameters // or response_body_parameters
+-->
+<!-- spec_insert_end -->
+```
+
+**Note:**: These components are still a work in progress and may not render correctly for all APIs.
+
+## Spec insert coverage report
+To generate a coverage report of the API components that are being used in the documentation, run the following command:
+
+```shell
+cd spec-insert
+bundle exec rake generate_utilization_coverage
+```
+
+The coverage report will be generated in the `spec-insert/utilization_coverage.md` by default.
+

spec-insert/.gitignore

@@ -1,2 +1,3 @@
 opensearch-openapi.yaml
 rspec_examples.txt
+utilization_coverage.md

spec-insert/Rakefile

@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+require 'rake'
+require 'active_support/all'
+require_relative 'lib/coverage/utilization_coverage'
+require_relative 'lib/utils'
+
+desc 'Generate utilization coverage of Spec-Insert components'
+task :generate_utilization_coverage do
+  Utils.load_spec
+  coverage = UtilizationCoverage.new.render
+  file = File.join(__dir__, 'utilization_coverage.md')
+  File.write(file, coverage)
+  puts "Utilization coverage written to #{file}"
+end

spec-insert/config.yml

@@ -1,4 +1,6 @@
 param_table:
+  parameter_column:
+    freeform_text: -- freeform field --
   default_column:
     empty_text: N/A
   required_column:

spec-insert/lib/api/action.rb

@@ -7,62 +7,104 @@
 # frozen_string_literal: true
 
 require_relative 'parameter'
+require_relative 'body'
 require_relative 'operation'
 
-# A collection of operations that comprise a single API Action
-# AKA operation-group
-class Action
-  # @param [SpecHash] spec Parsed OpenAPI spec
-  def self.actions=(spec)
-    operations = spec.paths.flat_map do |url, ops|
-      ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] }
+module Api
+  # A collection of operations that comprise a single API Action
+  # AKA operation-group
+  class Action
+    SUCCESS_CODES = %w[200 201 202 203 204 205 206 207 208 226].freeze
+
+    # @param [SpecHash] spec Parsed OpenAPI spec
+    def self.actions=(spec)
+      operations = spec.paths.flat_map do |url, ops|
+        ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] }
+      end
+      @actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }
     end
-    @actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }.index_by(&:full_name)
-  end
 
-  # @return [Hash<String, Action>] API Actions indexed by operation-group
-  def self.actions
-    raise 'Actions not set' unless @actions
-    @actions
-  end
+    # @return [Array<Action>] API Actions
+    def self.all
+      raise 'Actions not set' unless @actions
+      @actions
+    end
 
-  # @return [Array<Operation>] Operations in the action
-  attr_reader :operations
+    def self.by_full_name
+      @by_full_name ||= all.index_by(&:full_name).to_h
+    end
 
-  # @param [Array<Operation>] operations
-  def initialize(operations)
-    @operations = operations
-    @operation = operations.first
-    @spec = @operation&.spec
-  end
+    def self.by_namespace
+      @by_namespace ||= all.group_by(&:namespace)
+    end
 
-  # @return [Array<Parameter>] Input arguments.
-  def arguments; @arguments ||= Parameter.from_operations(@operations.map(&:spec)); end
+    # @return [Array<Api::Operation>] Operations in the action
+    attr_reader :operations
 
-  # @return [String] Full name of the action (i.e. namespace.action)
-  def full_name; @operation&.group; end
+    # @param [Array<Api::Operation>] operations
+    def initialize(operations)
+      @operations = operations
+      @operation = operations.first || {}
+      @spec = @operation&.spec
+    end
 
-  # return [String] Name of the action
-  def name; @operation&.action; end
+    def query_parameters
+      @operations.map(&:spec).flat_map(&:parameters).filter { |param| !param['x-global'] && param.in == 'query' }
+                 .group_by(&:name).values
+                 .map { |params| Parameter.from_param_specs(params, @operations.size) }
+    end
 
-  # @return [String] Namespace of the action
-  def namespace; @operation&.namespace; end
+    def path_parameters
+      @operations.map(&:spec).flat_map(&:parameters).filter { |param| param.in == 'path' }
+                 .group_by(&:name).values
+                 .map { |params| Parameter.from_param_specs(params, @operations.size) }
+    end
 
-  # @return [Array<String>] Sorted unique HTTP verbs
-  def http_verbs; @operations.map(&:http_verb).uniq.sort; end
+    # @return [Api::Body, nil] Request body
+    def request_body
+      @request_body ||=
+        begin
+          operation = @operations.find { |op| op.spec.requestBody.present? }
+          required = @operations.all? { |op| op.spec.requestBody.required }
+          operation.nil? ? nil : Body.new(operation.spec.requestBody.content, required:)
+        end
+    end
+
+    # @return [Api::Body] Response body
+    def response_body
+      @response_body ||=
+        begin
+          spec = @operations.first.spec
+          code = SUCCESS_CODES.find { |c| spec.responses[c].present? }
+          Body.new(@operations.first.spec.responses[code].content, required: nil)
+        end
+    end
+
+    # @return [String] Full name of the action (i.e. namespace.action)
+    def full_name; @operation.group; end
 
-  # @return [Array<String>] Unique URLs
-  def urls; @operations.map(&:url).uniq; end
+    # return [String] Name of the action
+    def name; @operation.action; end
 
-  # @return [String] Description of the action
-  def description; @spec&.description; end
+    # @return [String] Namespace of the action
+    def namespace; @operation.namespace || ''; end
 
-  # @return [Boolean] Whether the action is deprecated
-  def deprecated; @spec&.deprecated; end
+    # @return [Array<String>] Sorted unique HTTP verbs
+    def http_verbs; @operations.map(&:http_verb).uniq.sort; end
 
-  # @return [String] Deprecation message
-  def deprecation_message; @spec['x-deprecation-message']; end
+    # @return [Array<String>] Unique URLs
+    def urls; @operations.map(&:url).uniq; end
 
-  # @return [String] API reference
-  def api_reference; @operation&.external_docs&.url; end
+    # @return [String] Description of the action
+    def description; @spec.description; end
+
+    # @return [Boolean] Whether the action is deprecated
+    def deprecated; @spec.deprecated; end
+
+    # @return [String] Deprecation message
+    def deprecation_message; @spec['x-deprecation-message']; end
+
+    # @return [String] API reference
+    def api_reference; @operation.external_docs.url; end
+  end
 end

spec-insert/lib/api/body.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require_relative 'parameter'
+require_relative 'body_parameter'
+
+module Api
+  # Request or response body
+  class Body
+    # @return [Boolean] Whether the body is in NDJSON format
+    attr_reader :ndjson
+
+    # @return [Boolean]
+    attr_reader :required
+
+    # @return [Array<Api::BodyParameterGroup>]
+    attr_reader :params_group
+
+    # @param [SpecHash] content
+    # @param [Boolean, nil] required
+    def initialize(content, required:)
+      @required = required
+      @ndjson = content['application/json'].nil?
+      spec = content['application/json'] || content['application/x-ndjson']
+      @params_group = BodyParameterGroup.from_schema(
+        flatten_schema(spec.schema),
+        description: spec.description || spec.schema.description,
+        ancestors: []
+      )
+    end
+
+    # @param [SpecHash] schema
+    # @return [SpecHash] a schema with allOf flattened
+    def flatten_schema(schema)
+      return schema if schema.type.present? && schema.type != 'object'
+      return schema if schema.properties.present?
+      return schema if schema.additionalProperties.present?
+      return schema.anyOf.map { |sch| flatten_schema(sch) } if schema.anyOf.present?
+      return schema.oneOf.map { |sch| flatten_schema(sch) } if schema.oneOf.present?
+      return schema if schema.allOf.blank?
+
+      schema = schema.allOf.each_with_object({ properties: {}, required: [] }) do |sch, h|
+        sch = flatten_schema(sch)
+        h[:properties].merge!(sch.properties || {})
+        h[:required] += sch.required || []
+        h[:additionalProperties] ||= sch.additionalProperties
+      end
+
+      SpecHash.new(schema, fully_parsed: true)
+    end
+  end
+end