Skip to content

Commit b6b3aff

Browse files
authored
Merge branch 'main' into performance-testing-practices
2 parents 401c0ce + aa79358 commit b6b3aff

36 files changed

+1169
-227
lines changed

DEVELOPER_GUIDE.md

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
The `.md` documents in this repository are rendered into HTML pages using [Jekyll](https://jekyllrb.com/). These HTML pages are hosted on [opensearch.org](https://opensearch.org/docs/latest/).
1515

1616
## Starting the Jekyll server locally
17+
1718
You can run the Jekyll server locally to view the rendered HTML pages using the following steps:
1819

1920
1. Install [Ruby](https://www.ruby-lang.org/en/documentation/installation/) 3.1.0 or later for your operating system.
@@ -22,6 +23,7 @@ You can run the Jekyll server locally to view the rendered HTML pages using the
2223
4. Open your browser and navigate to `http://localhost:4000` to view the rendered HTML pages.
2324

2425
## Using the `spec-insert` Jekyll plugin
26+
2527
The `spec-insert` Jekyll plugin is used to insert API components into Markdown files. The plugin downloads the [latest OpenSearch specification](https://github.com/opensearch-project/opensearch-api-specification) and renders the API components from the spec. This aims to reduce the manual effort required to keep the documentation up to date.
2628

2729
To use this plugin, make sure that you have installed Ruby 3.1.0 or later and the required gems by running `bundle install`.
@@ -67,24 +69,28 @@ bundle exec jekyll spec-insert --refresh-spec
6769
```
6870

6971
### Ignoring files and folders
72+
7073
The `spec-insert` plugin ignores all files and folders listed in the [./_config.yml#exclude](./_config.yml) list, which is also the list of files and folders that Jekyll ignores.
7174

7275
### Configuration
76+
7377
You can update the configuration settings for this plugin through the [config.yml](./spec-insert/config.yml) file.
7478

75-
_Note that tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run._
79+
**Note:** The tests for this plugin use a mock configuration [file](./spec-insert/spec/mock_config.yml) to assure that the tests still pass when the config file is altered. The expected output for the tests is based on the mock configuration file and will look different from the actual output when the plugin is run.
7680

7781
## CI/CD
7882
The `spec-insert` plugin is run as part of the CI/CD pipeline to ensure that the API components are up to date in the documentation. This is performed through the [update-api-components.yml](.github/workflows/update-api-components.yml) GitHub Actions workflow, which creates a pull request containing the updated API components every Sunday.
7983

8084
## Spec insert components
8185
All spec insert components accept the following arguments:
86+
8287
- `api` (String; required): The name of the API to render the component from. This is equivalent to the `x-operation-group` field in the OpenSearch OpenAPI Spec.
8388
- `component` (String; required): The name of the component to render, such as `query_parameters`, `path_parameters`, or `endpoints`.
8489
- `omit_header` (Boolean; Default is `false`): If set to `true`, the markdown header of the component will not be rendered.
8590

8691
### Endpoints
8792
To insert endpoints for the `search` API, use the following snippet:
93+
8894
```markdown
8995
<!-- spec_insert_start
9096
api: search
@@ -104,10 +110,13 @@ component: path_parameters
104110
-->
105111
<!-- spec_insert_end -->
106112
```
113+
107114
This table accepts the same arguments as the query parameters table except the `include_global` argument.
108115

109116
### Query parameters
117+
110118
To insert the API query parameters table of the `cat.indices` API, use the following snippet:
119+
111120
```markdown
112121
<!-- spec_insert_start
113122
api: cat.indices
@@ -144,3 +153,28 @@ pretty: true
144153
-->
145154
<!-- spec_insert_end -->
146155
```
156+
157+
### Request and response bodies (Beta)
158+
159+
To insert the request and response body tables of the `indices.create` API, use the following snippet:
160+
161+
```markdown
162+
<!-- spec_insert_start
163+
api: indices.create
164+
component: request_body_parameters // or response_body_parameters
165+
-->
166+
<!-- spec_insert_end -->
167+
```
168+
169+
**Note:**: These components are still a work in progress and may not render correctly for all APIs.
170+
171+
## Spec insert coverage report
172+
To generate a coverage report of the API components that are being used in the documentation, run the following command:
173+
174+
```shell
175+
cd spec-insert
176+
bundle exec rake generate_utilization_coverage
177+
```
178+
179+
The coverage report will be generated in the `spec-insert/utilization_coverage.md` by default.
180+

_dashboards/dql.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ By default, OpenSearch Dashboards uses DQL syntax. To switch to query string que
1919

2020
The syntax changes to **Lucene**. To switch back to DQL, select the **Lucene** button and toggle the **Off** switch.
2121

22+
## Queries on analyzed text
23+
24+
When running queries, understanding whether your fields are analyzed ([`text`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/text/) type) or non-analyzed ([`keyword`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/keyword/) type) is crucial because it significantly impacts search behavior. In analyzed fields, text undergoes tokenization and filtering, while non-analyzed fields store exact values. For simple field queries like `wind`, searches against analyzed fields match documents containing `wind` regardless of case, while the same query on keyword fields requires exact matching of the full string. For more information about analyzed fields, see [Text analysis]({{site.url}}{{site.baseurl}}/analyzers/).
25+
2226
## Setup
2327

2428
To follow this tutorial in OpenSearch Dashboards, expand the following setup steps.
@@ -142,7 +146,7 @@ The following table provides a quick reference for both query language commands.
142146
| Boolean `NOT` | `NOT media_type: article` <br><br> `not media_type: article` | `NOT media_type:article` <br><br> `-media_type:article` |
143147
| Boolean `OR` | `title: wind OR description: film` <br><br> `title: wind or description: film` | `title: wind OR description: film` |
144148
| Required/Prohibited operators | Not supported | Supports both `+` (required operator) and `-` (prohibited operator) <br><br> `+title:wind -media_type:article` (returns documents in which `title` contains `wind` but `media_type` does not contain `article`) |
145-
| Wildcards | `title: wind*`<br><br> `titl*: wind` <br><br> Only supports `*` (multiple characters) | `title:wind*` or `title:w?nd` <br><br> Does not support wildcards in field names <br><br> Supports `*` (multiple characters) and `?` (single character) |
149+
| Wildcards | `title: wind*`<br><br> `titl*: wind` <br><br> Does not support wildcards in phrase searches (within quotation marks) <br><br> Only supports `*` (multiple characters) | `title:wind*` or `title:w?nd` <br><br> Does not support wildcards in field names <br><br> Does not support wildcards in phrase searches (within quotation marks) <br><br> Supports `*` (multiple characters) and `?` (single character) |
146150
| Regular expressions | Not supported | `title:/w[a-z]nd/` |
147151
| Fuzzy search | Not supported | `title:wind~2` |
148152
| Proximity search | Not supported | `"wind rises"~2` |

_install-and-configure/install-opensearch/docker.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ docker compose logs <serviceName>
297297
```
298298
{% include copy.html %}
299299

300-
Verify access to OpenSearch Dashboards by connecting to http://localhost:5601 from a browser. The default username and password are `admin`. We do not recommend using this configuration on hosts that are accessible from the public internet until you have customized the security configuration of your deployment.
300+
Verify access to OpenSearch Dashboards by connecting to http://localhost:5601 from a browser. For OpenSearch 2.12 and later, you must use your configured username and password. For earlier versions, the default username and password are `admin`. We do not recommend using this configuration on hosts that are accessible from the public internet until you have customized the security configuration of your deployment.
301301

302302
Remember that `localhost` cannot be accessed remotely. If you are deploying these containers to a remote host, then you will need to establish a network connection and replace `localhost` with the IP or DNS record corresponding to the host.
303303
{: .note}

_security/access-control/field-masking.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,16 @@ See [Create role]({{site.url}}{{site.baseurl}}/security/access-control/api/#crea
7979

8080
By default, the Security plugin uses the BLAKE2b algorithm, but you can use any hashing algorithm that your JVM provides. This list typically includes MD5, SHA-1, SHA-384, and SHA-512.
8181

82-
You can override the default algorithm in `opensearch.yml` using the option default masking algorithm setting `plugins.security.masked_fields.algorithm.default`, as shown in the following example:
82+
You can override the default algorithm in `opensearch.yml` using the optional default masking algorithm setting `plugins.security.masked_fields.algorithm.default`, as shown in the following example:
8383

8484
```yml
8585
plugins.security.masked_fields.algorithm.default: SHA-256
8686
```
87+
OpenSearch 3.x contains a bug fix to apply the default BLAKE2b algorithm correctly. You can override the default algorithm in OpenSearch 3.x to continue to produce the same masked values as OpenSearch 1.x and 2.x in `opensearch.yml` using the optional default masking algorithm setting `plugins.security.masked_fields.algorithm.default`, as shown in the following example:
88+
89+
```yml
90+
plugins.security.masked_fields.algorithm.default: BLAKE2B_LEGACY_DEFAULT
91+
```
8792

8893
To specify a different algorithm, add it after the masked field in `roles.yml`, as shown in the following:
8994

spec-insert/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
opensearch-openapi.yaml
22
rspec_examples.txt
3+
utilization_coverage.md

spec-insert/Rakefile

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# The OpenSearch Contributors require contributions made to
4+
# this file be licensed under the Apache-2.0 license or a
5+
# compatible open source license.
6+
7+
# frozen_string_literal: true
8+
9+
require 'rake'
10+
require 'active_support/all'
11+
require_relative 'lib/coverage/utilization_coverage'
12+
require_relative 'lib/utils'
13+
14+
desc 'Generate utilization coverage of Spec-Insert components'
15+
task :generate_utilization_coverage do
16+
Utils.load_spec
17+
coverage = UtilizationCoverage.new.render
18+
file = File.join(__dir__, 'utilization_coverage.md')
19+
File.write(file, coverage)
20+
puts "Utilization coverage written to #{file}"
21+
end

spec-insert/config.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
param_table:
2+
parameter_column:
3+
freeform_text: -- freeform field --
24
default_column:
35
empty_text: N/A
46
required_column:

spec-insert/lib/api/action.rb

Lines changed: 84 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -7,62 +7,104 @@
77
# frozen_string_literal: true
88

99
require_relative 'parameter'
10+
require_relative 'body'
1011
require_relative 'operation'
1112

12-
# A collection of operations that comprise a single API Action
13-
# AKA operation-group
14-
class Action
15-
# @param [SpecHash] spec Parsed OpenAPI spec
16-
def self.actions=(spec)
17-
operations = spec.paths.flat_map do |url, ops|
18-
ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] }
13+
module Api
14+
# A collection of operations that comprise a single API Action
15+
# AKA operation-group
16+
class Action
17+
SUCCESS_CODES = %w[200 201 202 203 204 205 206 207 208 226].freeze
18+
19+
# @param [SpecHash] spec Parsed OpenAPI spec
20+
def self.actions=(spec)
21+
operations = spec.paths.flat_map do |url, ops|
22+
ops.filter_map { |verb, op| Operation.new(op, url, verb) unless op['x-ignorable'] }
23+
end
24+
@actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }
1925
end
20-
@actions = operations.group_by(&:group).values.map { |ops| Action.new(ops) }.index_by(&:full_name)
21-
end
2226

23-
# @return [Hash<String, Action>] API Actions indexed by operation-group
24-
def self.actions
25-
raise 'Actions not set' unless @actions
26-
@actions
27-
end
27+
# @return [Array<Action>] API Actions
28+
def self.all
29+
raise 'Actions not set' unless @actions
30+
@actions
31+
end
2832

29-
# @return [Array<Operation>] Operations in the action
30-
attr_reader :operations
33+
def self.by_full_name
34+
@by_full_name ||= all.index_by(&:full_name).to_h
35+
end
3136

32-
# @param [Array<Operation>] operations
33-
def initialize(operations)
34-
@operations = operations
35-
@operation = operations.first
36-
@spec = @operation&.spec
37-
end
37+
def self.by_namespace
38+
@by_namespace ||= all.group_by(&:namespace)
39+
end
3840

39-
# @return [Array<Parameter>] Input arguments.
40-
def arguments; @arguments ||= Parameter.from_operations(@operations.map(&:spec)); end
41+
# @return [Array<Api::Operation>] Operations in the action
42+
attr_reader :operations
4143

42-
# @return [String] Full name of the action (i.e. namespace.action)
43-
def full_name; @operation&.group; end
44+
# @param [Array<Api::Operation>] operations
45+
def initialize(operations)
46+
@operations = operations
47+
@operation = operations.first || {}
48+
@spec = @operation&.spec
49+
end
4450

45-
# return [String] Name of the action
46-
def name; @operation&.action; end
51+
def query_parameters
52+
@operations.map(&:spec).flat_map(&:parameters).filter { |param| !param['x-global'] && param.in == 'query' }
53+
.group_by(&:name).values
54+
.map { |params| Parameter.from_param_specs(params, @operations.size) }
55+
end
4756

48-
# @return [String] Namespace of the action
49-
def namespace; @operation&.namespace; end
57+
def path_parameters
58+
@operations.map(&:spec).flat_map(&:parameters).filter { |param| param.in == 'path' }
59+
.group_by(&:name).values
60+
.map { |params| Parameter.from_param_specs(params, @operations.size) }
61+
end
5062

51-
# @return [Array<String>] Sorted unique HTTP verbs
52-
def http_verbs; @operations.map(&:http_verb).uniq.sort; end
63+
# @return [Api::Body, nil] Request body
64+
def request_body
65+
@request_body ||=
66+
begin
67+
operation = @operations.find { |op| op.spec.requestBody.present? }
68+
required = @operations.all? { |op| op.spec.requestBody.required }
69+
operation.nil? ? nil : Body.new(operation.spec.requestBody.content, required:)
70+
end
71+
end
72+
73+
# @return [Api::Body] Response body
74+
def response_body
75+
@response_body ||=
76+
begin
77+
spec = @operations.first.spec
78+
code = SUCCESS_CODES.find { |c| spec.responses[c].present? }
79+
Body.new(@operations.first.spec.responses[code].content, required: nil)
80+
end
81+
end
82+
83+
# @return [String] Full name of the action (i.e. namespace.action)
84+
def full_name; @operation.group; end
5385

54-
# @return [Array<String>] Unique URLs
55-
def urls; @operations.map(&:url).uniq; end
86+
# return [String] Name of the action
87+
def name; @operation.action; end
5688

57-
# @return [String] Description of the action
58-
def description; @spec&.description; end
89+
# @return [String] Namespace of the action
90+
def namespace; @operation.namespace || ''; end
5991

60-
# @return [Boolean] Whether the action is deprecated
61-
def deprecated; @spec&.deprecated; end
92+
# @return [Array<String>] Sorted unique HTTP verbs
93+
def http_verbs; @operations.map(&:http_verb).uniq.sort; end
6294

63-
# @return [String] Deprecation message
64-
def deprecation_message; @spec['x-deprecation-message']; end
95+
# @return [Array<String>] Unique URLs
96+
def urls; @operations.map(&:url).uniq; end
6597

66-
# @return [String] API reference
67-
def api_reference; @operation&.external_docs&.url; end
98+
# @return [String] Description of the action
99+
def description; @spec.description; end
100+
101+
# @return [Boolean] Whether the action is deprecated
102+
def deprecated; @spec.deprecated; end
103+
104+
# @return [String] Deprecation message
105+
def deprecation_message; @spec['x-deprecation-message']; end
106+
107+
# @return [String] API reference
108+
def api_reference; @operation.external_docs.url; end
109+
end
68110
end

spec-insert/lib/api/body.rb

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# frozen_string_literal: true
2+
3+
require_relative 'parameter'
4+
require_relative 'body_parameter'
5+
6+
module Api
7+
# Request or response body
8+
class Body
9+
# @return [Boolean] Whether the body is in NDJSON format
10+
attr_reader :ndjson
11+
12+
# @return [Boolean]
13+
attr_reader :required
14+
15+
# @return [Array<Api::BodyParameterGroup>]
16+
attr_reader :params_group
17+
18+
# @param [SpecHash] content
19+
# @param [Boolean, nil] required
20+
def initialize(content, required:)
21+
@required = required
22+
@ndjson = content['application/json'].nil?
23+
spec = content['application/json'] || content['application/x-ndjson']
24+
@params_group = BodyParameterGroup.from_schema(
25+
flatten_schema(spec.schema),
26+
description: spec.description || spec.schema.description,
27+
ancestors: []
28+
)
29+
end
30+
31+
# @param [SpecHash] schema
32+
# @return [SpecHash] a schema with allOf flattened
33+
def flatten_schema(schema)
34+
return schema if schema.type.present? && schema.type != 'object'
35+
return schema if schema.properties.present?
36+
return schema if schema.additionalProperties.present?
37+
return schema.anyOf.map { |sch| flatten_schema(sch) } if schema.anyOf.present?
38+
return schema.oneOf.map { |sch| flatten_schema(sch) } if schema.oneOf.present?
39+
return schema if schema.allOf.blank?
40+
41+
schema = schema.allOf.each_with_object({ properties: {}, required: [] }) do |sch, h|
42+
sch = flatten_schema(sch)
43+
h[:properties].merge!(sch.properties || {})
44+
h[:required] += sch.required || []
45+
h[:additionalProperties] ||= sch.additionalProperties
46+
end
47+
48+
SpecHash.new(schema, fully_parsed: true)
49+
end
50+
end
51+
end

0 commit comments

Comments
 (0)