Skip to content

Commit 3f5742f

Browse files
authored
[chore] Refresh refcache, fix external links with invalid fragments (#6206)
1 parent fe62371 commit 3f5742f

File tree

17 files changed

+1302
-1162
lines changed

17 files changed

+1302
-1162
lines changed

content/en/blog/2023/jmx-metric-insight/index.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,9 @@ that our Kafka installation is working as expected.
111111
### Export metrics to Prometheus
112112

113113
The metrics can be exported by any of the supported metric exporters, to a
114-
backend of your choice. The full list of exporters and their configuration
115-
options can be found
116-
[here](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#exporters).
114+
backend of your choice. For the full list of exporters and their configuration
115+
options, see
116+
[Properties: exporters](/docs/languages/java/configuration/#properties-exporters).
117117
For instance, you can export the metrics to an OTel collector using the OTLP
118118
exporter, perform some processing and then consume the metrics on a backend of
119119
your choice. In this example for the sake of simplicity, we are directly

content/en/blog/2024/collector-roadmap/index.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ wanted to focus on:
6565
OTLP exporter.
6666
2. Individual Go modules that the Collector components rely upon must also be
6767
marked as stable as per the project's
68-
[versioning guidelines](https://github.com/open-telemetry/opentelemetry-collector/blob/main/VERSIONING.md#public-api-expectations).
68+
[versioning guidelines](https://github.com/open-telemetry/opentelemetry-collector/blob/main/VERSIONING.md#general-go-api-considerations).
6969

7070
Aside from this, there were a few areas the contributors wanted to improve based
7171
on user feedback:

content/en/blog/2024/go-contrib-removal.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ To become a code owner of one of the modules, you need to be a member of the
4747
OpenTelemetry organization and have a good working knowledge of the code you
4848
seek to maintain. To become a member of OpenTelemetry in GitHub, see the
4949
requirements in
50-
[Community membership](https://github.com/open-telemetry/community/blob/main/community-membership.md#requirements).
50+
[Community membership](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#requirements).
5151

5252
If you satisfy all requirements,
5353
[open an issue](https://github.com/open-telemetry/opentelemetry-go-contrib/issues/new?assignees=&labels=&projects=&template=owner.md&title=).

content/en/blog/2024/java-metric-systems-compared/index.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -285,7 +285,7 @@ The result is a configurable option unique to OpenTelemetry Java called
285285
what their memory mode is based on whether they read metric state concurrently
286286
or not. Right now you opt into the optimized memory behavior (which we call
287287
`MemoryMode.reusable_data`) via an
288-
[environment variable](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#exporters).
288+
[environment variable](/docs/languages/java/configuration/#properties-exporters).
289289
In the future, the optimized memory mode will be enabled by default, since only
290290
exceptional cases need concurrent access to the metric state. It turns out that
291291
the objects holding the metric state (`MetricData` in OpenTelemetry Java terms)

content/en/docs/demo/docker-deployment.md

+1-2
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,7 @@ cSpell:ignore: otlphttp spanmetrics tracetest tracetesting
1010
## Prerequisites
1111

1212
- Docker
13-
- [Docker Compose](https://docs.docker.com/compose/install/#install-compose)
14-
v2.0.0+
13+
- [Docker Compose](https://docs.docker.com/compose/install/) v2.0.0+
1514
- Make (optional)
1615
- 6 GB of RAM for the application
1716

content/en/docs/kubernetes/operator/troubleshooting/target-allocator.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -329,7 +329,7 @@ See the full `OpenTelemetryCollector`
329329
### Did you configure a ServiceMonitor (or PodMonitor) selector?
330330

331331
If you configured a
332-
[`ServiceMonitor`](https://observability.thomasriley.co.uk/prometheus/configuring-prometheus/using-service-monitors/#:~:text=The%20ServiceMonitor%20is%20used%20to,build%20the%20required%20Prometheus%20configuration.)
332+
[`ServiceMonitor`](https://observability.thomasriley.co.uk/prometheus/configuring-prometheus/using-service-monitors/)
333333
selector, it means that the Target Allocator only looks for `ServiceMonitors`
334334
having a `metadata.label` that matches the value in
335335
[`serviceMonitorSelector`](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#opentelemetrycollectorspectargetallocatorprometheuscr-1).

content/en/docs/languages/java/getting-started.md

+4-6
Original file line numberDiff line numberDiff line change
@@ -245,10 +245,10 @@ value=8192, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}
245245

246246
For more:
247247

248-
- Run this example with another [exporter][] for telemetry data.
248+
- Run this example with another [exporter] for telemetry data.
249249
- Try [zero-code instrumentation](/docs/zero-code/java/agent/) on one of your
250250
own apps.
251-
- For light-weight customized telemetry, try [annotations][].
251+
- For light-weight customized telemetry, try [annotations].
252252
- Learn about [manual instrumentation][] and try out more
253253
[examples](../examples/).
254254
- Take a look at the [OpenTelemetry Demo](/docs/demo/), which includes Java
@@ -260,10 +260,8 @@ For more:
260260
[logs]: /docs/concepts/signals/logs/
261261
[annotations]: /docs/zero-code/java/agent/annotations/
262262
[configure the java agent]: /docs/zero-code/java/agent/configuration/
263-
[console exporter]:
264-
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#logging-exporter
265-
[exporter]:
266-
https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#exporters
263+
[console exporter]: /docs/languages/java/configuration/#properties-exporters
264+
[exporter]: /docs/languages/java/configuration/#properties-exporters
267265
[java-vers]:
268266
https://github.com/open-telemetry/opentelemetry-java/blob/main/VERSIONING.md#language-version-compatibility
269267
[manual instrumentation]: ../instrumentation

content/en/docs/languages/php/_index.md

-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ cSpell:ignore: mbstring opcache
1212
## Further Reading
1313

1414
- [OpenTelemetry for PHP on GitHub](https://github.com/open-telemetry/opentelemetry-php)
15-
- [Installation](https://github.com/open-telemetry/opentelemetry-php#installation)
1615
- [Examples](https://github.com/open-telemetry/opentelemetry-php/tree/main/examples)
1716

1817
## Requirements

content/en/docs/languages/ruby/getting-started.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ a few more features that will allow you gain even deeper insights!
189189

190190
[traces]: /docs/concepts/signals/traces/
191191
[instrumentations]:
192-
https://github.com/open-telemetry/opentelemetry-ruby#instrumentation-libraries
192+
https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation
193193
[config]: ../libraries/#configuring-specific-instrumentation-libraries
194194
[exporters]: ../exporters/
195195
[context propagation]: ../instrumentation/#context-propagation

content/en/docs/zero-code/java/spring-boot-starter/getting-started.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ project. For example, if you import the `spring-boot-dependencies` BOM, you have
3636
to declare it after the OpenTelemetry BOMs.
3737

3838
Gradle selects the
39-
[latest version](https://docs.gradle.org/current/userguide/dependency_resolution.html#sec:version-conflict)
39+
[latest version](https://docs.gradle.org/current/userguide/dependency_resolution.html#2_perform_conflict_resolution)
4040
of a dependency when multiple BOMs, so the order of BOMs is not important.
4141

4242
{{% /alert %}}
@@ -106,7 +106,7 @@ with the `io.spring.dependency-management` plugin.
106106
Add the dependency given below to enable the OpenTelemetry starter.
107107

108108
The OpenTelemetry starter uses OpenTelemetry Spring Boot
109-
[autoconfiguration](https://docs.spring.io/spring-boot/docs/current/reference/html/using.html#using.auto-configuration).
109+
[autoconfiguration](https://docs.spring.io/spring-boot/reference/using/auto-configuration.html).
110110

111111
{{< tabpane text=true >}} {{% tab header="Maven (`pom.xml`)" lang=Maven %}}
112112

content/en/docs/zero-code/net/troubleshooting.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ that require the assemblies used to instrument .NET Framework applications, the
157157
ones under the `netfx` folder of the installation directory, to be also
158158
installed into the Global Assembly Cache (GAC):
159159

160-
1. [**Monkey patch instrumentation**](https://en.wikipedia.org/wiki/Monkey_patch#:~:text=Monkey%20patching%20is%20a%20technique,Python%2C%20Groovy%2C%20etc.)
160+
1. [**Monkey patch instrumentation**](https://en.wikipedia.org/wiki/Monkey_patch)
161161
of assemblies loaded as domain-neutral.
162162
2. Assembly redirection for strong-named applications if the app also ships
163163
different versions of some assemblies also shipped in the `netfx` folder.

data/ecosystem/vendors.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@
212212
commercial: true
213213
- name: Logz.io
214214
nativeOTLP: false
215-
url: https://docs.logz.io/shipping/tracing-sources/opentelemetry.html#overview
215+
url: https://docs.logz.io/docs/shipping/other/opentelemetry-data/
216216
contact:
217217
oss: false
218218
commercial: true
@@ -344,7 +344,7 @@
344344
commercial: true
345345
- name: SolarWinds
346346
nativeOTLP: true
347-
url: https://documentation.solarwinds.com/en/success_center/observability/default.htm#cshid=third-otel-integration
347+
url: https://documentation.solarwinds.com/en/success_center/observability/content/intro/otel.htm
348348
contact:
349349
oss: false
350350
commercial: true

data/registry/application-integration-python-cisco-nso.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,6 @@ authors:
1616
url: https://www.cisco.com/
1717
urls:
1818
website: https://www.cisco.com/c/en/us/products/cloud-systems-management/network-services-orchestrator/index.html
19-
docs: https://developer.cisco.com/docs/nso/#!observability-exporter/
19+
docs: https://developer.cisco.com/docs/nso/observability-exporter/
2020
createdAt: '2024-08-06'
2121
isFirstParty: true

scripts/content-modules/adjust-pages.pl

+1
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ ()
118118
s|(docs/specs/otel/logs/api.md#emit-a)n-event|$1-logrecord|;
119119
s|\[semantic-convention-groups\]|[group-stability]|;
120120
s|\Q../../docs/|../|g; # https://github.com/open-telemetry/semantic-conventions/pull/1843
121+
s|\Qhttps://wikipedia.org/wiki/Where_(SQL)#IN|https://wikipedia.org/wiki/SQL_syntax#Operators|g;
121122
}
122123

123124
sub getVersFromSubmodule() {

scripts/double-check-refcache-400s.mjs

+152-13
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,37 @@
22

33
import fs from 'fs/promises';
44
import { getUrlStatus, isHttp2XX } from './get-url-status.mjs';
5+
import { exit } from 'process';
56

67
const CACHE_FILE = 'static/refcache.json';
8+
const GOOGLE_DOCS_URL = 'https://docs.google.com/';
9+
let checkForFragments = false;
10+
let maxNumEntriesToUpdate = 3;
711
const cratesIoURL = 'https://crates.io/crates/';
812

13+
// Magic numbers that we use to determine if a URL with a fragment has been
14+
// checked with this script. Since we can't add new fields to the cache, we
15+
// encode "magic" values in the LastSeen field.
16+
const fragSecondsOk = 12;
17+
const fragMillisecondsOk = 345;
18+
const fragSecondsInvalid = 59;
19+
const fragMillisecondsInvalid = 999;
20+
21+
function isHttp2XXForFragments(StatusCode, lastSeenDate) {
22+
return (
23+
isHttp2XX(StatusCode) &&
24+
lastSeenDate.getSeconds() === fragSecondsOk &&
25+
lastSeenDate.getMilliseconds() === fragMillisecondsOk
26+
);
27+
}
28+
29+
function is4XXForFragments(StatusCode, lastSeenDate) {
30+
return (
31+
lastSeenDate.getSeconds() === fragSecondsInvalid &&
32+
lastSeenDate.getMilliseconds() === fragMillisecondsInvalid
33+
);
34+
}
35+
936
async function readRefcache() {
1037
try {
1138
const data = await fs.readFile(CACHE_FILE, 'utf8');
@@ -18,42 +45,154 @@ async function readRefcache() {
1845

1946
async function writeRefcache(cache) {
2047
await fs.writeFile(CACHE_FILE, JSON.stringify(cache, null, 2) + '\n', 'utf8');
21-
console.log(`Updated ${CACHE_FILE} with fixed links.`);
48+
console.log(`Wrote updated ${CACHE_FILE}.`);
2249
}
2350

2451
// Retry HTTP status check for refcache URLs with non-200s and not 404
2552
async function retry400sAndUpdateCache() {
53+
console.log(`Checking ${CACHE_FILE} for 4XX status URLs ...`);
2654
const cache = await readRefcache();
27-
let updated = false;
55+
let updatedCount = 0;
56+
let entriesCount = 0;
57+
let urlWithFragmentCount = 0;
58+
let urlWithInvalidFragCount = 0;
59+
let statusCounts = {};
2860

2961
for (const [url, details] of Object.entries(cache)) {
62+
entriesCount++;
63+
const parsedUrl = new URL(url);
64+
if (parsedUrl.hash) urlWithFragmentCount++;
3065
const { StatusCode, LastSeen } = details;
31-
if (isHttp2XX(StatusCode)) continue;
32-
if (StatusCode === 404 && !url.startsWith(cratesIoURL)) {
33-
console.log(`Skipping 404: ${url} (last seen ${LastSeen}).`);
66+
const lastSeenDate = new Date(LastSeen);
67+
68+
countStatuses(StatusCode, parsedUrl, lastSeenDate, statusCounts);
69+
70+
if (
71+
checkForFragments && parsedUrl.hash
72+
? isHttp2XXForFragments(StatusCode, lastSeenDate)
73+
: isHttp2XX(StatusCode)
74+
) {
75+
// process.stdout.write('.');
3476
continue;
3577
}
3678

37-
process.stdout.write(`Checking: ${url} (was ${StatusCode}) ... `);
38-
const verbose = false;
39-
const status = await getUrlStatus(url, verbose);
79+
if (
80+
(StatusCode === 404 &&
81+
// Handles special case of crates.io. For details, see:
82+
// https://github.com/rust-lang/crates.io/issues/788
83+
!url.startsWith(cratesIoURL)) ||
84+
(parsedUrl.hash && is4XXForFragments(StatusCode, lastSeenDate))
85+
) {
86+
console.log(
87+
`Skipping ${StatusCode}: ${url} (last seen ${lastSeenDate.toLocaleDateString()})${
88+
is4XXForFragments(StatusCode, lastSeenDate) ? ' INVALID FRAGMENT' : ''
89+
}`,
90+
);
91+
if (parsedUrl.hash) urlWithInvalidFragCount++;
92+
continue;
93+
}
94+
95+
if (url.startsWith(GOOGLE_DOCS_URL)) {
96+
// console.log(`Skipping Google Docs URL (for now): ${url}.`);
97+
// process.stdout.write('.');
98+
continue;
99+
/*
100+
URLs are of the form:
101+
https://docs.google.com/document/d/15vR7D1x2tKd7u3zaTF0yH1WaHkUr2T4hhr7OyiZgmBg/edit?tab=t.0#heading=h.4xuru5ljcups
102+
We can simply check for the presence of the heading query parameter value in the page.
103+
"ps_hdid":"h.4xuru5ljcups" # cSpell:disable-line
104+
*/
105+
}
106+
107+
if (maxNumEntriesToUpdate && updatedCount >= maxNumEntriesToUpdate) {
108+
console.log(`Updated max of ${maxNumEntriesToUpdate} entries, exiting.`);
109+
break;
110+
}
111+
112+
process.stdout.write(
113+
`Checking${
114+
parsedUrl.hash ? ` for fragment in` : `:`
115+
} ${url} (was ${StatusCode}) ... `,
116+
);
117+
118+
let status = await getUrlStatus(url);
40119
console.log(`${status}.`);
41120

42-
if (!isHttp2XX(status)) continue;
121+
let now = new Date();
122+
if (parsedUrl.hash) {
123+
if (isHttp2XX(status)) {
124+
// Encore that the fragment was checked and is valid.
125+
now.setSeconds(fragSecondsOk);
126+
now.setMilliseconds(fragMillisecondsOk);
127+
} else {
128+
status = StatusCode; // Keep the original status, rather than our custom 4XX status.
129+
now.setSeconds(fragSecondsInvalid);
130+
now.setMilliseconds(fragMillisecondsInvalid);
131+
urlWithInvalidFragCount++;
132+
}
133+
} else if (!isHttp2XX(status)) {
134+
continue;
135+
}
43136

44137
cache[url] = {
45138
StatusCode: status,
46-
LastSeen: new Date().toISOString(),
139+
LastSeen: now.toISOString(),
47140
};
48-
49-
updated = true;
141+
updatedCount++;
50142
}
51143

52-
if (updated) {
144+
if (updatedCount) {
53145
await writeRefcache(cache);
54146
} else {
55147
console.log(`No updates needed.`);
56148
}
149+
150+
console.log(
151+
`Processed ${entriesCount} URLs${
152+
checkForFragments
153+
? ` (${urlWithFragmentCount} with fragments, ${urlWithInvalidFragCount} are invalid)`
154+
: ''
155+
}`,
156+
);
157+
for (const [status, count] of Object.entries(statusCounts)) {
158+
console.log(`Status ${status}: ${count}`);
159+
}
57160
}
58161

162+
function countStatuses(StatusCode, parsedUrl, lastSeenDate, statusCounts) {
163+
let sc = StatusCode;
164+
if (checkForFragments) {
165+
sc += parsedUrl.hash
166+
? ' frag ' +
167+
(isHttp2XXForFragments(StatusCode, lastSeenDate) ? 'ok' : 'er')
168+
: ' no frag';
169+
}
170+
statusCounts[sc] = (statusCounts[sc] || 0) + 1;
171+
}
172+
173+
function getNumericFlagValue(flagName) {
174+
const flagArg = process.argv.find((arg) => arg.startsWith(flagName));
175+
if (!flagArg) return;
176+
177+
const valueArg = flagArg.includes('=')
178+
? flagArg.split('=')[1]
179+
: process.argv[process.argv.indexOf(flagName) + 1];
180+
let value = parseInt(valueArg);
181+
182+
if (value < 0) {
183+
console.error(
184+
`ERROR: invalid value for ${flagName}: ${valueArg}. ` +
185+
`Must be a number > 0. Using default ${maxNumEntriesToUpdate}.`,
186+
);
187+
exit(1);
188+
}
189+
return value;
190+
}
191+
192+
const _maxNumEntriesToUpdateFlag = getNumericFlagValue('--max-num-to-update');
193+
if (_maxNumEntriesToUpdateFlag >= 0)
194+
maxNumEntriesToUpdate = _maxNumEntriesToUpdateFlag;
195+
checkForFragments =
196+
process.argv.includes('--check-for-fragments') || process.argv.includes('-f');
197+
59198
await retry400sAndUpdateCache();

0 commit comments

Comments
 (0)