feat(cloud): add ENGRAM_CLOUD_EXTRA_HEADERS support - #645
Conversation
Adds opt-in support for injecting arbitrary HTTP headers into every outgoing cloud sync request via ENGRAM_CLOUD_EXTRA_HEADERS env var. Comma-separated Key: Value pairs are parsed once at construction and stored on both RemoteTransport and MutationTransport. Authorization key is rejected at parse time to prevent token shadowing. Malformed pairs are skipped with a warning log. Header values are never logged. Enables compatibility with Cloudflare tunnels, reverse proxies, and access systems that require custom headers beyond Authorization.
📝 WalkthroughWalkthroughAdds ChangesCloud extra headers
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant SyncCaller
participant RemoteTransport
participant CloudEndpoint
SyncCaller->>RemoteTransport: Start cloud sync request
RemoteTransport->>RemoteTransport: Set bearer Authorization and apply extra headers
RemoteTransport->>CloudEndpoint: Send cloud request
CloudEndpoint-->>RemoteTransport: Return response
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR adds opt-in support for injecting additional HTTP headers into all outgoing cloud sync requests via ENGRAM_CLOUD_EXTRA_HEADERS, intended for environments like Cloudflare Access / zero-trust proxies, while preventing Authorization header shadowing.
Changes:
- Parse
ENGRAM_CLOUD_EXTRA_HEADERSonce duringRemoteTransport/MutationTransportconstruction and apply headers to all relevant requests. - Add unit tests covering header parsing edge cases and request header propagation.
- Document the new env var in CLI help text and
DOCS.mdautosync documentation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/cloud/remote/transport.go | Adds parsing + application of extra headers, with Authorization guardrail, and wires header application into transports. |
| internal/cloud/remote/transport_headers_test.go | Adds parsing-focused tests (including malformed input + Authorization rejection). |
| internal/cloud/remote/transport_extra_test.go | Adds integration-style tests verifying headers are present on outgoing HTTP requests from both transports. |
| cmd/engram/main.go | Updates CLI environment/help text to include ENGRAM_CLOUD_EXTRA_HEADERS. |
| DOCS.md | Documents the new env var format/rules for autosync deployments. |
Comments suppressed due to low confidence (1)
internal/cloud/remote/transport_headers_test.go:229
- The custom
contains/containsStrhelpers plus theos.Stderrsentinel add noise and an inaccurate comment about “other tests”. Usingstrings.Containsis simpler and removes the need for the unused import workaround.
// contains is a helper to avoid importing strings in the test name collision.
func contains(s, substr string) bool {
return len(s) >= len(substr) && (s == substr || len(substr) == 0 || containsStr(s, substr))
}
func containsStr(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
}
// Suppress unused import warning - os is used by other tests that may reference it.
var _ = os.Stderr
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| pair = strings.TrimSpace(pair) | ||
| idx := strings.IndexByte(pair, ':') | ||
| if idx < 0 { | ||
| log.Printf("[cloud] WARN ENGRAM_CLOUD_EXTRA_HEADERS: skipping malformed pair %q (missing colon)", pair) |
| // The log should reference the malformed pair "badpair" | ||
| if !contains(logOut, "badpair") { | ||
| t.Fatalf("expected log to mention 'badpair', got: %q", logOut) | ||
| } |
| import ( | ||
| "bytes" | ||
| "log" | ||
| "os" | ||
| "testing" | ||
| ) |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/cloud/remote/transport_headers_test.go`:
- Around line 61-64: In internal/cloud/remote/transport_headers_test.go at lines
61-64, add a mutex-protected safeBuffer implementing Write and String, importing
sync as needed; replace the local bytes.Buffer with safeBuffer at lines 61-64,
91-94, 105-108, 122-125, 139-142, and 186-189, preserving the existing global
log interception and assertions.
- Around line 213-228: Replace the custom contains and containsStr helpers in
the test file with the standard strings.Contains function, importing strings or
an alias if needed for any naming collision. Update all call sites accordingly,
then remove the unused os import and the var _ = os.Stderr suppression.
In `@internal/cloud/remote/transport.go`:
- Around line 37-60: Update the extra-header parsing loop in transport.go to
accumulate comma-separated segments without a colon into the current header
value instead of discarding them or logging malformed-pair warnings, while
preserving key canonicalization and Authorization protection. Add an edge-case
test in internal/cloud/remote/transport_headers_test.go covering a header value
containing commas and asserting the complete value is retained.
- Around line 77-84: Replace the call to sortStrings with sort.Strings, add the
standard library sort import, and remove the custom sortStrings insertion-sort
function. Preserve the existing in-place sorting behavior for the string slice.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 973ab9ca-b969-41e9-ba8b-2a8e825fbcc5
📒 Files selected for processing (5)
DOCS.mdcmd/engram/main.gointernal/cloud/remote/transport.gointernal/cloud/remote/transport_extra_test.gointernal/cloud/remote/transport_headers_test.go
| orig := log.Writer() | ||
| var buf bytes.Buffer | ||
| log.SetOutput(&buf) | ||
| defer log.SetOutput(orig) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Data race hazard: bytes.Buffer is not thread-safe for global log interception.
These tests intercept the global log output using log.SetOutput(&buf) with a standard bytes.Buffer. If any background goroutine (such as those spun up by httptest.NewServer in other tests) or a concurrent test logs a message, it will race on the unprotected bytes.Buffer.Write, causing the test suite to panic.
internal/cloud/remote/transport_headers_test.go#L61-L64: Wrap thebytes.Bufferin a thread-safe struct with async.Mutexto safely intercept the global logger.internal/cloud/remote/transport_headers_test.go#L91-L94: Use the thread-safe buffer here.internal/cloud/remote/transport_headers_test.go#L105-L108: Use the thread-safe buffer here.internal/cloud/remote/transport_headers_test.go#L122-L125: Use the thread-safe buffer here.internal/cloud/remote/transport_headers_test.go#L139-L142: Use the thread-safe buffer here.internal/cloud/remote/transport_headers_test.go#L186-L189: Use the thread-safe buffer here.
🛠️ Proposed thread-safe buffer helper
Add this helper to transport_headers_test.go and replace var buf bytes.Buffer with var buf safeBuffer in the affected tests:
type safeBuffer struct {
mu sync.Mutex
b bytes.Buffer
}
func (s *safeBuffer) Write(p []byte) (n int, err error) {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.Write(p)
}
func (s *safeBuffer) String() string {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.String()
}📍 Affects 1 file
internal/cloud/remote/transport_headers_test.go#L61-L64(this comment)internal/cloud/remote/transport_headers_test.go#L91-L94internal/cloud/remote/transport_headers_test.go#L105-L108internal/cloud/remote/transport_headers_test.go#L122-L125internal/cloud/remote/transport_headers_test.go#L139-L142internal/cloud/remote/transport_headers_test.go#L186-L189
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/cloud/remote/transport_headers_test.go` around lines 61 - 64, In
internal/cloud/remote/transport_headers_test.go at lines 61-64, add a
mutex-protected safeBuffer implementing Write and String, importing sync as
needed; replace the local bytes.Buffer with safeBuffer at lines 61-64, 91-94,
105-108, 122-125, 139-142, and 186-189, preserving the existing global log
interception and assertions.
| // contains is a helper to avoid importing strings in the test name collision. | ||
| func contains(s, substr string) bool { | ||
| return len(s) >= len(substr) && (s == substr || len(substr) == 0 || containsStr(s, substr)) | ||
| } | ||
|
|
||
| func containsStr(s, substr string) bool { | ||
| for i := 0; i <= len(s)-len(substr); i++ { | ||
| if s[i:i+len(substr)] == substr { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } | ||
|
|
||
| // Suppress unused import warning - os is used by other tests that may reference it. | ||
| var _ = os.Stderr |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Remove custom contains helpers and unused import suppressions.
Reinventing strings.Contains to avoid an import is an anti-pattern. If there were an actual naming collision with a local strings variable, the import could be aliased (e.g., import stdstrings "strings"). Additionally, bypassing the Go compiler's unused import checks with a dummy variable (var _ = os.Stderr) obfuscates dependencies.
Please import "strings", use strings.Contains throughout the test file, and remove the unused os import entirely.
♻️ Proposed refactor
-// contains is a helper to avoid importing strings in the test name collision.
-func contains(s, substr string) bool {
- return len(s) >= len(substr) && (s == substr || len(substr) == 0 || containsStr(s, substr))
-}
-
-func containsStr(s, substr string) bool {
- for i := 0; i <= len(s)-len(substr); i++ {
- if s[i:i+len(substr)] == substr {
- return true
- }
- }
- return false
-}
-
-// Suppress unused import warning - os is used by other tests that may reference it.
-var _ = os.Stderr🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/cloud/remote/transport_headers_test.go` around lines 213 - 228,
Replace the custom contains and containsStr helpers in the test file with the
standard strings.Contains function, importing strings or an alias if needed for
any naming collision. Update all call sites accordingly, then remove the unused
os import and the var _ = os.Stderr suppression.
| // sortStrings sorts a slice of strings in-place (insertion sort — small N, avoids importing sort). | ||
| func sortStrings(ss []string) { | ||
| for i := 1; i < len(ss); i++ { | ||
| for j := i; j > 0 && ss[j] < ss[j-1]; j-- { | ||
| ss[j], ss[j-1] = ss[j-1], ss[j] | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Prefer standard library sorting.
The net/http package already transitively imports sort (and slices), so avoiding the standard sort package provides no binary size benefit. Please use sort.Strings(keys) or slices.Sort(keys) (Go 1.21+) instead of maintaining a custom sorting implementation.
♻️ Proposed refactor
Replace sortStrings(keys) on line 71 with sort.Strings(keys), remove the sortStrings function entirely, and ensure sort is imported.
-// sortStrings sorts a slice of strings in-place (insertion sort — small N, avoids importing sort).
-func sortStrings(ss []string) {
- for i := 1; i < len(ss); i++ {
- for j := i; j > 0 && ss[j] < ss[j-1]; j-- {
- ss[j], ss[j-1] = ss[j-1], ss[j]
- }
- }
-}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/cloud/remote/transport.go` around lines 77 - 84, Replace the call to
sortStrings with sort.Strings, add the standard library sort import, and remove
the custom sortStrings insertion-sort function. Preserve the existing in-place
sorting behavior for the string slice.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
internal/cloud/remote/transport.go:52
- Malformed segments (missing
:) are currently appended to the previous header value whencurrentKeyis set. This contradicts the documented/expected “skip malformed pairs” behavior and can also cause accidental logging of secrets if a user forgets a colon. Consider always treating missing-colon segments as malformed: log a generic warning (no segment content) and skip without mutating any prior header value.
if idx < 0 {
if currentKey != "" {
result[currentKey] += ", " + part
} else {
log.Printf("[cloud] WARN ENGRAM_CLOUD_EXTRA_HEADERS: skipping malformed segment %q (missing colon)", part)
internal/cloud/remote/transport_headers_test.go:86
- This test currently expects malformed segments to be echoed back in logs (e.g.
badpair). If logging is adjusted to avoid leaking potentially-sensitive raw segments/values, update the assertion to look for a generic warning message instead of the raw segment contents.
// The log should reference the malformed pair "badpair"
if !contains(logOut, "badpair") {
t.Fatalf("expected log to mention 'badpair', got: %q", logOut)
}
internal/cloud/remote/transport_headers_test.go:7
osis imported only to satisfyvar _ = os.Stderrat the bottom of the file, but imports are file-scoped in Go, so this doesn’t help “other tests”. This adds noise and can confuse future maintenance; please drop theosimport and the dummy reference.
import (
"bytes"
"log"
"os"
"testing"
internal/cloud/remote/transport_headers_test.go:228
- The dummy
os.Stderrreference (and its comment) should be removed once theosimport is dropped; keeping it would break compilation.
// Suppress unused import warning - os is used by other tests that may reference it.
var _ = os.Stderr
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/cloud/remote/transport.go`:
- Around line 37-54: Update the extra-header parsing loop around currentKey so
colonless segments are always treated as malformed: skip them and emit the
existing warning instead of appending them to the previous header value.
Preserve parsing of subsequent valid pairs and ensure
TestParseExtraHeaders_MalformedPairSkipped still yields X-A as "1"; document
that unescaped commas cannot appear in values unless an explicit quoting or
escaping rule is implemented.
- Around line 57-71: Update the extra-header parsing flow around currentKey and
result to validate both trimmed header names and values with net/http’s header
validation before storing them for Header.Set. Skip malformed entries, including
names or values containing control characters, spaces, or invalid bytes, and
emit only a key-focused warning without logging values; preserve the existing
Authorization override protection.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3a1b830f-dae1-4ff5-8b3b-0b589a16f4b8
📒 Files selected for processing (1)
internal/cloud/remote/transport.go
| parts := strings.Split(raw, ",") | ||
| result := make(map[string]string) | ||
| var currentKey string | ||
|
|
||
| for _, part := range parts { | ||
| part = strings.TrimSpace(part) | ||
| if part == "" { | ||
| continue | ||
| } | ||
|
|
||
| idx := strings.IndexByte(part, ':') | ||
| if idx < 0 { | ||
| if currentKey != "" { | ||
| result[currentKey] += ", " + part | ||
| } else { | ||
| log.Printf("[cloud] WARN ENGRAM_CLOUD_EXTRA_HEADERS: skipping malformed segment %q (missing colon)", part) | ||
| } | ||
| continue |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
This continuation heuristic breaks the malformed-pair contract.
For X-A: 1, badpair, X-B: 2, currentKey remains X-A, so badpair is appended to the value and no warning is emitted. The supplied TestParseExtraHeaders_MalformedPairSkipped expects X-A == "1" plus a warning, so this change makes the existing test fail. Define an explicit quoting/escaping rule for comma-containing values, or preserve skip-and-warn behavior and document the limitation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/cloud/remote/transport.go` around lines 37 - 54, Update the
extra-header parsing loop around currentKey so colonless segments are always
treated as malformed: skip them and emit the existing warning instead of
appending them to the previous header value. Preserve parsing of subsequent
valid pairs and ensure TestParseExtraHeaders_MalformedPairSkipped still yields
X-A as "1"; document that unescaped commas cannot appear in values unless an
explicit quoting or escaping rule is implemented.
| key := strings.TrimSpace(part[:idx]) | ||
| if key == "" { | ||
| currentKey = "" | ||
| continue | ||
| } | ||
|
|
||
| canonical := http.CanonicalHeaderKey(key) | ||
| if canonical == "Authorization" { | ||
| log.Printf("[cloud] WARN ENGRAM_CLOUD_EXTRA_HEADERS: refusing to override Authorization header") | ||
| currentKey = "" // Prevent appending trailing segments to a rejected header | ||
| continue | ||
| } | ||
|
|
||
| currentKey = canonical | ||
| result[currentKey] = strings.TrimSpace(part[idx+1:]) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate transport.go"
fd -a 'transport.go$' . | sed 's#^\./##'
echo
echo "Show internal/cloud/remote/transport.go around parser and callers"
sed -n '1,130p' internal/cloud/remote/transport.go
echo
echo "Search Env var usage and parsing"
rg -n "ENGRAM_CLOUD_EXTRA_HEADERS|ExtraHeaders|extra.headers|extra_headers|extra headers" internal/cloud/remote . -g '!vendor' | head -200
echo
echo "Go parser details for Header Values / Header validation across version if available"
go version 2>/dev/null || true
if command -v go >/dev/null 2>&1; then
python3 - <<'PY'
import os, subprocess, tempfile
src = '''
package main
import (
"net/http"
"reflect"
"regexp"
"strings"
"fmt"
)
func main() {
cases := []struct{ name string }{
{""}, {"A B"}, {"Bad\tHeader"}, {"Bad\rHeader"},
{"Authorization Bad"}, {"Authorization", "A B C"},
}
for _, c := range cases {
fmt.Printf("name:%q canonical:%q\n", c.name, http.CanonicalHeaderKey(c.name))
}
for _, name := range []string{"A B", "Bad\tHeader", "Bad\rHeader", "Authorization B"} {
req, err := http.NewRequest("GET", "http://example.test", nil)
req.Header[model.String] = map[string][]string{}
req.Header[model.String][name] = []string{"value"}
fmt.Printf("header %q err=%v req=%v\n", name, err != nil, req)
}
h := make(http.Header)
h["Value\tBad"] = []string{"v"}
h["Val\rue"] = []string{"v"}
fmt.Printf("Header: %q\n", h.Get("Value\tBad"))
}
'''
# This won't run if dependencies missing; just avoid.
PY
fi
echo
echo "Parse behavior probe using repository copy as data? No runtime - use source references"
rg -n "http.CanonicalHeaderKey|addHeader|Header\." internal/cloud/remote/transport.goRepository: Gentleman-Programming/engram
Length of output: 21269
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Behavioral probe with the Go stdlib if available, but keep it read-only and avoid running repository code.
if command -v go >/dev/null 2>&1; then
tmp="$(mktemp -d)"
cat > "$tmp/probe.go" <<'EOF'
package main
import (
"fmt"
"net/http"
)
func main() {
cases := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range cases {
fmt.Printf("name=%q canonical=%q\n", name, http.CanonicalHeaderKey(name))
}
var values = []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue"}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
if err != nil {
panic(err)
}
req.Header.Add("X-Test", v)
b, _ := req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
fmt.Printf("value=%q\n", v)
fmt.Printf("bytes contain tab=%v cr=%v\n", strings.Contains(string(b), "\t"), strings.Contains(string(b), "\r"))
}
}
EOF
(cd "$tmp" && go run .)
rm -rf "$tmp"
fiRepository: Gentleman-Programming/engram
Length of output: 260
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
cat > "$tmp/module/probe.go" <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
fmt.Printf("name=%q canonical=%q empty=%v\n", name, http.CanonicalHeaderKey(name), name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
if err != nil {
fmt.Printf("request_new_value=%q err=%v\n", v, err)
continue
}
req.Header.Set("X-Test", v)
b, _ := req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
wire := string(b)
fmt.Printf("value=%q reqerr=%v wire_contains=%v\n", v, (err != nil), strings.Contains(wire, "\t"))
}
}
EOF
go run "$tmp/module/probe.go"
rm -rf "$tmp"Repository: Gentleman-Programming/engram
Length of output: 243
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d || exit 1)"
mkdir -p "$tmp/module"
cat > "$tmp/module/probe.go" <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
fmt.Printf("name=%q canonical=%q empty=%v\n", name, http.CanonicalHeaderKey(name), name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
if err != nil {
fmt.Printf("request_new_value=%q err=%v\n", v, err)
continue
}
req.Header.Set("X-Test", v)
b, err := req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
wire := string(b)
marshaledErr := ""
if err != nil {
marshaledErr = err.Error()
}
fmt.Printf("value=%q reqerr=%v writeerr=%s wire=%q contains_tab=%v contains_cr=%v\n",
v, err != nil, marshaledErr, wire, strings.Contains(wire, "\t"), strings.Contains(wire, "\r"))
}
}
EOF
cd "$tmp/module"
go run .Repository: Gentleman-Programming/engram
Length of output: 260
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d || exit 1)"
mkdir -p "$tmp/module"
cd "$tmp/module" || exit 1
cat > go.mod <<'EOF'
module probe
go 1.26
EOF
cat > probe.go <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
fmt.Printf("name=%q canonical=%q empty=%v\n", name, http.CanonicalHeaderKey(name), name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
if err != nil {
fmt.Printf("request_new_value=%q err=%v\n", v, err)
continue
}
req.Header.Set("X-Test", v)
b, err := req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
wire := string(b)
marshaledErr := ""
if err != nil {
marshaledErr = err.Error()
}
fmt.Printf("value=%q reqerr=%v writeerr=%s wire=%q contains_tab=%v contains_cr=%v\n",
v, err != nil, marshaledErr, wire, strings.Contains(wire, "\t"), strings.Contains(wire, "\r"))
}
}
EOF
go run .Repository: Gentleman-Programming/engram
Length of output: 268
🌐 Web query:
Go net/http Header WriteSubset request invalid header value tab NewRequest documentation
💡 Result:
In Go's net/http package, HTTP header values are strictly validated to ensure compliance with RFC standards [1][2]. While header values generally allow visible ASCII characters (0x21-0x7e) as well as horizontal tabs (HTAB, 0x09) and spaces (SP, 0x20), they must not contain control characters such as newlines (LF, 0x0a), carriage returns (CR, 0x0d), or null bytes (NUL, 0x00) [3][1]. If you attempt to set an HTTP header with an invalid character—such as a tab, which might be restricted depending on the specific validation context or Go version—the net/http library will reject it [2][4]. Specifically, methods like http.NewRequest and Header.Add/Set validate inputs; providing an invalid value will typically result in an error such as: net/http: invalid header field value [2][4]. Regarding Header.WriteSubset: 1. Purpose: This method is used to write header keys and values to an io.Writer in wire format, optionally excluding a subset of headers [5][6]. It is commonly used by internal tools like httputil.DumpRequest or proxies to filter out hop-by-hop headers [6]. 2. Validation: Because WriteSubset relies on the underlying header map, if you have bypassed initial validation (e.g., by manually manipulating the map) to insert invalid characters, the writing process may fail or produce malformed HTTP output [3][1]. 3. Documentation: Official documentation for Header.WriteSubset is available on the Go packages site [5][7]. If you find that your use case requires sending headers that are otherwise considered invalid by the standard library (such as including specific control characters or unconventional formatting), you cannot do so via the standard net/http request construction methods [4]. The common workaround is to manually construct the raw HTTP request byte stream and send it over a low-level net.Conn, bypassing the high-level net/http request validation logic entirely [4].
Citations:
- 1: https://github.com/golang/net/blob/master/http/httpguts/httplex.go
- 2: net/http: client should not include raw header values in error messages golang/go#43631
- 3: net/http: allows invalid characters in header values golang/go#11207
- 4: https://stackoverflow.com/questions/70678016/how-to-bypass-golangs-http-request-net-http-rfc-compliance
- 5: https://pkg.go.dev/net/http
- 6: https://stackoverflow.com/questions/48976837/what-is-an-idiomatic-usage-of-header-write-writesubset
- 7: https://tip.golang.org/pkg/http
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d || exit 1)"
mkdir -p "$tmp/module"
cd "$tmp/module" || exit 1
cat > go.mod <<'EOF'
module probe
go 1.26
EOF
cat > probe.go <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
fmt.Printf("name=%q canonical=%q empty=%v\n", name, http.CanonicalHeaderKey(name), name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
if err != nil {
fmt.Printf("request_new_value=%q err=%v\n", v, err)
continue
}
req.Header.Set("X-Test", v)
b := req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
wire := string(b)
fmt.Printf("value=%q reqerr=%v writeerr=%v wire=%q contains_tab=%v contains_cr=%v\n",
v, err != nil, (err != nil), wire, strings.Contains(wire, "\t"), strings.Contains(wire, "\r"))
}
}
EOF
go run .Repository: Gentleman-Programming/engram
Length of output: 259
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d || exit 1)"
mkdir -p "$tmp/module"
cd "$tmp/module" || exit 1
cat > go.mod <<'EOF'
module probe
go 1.26
EOF
cat > probe.go <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
fmt.Printf("name=%q canonical=%q empty=%v\n", name, http.CanonicalHeaderKey(name), name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
fmt.Printf("request_new_value=%q reqerr=%v\n", v, (err != nil))
if err != nil {
continue
}
req.Header.Set("X-Test", v)
err = req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
fmt.Printf("value=%q writeerr=%v\n", v, err != nil)
}
}
EOF
go run .Repository: Gentleman-Programming/engram
Length of output: 222
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d || exit 1)"
mkdir -p "$tmp/module"
cd "$tmp/module" || exit 1
cat > go.mod <<'EOF'
module probe
go 1.26
EOF
cat > probe.go <<'EOF'
package main
import (
"fmt"
"net/http"
"strings"
)
func main() {
names := []string{"", "A B", "Bad\tHeader", "Bad\rHeader"}
for _, name := range names {
canonical := http.CanonicalHeaderKey(name)
fmt.Printf("name=%q canonical=%q empty=%v\n", name, canonical, name == "")
}
values := []string{"ok", "bad\tvalue", "bad\rv alue", "bad\t\rvalue", "bad \"quote\""}
for _, v := range values {
req, err := http.NewRequest("GET", "http://example.test", nil)
fmt.Printf("request_new_value=%q reqerr=%v\n", v, (err != nil))
if err != nil {
continue
}
req.Header.Set("X-Test", v)
err = req.Header.WriteSubset(nil, map[string]bool{"Connection": true})
wire := strings.TrimSpace(fmt.Sprint(req.Header))
fmt.Printf("value=%q writeerr=%v wire_contains_tab=%v wire_has_cr=%v\n",
v, (err != nil), strings.Contains(wire, "\t"), strings.Contains(wire, "\r"))
}
}
EOF
go run .Repository: Gentleman-Programming/engram
Length of output: 904
Validate extra headers before applying them.
TrimSpace does not catch tab/carriage-return control characters in header names or values, and http.CanonicalHeaderKey leaves spaces or invalid bytes unchanged. Malformed ENGMT_CLOUD_EXTRA_HEADERS entries can make net/http fail to send the request; validate both parts before Header.Set, skip malformed entries with a key-only warning, and do not log values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/cloud/remote/transport.go` around lines 57 - 71, Update the
extra-header parsing flow around currentKey and result to validate both trimmed
header names and values with net/http’s header validation before storing them
for Header.Set. Skip malformed entries, including names or values containing
control characters, spaces, or invalid bytes, and emit only a key-focused
warning without logging values; preserve the existing Authorization override
protection.
🔗 Linked Issue
Closes #355
🏷️ PR Type
type:feature— New feature📝 Summary
ENGRAM_CLOUD_EXTRA_HEADERSenv var to inject custom HTTP headers into every outgoing cloud sync request (Cloudflare Access, reverse proxies, zero-trust gateways)Authorizationis rejected at parse time regardless of casing, preventing bearer-token shadowing; malformed pairs are skipped with a warning, values are never loggedRemoteTransportandMutationTransport; zero per-request overhead when unset📂 Changes
internal/cloud/remote/transport.goparseExtraHeaders/applyExtraHeaders,extraHeadersfield on both transport structs, wired intoReadManifest,WriteChunk,ReadChunk,PullMutationsinternal/cloud/remote/transport_extra_test.gointernal/cloud/remote/transport_headers_test.gocmd/engram/main.goDOCS.mdENGRAM_CLOUD_EXTRA_HEADERSin the autosync env var table🧪 Test Plan
go test ./...go test -tags e2e ./internal/server/...Design was reviewed and approved by @Alan-TheGentleman on the issue thread, including the
strings.TrimSpacecondition on both key and value after splitting (closes a leading-space bypass of theAuthorizationguardrail) — confirmed present inparseExtraHeaders.✅ Contributor Checklist
Closes #355)type:*label to this PRgo test ./...go test -tags e2e ./internal/server/...Co-Authored-Bytrailers in commits💬 Notes for Reviewers
Rebased onto latest
mainbefore opening (fork'smainwas ~65 commits behind upstream); resolved one trivial table-formatting conflict inDOCS.md.Summary by CodeRabbit
New Features
ENGRAM_CLOUD_EXTRA_HEADERS(comma-separatedKey: Valuepairs).Authorizationheader; canonicalizes header names and uses last-value-wins for duplicates.Documentation
Tests