Skip to content

Commit 8ba79f5

Browse files
alex-clickhousealex-clickhouse
authored
add support for selecting protocol version in native format (#17)
* fix release workflow * fix release 2 * agents.md * docs * add support for selecting protocol version in native format * fix test failures
1 parent 0b71499 commit 8ba79f5

23 files changed

+1815
-390
lines changed

‎AGENTS.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# CLAUDE.md - ClickHouse Format Explorer
2+
3+
## Project Overview
4+
5+
A tool for visualizing ClickHouse RowBinary and Native wire format data. Features an interactive hex viewer with AST-based type visualization, similar to ImHex. Available as a web app (Docker) or an Electron desktop app that connects to your existing ClickHouse server.
6+
7+
**Current scope**: RowBinaryWithNamesAndTypes and Native formats.
8+
9+
## Tech Stack
10+
11+
- **Frontend**: React 18 + TypeScript + Vite
12+
- **State**: Zustand
13+
- **UI**: react-window (virtualized hex viewer), react-resizable-panels (split panes)
14+
- **Desktop**: Electron (optional, connects to user's ClickHouse)
15+
- **Testing**: Vitest + testcontainers (integration), Playwright (Electron e2e)
16+
- **Deployment**: Docker (bundles ClickHouse + nginx) or Electron desktop app
17+
18+
## Commands
19+
20+
```bash
21+
npm run dev # Start web dev server (requires ClickHouse at localhost:8123)
22+
npm run build # Build web app for production
23+
npm run test # Run integration tests (uses testcontainers)
24+
npm run lint # ESLint check
25+
npm run test:e2e # Build Electron + run Playwright e2e tests
26+
27+
# Electron desktop app
28+
npm run electron:dev # Dev mode with hot reload
29+
npm run electron:build # Package desktop installer for current platform
30+
31+
# Docker (self-contained with bundled ClickHouse)
32+
docker build -t rowbinary-explorer .
33+
docker run -d -p 8080:80 rowbinary-explorer
34+
```
35+
36+
## Directory Structure
37+
38+
```
39+
src/
40+
├── components/ # React components
41+
│ ├── App.tsx # Main layout with resizable panels
42+
│ ├── QueryInput.tsx # SQL query input + run button + connection settings
43+
│ ├── HexViewer/ # Virtualized hex viewer with highlighting
44+
│ └── AstTree/ # Collapsible AST tree view
45+
├── core/
46+
│ ├── types/
47+
│ │ ├── ast.ts # AstNode, ByteRange, ParsedData interfaces
48+
│ │ └── clickhouse-types.ts # ClickHouseType discriminated union
49+
│ ├── decoder/
50+
│ │ ├── rowbinary-decoder.ts # RowBinaryWithNamesAndTypes decoder
51+
│ │ ├── native-decoder.ts # Native format decoder
52+
│ │ ├── reader.ts # BinaryReader with byte-range tracking
53+
│ │ ├── leb128.ts # LEB128 varint decoder
54+
│ │ ├── test-helpers.ts # Shared test utilities
55+
│ │ ├── smoke-cases.ts # Smoke test case definitions
56+
│ │ └── validation-cases.ts # Validation test case definitions
57+
│ ├── parser/
58+
│ │ ├── type-lexer.ts # Tokenizer for type strings
59+
│ │ └── type-parser.ts # Parser: string -> ClickHouseType
60+
│ └── clickhouse/
61+
│ └── client.ts # HTTP client (fetch for web, IPC for Electron)
62+
├── store/
63+
│ └── store.ts # Zustand store (query, parsed data, UI state)
64+
└── styles/ # CSS files
65+
electron/
66+
├── main.ts # Electron main process (window, IPC handlers)
67+
└── preload.ts # Preload script (contextBridge → electronAPI)
68+
e2e/
69+
└── electron.spec.ts # Playwright Electron e2e tests
70+
docs/
71+
├── rowbinaryspec.md # RowBinary wire format specification
72+
├── nativespec.md # Native wire format specification
73+
└── jsonspec.md # JSON type specification
74+
docker/
75+
├── nginx.conf # Proxies /clickhouse to ClickHouse server
76+
├── users.xml # Read-only ClickHouse user
77+
└── supervisord.conf # Runs nginx + ClickHouse together
78+
```
79+
80+
## Wire Format Docs
81+
82+
* RowBinary: docs/rowbinaryspec.md
83+
* Native: docs/nativespec.md
84+
* JSON: docs/jsonspec.md
85+
86+
## Key Concepts
87+
88+
### AstNode
89+
Every decoded value is represented as an `AstNode` (`src/core/types/ast.ts:12`):
90+
- `id` - Unique identifier for selection/highlighting
91+
- `type` - ClickHouse type name string
92+
- `byteRange` - `{start, end}` byte offsets (exclusive end)
93+
- `value` - Decoded JavaScript value
94+
- `displayValue` - Human-readable string
95+
- `children` - Child nodes for composite types (Array, Tuple, etc.)
96+
97+
### ClickHouseType
98+
A discriminated union representing all ClickHouse types (`src/core/types/clickhouse-types.ts:4`):
99+
- Primitives: `UInt8`-`UInt256`, `Int8`-`Int256`, `Float32/64`, `String`, etc.
100+
- Composites: `Array`, `Tuple`, `Map`, `Nullable`, `LowCardinality`
101+
- Advanced: `Variant`, `Dynamic`, `JSON`
102+
- Geo: `Point`, `Ring`, `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`, `Geometry`
103+
- Intervals: `IntervalSecond`, `IntervalMinute`, `IntervalHour`, `IntervalDay`, `IntervalWeek`, `IntervalMonth`, `IntervalQuarter`, `IntervalYear` (stored as Int64)
104+
- Other: `Enum8/16`, `Nested`, `QBit`, `AggregateFunction`
105+
106+
### Decoding Flow
107+
1. User enters SQL query, clicks "Run Query"
108+
2. `ClickHouseClient` (`src/core/clickhouse/client.ts`) sends query:
109+
- **Web mode**: `fetch()` via Vite proxy or nginx
110+
- **Electron mode**: IPC to main process → `fetch()` to ClickHouse (no CORS)
111+
3. Decoder parses the binary response:
112+
- **RowBinary** (`rowbinary-decoder.ts`): Row-oriented, header + rows
113+
- **Native** (`native-decoder.ts`): Column-oriented with blocks
114+
4. Type strings parsed via `parseType()` into `ClickHouseType`
115+
5. Each decoded value returns an `AstNode` with byte tracking
116+
6. UI renders hex view (left) and AST tree (right)
117+
118+
### Electron Architecture
119+
```
120+
Renderer (React) Main Process (Node.js)
121+
│ │
122+
├─ window.electronAPI │
123+
│ .executeQuery(opts) ────────►├─ fetch(clickhouseUrl + query)
124+
│ │ → ArrayBuffer
125+
│◄── IPC response ──────────────┤
126+
│ │
127+
├─ Uint8Array → decoders │
128+
└─ render hex view + AST tree │
129+
```
130+
131+
- Runtime detection: `window.electronAPI` exists → IPC path, otherwise → `fetch()`
132+
- `vite-plugin-electron` activates only when `ELECTRON=true` env var is set
133+
- Connection config in `config.json` (project root in dev, next to executable in prod)
134+
- Experimental ClickHouse settings (Variant, Dynamic, JSON, etc.) sent as query params
135+
136+
### Interactive Highlighting
137+
- Click a node in AST tree → highlights corresponding bytes in hex view
138+
- Click a byte in hex view → selects the deepest AST node containing that byte
139+
- State managed in Zustand store: `activeNodeId`, `hoveredNodeId`
140+
141+
## Adding a New ClickHouse Type
142+
143+
1. Add type variant to `ClickHouseType` in `src/core/types/clickhouse-types.ts`
144+
2. Add `typeToString()` case for serialization back to string
145+
3. Add `getTypeColor()` case for UI coloring
146+
4. Add parser case in `src/core/parser/type-parser.ts`
147+
5. Add decoder method in `RowBinaryDecoder` (`src/core/decoder/rowbinary-decoder.ts`):
148+
- Add case in `decodeValue()` switch
149+
- Implement `decode{TypeName}()` method returning `AstNode`
150+
6. Add decoder method in `NativeDecoder` (`src/core/decoder/native-decoder.ts`):
151+
- Add case in `decodeValue()` switch
152+
- For columnar types, may need `decode{TypeName}Column()` method
153+
7. If type has binary type index (for Dynamic), add to `decodeDynamicType()`
154+
8. Add test cases to `smoke-cases.ts` and `validation-cases.ts`
155+
156+
## Important Implementation Details
157+
158+
- **LEB128**: Variable-length integers used for string lengths, array sizes, column counts
159+
- **UUID byte order**: ClickHouse uses a special byte ordering (see `decodeUUID()` at `decoder.ts:629`)
160+
- **IPv4**: Stored as little-endian UInt32, displayed in reverse order
161+
- **Dynamic type**: Uses BinaryTypeIndex encoding; type is encoded in the data itself
162+
- **LowCardinality**: Does not affect wire format in RowBinary (transparent wrapper)
163+
- **Nested**: Encoded as parallel arrays, one per field
164+
165+
## Testing
166+
167+
### Integration Tests (Vitest + testcontainers)
168+
169+
Tests use testcontainers to spin up a real ClickHouse instance:
170+
```bash
171+
npm run test # Runs all integration tests
172+
```
173+
174+
Tests are organized into three categories with shared test case definitions:
175+
176+
1. **Smoke Tests** (`smoke.integration.test.ts`)
177+
- Verify parsing succeeds without value validation
178+
- Test cases defined in `smoke-cases.ts`
179+
- Parametrized for both RowBinary and Native formats
180+
181+
2. **Validation Tests** (`validation.integration.test.ts`)
182+
- Verify decoded values and AST structure
183+
- Test cases defined in `validation-cases.ts` with format-specific callbacks
184+
- Check values, children counts, byte ranges, metadata
185+
186+
3. **Coverage Tests** (`coverage.integration.test.ts`)
187+
- Analyze byte coverage of AST leaf nodes
188+
- Report uncovered byte ranges
189+
190+
### Electron e2e Tests (Playwright)
191+
192+
```bash
193+
npm run test:e2e # Builds Electron app + runs Playwright tests
194+
```
195+
196+
Tests in `e2e/electron.spec.ts` launch the actual Electron app and verify:
197+
- App window opens and UI renders
198+
- Host input is visible (Electron mode) and Share button is hidden
199+
- Connection settings can be edited
200+
- Upload button is present and functional
201+
202+
### Test Case Interface
203+
```typescript
204+
interface ValidationTestCase {
205+
name: string;
206+
query: string;
207+
settings?: Record<string, string | number>;
208+
rowBinaryValidator?: (result: DecodedResult) => void;
209+
nativeValidator?: (result: DecodedResult) => void;
210+
}
211+
```
212+
213+
### Adding New Test Cases
214+
1. Add query to `smoke-cases.ts` for basic parsing verification
215+
2. Add to `validation-cases.ts` with validator callbacks for detailed checks
216+
3. Use `bothFormats(validator)` helper when validation logic is identical

0 commit comments

Comments
 (0)