Skip to content

Commit bfce99d

Browse files
Refactor tests into parametrized 2x2 structure and add Interval types
Test refactoring: - Consolidate 4 test files (~4600 lines) into parametrized structure - smoke-cases.ts: 159 query-only test cases - validation-cases.ts: 74 test cases with format-specific validators - coverage.integration.test.ts: byte coverage analysis for AST nodes - test-helpers.ts: shared utilities (TestContext, DecodedResult wrapper) - Delete old: decoder, native-decoder, dynamic-exhaustive, qa-edge-cases tests New features: - Add all 11 Interval types (IntervalSecond through IntervalYear) - Support in both RowBinary and Native decoders - Intervals stored as Int64, displayed with unit (e.g., "45 seconds") 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent a212ab8 commit bfce99d

15 files changed

+5903
-8596
lines changed

CLAUDE.md

Lines changed: 56 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
A web-based tool for visualizing ClickHouse RowBinary wire format data. Features an interactive hex viewer with AST-based type visualization, similar to ImHex. The tool queries a local ClickHouse database and presents the raw binary data alongside a decoded AST tree with bidirectional highlighting.
66

7-
**Current scope**: RowBinaryWithNamesAndTypes format only, with plans for expansion to other ClickHouse wire formats.
7+
**Current scope**: RowBinaryWithNamesAndTypes and Native formats.
88

99
## Tech Stack
1010

@@ -41,9 +41,13 @@ src/
4141
│ │ ├── ast.ts # AstNode, ByteRange, ParsedData interfaces
4242
│ │ └── clickhouse-types.ts # ClickHouseType discriminated union
4343
│ ├── decoder/
44-
│ │ ├── decoder.ts # Main RowBinaryDecoder - decodes all types
45-
│ │ ├── reader.ts # BinaryReader with byte-range tracking
46-
│ │ └── leb128.ts # LEB128 varint decoder
44+
│ │ ├── rowbinary-decoder.ts # RowBinaryWithNamesAndTypes decoder
45+
│ │ ├── native-decoder.ts # Native format decoder
46+
│ │ ├── reader.ts # BinaryReader with byte-range tracking
47+
│ │ ├── leb128.ts # LEB128 varint decoder
48+
│ │ ├── test-helpers.ts # Shared test utilities
49+
│ │ ├── smoke-cases.ts # Smoke test case definitions
50+
│ │ └── validation-cases.ts # Validation test case definitions
4751
│ ├── parser/
4852
│ │ ├── type-lexer.ts # Tokenizer for type strings
4953
│ │ └── type-parser.ts # Parser: string -> ClickHouseType
@@ -80,17 +84,18 @@ A discriminated union representing all ClickHouse types (`src/core/types/clickho
8084
- Composites: `Array`, `Tuple`, `Map`, `Nullable`, `LowCardinality`
8185
- Advanced: `Variant`, `Dynamic`, `JSON`
8286
- Geo: `Point`, `Ring`, `Polygon`, `MultiPolygon`, `LineString`, `MultiLineString`, `Geometry`
83-
- Other: `Enum8/16`, `Nested`, `QBit`
87+
- Intervals: `IntervalSecond`, `IntervalMinute`, `IntervalHour`, `IntervalDay`, `IntervalWeek`, `IntervalMonth`, `IntervalQuarter`, `IntervalYear` (stored as Int64)
88+
- Other: `Enum8/16`, `Nested`, `QBit`, `AggregateFunction`
8489

8590
### Decoding Flow
8691
1. User enters SQL query, clicks "Run Query"
87-
2. `ClickHouseClient` (`src/core/clickhouse/client.ts`) POSTs query with `default_format=RowBinaryWithNamesAndTypes`
88-
3. `RowBinaryDecoder` (`src/core/decoder/decoder.ts:10`) decodes:
89-
- Header: column count (LEB128), column names, column types
90-
- Type strings parsed via `parseType()` into `ClickHouseType`
91-
- Rows: for each row, decode each column value based on its type
92-
4. Each decoded value returns an `AstNode` with byte tracking
93-
5. UI renders hex view (left) and AST tree (right)
92+
2. `ClickHouseClient` (`src/core/clickhouse/client.ts`) POSTs query with selected format
93+
3. Decoder parses the binary response:
94+
- **RowBinary** (`rowbinary-decoder.ts`): Row-oriented, header + rows
95+
- **Native** (`native-decoder.ts`): Column-oriented with blocks
96+
4. Type strings parsed via `parseType()` into `ClickHouseType`
97+
5. Each decoded value returns an `AstNode` with byte tracking
98+
6. UI renders hex view (left) and AST tree (right)
9499

95100
### Interactive Highlighting
96101
- Click a node in AST tree → highlights corresponding bytes in hex view
@@ -103,10 +108,14 @@ A discriminated union representing all ClickHouse types (`src/core/types/clickho
103108
2. Add `typeToString()` case for serialization back to string
104109
3. Add `getTypeColor()` case for UI coloring
105110
4. Add parser case in `src/core/parser/type-parser.ts`
106-
5. Add decoder method in `RowBinaryDecoder` (`src/core/decoder/decoder.ts`):
111+
5. Add decoder method in `RowBinaryDecoder` (`src/core/decoder/rowbinary-decoder.ts`):
107112
- Add case in `decodeValue()` switch
108113
- Implement `decode{TypeName}()` method returning `AstNode`
109-
6. If type has binary type index (for Dynamic), add to `decodeDynamicType()`
114+
6. Add decoder method in `NativeDecoder` (`src/core/decoder/native-decoder.ts`):
115+
- Add case in `decodeValue()` switch
116+
- For columnar types, may need `decode{TypeName}Column()` method
117+
7. If type has binary type index (for Dynamic), add to `decodeDynamicType()`
118+
8. Add test cases to `smoke-cases.ts` and `validation-cases.ts`
110119

111120
## Important Implementation Details
112121

@@ -121,7 +130,38 @@ A discriminated union representing all ClickHouse types (`src/core/types/clickho
121130

122131
Integration tests use testcontainers to spin up a real ClickHouse instance:
123132
```bash
124-
npm run test # Runs src/core/decoder/decoder.integration.test.ts
133+
npm run test # Runs all integration tests
134+
```
135+
136+
### Test Structure
137+
Tests are organized into three categories with shared test case definitions:
138+
139+
1. **Smoke Tests** (`smoke.integration.test.ts`)
140+
- Verify parsing succeeds without value validation
141+
- Test cases defined in `smoke-cases.ts`
142+
- Parametrized for both RowBinary and Native formats
143+
144+
2. **Validation Tests** (`validation.integration.test.ts`)
145+
- Verify decoded values and AST structure
146+
- Test cases defined in `validation-cases.ts` with format-specific callbacks
147+
- Check values, children counts, byte ranges, metadata
148+
149+
3. **Coverage Tests** (`coverage.integration.test.ts`)
150+
- Analyze byte coverage of AST leaf nodes
151+
- Report uncovered byte ranges
152+
153+
### Test Case Interface
154+
```typescript
155+
interface ValidationTestCase {
156+
name: string;
157+
query: string;
158+
settings?: Record<string, string | number>;
159+
rowBinaryValidator?: (result: DecodedResult) => void;
160+
nativeValidator?: (result: DecodedResult) => void;
161+
}
125162
```
126163

127-
Tests verify decoding of various type combinations against actual ClickHouse output.
164+
### Adding New Test Cases
165+
1. Add query to `smoke-cases.ts` for basic parsing verification
166+
2. Add to `validation-cases.ts` with validator callbacks for detailed checks
167+
3. Use `bothFormats(validator)` helper when validation logic is identical
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
2+
import {
3+
TestContext,
4+
decodeRowBinary,
5+
decodeNative,
6+
analyzeByteRange,
7+
formatUncoveredRanges,
8+
} from './test-helpers';
9+
import { SMOKE_TEST_CASES } from './smoke-cases';
10+
11+
/**
12+
* Byte coverage tests - verify that the AST leaf nodes cover all bytes in the data
13+
*
14+
* These tests ensure that every byte in the decoded data is accounted for by
15+
* at least one leaf node in the AST tree.
16+
*/
17+
describe('Byte Coverage Tests', () => {
18+
const ctx = new TestContext();
19+
20+
beforeAll(async () => {
21+
await ctx.start();
22+
}, 120000);
23+
24+
afterAll(async () => {
25+
await ctx.stop();
26+
});
27+
28+
// Test a representative subset of cases for coverage
29+
const coverageTestCases = SMOKE_TEST_CASES.filter(c =>
30+
// Focus on diverse type categories
31+
c.name.includes('UInt8') ||
32+
c.name.includes('String basic') ||
33+
c.name.includes('Array integers') ||
34+
c.name.includes('Tuple simple') ||
35+
c.name.includes('Map with entries') ||
36+
c.name.includes('Nullable non-null') ||
37+
c.name.includes('Multiple columns') ||
38+
c.name.includes('IntervalSecond')
39+
);
40+
41+
describe('RowBinary Format', () => {
42+
it.each(coverageTestCases)(
43+
'$name - byte coverage',
44+
async ({ query, settings, skipRowBinary }) => {
45+
if (skipRowBinary) return;
46+
47+
const data = await ctx.queryRowBinary(query, settings);
48+
const parsed = decodeRowBinary(data);
49+
const coverage = analyzeByteRange(parsed, data.length);
50+
51+
if (!coverage.isComplete) {
52+
const details = formatUncoveredRanges(coverage, data);
53+
console.log(`[RowBinary] ${query}\n${details}`);
54+
}
55+
56+
// Allow some header bytes to be uncovered (column count LEB128)
57+
// The header structure doesn't have leaf nodes for everything
58+
expect(coverage.coveragePercent).toBeGreaterThan(80);
59+
},
60+
);
61+
});
62+
63+
describe('Native Format', () => {
64+
it.each(coverageTestCases)(
65+
'$name - byte coverage',
66+
async ({ query, settings, skipNative }) => {
67+
if (skipNative) return;
68+
69+
const data = await ctx.queryNative(query, settings);
70+
const parsed = decodeNative(data);
71+
const coverage = analyzeByteRange(parsed, data.length);
72+
73+
if (!coverage.isComplete) {
74+
const details = formatUncoveredRanges(coverage, data);
75+
console.log(`[Native] ${query}\n${details}`);
76+
}
77+
78+
// Native format has block headers that may not be fully covered
79+
expect(coverage.coveragePercent).toBeGreaterThan(70);
80+
},
81+
);
82+
});
83+
84+
describe('Full Coverage Sanity Checks', () => {
85+
it('simple UInt8 value has reasonable coverage (RowBinary)', async () => {
86+
const data = await ctx.queryRowBinary('SELECT 42::UInt8 as val');
87+
const parsed = decodeRowBinary(data);
88+
const coverage = analyzeByteRange(parsed, data.length);
89+
90+
// Should cover most of the data
91+
expect(coverage.coveragePercent).toBeGreaterThan(50);
92+
93+
// Log uncovered if any
94+
if (!coverage.isComplete) {
95+
console.log('Uncovered ranges:', coverage.uncoveredRanges);
96+
}
97+
});
98+
99+
it('simple UInt8 value has reasonable coverage (Native)', async () => {
100+
const data = await ctx.queryNative('SELECT 42::UInt8 as val');
101+
const parsed = decodeNative(data);
102+
const coverage = analyzeByteRange(parsed, data.length);
103+
104+
// Should cover most of the data
105+
expect(coverage.coveragePercent).toBeGreaterThan(50);
106+
107+
// Log uncovered if any
108+
if (!coverage.isComplete) {
109+
console.log('Uncovered ranges:', coverage.uncoveredRanges);
110+
}
111+
});
112+
});
113+
}, 300000);

0 commit comments

Comments
 (0)