Skip to content

Commit be51f87

Browse files
pcarletonclaudeochafik
authored
Refactor client testing to use 'everything' client with scenario routing (#54)
* Refactor client testing to pass scenario name and server URL The CLI now passes both scenario name and server URL as arguments to the client command (instead of just the server URL). This enables a single "everything" client that can handle all scenarios based on the scenario name it receives. Changes: - Update executeClient() to pass scenarioName as first argument to client - Add everything-client.ts that routes to appropriate behavior based on scenario name - Update README with new usage examples and documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Use env var for scenario name instead of positional arg Pass scenario name via MCP_CONFORMANCE_SCENARIO env var instead of as a positional argument. This is non-breaking for existing clients that just accept <server-url>. The env var approach is consistent with how context (including private keys for JWT auth) is already passed via MCP_CONFORMANCE_CONTEXT. * Add comment explaining env var design decision --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Olivier Chafik <ochafik@anthropic.com>
1 parent 053e9bd commit be51f87

3 files changed

Lines changed: 240 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ A framework for testing MCP (Model Context Protocol) client and server implement
99
### Testing Clients
1010

1111
```bash
12-
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
12+
# Using the everything-client (recommended)
13+
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --scenario initialize
14+
15+
# Run an entire suite of tests
16+
npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/everything-client.ts" --suite auth
1317
```
1418

1519
### Testing Servers
@@ -59,10 +63,11 @@ npx @modelcontextprotocol/conformance client --command "<client-command>" --scen
5963

6064
- `--command` - The command to run your MCP client (can include flags)
6165
- `--scenario` - The test scenario to run (e.g., "initialize")
66+
- `--suite` - Run a suite of tests in parallel (e.g., "auth")
6267
- `--timeout` - Timeout in milliseconds (default: 30000)
6368
- `--verbose` - Show verbose output
6469

65-
The framework appends the server URL as the final argument to your command.
70+
The framework appends `<server-url>` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data.
6671

6772
### Server Testing
6873

@@ -89,8 +94,9 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
8994

9095
## Example Clients
9196

92-
- `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
93-
- `examples/clients/typescript/test-broken.ts` - Invalid client missing required fields (fails checks)
97+
- `examples/clients/typescript/everything-client.ts` - Single client that handles all scenarios based on scenario name (recommended)
98+
- `examples/clients/typescript/test1.ts` - Simple MCP client (for reference)
99+
- `examples/clients/typescript/auth-test.ts` - Well-behaved OAuth client (for reference)
94100

95101
## Available Scenarios
96102

‎examples/clients/typescript/everything-client.ts‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
#!/usr/bin/env node
2+
3+
/**
4+
* Everything client - a single conformance test client that handles all scenarios.
5+
*
6+
* Usage: everything-client <server-url>
7+
*
8+
* The scenario name is read from the MCP_CONFORMANCE_SCENARIO environment variable,
9+
* which is set by the conformance test runner.
10+
*
11+
* This client routes to the appropriate behavior based on the scenario name,
12+
* consolidating all the individual test clients into one.
13+
*/
14+
15+
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
16+
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
17+
import { ElicitRequestSchema } from '@modelcontextprotocol/sdk/types.js';
18+
import { withOAuthRetry } from './helpers/withOAuthRetry.js';
19+
import { logger } from './helpers/logger.js';
20+
21+
// Scenario handler type
22+
type ScenarioHandler = (serverUrl: string) => Promise<void>;
23+
24+
// Registry of scenario handlers
25+
const scenarioHandlers: Record<string, ScenarioHandler> = {};
26+
27+
// Helper to register a scenario handler
28+
function registerScenario(name: string, handler: ScenarioHandler): void {
29+
scenarioHandlers[name] = handler;
30+
}
31+
32+
// Helper to register multiple scenarios with the same handler
33+
function registerScenarios(names: string[], handler: ScenarioHandler): void {
34+
for (const name of names) {
35+
scenarioHandlers[name] = handler;
36+
}
37+
}
38+
39+
// ============================================================================
40+
// Basic scenarios (initialize, tools-call)
41+
// ============================================================================
42+
43+
async function runBasicClient(serverUrl: string): Promise<void> {
44+
const client = new Client(
45+
{ name: 'test-client', version: '1.0.0' },
46+
{ capabilities: {} }
47+
);
48+
49+
const transport = new StreamableHTTPClientTransport(new URL(serverUrl));
50+
51+
await client.connect(transport);
52+
logger.debug('Successfully connected to MCP server');
53+
54+
await client.listTools();
55+
logger.debug('Successfully listed tools');
56+
57+
await transport.close();
58+
logger.debug('Connection closed successfully');
59+
}
60+
61+
registerScenarios(['initialize', 'tools-call'], runBasicClient);
62+
63+
// ============================================================================
64+
// Auth scenarios - well-behaved client
65+
// ============================================================================
66+
67+
async function runAuthClient(serverUrl: string): Promise<void> {
68+
const client = new Client(
69+
{ name: 'test-auth-client', version: '1.0.0' },
70+
{ capabilities: {} }
71+
);
72+
73+
const oauthFetch = withOAuthRetry(
74+
'test-auth-client',
75+
new URL(serverUrl)
76+
)(fetch);
77+
78+
const transport = new StreamableHTTPClientTransport(new URL(serverUrl), {
79+
fetch: oauthFetch
80+
});
81+
82+
await client.connect(transport);
83+
logger.debug('Successfully connected to MCP server');
84+
85+
await client.listTools();
86+
logger.debug('Successfully listed tools');
87+
88+
await client.callTool({ name: 'test-tool', arguments: {} });
89+
logger.debug('Successfully called tool');
90+
91+
await transport.close();
92+
logger.debug('Connection closed successfully');
93+
}
94+
95+
// Register all auth scenarios that should use the well-behaved auth client
96+
registerScenarios(
97+
[
98+
'auth/basic-dcr',
99+
'auth/basic-metadata-var1',
100+
'auth/basic-metadata-var2',
101+
'auth/basic-metadata-var3',
102+
'auth/2025-03-26-oauth-metadata-backcompat',
103+
'auth/2025-03-26-oauth-endpoint-fallback',
104+
'auth/scope-from-www-authenticate',
105+
'auth/scope-from-scopes-supported',
106+
'auth/scope-omitted-when-undefined',
107+
'auth/scope-step-up'
108+
],
109+
runAuthClient
110+
);
111+
112+
// ============================================================================
113+
// Elicitation defaults scenario
114+
// ============================================================================
115+
116+
async function runElicitationDefaultsClient(serverUrl: string): Promise<void> {
117+
const client = new Client(
118+
{ name: 'elicitation-defaults-test-client', version: '1.0.0' },
119+
{
120+
capabilities: {
121+
elicitation: {
122+
applyDefaults: true
123+
}
124+
}
125+
}
126+
);
127+
128+
// Register elicitation handler that returns empty content
129+
// The SDK should fill in defaults for all omitted fields
130+
client.setRequestHandler(ElicitRequestSchema, async (request) => {
131+
logger.debug(
132+
'Received elicitation request:',
133+
JSON.stringify(request.params, null, 2)
134+
);
135+
logger.debug('Accepting with empty content - SDK should apply defaults');
136+
137+
// Return empty content - SDK should merge in defaults
138+
return {
139+
action: 'accept' as const,
140+
content: {}
141+
};
142+
});
143+
144+
const transport = new StreamableHTTPClientTransport(new URL(serverUrl));
145+
146+
await client.connect(transport);
147+
logger.debug('Successfully connected to MCP server');
148+
149+
// List available tools
150+
const tools = await client.listTools();
151+
logger.debug(
152+
'Available tools:',
153+
tools.tools.map((t) => t.name)
154+
);
155+
156+
// Call the test tool which will trigger elicitation
157+
const testTool = tools.tools.find(
158+
(t) => t.name === 'test_client_elicitation_defaults'
159+
);
160+
if (!testTool) {
161+
throw new Error('Test tool not found: test_client_elicitation_defaults');
162+
}
163+
164+
logger.debug('Calling test_client_elicitation_defaults tool...');
165+
const result = await client.callTool({
166+
name: 'test_client_elicitation_defaults',
167+
arguments: {}
168+
});
169+
170+
logger.debug('Tool result:', JSON.stringify(result, null, 2));
171+
172+
await transport.close();
173+
logger.debug('Connection closed successfully');
174+
}
175+
176+
registerScenario('elicitation-defaults', runElicitationDefaultsClient);
177+
178+
// ============================================================================
179+
// Main entry point
180+
// ============================================================================
181+
182+
async function main(): Promise<void> {
183+
const scenarioName = process.env.MCP_CONFORMANCE_SCENARIO;
184+
const serverUrl = process.argv[2];
185+
186+
if (!scenarioName || !serverUrl) {
187+
console.error(
188+
'Usage: MCP_CONFORMANCE_SCENARIO=<scenario> everything-client <server-url>'
189+
);
190+
console.error(
191+
'\nThe MCP_CONFORMANCE_SCENARIO env var is set automatically by the conformance runner.'
192+
);
193+
console.error('\nAvailable scenarios:');
194+
for (const name of Object.keys(scenarioHandlers).sort()) {
195+
console.error(` - ${name}`);
196+
}
197+
process.exit(1);
198+
}
199+
200+
const handler = scenarioHandlers[scenarioName];
201+
if (!handler) {
202+
console.error(`Unknown scenario: ${scenarioName}`);
203+
console.error('\nAvailable scenarios:');
204+
for (const name of Object.keys(scenarioHandlers).sort()) {
205+
console.error(` - ${name}`);
206+
}
207+
process.exit(1);
208+
}
209+
210+
try {
211+
await handler(serverUrl);
212+
process.exit(0);
213+
} catch (error) {
214+
console.error('Error:', error);
215+
process.exit(1);
216+
}
217+
}
218+
219+
main().catch((error) => {
220+
console.error('Unhandled error:', error);
221+
process.exit(1);
222+
});

‎src/runner/client.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ export interface ClientExecutionResult {
1414

1515
async function executeClient(
1616
command: string,
17+
scenarioName: string,
1718
serverUrl: string,
1819
timeout: number = 30000,
1920
context?: Record<string, unknown>
@@ -26,8 +27,13 @@ async function executeClient(
2627
let stderr = '';
2728
let timedOut = false;
2829

29-
// Build environment with optional context
30+
// Build environment with scenario name and optional context.
31+
// We use separate env vars rather than putting scenario in context because:
32+
// 1. Scenario is always set, context is only set when there's scenario-specific data
33+
// 2. Simpler to read a string vs parsing JSON just to get the scenario name
34+
// 3. Semantic separation: scenario identifies "which test", context provides "test data"
3035
const env = { ...process.env };
36+
env.MCP_CONFORMANCE_SCENARIO = scenarioName;
3137
if (context) {
3238
env.MCP_CONFORMANCE_CONTEXT = JSON.stringify(context);
3339
}
@@ -105,6 +111,7 @@ export async function runConformanceTest(
105111
try {
106112
const clientOutput = await executeClient(
107113
clientCommand,
114+
scenarioName,
108115
urls.serverUrl,
109116
timeout,
110117
urls.context

0 commit comments

Comments
 (0)