Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 76 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,13 @@ ldbc.finbench.transaction.queries.updates_dir=src/main/resources/example/data/in

### 1.4 Mode

The driver operates in three modes:
The driver supports the following user-facing modes:

- CREATE_VALIDATION
- VALIDATE_DATABASE
- EXECUTE_BENCHMARK
- AUTOMATIC_TEST
- OPTIMIZATION_RECOMMENDATION

#### CREATE_VALIDATION

Expand Down Expand Up @@ -93,6 +95,67 @@ warmup=5
operation_count=10000
```


#### AUTOMATIC_TEST

Perform the performance test automatically with the `AUTOMATIC_TEST` mode, utilizing an adaptive testing approach. This approach leverages pre-execution on hardware devices for initial parameter estimation and achieves optimal performance, meeting requirements through automatic tuning. The entire process is divided into two phases: **(1) Rapid Estimation Phase** and **(2) Precision Tuning Phase**.

1. **estimate**: Quickly calculate the duration of each test phase (in milliseconds). If set to -1, it indicates the completion of operations to achieve the specified `warmup` value. If the parameter is not used, the default is 300,000 ms.
2. **accurate**: Specifies the duration of each test in the precise tuning phase (in milliseconds). If set to -1, it indicates the completion of operations to reach the specified `operation_count`. If the parameter is not used, the default is 7,200,000 ms.
3. **error_range**: Binary end condition, indicating the tolerance range. If the parameter is not used, the default is 1E-5.
4. **tcr_min**: Minimum time compression ratio limit. If the parameter is not used, the default is 1E-9.
5. **tcr_max**: Maximum time compression ratio limit. If the parameter is not used, the default is 1.
6. **timeout_rate**: Specifies the ratio of timeouts to total operations at which the threshold is allowed to be exceeded. If the parameter is not used, the default is 0.05.
7. **time_compression_ratio**: When utilized in the initial testing round, setting an appropriate value for the machine can significantly reduce the overall test duration. If the parameter is not used, the default is the average of (tcr_min + tcr_max) / 2.

```shell
mode=AUTOMATIC_TEST
estimate=300000
accurate=7200000
error_range=1E-5
tcr_min=1E-9
tcr_max=1
timeout_rate=0.05
time_compression_ratio=0.1
```

#### OPTIMIZATION_RECOMMENDATION

Runs a normal benchmark as a baseline, ranks operation types by optimization recommendation score,
profiles each selected operation through the optional SUT `OptimizationSupport` capability, then
reruns the full workload for every selected operation and reduction step. During each scenario run,
only the selected operation is replaced by the SUT synthetic execution hook; all other operations
use their normal handlers. Existing benchmark modes do not require this capability.

The driver captures one real operation, including its parameters, for every operation type during
the baseline. It executes the selected operations through their normal database handlers while
sampling resources. A SUT only needs to implement `getCurrentCpuCores`,
`getCurrentMemoryBytes`, and `executeSyntheticLoad`;
it does not need to construct or dispatch FinBench operations.

CPU is expressed as cores, memory as bytes, and synthetic duration as nanoseconds. The SUT must
release requested resources before `executeSyntheticLoad` returns. Generated scenario
configuration stores duration in `optimization.simulation.target_duration_nanos`, while reports
display fractional milliseconds. See the Dummy implementation for an integration example.

```properties
mode=OPTIMIZATION_RECOMMENDATION
results_dir=results/optimization
operation_count=10000
optimization.top_n=3
optimization.reduction_steps=0.05,0.10,0.20,0.30
optimization.profile_repetitions=3
optimization.repetitions=3
optimization.sample_interval_millis=50
```

The mode writes `optimization-report.json` and `optimization-report.md`. Scenario throughput
gains are measured from those full-workload reruns and should be verified again after implementing
the real SUT optimization.

The DummyDB example configuration is available at
`src/main/resources/example/ldbc_finbench_optimization_recommendation_dummy.properties`.

## 2. Quick Start

To get started, clone the repository and build it with Maven:
Expand All @@ -108,3 +171,15 @@ For a quick trial of the driver, utilize the DummyDB shipped with it by running
```bash
java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_driver_dummy.properties
```

To automatically find the suitable time_compression_ratio for your system on this machine, execute the following command:

```bash
java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_automatic_test_dummy.properties
```

To run the optimization recommendation example:

```bash
java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_optimization_recommendation_dummy.properties
```
78 changes: 62 additions & 16 deletions src/main/java/org/ldbcouncil/finbench/driver/Db.java
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,13 @@
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Consumer;
import org.ldbcouncil.finbench.driver.log.LoggingService;
import org.ldbcouncil.finbench.driver.optimization.OptimizationSimulationConfig;
import org.ldbcouncil.finbench.driver.optimization.OptimizationSupport;
import org.ldbcouncil.finbench.driver.optimization.SyntheticOptimizationOperationHandler;
import org.ldbcouncil.finbench.driver.util.ClassLoaderHelper;

public abstract class Db implements Closeable {
Expand All @@ -34,19 +39,24 @@ public abstract class Db implements Closeable {
private Map<Class<? extends Operation>, OperationHandler> operationHandlers = new HashMap<>();
private OperationHandler[] operationHandlersArray = null;
private OperationHandlerRunnerFactory operationHandlerRunnableContextFactory = null;
private Optional<OptimizationSimulationConfig> optimizationSimulationConfig = Optional.empty();
private Optional<OptimizationSupport> optimizationSupport = Optional.empty();
private Consumer<Operation> operationExecutionListener = null;

public final synchronized void init(
Map<String, String> params,
LoggingService loggingService,
Map<Integer, Class<? extends Operation>> operationTypeToClassMapping)
throws DbException {
Map<String, String> params,
LoggingService loggingService,
Map<Integer, Class<? extends Operation>> operationTypeToClassMapping)
throws DbException {
if (isInitialized) {
throw new DbException("DB may be initialized only once");
}
onInit(params, loggingService);
optimizationSimulationConfig = OptimizationSimulationConfig.from(params);
optimizationSupport = optimizationSimulationConfig.isPresent() ? optimizationSupport() : Optional.empty();
dbConnectionState = getConnectionState();
operationHandlerRunnableContextFactory = new PoolingOperationHandlerRunnerFactory(
new InstantiatingOperationHandlerRunnerFactory()
new InstantiatingOperationHandlerRunnerFactory()
);
operationHandlersArray = toOperationHandlerArray(operationTypeToClassMapping, operationHandlers);
operationHandlers = null;
Expand Down Expand Up @@ -81,7 +91,18 @@ public final synchronized void reInit() throws DbException {
throw new DbException("Error shutting down operation handler runnable factory", e);
}
operationHandlerRunnableContextFactory = new PoolingOperationHandlerRunnerFactory(
new InstantiatingOperationHandlerRunnerFactory()
new InstantiatingOperationHandlerRunnerFactory()
);
}

public final synchronized void reInitAutomatic() throws DbException {
try {
((PoolingOperationHandlerRunnerFactory) operationHandlerRunnableContextFactory).shutdownAutomatic();
} catch (OperationException e) {
throw new DbException("Error shutting down operation handler runnable factory", e);
}
operationHandlerRunnableContextFactory = new PoolingOperationHandlerRunnerFactory(
new InstantiatingOperationHandlerRunnerFactory()
);
}

Expand All @@ -91,7 +112,7 @@ public final synchronized void reInit() throws DbException {
protected abstract void onClose() throws IOException;

public final <A extends Operation, H extends OperationHandler<A, ?>> void registerOperationHandler(
Class<A> operationType, Class<H> operationHandlerType) throws DbException {
Class<A> operationType, Class<H> operationHandlerType) throws DbException {
if (operationHandlers.containsKey(operationType)) {
throw new DbException(format("Client already has handler registered for %s", operationType.getClass()));
}
Expand All @@ -100,23 +121,34 @@ public final synchronized void reInit() throws DbException {
operationHandlers.put(operationType, operationHandler);
} catch (OperationException e) {
throw new DbException(
format("%s could not instantiate instance of %s",
getClass().getSimpleName(),
operationHandlerType.getSimpleName()
),
e);
format("%s could not instantiate instance of %s",
getClass().getSimpleName(),
operationHandlerType.getSimpleName()
),
e);
}
}

public final OperationHandlerRunnableContext getOperationHandlerRunnableContext(Operation operation)
throws DbException {
throws DbException {
if (operationExecutionListener != null) {
operationExecutionListener.accept(operation);
}
OperationHandler operationHandler = operationHandlersArray[operation.type()];
if (null == operationHandler) {
throw new DbException(format("No handler registered for %s", operation.getClass()));
}
if (optimizationSimulationConfig.isPresent()
&& optimizationSupport.isPresent()
&& optimizationSimulationConfig.get().matches(operation.getClass().getSimpleName())) {
operationHandler = new SyntheticOptimizationOperationHandler(
optimizationSupport.get(),
optimizationSimulationConfig.get()
);
}
try {
OperationHandlerRunnableContext operationHandlerRunnableContext =
operationHandlerRunnableContextFactory.newOperationHandlerRunner();
operationHandlerRunnableContextFactory.newOperationHandlerRunner();
operationHandlerRunnableContext.setOperationHandler(operationHandler);
operationHandlerRunnableContext.setDbConnectionState(dbConnectionState);
return operationHandlerRunnableContext;
Expand All @@ -126,8 +158,8 @@ public final OperationHandlerRunnableContext getOperationHandlerRunnableContext(
}

private static OperationHandler[] toOperationHandlerArray(
Map<Integer, Class<? extends Operation>> operationTypeToClassMapping,
Map<Class<? extends Operation>, OperationHandler> operationHandlers) throws DbException {
Map<Integer, Class<? extends Operation>> operationTypeToClassMapping,
Map<Class<? extends Operation>, OperationHandler> operationHandlers) throws DbException {
if (operationTypeToClassMapping.isEmpty()) {
return new OperationHandler[] {};
} else {
Expand All @@ -153,4 +185,18 @@ private static OperationHandler[] toOperationHandlerArray(
* reused by all operation handlers
*/
protected abstract DbConnectionState getConnectionState() throws DbException;

/**
* Optional SUT capability used by optimization recommendation mode.
*/
public Optional<OptimizationSupport> optimizationSupport() {
return Optional.empty();
}

/**
* Registers a listener invoked whenever an operation is dispatched to its handler.
*/
public final void setOperationExecutionListener(Consumer<Operation> listener) {
operationExecutionListener = listener;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,11 @@ public void shutdown() throws OperationException {
}
}

public void shutdownAutomatic() throws OperationException {
innerOperationHandlerRunnerFactory.shutdown();
operationHandlerRunnerPool.shutdown();
}

@Override
public String toString() {
return PoolingOperationHandlerRunnerFactory.class.getSimpleName() + "{"
Expand Down
61 changes: 42 additions & 19 deletions src/main/java/org/ldbcouncil/finbench/driver/Workload.java
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,41 @@ public abstract class Workload implements Closeable {
* @return
*/
public ResultsLogValidationTolerances resultsLogValidationTolerances(
DriverConfiguration configuration,
boolean warmup
DriverConfiguration configuration,
boolean warmup
) {
long excessiveDelayThresholdAsMilli = TimeUnit.SECONDS.toMillis(1);
double toleratedExcessiveDelayCountPercentage = 0.05; // 95% of the queries must run below delay threshold
// Specifies the fraction of the delay threshold that is allowed to be exceeded
double toleratedExcessiveDelayCountPercentage = configuration.timeoutRate();
long toleratedExcessiveDelayCount = // Total tolerated excessive delay count
(warmup) ? Math.round(configuration.warmupCount() * toleratedExcessiveDelayCountPercentage)
: Math.round(configuration.operationCount() * toleratedExcessiveDelayCountPercentage);
(warmup) ? Math.round(configuration.warmupCount() * toleratedExcessiveDelayCountPercentage)
: Math.round(configuration.operationCount() * toleratedExcessiveDelayCountPercentage);
return new ResultsLogValidationTolerances(
excessiveDelayThresholdAsMilli,
toleratedExcessiveDelayCount,
toleratedExcessiveDelayCountPercentage
excessiveDelayThresholdAsMilli,
toleratedExcessiveDelayCount,
toleratedExcessiveDelayCountPercentage
);
}

/**
* resultsLogValidationTolerances Automated beta version of the method
*
* @param operationCount
* @return
*/
public ResultsLogValidationTolerances resultsLogValidationTolerancesAutomatic(
DriverConfiguration configuration,
long operationCount
) {
long excessiveDelayThresholdAsMilli = TimeUnit.SECONDS.toMillis(1);
// Specifies the fraction of the delay threshold that is allowed to be exceeded
double toleratedExcessiveDelayCountPercentage = configuration.timeoutRate();
// Total tolerated excessive delay count
long toleratedExcessiveDelayCount = Math.round(operationCount * toleratedExcessiveDelayCountPercentage);
return new ResultsLogValidationTolerances(
excessiveDelayThresholdAsMilli,
toleratedExcessiveDelayCount,
toleratedExcessiveDelayCountPercentage
);
}

Expand Down Expand Up @@ -97,7 +120,7 @@ public final WorkloadStreams streams(GeneratorFactory gf, boolean hasDbConnected
}

protected abstract WorkloadStreams getStreams(GeneratorFactory generators, boolean hasDbConnected)
throws WorkloadException;
throws WorkloadException;

public DbValidationParametersFilter dbValidationParametersFilter(final Integer requiredValidationParameterCount) {
return new DbValidationParametersFilter() {
Expand All @@ -111,18 +134,18 @@ public boolean useOperation(Operation operation) {

@Override
public DbValidationParametersFilterResult useOperationAndResultForValidation(
Operation operation,
Object operationResult) {
Operation operation,
Object operationResult) {
if (validationParameterCount < requiredValidationParameterCount) {
validationParameterCount++;
return new DbValidationParametersFilterResult(
DbValidationParametersFilterAcceptance.ACCEPT_AND_CONTINUE,
injectedOperations
DbValidationParametersFilterAcceptance.ACCEPT_AND_CONTINUE,
injectedOperations
);
} else {
return new DbValidationParametersFilterResult(
DbValidationParametersFilterAcceptance.REJECT_AND_FINISH,
injectedOperations
DbValidationParametersFilterAcceptance.REJECT_AND_FINISH,
injectedOperations
);
}
}
Expand All @@ -139,8 +162,8 @@ public interface DbValidationParametersFilter {
boolean useOperation(Operation operation);

DbValidationParametersFilterResult useOperationAndResultForValidation(
Operation operation,
Object operationResult);
Operation operation,
Object operationResult);
}

public enum DbValidationParametersFilterAcceptance {
Expand All @@ -155,8 +178,8 @@ public static class DbValidationParametersFilterResult {
private final List<Operation> injectedOperations;

public DbValidationParametersFilterResult(
DbValidationParametersFilterAcceptance acceptance,
List<Operation> injectedOperations) {
DbValidationParametersFilterAcceptance acceptance,
List<Operation> injectedOperations) {
this.acceptance = acceptance;
this.injectedOperations = injectedOperations;
}
Expand Down
Loading
Loading