tuning for high high concurrency when most urls are invalid #2108
Description
I have like 100M urls to visit, but 70% of them are invalid. how to tune it to get high speed? right now it performance at only 300k/s, although if all the urls are valid, the speed can achieve 3M/s.
I want 5M/s at least(when 70% of the urls are invalid).. when I use apache httpclient4 with multi threads, the speed can be 3M/s. I expected ACH to be more effective.
what should I do to improve?
public AsyncBatchHttpClient() {
DefaultAsyncHttpClientConfig.Builder configBuilder =
new DefaultAsyncHttpClientConfig.Builder()
.setHandshakeTimeout(1000)
.setConnectTimeout(Duration.ofSeconds(1))
.setConnectionTtl(Duration.ofSeconds(15))
.setRequestTimeout(Duration.ofSeconds(10))
.setReadTimeout(Duration.ofSeconds(10))
////.setShutdownQuietPeriod(Duration.ofSeconds(15))
.setMaxConnections(100000)
.setIoThreadsCount(10000)
.setPooledConnectionIdleTimeout(Duration.ofSeconds(60))
.setSoLinger(0)
.setTcpNoDelay(true)
.setMaxConnectionsPerHost(5)
.setFollowRedirect(true)
.setMaxRedirects(3)
.setMaxRequestRetry(1)
.setKeepAlive(false)
.setUseInsecureTrustManager(true)
.setDisableHttpsEndpointIdentificationAlgorithm(true);
this.httpClient = Dsl.asyncHttpClient(configBuilder);
System.out.println("MaxConnections: "+httpClient.getConfig().getMaxConnections());
}