Skip to content

Commit ad83682

Browse files
committed
HDFS-17715. [ARR] Add documentation for asynchronous router. (#7318). Contributed by Jian Zhang.
Signed-off-by: He Xiaoqiao <[email protected]>
1 parent bdff83a commit ad83682

File tree

1 file changed

+33
-1
lines changed

1 file changed

+33
-1
lines changed

hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md

+33-1
Original file line numberDiff line numberDiff line change
@@ -557,4 +557,36 @@ Enable Router to rename across namespaces. Currently it is implemented based on
557557
| dfs.federation.router.federation.rename.bandwidth | | Specify bandwidth per map in MB.|
558558
| dfs.federation.router.federation.rename.delay | 1000 | Specify the delayed duration(millie seconds) when the job needs to retry.|
559559
| dfs.federation.router.federation.rename.diff | 0 | Specify the threshold of the diff entries that used in incremental copy stage.|
560-
| dfs.federation.router.federation.rename.trash | trash | This options has 3 values: trash (move the source path to trash), delete (delete the source path directly) and skip (skip both trash and deletion).|
560+
| dfs.federation.router.federation.rename.trash | trash | This options has 3 values: trash (move the source path to trash), delete (delete the source path directly) and skip (skip both trash and deletion).|
561+
562+
Asynchronous Router RPC
563+
-------
564+
Asynchronous router RPC is designed to address the performance bottlenecks of synchronous router RPC in scenarios with high concurrency and multiple named services.
565+
By introducing an asynchronous processing mechanism, it optimizes the request-handling process, enhances the system's concurrency capacity and resource utilization efficiency.
566+
567+
See the Apache JIRA ticket [HDFS-17531](https://issues.apache.org/jira/browse/HDFS-17531) for more information on this feature.
568+
569+
### Asynchronous router RPC threads
570+
571+
- **Handler**: Retrieves ```RpcCall``` from the ```CallQueue``` for preliminary processing. In case of exceptions (such as the non-existence of the mount point),
572+
it directly places the response into the response queue. Otherwise, it forwards the ```RpcCall``` to the **Async-Handler**.
573+
- **Async-Handler**: Puts the ```RpcCall``` into the ```connection.calls``` of the connection thread and returns immediately without blocking and waiting.
574+
- **Async-Responder**: Is responsible for processing the response received by the connection thread. If the ```RpcCall``` needs to be retried (such as the downstream service returns a ```StandbyException```),
575+
it re-adds the ```RpcCall``` to the ```connection.calls```; otherwise, it puts the response into the ```ResponseQueue```.
576+
- **Responder**: Retrieves the response from the ```ResponseQueue``` and returns it to the client.
577+
578+
### Advantages of the Asynchronous Router Rpc
579+
580+
- **High Processing Performance**: Benefiting from the asynchronous RPC (Remote Procedure Call) processing mechanism, the asynchronous router is capable of handling a large number of requests simultaneously.
581+
This not only significantly enhances the system's concurrent processing capacity but also optimizes the overall throughput. This mechanism enables the system to respond rapidly to high - traffic requests and maintain efficient operation even under high - load conditions.
582+
- **High Resource Utilization**: The asynchronous design effectively reduces thread blocking and frequent thread switching. As a result, it minimizes the resource waste associated with threads, thereby improving the overall efficiency of the system and reducing CPU idle time.
583+
- **Isolation**: Different name-services employ distinct asynchronous processor thread pools. This architecture achieves isolation among name-services. If a particular name-service experiences a performance degradation,
584+
it will not impact the processing capabilities of other name-services, ensuring the stability and reliability of the entire system.
585+
586+
### Asynchronous Router Rpc configuration
587+
| Property | Default | Description|
588+
|:---- |:---- |:---- |
589+
| dfs.federation.router.async.rpc.enable | false | If true, router will process the ```RpcCall``` asynchronously. |
590+
| dfs.federation.router.async.rpc.ns.handler.count | | The number of async-handlers per nameservice, separated by commas, internally separated by colons. The identifier of nameservice is in dfs.nameservices configuration entry. Such as: ns1:count1,ns2:count2,ns3:count3. |
591+
| dfs.federation.router.async.rpc.handler.count | 10 | For those nameservices not in dfs.federation.router.async.rpc.ns.handler.count configuration entry, use this value as the async-handler counts. |
592+
| dfs.federation.router.async.rpc.responder.count | 10 | The thread counts of async-responders. |

0 commit comments

Comments
 (0)