Duplicate datanodes after a disk failure

[soenkeliebau]

Affected version

0.6.0

Current and expected behavior

In an HDFS cluster with 2 namenodes and 3 datanodes using local disks, one of the datanodes had a disk failure.
After fixing the disk issue and restarting the datanode, this node showed up multiple times in the datanode list.

The node did register and become an active datanode, but had two rogue identities remaining listed as dead in the datanode list.

4.4$ /stackable/hadoop/bin/hdfs dfsadmin -report
Configured Capacity: 643930521600 (599.71 GB)
Present Capacity: 637323175133 (593.55 GB)
DFS Remaining: 637294691549 (593.53 GB)
DFS Used: 28483584 (27.16 MB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
Erasure Coded Block Groups:
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Low redundancy blocks with highest priority to recover: 0
        Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 100.64.13.117:9866 (simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9478144 (9.04 MB)
Non DFS Used: 1531338752 (1.43 GB)
DFS Remaining: 212431602079 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 08:07:50 UTC 2022
Num of Blocks: 861
Name: 100.64.43.71:9866 (100-64-43-71.simple-hdfs-datanode-default-2.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-2.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 05:26:22 UTC 2022
Num of Blocks: 861
Name: 100.64.45.118:9866 (100-64-45-118.simple-hdfs-datanode-default-0.stack-dev.svc.cluster.local)
Hostname: simple-hdfs-datanode-default-0.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 9502720 (9.06 MB)
Non DFS Used: 1531371520 (1.43 GB)
DFS Remaining: 212431544735 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 10
Last contact: Mon Dec 12 10:14:04 UTC 2022
Last Block Report: Mon Dec 12 06:52:28 UTC 2022
Num of Blocks: 861
Dead datanodes (2):
Name: 100.64.13.116:9866 (100.64.13.116)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 53775 (52.51 KB)
Non DFS Used: 1533402609 (1.43 GB)
DFS Remaining: 213110050816 (198.47 GB)
DFS Used%: 0.00%
DFS Remaining%: 99.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Dec 12 08:07:09 UTC 2022
Last Block Report: Mon Dec 12 08:05:21 UTC 2022
Num of Blocks: 0
Name: 100.64.48.102:9866 (100.64.48.102)
Hostname: simple-hdfs-datanode-default-1.simple-hdfs-datanode-default.stack-dev.svc.cluster.local
Decommission Status : Normal
Configured Capacity: 214643507200 (199.90 GB)
DFS Used: 7147520 (6.82 MB)
Non DFS Used: 1531076608 (1.43 GB)
DFS Remaining: 212434194847 (197.84 GB)
DFS Used%: 0.00%
DFS Remaining%: 98.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Fri Dec 09 10:28:20 UTC 2022
Last Block Report: Fri Dec 09 09:13:23 UTC 2022
Num of Blocks: 0

To a certain extent this is expected behavior, as the datanode generates a new datanode uid when it is started up with an empty data directory, as can easily be shown by shutting down a datanode, removing the persistent volume it stores its data on and then restarting it:

# Datanode ID before restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 11:59:14 UTC 2022
datanodeUuid=f6ee6a85-1ef1-427c-8a59-b1025ce7560f
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-976c3639-f61b-4acb-bbd0-6e4d7af2da92
bash-4.4$

# Datanode ID after restart with empty PV
bash-4.4$ cat /data/data/current/VERSION
#Tue Dec 13 12:24:05 UTC 2022
datanodeUuid=0ea49c48-09c3-4443-9d91-c767ed5e0426
storageType=DATA_NODE
cTime=0
clusterID=CID-d3fe3677-072d-4699-b2cf-d4d24414c62c
layoutVersion=-57
storageID=DS-9c219b84-bceb-4e72-900f-c569442978f2
bash-4.4$

Here you can see that the datanodeUuid changes over the restart, and since this uid is used by the namenode to look up if this datanode is known to the cluster, the node registers as a new node after the restart.

This can be mitigated by using multiple data directories, as the uuid is stored in all of the directories and recovered to an empty one in the case of disk failure.

Possible solution

Document this behavior, especially the effect of having multiple data directories and workarounds for removing the dead datanodes from the nodelist (restart namenodes).

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[sbernauer]

I would consider this expected behavior and remove the type/bug label. But also fine to leave it as is

[soenkeliebau]

I have removed the bug label and would tend to just close this as "works as designed".

The issue was, that the datanodes were configured with only one physical disk, and when that disk failed the datenode lost all its metadata - which also contains its id.
So at the next startup it generated a new id and registered with that on the namenode, hence creating a new datanode.

This shouldn't happen if a datanode has multiple physical disks configured, unless the entire node is rotated and all disks are empty during startup.
If this is expected behavior, the suggested workaround is to mount one small network based pvc alongside the physical disks, as this would be reattached after rotating the node and provide the datanode id. Data on the actual disks can then be replicated from other datanodes and the old state fully restored.