Unable to use harbor p2p distribution feature to preheat images with dragonfly

[bmmadhuprasad]

I have deployed Harbor v2.0 and dragonfly latest version trying to use harbor p2p image distribution to dragonfly but getting below when applied the preheat policy to preheat image
Check preheat progress: Task is failing and distribution is not happening

2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/test:0.1@sha256:54c9b521ae50156029ea48394041885f33414c1dad2a5deefcaa46ddf22079a3' to the target preheat provider: dragonfly newdragon:http://dragonflynew-supernode.dfn.svc.cluster.local:8002
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done
2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m)
2021-03-31T13:19:49Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:19:59Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:09Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:19Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z
2021-03-31T13:20:29Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z

Below is logs from dragonfly

2021-03-31 14:17:50.496 DEBU sign:9 : GET /_ping remote:192.168.71.183:40028 cost:2.901µs err:
2021-03-31 14:17:50.498 DEBU sign:9 : create preheat: Type[image] URL[https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1] Filter[] Identifier[] Headers[map[Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTcyMDIwNjcsImp0aSI6Imc3TFhnUmJidmkzOFV4bFUiLCJpYXQiOjE2MTcyMDAyNjcsImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTcyMDAyNjcsInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.DFeugff4V8VplCOGrW1AqAUhUTRo8F4pCwPAKa06zP8A67Euls4GC6MidfXpzAazU5QndtHBMkKdh-CBAUgzKLv73F3Kz1tdhXBAcyGXJHX7r8otQ4J8wNf0TwOHesmoW1sKc0IUBmlsA_a4fXR_7DAspzJmFiBchNin5qGCvhvn6449Tr_kF5HyoknOrw1NoK5_mG0mcBWLSC6E_16P1YIeAWaisfbuPurfnd-Zs2MLfSltWDjgibPXU02csiBmfGgy1mES0icDKkgeuqiLlFz90pgV61cZcQieqAeB5k3quMN20HWC3EaYTqPyspo0ZMjc_w1WItldMFDusncH8kqak7iKfQYSISf2MlD7WSmHO77_jgePWsEYKvdQIuLOcY-PZ-dswISb1DE72hJ6ZvNEnOna7LKe67VfFgfBvar4UEtwCzMCyJt0aAUzepgvFbdA1ajUQagpeAlco5o-oZmvmbfay_xiBRxEsBa0n9KIKrxQ3ceLhx5xCFau3OH9Poa9ch2QeMI7bYe5BAkL2xQKzZdk4LNkG6hpQtucLNlCzMTOZ_HO7E4lFXe2LQlYcNy6XSSv16g1R5qjM2W4UBf_pmGGo1E4wTC0WSVlgIXdlCkVo8P1h6xfVhmOfnZWiZgVmPynR21urj1pW-M0FEbarBefU-JTvQU4Qgt25Q0]]
2021-03-31 14:17:50.498 DEBU sign:9 : POST /preheats remote:192.168.71.183:40028 cost:193.662µs err:
2021-03-31 14:18:00.502 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.646µs err:
2021-03-31 14:18:10.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:24.602µs err:
2021-03-31 14:18:20.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:37.729µs err:
2021-03-31 14:18:30.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:25.405µs err:
2021-03-31 14:18:40.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:23.134µs err:
2021-03-31 14:18:50.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:28.289µs err:
2021-03-31 14:19:00.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.542µs err:

Can anybody let me know anything i am missing what may be the cause for the above issue

[bmmadhuprasad]

even i have deployed dragonfly supernode in VM directly from source code without using docker as per document and tried to use that as end point in harbor but i see same error when i created p2p preheat policy and apply to initiate image preheat to dragonfly
Below is the error i see from harbor
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'test/mem:0.1@sha256:9871d56e046e38e9daaef6b592e67282c1e286a85e84797ed18b942a8263eadf' to the target preheat provider: dragonfly vm:http://54.215.243.61:8002
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done
2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m)
2021-04-05T13:49:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:49:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:50:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:51:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00
2021-04-05T13:52:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check p
below is the logs from dragonfly supernode
1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhayenlv3u999yAnLnAgjGaYijNWv6r2IMeO6mFnI6AZSzlKH2ig3N25gPbxh1UT7PC7jd-mJEuxVd_BBMFhS2BttRbLqyJf6Bkr5T4gKOYyVjAm0tBG0zdhulrjakAt97BW6v3IFbP2cI0aL1-vk"},"type":"image","url":"https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1"}, res:{"body":{"id":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735"},"headers":{},"statusCode":"OK","statusCodeValue":200}
2021-04-05 13:49:00.958 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:c1c1ba809094891c97d8ca9f34817f54c72cfab868b6914d85f2a156a6ac26a9 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.969 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.970 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:79f39971f65f5d879d2ed9e4402b5fba9a825803c4c76b8a09e4926ea58185c2 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.973 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e72295e4b7adb87d61408b4101c0e4f8ccc15c84ccbfdbfe756e410f83b0fcaf parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.974 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e38d1a0197a41b27b23313a6ae0ec7f4219c31fdb55e5f5b9c43ee781237312f parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
2021-04-05 13:49:00.980 INFO [preheat-12] c.d.d.s.s.p.PreheatServiceImpl - command: /usr/local/bin/dfget -u https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 -o /home/admin/supernode/repo/preheat/87626317-ef0f-448a-8a6e-78e60f72cc6c --callsystem dragonfly_preheat --totallimit 90M -s 90M --notbs --header Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhaye
Please help us to fix the issue and let us know anything i am missing

[ansinlee]

Is the image public?

[bmmadhuprasad]

yes project created in harbor where this image pushed is public

[bmmadhuprasad]

dragonfly supernode image i am using is dragonflyoss/supernode:1.0.6,I have also tried with latest image of supernode but same issue.

[jim3ma]

Thanks for report this issue.
I will verify it ASAP.

[ansinlee]

query the detail error message from dragonfly server of the task as the following command
curl http://{dragonfly-server-address}/preheat/{task-id}

for example:
curl http://192.168.0.180:8002/preheats/51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127

{"ID":"51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127","finishTime":"2021-04-07T08:42:52.682Z","startTime":"2021-04-07T08:42:49.681Z","status":"FAILED","errorMsg":"Get http://xx.xx.xx.xx/v2/library/nginx/manifests/latest: dial tcp xx.xx.xx.xx:80: i/o timeout"}

[bmmadhuprasad]

@ansinlee
curl http://13.56.114.60:8002/preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735
{"ID":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735","finish
Time":"2021-04-08T09:28:33.656Z","startTime":"2021-04-08T09:28:33.400Z","status"
:"FAILED","errorMsg":"Get https://harbor.madhu.mwpdemos.com/v2/test/mem/manifest
s/0.1: x509: certificate signed by unknown authority"}

[jim3ma]

@bmmadhuprasad It seems your harbor cert is not valid for dragonfly supernode.

[liang-junwei]

I also met the same problem, how to solve it?

[huangpxs]

I also met the same problem，my harbor version is v2.2.1

[jim3ma]

Follow https://blog.jim.plus/blog/post/jim/selfsign-ca-and-update-certficates to update ca chains.

[zkking]

@bmmadhuprasad Obviously, this is a bug of Dragonfly 1.x. In the file https://github.com/dragonflyoss/Dragonfly/tree/master/supernode/daemon/mgr/preheat/image_preaheater.go, the getLayers method does not support TLS to obtain image layer information.

Therefore, before fixing the problem, you need to manually set the certificate trust for the supernode container. The operation is as follows:
docker exec -it supernode bash
cp /certs/ca.crt /usr/local/share/ca-certificates/
update-ca-certificates
docker restart supernode