tests fail when run in chroot (during package building)

[dvzrv]

Helloes!
I'm currently pushing python-ethtool to Arch Linux's [community].
On Arch, packages are built in a clean chroot (to ensure package and dependency integrity). There however, the tests fail:

running test                                                                                                                                                                                                                                                          
running egg_info
creating ethtool.egg-info                     
writing ethtool.egg-info/PKG-INFO                                                        
writing dependency_links to ethtool.egg-info/dependency_links.txt
writing top-level names to ethtool.egg-info/top_level.txt                   
writing manifest file 'ethtool.egg-info/SOURCES.txt'            
reading manifest file 'ethtool.egg-info/SOURCES.txt'                  
reading manifest template 'MANIFEST.in'                         
writing manifest file 'ethtool.egg-info/SOURCES.txt'          
running build_ext                            
copying build/lib.linux-x86_64-3.6/ethtool.cpython-36m-x86_64-linux-gnu.so ->       
test_driver_eth (tests.test_scripts.ScriptsTests) ... ok
test_driver_lo (tests.test_scripts.ScriptsTests) ... ok                                                              
test_flags2str (tests.test_scripts.ScriptsTests) ... ok
test_show_coalesce_eth (tests.test_scripts.ScriptsTests) ... ok
test_show_coalesce_lo (tests.test_scripts.ScriptsTests) ... ok
test_show_offload_eth (tests.test_scripts.ScriptsTests) ... ok                                                                                                                                                                                                         
test_show_offload_lo (tests.test_scripts.ScriptsTests) ... FAIL
test_show_ring_eth (tests.test_scripts.ScriptsTests) ... ok
test_show_ring_lo (tests.test_scripts.ScriptsTests) ... ok
test_etherinfo_objects (tests.test_ethtool.EthtoolTests) ... FAIL
test_get_active_devices (tests.test_ethtool.EthtoolTests) ... ERROR            
test_get_interface_info_active (tests.test_ethtool.EthtoolTests) ... FAIL      
test_get_interface_info_all (tests.test_ethtool.EthtoolTests) ... FAIL
test_get_interface_info_invalid (tests.test_ethtool.EthtoolTests) ... ok        
test_invalid_devices (tests.test_ethtool.EthtoolTests) ... ok                                                                                                                                                                                                           
test_full (tests.parse_ifconfig.ParserTests) ... ok
test_local (tests.parse_ifconfig.ParserTests) ... ok
test_parse_ip4addr (tests.parse_ifconfig.ParserTests) ... ok                              
test_parsing_device_line (tests.parse_ifconfig.ParserTests) ... ok
test_parsing_flags_line (tests.parse_ifconfig.ParserTests) ... ok           
test_parsing_ip (tests.parse_ifconfig.ParserTests) ... ok       
                                                                      
======================================================================
ERROR: test_get_active_devices (tests.test_ethtool.EthtoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):                                                   
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 245, in test_get_active_devices
    self._functions_accepting_devnames(devname)                                                                       
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 147, in _functions_accepting_devnames
    self.assertIsInt(ethtool.get_ufo(devname))
PermissionError: [Errno 1] Operation not permitted           
                                                                                                                                                                                                                                                                        
======================================================================
FAIL: test_show_offload_lo (tests.test_scripts.ScriptsTests)
----------------------------------------------------------------------
Traceback (most recent call last):                       
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_scripts.py", line 80, in test_show_offload_lo
    '''                
AssertionError: 'scat[60 chars]ion offload: not supported\ngeneric segmentation offload: on\n' != 'scat[60 chars]ion offload: on\ngeneric segmentation offload: on\n'
  scatter-gather: on
  tcp segmentation offload: on
- udp fragmentation offload: not supported
?                             ------------                       
+ udp fragmentation offload: on                          
?                            +                      
  generic segmentation offload: on                  

======================================================================
FAIL: test_etherinfo_objects (tests.test_ethtool.EthtoolTests)                                                       
----------------------------------------------------------------------
Traceback (most recent call last):                             
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 251, in test_etherinfo_objects
    self._verify_etherinfo_object(ei)                                                                                                                                                                                                                                  
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 168, in _verify_etherinfo_object
    self.assertEqual(ei.ipv4_address, scraped.inet)        
AssertionError: '10.10.23.10' != '10.10.23.150'           
- 10.10.23.10                                                    
+ 10.10.23.150                                                                 
?           +                                                                  
                                                                      
                                                                                
======================================================================                                                                                                                                                                                                  
FAIL: test_get_interface_info_active (tests.test_ethtool.EthtoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):                                                        
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 233, in test_get_interface_info_active
    self._verify_etherinfo_object(ei)                                       
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 168, in _verify_etherinfo_object
    self.assertEqual(ei.ipv4_address, scraped.inet)                   
AssertionError: '10.10.23.10' != '10.10.23.150'                       
- 10.10.23.10                                                   
+ 10.10.23.150                                                        
?           +                                                                        
                                                                                                                  
                                                                                                                      
======================================================================                                                  
FAIL: test_get_interface_info_all (tests.test_ethtool.EthtoolTests)
----------------------------------------------------------------------
Traceback (most recent call last):                                                                                                                                                                                                                                      
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 238, in test_get_interface_info_all
    self._verify_etherinfo_object(ei)                       
  File "/build/python-ethtool/src/python-ethtool-0.13/tests/test_ethtool.py", line 168, in _verify_etherinfo_object
    self.assertEqual(ei.ipv4_address, scraped.inet)      
AssertionError: '10.10.23.10' != '10.10.23.150'                                                               
- 10.10.23.10          
+ 10.10.23.150                                                                                                                                                       
?           +       
                              
                                          
----------------------------------------------------------------------
Ran 21 tests in 0.015s                                   
                                                    
FAILED (failures=4, errors=1)                       
Test failed: <unittest.runner.TextTestResult run=21 errors=1 failures=4>
Device(name='enp0s25', flagsint=4163, flagsstr='UP,BROADCAST,RUNNING,MULTICAST', mtu=1500)
Device(name='lo', flagsint=73, flagsstr='UP,LOOPBACK,RUNNING', mtu=65536)
Device(name='tap0', flagsint=4163, flagsstr='UP,BROADCAST,RUNNING,MULTICAST', mtu=1500)
Device(name='wlp3s0', flagsint=4099, flagsstr='UP,BROADCAST,MULTICAST', mtu=1500)
error: Test failed: <unittest.runner.TextTestResult run=21 errors=1 failures=4>

Within the chroot, the user, that's building the package is not allowed root privileges. This might lead to the tests failing, but maybe you can provide some more insight?
I'd like to be able to use the tests during packaging for python{2,3}.

[frenzymadness]

Hello.

Thank you for filling this issue. In points:

• I'll fix the errored test. This test may fail due to insufficient privileges.
• The second one is caused by the same problem as the first one but the fix won't be simple here.
• The last three tests are failing because there are differences between device info obtained by ethtool and device info obtained by ifconfig. Could you please send us an output of ifconfig and ip a commands executed in chroot?

Have a nice day.
Lumír

[dvzrv]

Hey @frenzymadness!
Thanks for looking into this.

ifconfig:

enp0s25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.23.150  netmask 255.255.255.0  broadcast 10.10.23.255
        inet6 fe80::3e97:eff:fede:1de7  prefixlen 64  scopeid 0x20<link>
        ether 3c:97:0e:de:1d:e7  txqueuelen 1000  (Ethernet)
        RX packets 22266  bytes 20715791 (19.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14937  bytes 1750192 (1.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xb4a00000-b4a20000

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 29261  bytes 4823456 (4.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29261  bytes 4823456 (4.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.23.1.2  netmask 255.255.255.0  broadcast 172.23.1.255
        inet6 fe80::d854:ebff:fe99:a603  prefixlen 64  scopeid 0x20<link>
        ether da:54:eb:99:a6:03  txqueuelen 100  (Ethernet)
        RX packets 5466  bytes 2708136 (2.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4743  bytes 338255 (330.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp3s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 5c:51:4f:9e:de:af  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip a:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 3c:97:0e:de:1d:e7 brd ff:ff:ff:ff:ff:ff
    inet 10.10.23.150/24 brd 10.10.23.255 scope global enp0s25
       valid_lft forever preferred_lft forever
    inet 10.10.23.10/24 brd 10.10.23.255 scope global secondary dynamic enp0s25
       valid_lft 312sec preferred_lft 312sec
    inet6 fe80::3e97:eff:fede:1de7/64 scope link 
       valid_lft forever preferred_lft forever
3: wlp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 5c:51:4f:9e:de:af brd ff:ff:ff:ff:ff:ff
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/ether da:54:eb:99:a6:03 brd ff:ff:ff:ff:ff:ff
    inet 172.23.1.2/24 brd 172.23.1.255 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::d854:ebff:fe99:a603/64 scope link 
       valid_lft forever preferred_lft forever

[frenzymadness]

Now it is obvious where the problem comes from. ifconfig is deprecated and it cannot show more than one IPv4 address. There is one way how to do it but it requires that each secondary IPv4 address has to have a label (known as subinterface) associated. For example enp0s25:0. But we cannot change setting of your environment.

In the test suite, ifconfig is used as a secondary source for info about network interfaces because we have to have something we can compare results from ethtool with.

The solution here is obvious. We can implement Python wrapper for ip command which is not obsolete and which can provide all necessary information about network interfaces with all new kernel features. But honestly, I don't think that it worth. We became maintainers of python-ethtool to make it Python 3 compatible and I don't think that we have enough time to make that big change and implement new stuff.

@hroncok @Sanqui What do you think guys?

I'll try to find some way to fix all your issues.

[hroncok]

Adding new features to python-ethtool is very unlikely.

[frenzymadness]

Hello. I've implemented a fix for first two issues. Tests now expecting being run without root privileges. Take a look or try it if you want.

I'll take a look also at the last problem. We don't have time to implement ip wrapper but we might find another solution.