Skip to content

Conversation

@norio-nomura
Copy link
Contributor

@norio-nomura norio-nomura commented Oct 10, 2025

  • pkg/hostagent: Detect the VM’s IP address on the same subnet as the host OS
  • pkg/hostagent: Add GuestIPAddressGuestIP field to GET /v1/info endpoint`
  • pkg/hostagent/events: Add GuestIPAddress string GuestIP string field to Status
  • pkg/hostagent: Update all ssh execution to support SSH address other than "127.0.0.1"
    already merged as pkg/hostagent: Update all ssh execution to support SSH address other than "127.0.0.1" #4241
  • Support port forwarding via direct access to guest IP
  • Support static port forwarding via direct access to guest IP
  • Add test using default.yaml with --network=vzNAT
  • Add the _LIMA_DIRECT_IP_PORT_FORWARDER environment variable to enable direct IP port forwarding to the VM

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 9c6d7cb to 912ed32 Compare October 10, 2025 10:01
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 89e8a70 to dee7e4e Compare October 11, 2025 08:36
@norio-nomura
Copy link
Contributor Author

norio-nomura commented Oct 11, 2025

Now, SSH port forwarding works via accessing guest IP directly.
iperf3 comparison:

edited: benchmark result is updated at #4175 (comment), following details are obsoleted.

- SSH port forwarding over direct IP ```console $ iperf3 -c 127.0.0.1 Connecting to host 127.0.0.1, port 5201 [ 5] local 127.0.0.1 port 62915 connected to 127.0.0.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 518 MBytes 4.32 Gbits/sec ... [ 5] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 6.07 GBytes 5.22 Gbits/sec sender [ 5] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec receiver

iperf Done.

- Direct IP
```console
$ iperf3 -c 192.168.64.3
Connecting to host 192.168.64.3, port 5201
[  5] local 192.168.64.1 port 62913 connected to 192.168.64.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  2.76 GBytes  23.6 Gbits/sec                  
...
[  5]   9.01-10.01  sec  2.82 GBytes  24.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  28.1 GBytes  24.1 Gbits/sec                  sender
[  5]   0.00-9.99   sec  28.1 GBytes  24.1 Gbits/sec                  receiver

iperf Done.
  • SSH port forwarding over VSOCK
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 62929 connected to 127.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   294 MBytes  2.46 Gbits/sec                  
...
[  5]   9.00-10.01  sec   301 MBytes  2.52 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  2.94 GBytes  2.52 Gbits/sec                  sender
[  5]   0.00-10.14  sec  2.94 GBytes  2.49 Gbits/sec                  receiver

iperf Done.

@norio-nomura norio-nomura marked this pull request as ready for review October 11, 2025 08:59
@norio-nomura
Copy link
Contributor Author

norio-nomura commented Oct 11, 2025

  • port forward via socat

edited: benchmark result is updated at #4175 (comment), following details are obsoleted.

$ socat TCP-LISTEN:5202,fork TCP:192.168.64.3:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[  5] local 127.0.0.1 port 63300 connected to 127.0.0.1 port 5202
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.14 GBytes  9.72 Gbits/sec                  
...
[  5]   9.01-10.01  sec  1.16 GBytes  9.95 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  11.6 GBytes  9.93 Gbits/sec                  sender
[  5]   0.00-10.01  sec  11.6 GBytes  9.93 Gbits/sec                  receiver

iperf Done.
2025/10/11 18:30:02 socat[15240] E write(5, 0x78f040000, 8192): Broken pipe

Will limactl also be able to go this far?

@norio-nomura norio-nomura changed the title [WIP] Use the VM’s IP address on the same subnet as the host OS. Use the VM’s IP address on the same subnet as the host OS. Oct 11, 2025
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 2 times, most recently from 83ede84 to cc08d05 Compare October 12, 2025 04:33
@norio-nomura
Copy link
Contributor Author

iperf3 comparison

Access method sender receiver
GRPC forwarder 2.88 2.89
SSH forwarder 2.31 2.31
SSH forwarder over VSOCK 3.67 3.66
[New] SSH fwrdr via vzNAT 8.76 8.75
[New] Forwarder via vzNAT 22.0 22.0
(e.g.) socat via vzNAT 8.93 8.93
(e.g.) Direct IP via vzNAT 40.6 40.6
Gbits/sec
How to measure

Setup

MacBook Pro 14 inch, 2023
cpu: Apple M2 Pro
mem: 16GB

$ sw_vers
ProductName:		macOS
ProductVersion:		26.0.1
BuildVersion:		25A362
$ limactl --version
limactl version 2.0.0-alpha.2-89-gcc08d051
$ limactl start template://ubuntu --name test --containerd=none --log-level error
$ yes|limactl shell test DEBIAN_FRONTEND=noninteractive sudo apt-get -U install -qqq -y iperf3 &>/dev/null

GRPC port forwarder

$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57887 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.36 GBytes  2.88 Gbits/sec                  sender
[  5]   0.00-9.97   sec  3.36 GBytes  2.89 Gbits/sec                  receiver

iperf Done.

SSH port forwarder

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57958 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec                  sender
[  5]   0.00-10.01  sec  2.69 GBytes  2.31 Gbits/sec                  receiver

iperf Done.

SSH port forwarder over VSOCK

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=true limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 57970 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.27 GBytes  3.67 Gbits/sec                  sender
[  5]   0.00-10.00  sec  4.27 GBytes  3.66 Gbits/sec                  receiver

iperf Done.

[NEW] SSH port forwarder via vzNAT

$ limactl stop test --log-level error
$ LIMA_SSH_PORT_FORWARDER=true limactl start test --log-level error --network vzNAT
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 58015 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.2 GBytes  8.76 Gbits/sec                  sender
[  5]   0.00-10.00  sec  10.2 GBytes  8.75 Gbits/sec                  receiver

iperf Done.

[NEW] Port forwarder via vzNAT

$ limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[  5] local 127.0.0.1 port 58025 connected to 127.0.0.1 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  25.7 GBytes  22.0 Gbits/sec                  sender
[  5]   0.00-10.00  sec  25.6 GBytes  22.0 Gbits/sec                  receiver

iperf Done.

(e.g.)P socat via vzNAT

$ socat TCP-LISTEN:5202,fork TCP:192.168.64.2:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[  5] local 127.0.0.1 port 58136 connected to 127.0.0.1 port 5202
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.4 GBytes  8.93 Gbits/sec                  sender
[  5]   0.00-10.00  sec  10.4 GBytes  8.93 Gbits/sec                  receiver

iperf Done.
2025/10/12 14:54:15 socat[24552] E write(5, 0x8dac10000, 8192): Broken pipe

(e.g.) Direct IP via vzNAT

$ limactl list test
NAME    STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
test    Running    192.168.64.2:22    vz        aarch64    4       4GiB      100GiB    ~/.lima/test
$ iperf3 -c 192.168.64.2
Connecting to host 192.168.64.2, port 5201
[  5] local 192.168.64.1 port 58127 connected to 192.168.64.2 port 5201
...
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  47.3 GBytes  40.6 Gbits/sec                  sender
[  5]   0.00-10.01  sec  47.3 GBytes  40.6 Gbits/sec                  receiver

iperf Done.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from cc08d05 to 4954852 Compare October 12, 2025 06:41
@norio-nomura
Copy link
Contributor Author

It seems that #3715 is related to the failure of the test using w3m in CI, but I want to know what to do.

@norio-nomura
Copy link
Contributor Author

Some tests failed because that requires forwarding port on 127.0.0.1 in the guest.
That cannot be supported by port forwarding to direct IP on the guest.

@norio-nomura
Copy link
Contributor Author

It seems that #3715 is related to the failure of the test using w3m in CI, but I want to know what to do.

Fixed by reverted to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy"

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from c81c411 to 9fd03ae Compare October 13, 2025 05:26
@norio-nomura
Copy link
Contributor Author

Some tests failed because that requires forwarding port on 127.0.0.1 in the guest. That cannot be supported by port forwarding to direct IP on the guest.

Changed to check if guest IP is a direct accessible IP or unspecified when using port forwarding to a direct IP.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 4 times, most recently from 29db5b5 to f22a2f0 Compare October 14, 2025 01:14
run: brew install bash coreutils w3m socat
- name: Uninstall qemu
run: brew uninstall --ignore-dependencies --force qemu
- name: "Allow limactl to firewall"
Copy link
Member

@AkihiroSuda AkihiroSuda Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the reason as a comment line.
Is this needed for all the port forwards, or just for a particular test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, it's test.
Changed to draft.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't resolve error on https://github.com/lima-vm/lima/actions/runs/18483981814/job/52663789633#step:11:532
The failure was recorded in ha.stderr.log as:

2025/10/14 03:06:50 tcpproxy: for incoming conn 127.0.0.1:49276, error dialing "0.0.0.0:3022": dial tcp 192.168.65.1:0->192.168.65.2:3022: connect: no route to host

I guessed "no route to host" was caused by firewall, but it seems to be different.
🤔

@norio-nomura norio-nomura marked this pull request as draft October 14, 2025 03:17
@AkihiroSuda
Copy link
Member

[New] Forwarder via vzNAT 22.0 22.0

Is this gRPC forwarder?

@norio-nomura
Copy link
Contributor Author

Is this gRPC forwarder?

No, it's using tcpproxy.DialProxy with DialContextToGuestIP as DialContext.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 2 times, most recently from 805e32c to f34aebb Compare October 15, 2025 00:34
@norio-nomura
Copy link
Contributor Author

I can't reproduce "connect: no route to host" errors on dialing with local Macs that are mentioned in #4175 (comment)
The test-port-forwarding.pl test passed on local Macs:

  • macOS 26.0.1 with Apple M2
  • macOS 13.7.8 with Intel

Firewall state seems not to affect the errors.

@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from f34aebb to dbe02f0 Compare October 15, 2025 10:14
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch 2 times, most recently from 47bef26 to cd577d0 Compare October 24, 2025 05:31
@AkihiroSuda
Copy link
Member

Is this still draft ?

@norio-nomura norio-nomura marked this pull request as ready for review October 27, 2025 07:17
### `_LIMA_DIRECT_IP_PORT_FORWARDER`

- **Description**: Specifies to use direct IP port forwarding instead of gRPC.
- **Default**: `false`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this planned to be true in future? (That's why the variable starts with an underscore?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be marked as an experimental ?


### Direct IP Port Forwarding

To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be marked as an experimental ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please document the version that supports this feature


### Direct IP Port Forwarding

To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before explaining how to "enable direct IP port forwarding", the document should explain what it is, w/ pros and cons

LimaVersion string `json:"limaVersion"`
Param map[string]string `json:"param,omitempty"`
// Guest IP address directly accessible from the host.
GuestIP net.IP `json:"guestIP,omitempty"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just call it IP ?

Suggested change
GuestIP net.IP `json:"guestIP,omitempty"`
IP net.IP `json:"ip,omitempty"`

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add HostAgent.HostIP to avoid HostAgent using 0.0.0.0 on binding Listener in the coming PR.
In that PR, Instance.HostIP will not be required, but I think it should keep this name for consistency.

uses: ./.github/actions/upload_failure_logs_if_exists
with:
suffix: ${{ matrix.template }}
suffix: ${{ matrix.template }}${{ matrix.create_arg }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs matrix.additional_env too?

@norio-nomura
Copy link
Contributor Author

I'll rewrite to remove HostAgent.detectGuestIfnameOnSameSubnetAtHost().

@norio-nomura norio-nomura marked this pull request as draft October 29, 2025 05:58
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from cd577d0 to 759b2ad Compare October 29, 2025 06:40
@norio-nomura
Copy link
Contributor Author

I'll rewrite to remove HostAgent.detectGuestIfnameOnSameSubnetAtHost().

done

@norio-nomura norio-nomura marked this pull request as ready for review October 29, 2025 06:40
@AkihiroSuda
Copy link
Member

Thanks, LGTM after updating the docs

}
var dialContext func(ctx context.Context, network string, guestAddress string) (net.Conn, error)
if useDirectIPPortForwarding {
logrus.Warn("Direct IP Port forwarding is enabled. It may fall back to GRPC Port Forwarding in some cases.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also support falling back to SSH ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we should rather remove the env var and define the YAML to specify the candidates of the forwarder implementation per protocols (TCP, UDP, ...)

portForwarderTypes:
  tcp: [direct, ssh]
  udp: none

Similar to:

Probably we have no time to cover this in v2.0, so let me postpone this PR to v2.1+, sorry

@AkihiroSuda AkihiroSuda modified the milestones: v2.0.0, v2.1.0 (?) Oct 30, 2025
…ost OS.

Guest IP address will be detected in requirement process.

`hostagent.go`:
- Add fields to `HostAgent`
  - `guestIfnameOnSameSubnetAsHost string`
  - `guestIPv4 net.IP`
  - `guestIPv6 net.IP`
- Add methods `GuestIP()`, `GuestIPv4()`, and `GuestIPv6()` to `HostAgent`
- Add `HostAgent.WriteSSHConfigFile()` helper to write SSHConfigFile
- Add `HostAgent.sshAddressPort()` helper to provide ipAddress and port for SSH

`requirements.go`:
- Add `stdoutParser func(string) error` field to `requirement`
- Add `HostAgent.detectGuestIfnameOnSameSubnetAtHost()` to parse `ip -j neighbor` command output
- Add `HostAgent.detectGuestIPAddress()` to parse `ip -j addr` command output
- Add two requirements to `HostAgent.essentialRequirements()`
  - "detect guest interface on same subnet as the host"
  - "detect guest IPv4 address"
  - "detect guest IPv6 address"

Signed-off-by: Norio Nomura <[email protected]>

# Conflicts:
#	pkg/hostagent/hostagent.go

pkg/hostagent: Remove `HostAgent.detectGuestIfnameOnSameSubnetAtHost()`

Signed-off-by: Norio Nomura <[email protected]>
Update to use guestIPAddress:
- `limactl shell`
- `limactl show-ssh`
- `limactl tunnel`

pkg/limatype:
- Add `GuestIP net.IP` to `Instance`
- Add `Instance.SSHAddressPort()` helper to provide ipAddress and port for SSH

Signed-off-by: Norio Nomura <[email protected]>
Support printing "Guest IP Address: %s" on `limactl start`

Signed-off-by: Norio Nomura <[email protected]>

# Conflicts:
#	pkg/hostagent/requirements.go
Change to use `github.com/inetaf/tcpproxy`:
- pkg/driver/vz/vsock_forwarder.go
- pkg/portfwd/client.go
- pkg/portfwdserver/server.go

pkg/portfwd:
- Use `dialContext` instead of `client` to use `net.Conn` other than `GrpcClientRW`.
- Change to create proxies from `dialContext` parameters instead of `conn`.
- Add `DialContextToGRPCTunnel()` to return `DialContext` function that connects to GRPC tunnel.

pkg/hostagent:
- Add `HostAgent.DialContextToGuestIP()` to return `DialContext` function that connects to the guest IP directly.
  If the guest IP is not known, it returns nil.
- Prefer `HostAgent.DialContextToGuestIP()` over `portfwd.DialContextToGRPCTunnel()`.

Signed-off-by: Norio Nomura <[email protected]>

pkg/hostagent: Use `HostAgent.DialContextToGuestIP()` if the IP is accessible directly.

Signed-off-by: Norio Nomura <[email protected]>

Revert to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy"

Signed-off-by: Norio Nomura <[email protected]>

# Conflicts:
#	pkg/hostagent/hostagent.go

pkg/hostagent: Aware forwarding guest address is IPv4 or IPv6

Signed-off-by: Norio Nomura <[email protected]>

pkg/hostagent: Fallback to GRPC forwarder if dialing to a direct ip fails

Signed-off-by: Norio Nomura <[email protected]>

# Conflicts:
#	pkg/hostagent/hostagent.go
To following tests will be run with enabled.

Signed-off-by: Norio Nomura <[email protected]>
…ariable to enable direct IP port forwarding to the VM.

### Direct IP Port Forwarding

To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`:

```bash
export _LIMA_DIRECT_IP_PORT_FORWARDER=true
```
This feature makes Lima to use direct IP port forwarding instead of gRPC port forwarding.
When this feature is enabled, Lima tries to connect to the guest's IP address directly for port forwarding.

#### Fallback to gRPC Port Forwarding
Lima may fall back to gRPC port forwarding in the following cases:
- If the guest's IP address is not available, Lima falls back to gRPC port forwarding.
- If the guest's IP address is available but the connection to the guest's IP address fails, Lima also falls back to gRPC port forwarding.
- If the guest's IP address is not accessible from the host (e.g. localhost on guest), Lima falls back to gRPC port forwarding as well.

Signed-off-by: Norio Nomura <[email protected]>
@norio-nomura norio-nomura force-pushed the Detect-the-VMs-IP-address-on-the-same-subnet-as-the-host-OS branch from 759b2ad to 7bb3eb4 Compare October 30, 2025 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report VM IP addresses to host agent

2 participants