-
Notifications
You must be signed in to change notification settings - Fork 716
Use the VM’s IP address on the same subnet as the host OS. #4175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Use the VM’s IP address on the same subnet as the host OS. #4175
Conversation
9c6d7cb to
912ed32
Compare
89e8a70 to
dee7e4e
Compare
|
Now, SSH port forwarding works via accessing guest IP directly. edited: benchmark result is updated at #4175 (comment), following details are obsoleted.
- SSH port forwarding over direct IP
```console
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 62915 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 518 MBytes 4.32 Gbits/sec
...
[ 5] 9.00-10.00 sec 650 MBytes 5.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 6.07 GBytes 5.22 Gbits/sec sender
[ 5] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec receiver
iperf Done.
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 62929 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 294 MBytes 2.46 Gbits/sec
...
[ 5] 9.00-10.01 sec 301 MBytes 2.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 2.94 GBytes 2.52 Gbits/sec sender
[ 5] 0.00-10.14 sec 2.94 GBytes 2.49 Gbits/sec receiver
iperf Done. |
edited: benchmark result is updated at #4175 (comment), following details are obsoleted. $ socat TCP-LISTEN:5202,fork TCP:192.168.64.3:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[ 5] local 127.0.0.1 port 63300 connected to 127.0.0.1 port 5202
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.14 GBytes 9.72 Gbits/sec
...
[ 5] 9.01-10.01 sec 1.16 GBytes 9.95 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 11.6 GBytes 9.93 Gbits/sec sender
[ 5] 0.00-10.01 sec 11.6 GBytes 9.93 Gbits/sec receiver
iperf Done.
2025/10/11 18:30:02 socat[15240] E write(5, 0x78f040000, 8192): Broken pipeWill |
83ede84 to
cc08d05
Compare
|
| Access method | sender | receiver |
|---|---|---|
| GRPC forwarder | 2.88 | 2.89 |
| SSH forwarder | 2.31 | 2.31 |
| SSH forwarder over VSOCK | 3.67 | 3.66 |
| [New] SSH fwrdr via vzNAT | 8.76 | 8.75 |
| [New] Forwarder via vzNAT | 22.0 | 22.0 |
(e.g.) socat via vzNAT |
8.93 | 8.93 |
| (e.g.) Direct IP via vzNAT | 40.6 | 40.6 |
| Gbits/sec |
How to measure
Setup
MacBook Pro 14 inch, 2023
cpu: Apple M2 Pro
mem: 16GB
$ sw_vers
ProductName: macOS
ProductVersion: 26.0.1
BuildVersion: 25A362
$ limactl --version
limactl version 2.0.0-alpha.2-89-gcc08d051
$ limactl start template://ubuntu --name test --containerd=none --log-level error
$ yes|limactl shell test DEBIAN_FRONTEND=noninteractive sudo apt-get -U install -qqq -y iperf3 &>/dev/nullGRPC port forwarder
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57887 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.36 GBytes 2.88 Gbits/sec sender
[ 5] 0.00-9.97 sec 3.36 GBytes 2.89 Gbits/sec receiver
iperf Done.SSH port forwarder
$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57958 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec sender
[ 5] 0.00-10.01 sec 2.69 GBytes 2.31 Gbits/sec receiver
iperf Done.SSH port forwarder over VSOCK
$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=true limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 57970 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 4.27 GBytes 3.67 Gbits/sec sender
[ 5] 0.00-10.00 sec 4.27 GBytes 3.66 Gbits/sec receiver
iperf Done.[NEW] SSH port forwarder via vzNAT
$ limactl stop test --log-level error
$ LIMA_SSH_PORT_FORWARDER=true limactl start test --log-level error --network vzNAT
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 58015 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.2 GBytes 8.76 Gbits/sec sender
[ 5] 0.00-10.00 sec 10.2 GBytes 8.75 Gbits/sec receiver
iperf Done.[NEW] Port forwarder via vzNAT
$ limactl restart test --log-level error
$ iperf3 -c 127.0.0.1
Connecting to host 127.0.0.1, port 5201
[ 5] local 127.0.0.1 port 58025 connected to 127.0.0.1 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 25.7 GBytes 22.0 Gbits/sec sender
[ 5] 0.00-10.00 sec 25.6 GBytes 22.0 Gbits/sec receiver
iperf Done.(e.g.)P socat via vzNAT
$ socat TCP-LISTEN:5202,fork TCP:192.168.64.2:5201&
$ iperf3 -c 127.0.0.1 -p 5202
Connecting to host 127.0.0.1, port 5202
[ 5] local 127.0.0.1 port 58136 connected to 127.0.0.1 port 5202
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec sender
[ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec receiver
iperf Done.
2025/10/12 14:54:15 socat[24552] E write(5, 0x8dac10000, 8192): Broken pipe(e.g.) Direct IP via vzNAT
$ limactl list test
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
test Running 192.168.64.2:22 vz aarch64 4 4GiB 100GiB ~/.lima/test
$ iperf3 -c 192.168.64.2
Connecting to host 192.168.64.2, port 5201
[ 5] local 192.168.64.1 port 58127 connected to 192.168.64.2 port 5201
...
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 47.3 GBytes 40.6 Gbits/sec sender
[ 5] 0.00-10.01 sec 47.3 GBytes 40.6 Gbits/sec receiver
iperf Done.cc08d05 to
4954852
Compare
|
It seems that #3715 is related to the failure of the test using |
|
Some tests failed because that requires forwarding port on 127.0.0.1 in the guest. |
Fixed by reverted to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy" |
c81c411 to
9fd03ae
Compare
Changed to check if guest IP is a direct accessible IP or unspecified when using port forwarding to a direct IP. |
29db5b5 to
f22a2f0
Compare
.github/workflows/test.yml
Outdated
| run: brew install bash coreutils w3m socat | ||
| - name: Uninstall qemu | ||
| run: brew uninstall --ignore-dependencies --force qemu | ||
| - name: "Allow limactl to firewall" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add the reason as a comment line.
Is this needed for all the port forwards, or just for a particular test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, it's test.
Changed to draft.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't resolve error on https://github.com/lima-vm/lima/actions/runs/18483981814/job/52663789633#step:11:532
The failure was recorded in ha.stderr.log as:
2025/10/14 03:06:50 tcpproxy: for incoming conn 127.0.0.1:49276, error dialing "0.0.0.0:3022": dial tcp 192.168.65.1:0->192.168.65.2:3022: connect: no route to host
I guessed "no route to host" was caused by firewall, but it seems to be different.
🤔
Is this gRPC forwarder? |
No, it's using |
805e32c to
f34aebb
Compare
|
I can't reproduce "connect: no route to host" errors on dialing with local Macs that are mentioned in #4175 (comment)
Firewall state seems not to affect the errors. |
f34aebb to
dbe02f0
Compare
47bef26 to
cd577d0
Compare
|
Is this still draft ? |
| ### `_LIMA_DIRECT_IP_PORT_FORWARDER` | ||
|
|
||
| - **Description**: Specifies to use direct IP port forwarding instead of gRPC. | ||
| - **Default**: `false` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this planned to be true in future? (That's why the variable starts with an underscore?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be marked as an experimental ?
|
|
||
| ### Direct IP Port Forwarding | ||
|
|
||
| To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be marked as an experimental ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also please document the version that supports this feature
|
|
||
| ### Direct IP Port Forwarding | ||
|
|
||
| To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before explaining how to "enable direct IP port forwarding", the document should explain what it is, w/ pros and cons
| LimaVersion string `json:"limaVersion"` | ||
| Param map[string]string `json:"param,omitempty"` | ||
| // Guest IP address directly accessible from the host. | ||
| GuestIP net.IP `json:"guestIP,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe just call it IP ?
| GuestIP net.IP `json:"guestIP,omitempty"` | |
| IP net.IP `json:"ip,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add HostAgent.HostIP to avoid HostAgent using 0.0.0.0 on binding Listener in the coming PR.
In that PR, Instance.HostIP will not be required, but I think it should keep this name for consistency.
| uses: ./.github/actions/upload_failure_logs_if_exists | ||
| with: | ||
| suffix: ${{ matrix.template }} | ||
| suffix: ${{ matrix.template }}${{ matrix.create_arg }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needs matrix.additional_env too?
|
I'll rewrite to remove |
cd577d0 to
759b2ad
Compare
done |
|
Thanks, LGTM after updating the docs |
| } | ||
| var dialContext func(ctx context.Context, network string, guestAddress string) (net.Conn, error) | ||
| if useDirectIPPortForwarding { | ||
| logrus.Warn("Direct IP Port forwarding is enabled. It may fall back to GRPC Port Forwarding in some cases.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should also support falling back to SSH ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably we should rather remove the env var and define the YAML to specify the candidates of the forwarder implementation per protocols (TCP, UDP, ...)
portForwarderTypes:
tcp: [direct, ssh]
udp: noneSimilar to:
- Implement "Dual" port forwarder (SSH for TCP, GRPC for UDP) #4210 (except the default values)
Probably we have no time to cover this in v2.0, so let me postpone this PR to v2.1+, sorry
…ost OS. Guest IP address will be detected in requirement process. `hostagent.go`: - Add fields to `HostAgent` - `guestIfnameOnSameSubnetAsHost string` - `guestIPv4 net.IP` - `guestIPv6 net.IP` - Add methods `GuestIP()`, `GuestIPv4()`, and `GuestIPv6()` to `HostAgent` - Add `HostAgent.WriteSSHConfigFile()` helper to write SSHConfigFile - Add `HostAgent.sshAddressPort()` helper to provide ipAddress and port for SSH `requirements.go`: - Add `stdoutParser func(string) error` field to `requirement` - Add `HostAgent.detectGuestIfnameOnSameSubnetAtHost()` to parse `ip -j neighbor` command output - Add `HostAgent.detectGuestIPAddress()` to parse `ip -j addr` command output - Add two requirements to `HostAgent.essentialRequirements()` - "detect guest interface on same subnet as the host" - "detect guest IPv4 address" - "detect guest IPv6 address" Signed-off-by: Norio Nomura <[email protected]> # Conflicts: # pkg/hostagent/hostagent.go pkg/hostagent: Remove `HostAgent.detectGuestIfnameOnSameSubnetAtHost()` Signed-off-by: Norio Nomura <[email protected]>
Update to use guestIPAddress: - `limactl shell` - `limactl show-ssh` - `limactl tunnel` pkg/limatype: - Add `GuestIP net.IP` to `Instance` - Add `Instance.SSHAddressPort()` helper to provide ipAddress and port for SSH Signed-off-by: Norio Nomura <[email protected]>
Support printing "Guest IP Address: %s" on `limactl start` Signed-off-by: Norio Nomura <[email protected]> # Conflicts: # pkg/hostagent/requirements.go
Change to use `github.com/inetaf/tcpproxy`: - pkg/driver/vz/vsock_forwarder.go - pkg/portfwd/client.go - pkg/portfwdserver/server.go pkg/portfwd: - Use `dialContext` instead of `client` to use `net.Conn` other than `GrpcClientRW`. - Change to create proxies from `dialContext` parameters instead of `conn`. - Add `DialContextToGRPCTunnel()` to return `DialContext` function that connects to GRPC tunnel. pkg/hostagent: - Add `HostAgent.DialContextToGuestIP()` to return `DialContext` function that connects to the guest IP directly. If the guest IP is not known, it returns nil. - Prefer `HostAgent.DialContextToGuestIP()` over `portfwd.DialContextToGRPCTunnel()`. Signed-off-by: Norio Nomura <[email protected]> pkg/hostagent: Use `HostAgent.DialContextToGuestIP()` if the IP is accessible directly. Signed-off-by: Norio Nomura <[email protected]> Revert to "github.com/containers/gvisor-tap-vsock/pkg/tcpproxy" Signed-off-by: Norio Nomura <[email protected]> # Conflicts: # pkg/hostagent/hostagent.go pkg/hostagent: Aware forwarding guest address is IPv4 or IPv6 Signed-off-by: Norio Nomura <[email protected]> pkg/hostagent: Fallback to GRPC forwarder if dialing to a direct ip fails Signed-off-by: Norio Nomura <[email protected]> # Conflicts: # pkg/hostagent/hostagent.go
…work=vzNAT` Signed-off-by: Norio Nomura <[email protected]>
Signed-off-by: Norio Nomura <[email protected]>
To following tests will be run with enabled. Signed-off-by: Norio Nomura <[email protected]>
…rt-forwarding.pl` Signed-off-by: Norio Nomura <[email protected]>
…ariable to enable direct IP port forwarding to the VM. ### Direct IP Port Forwarding To enable direct IP port forwarding, set the `_LIMA_DIRECT_IP_PORT_FORWARDER` environment variable to `true`: ```bash export _LIMA_DIRECT_IP_PORT_FORWARDER=true ``` This feature makes Lima to use direct IP port forwarding instead of gRPC port forwarding. When this feature is enabled, Lima tries to connect to the guest's IP address directly for port forwarding. #### Fallback to gRPC Port Forwarding Lima may fall back to gRPC port forwarding in the following cases: - If the guest's IP address is not available, Lima falls back to gRPC port forwarding. - If the guest's IP address is available but the connection to the guest's IP address fails, Lima also falls back to gRPC port forwarding. - If the guest's IP address is not accessible from the host (e.g. localhost on guest), Lima falls back to gRPC port forwarding as well. Signed-off-by: Norio Nomura <[email protected]>
…sts other than macOS Signed-off-by: Norio Nomura <[email protected]>
759b2ad to
7bb3eb4
Compare
GuestIPAddressGuestIPfield toGET /v1/infoendpoint`GuestIPAddress stringGuestIP stringfield toStatuspkg/hostagent: Update allsshexecution to support SSH address other than "127.0.0.1"already merged as pkg/hostagent: Update all
sshexecution to support SSH address other than "127.0.0.1" #4241Support static port forwarding via direct access to guest IPdefault.yamlwith--network=vzNAT_LIMA_DIRECT_IP_PORT_FORWARDERenvironment variable to enable direct IP port forwarding to the VM