File descriptors leak in undetected (UC) mode #3679
Replies: 7 comments 7 replies
-
|
Do you know how to find specifically the code where the file descriptors leaks occur? The specific locations would be helpful. Much of UC Mode / CDP Mode is based on |
Beta Was this translation helpful? Give feedback.
-
|
Hello @mdmintz, Thank you for your response. I spent several hours debugging and have identified a number of issues: 1. Session-based requests Original SeleniumBase/seleniumbase/undetected/cdp.py Line 104 in b9a89f7 Updated You can review all of my changes here: With these adjustments, the leak in uc_open_with_reconnect no longer occurs (even without invoking the garbage collector), and the number of leaked file descriptors in activate_cdp_mode is greatly reduced. 2. Remaining leak in activate_cdp_mode It appears that each of these functions establishes its own socket connection, which is never closed. Moreover, I have confirmed that the corresponding stop method (at line 569) is never invoked, which I believe is the underlying cause of the remaining leak:
FD=6 and FD=7 are the two socks that do not close, and this is the trace of when each is called |
Beta Was this translation helpful? Give feedback.
-
|
CDP Mode memory improvements will have to be figured out later. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @mdmintz, Thank you for the update. After running some additional tests I’ve traced the file‑descriptor leak to the Connection class. Although an aclose() coroutine exists, it’s never invoked. The aexit() method would normally trigger aclose(), but it only executes when Connection is used as an async context manager—which isn’t happening in the current workflow. To prevent the leak it will need to make sure every connection is explicitly closed, either by:
|
Beta Was this translation helpful? Give feedback.
-
|
Fixed in |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again for your help on this, @felipehertzer! |
Beta Was this translation helpful? Give feedback.
-
|
That is great, it seems totally fixed, Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @mdmintz, While deploying to production today I noticed that we’re still seeing a connection leak on the CDP side. It appears to originate around From what I can tell, the flow calls start() and then get(), which opens two separate WebSocket connections. The recent patch only closes the most recently opened socket via sb_config._cdp_aclose = self.aclose, leaving the first one dangling.for i in range(1, 4):
print(f"\nIteration {i}")
with SB(
uc=True,
) as browser:
browser.activate_cdp_mode("https://bloomberg.com/news/articles/2024-12-05/barclays-promotes-116-staffers-across-bank-to-managing-director")
browser.sleep(5)
browser.cdp.get_page_source()
inspect_fds() |
Beta Was this translation helpful? Give feedback.
-
|
@felipehertzer Did you run on import shutil
import os
import subprocess
import psutil
from seleniumbase import SB
# from seleniumbase import config as sb_config
def get_open_fds():
"""
Returns the number of open file descriptors for the current process.
"""
lsof_path = shutil.which("lsof")
if lsof_path is None:
raise NotImplementedError("lsof is not available on this system.")
try:
process = subprocess.run(
[lsof_path, "-w", "-Ff", "-p", str(os.getpid())],
capture_output=True,
text=True,
check=True,
)
output = process.stdout
fds = [
line for line in output.split(os.linesep)
if line.startswith("f") and line[1:].isdigit()
]
return len(fds)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Error executing lsof: {e}") from e
def inspect_fds():
p = psutil.Process()
# print(f"Count FD (psutil): {p.num_fds()}")
connections = p.net_connections()
if connections:
# print("Open connections:")
for conn in connections:
print(f" {conn}")
if __name__ == "__main__":
print("Start")
inspect_fds()
# print('Browser Running')
# print(get_open_fds())
for i in range(1, 4):
print(f"\nIteration {i}")
with SB(uc=True) as browser:
browser.activate_cdp_mode("https://example.com")
# browser.uc_open_with_reconnect("https://example.com", 1)
# print(get_open_fds())
inspect_fds()If you remove the following line, then the socket connections remain open: SeleniumBase/seleniumbase/plugins/sb_manager.py Line 1375 in 94b3fa6 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I am using the latest version. Interestingly, the issue only occurs when I add these lines. I not sure if it has anything to do with the time spent to render the website. browser.sleep(5) |
Beta Was this translation helpful? Give feedback.
-
|
Not sure, but just sleeping and getting the page source isn't causing any issue: import shutil
import os
import subprocess
import psutil
from seleniumbase import SB
# from seleniumbase import config as sb_config
def get_open_fds():
"""
Returns the number of open file descriptors for the current process.
"""
lsof_path = shutil.which("lsof")
if lsof_path is None:
raise NotImplementedError("lsof is not available on this system.")
try:
process = subprocess.run(
[lsof_path, "-w", "-Ff", "-p", str(os.getpid())],
capture_output=True,
text=True,
check=True,
)
output = process.stdout
fds = [
line for line in output.split(os.linesep)
if line.startswith("f") and line[1:].isdigit()
]
return len(fds)
except subprocess.CalledProcessError as e:
raise RuntimeError(f"Error executing lsof: {e}") from e
def inspect_fds():
p = psutil.Process()
# print(f"Count FD (psutil): {p.num_fds()}")
connections = p.net_connections()
if connections:
# print("Open connections:")
for conn in connections:
print(f" {conn}")
if __name__ == "__main__":
print("Start")
inspect_fds()
# print('Browser Running')
# print(get_open_fds())
for i in range(1, 4):
print(f"\nIteration {i}")
with SB(uc=True) as sb:
sb.activate_cdp_mode("https://example.com")
sb.sleep(1)
source = sb.get_page_source()
# sb.uc_open_with_reconnect("https://example.com", 1)
# print(get_open_fds())
inspect_fds() |
Beta Was this translation helpful? Give feedback.
-
|
Hey @mdmintz, I apologise for the delayed response. I was only able to replicate this issue with heavy websites as Bloomberg and WSJ, because they use a lot of iframes and background services. I’ve completed my investigation and implemented the following fixes:
With these changes, both the connection and FD leaks no longer occur in my tests. Replicate:Add a print after this line: print("aopen", self.websocket_url)And another after this line print("aclose", self.websocket_url)Output: Add this print after this line print(f"Tab: {websocket_url}")Output: AFTER THE FIX:My branch: master...felipehertzer:SeleniumBase:master |
Beta Was this translation helpful? Give feedback.
-
|
That creates a new bug with async errors. Here's the sample code that leads to the bug: from seleniumbase import SB
with SB(uc=True, test=True) as sb:
url1 = "https://seleniumbase.io/demo_page"
sb.activate_cdp_mode(url1)
driver1 = sb.driver
url2 = "https://seleniumbase.io/coffee/"
driver2 = sb.get_new_driver(undetectable=True)
sb.activate_cdp_mode(url2)
print(driver1.get_current_url())
print(driver2.get_current_url())
sb.switch_to_default_driver()
sb.assert_url_contains("demo_page")
print(sb.get_current_url())
sb.switch_to_driver(driver2)
sb.assert_url_contains("coffee")
print(sb.get_current_url()) |
Beta Was this translation helpful? Give feedback.
-
|
Hey, It appears the issue stems from a missing closure of the page socks on driver 1. Specifically, the connection isn’t being properly closed, and the loop continues listening. I’ll look into a reliable way to ensure that the page socket is always closed correctly. Let me know if you have any ideas or suggestions. Thanks! |
Beta Was this translation helpful? Give feedback.
-
Hi,
I’m seeing a steady increase in open file descriptors whenever I activate UC mode (uc=True), leading eventually to:
When UC mode is off the FD count remains flat.
When I use uc_open_with_reconnect and force a gc.collect() after teardown, the leak stops.
Below are three minimal repros—please let me know what else I can provide.
CDP
This script repeatedly opens a browser, calls activate_cdp_mode(...), then quits. You can see the number of FDs jump by ~8–10 each time, and connections in CLOSE_WAIT remain.
Output:
UC without GC
Here we use uc_open_with_reconnect(...) instead of CDP. The leak still occurs, growing by ~1 FD per iteration.
Output:
UC using GC
Adding a manual garbage collection after each teardown appears to reclaim all sockets—FD count stays constant. But it only works for uc_open_with_reconnect.
Output:
Environment:
SeleniumBase version: e.g. 4.37.2
Python version: e.g. 3.13
OS: MacOS and Linux
Chrome: 135.0.7049.85
Chromedriver: matching Chrome
Beta Was this translation helpful? Give feedback.
All reactions