Skip to content

App crash when events.js throws "socket hang up" even when I have error handle #1790

Closed
@datvm

Description

@datvm

Describe the bug

So I usually receive this error while the script is running:

events.js:292
      throw er; // Unhandled 'error' event
      ^

Error: socket hang up
    at connResetException (internal/errors.js:607:14)
    at TLSSocket.socketOnEnd (_http_client.js:493:23)
    at TLSSocket.emit (events.js:327:22)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketOnEnd (_http_client.js:493:9)
    at TLSSocket.emit (events.js:327:22)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'ECONNRESET'
}

My app subscribe to the block event (provider.on("block", ...)). I followed #1053 to add reconnection ability but somehow the exception still went throw and the whole app crashes. I even added extra try/catch and on("error") handle (see below).

const EXPECTED_PONG_BACK = 15000
const KEEP_ALIVE_CHECK_INTERVAL = 7500

export class ConnectionService {

    public pro: ethers.providers.WebSocketProvider;
    public signer: ethers.Wallet;

    private blockSubscriptions: BlockCallback[] = [];

    private pingTimeout: NodeJS.Timeout = null;
    private keepAliveInterval: NodeJS.Timeout = null;

    constructor(
        private s: IScriptSetting,
        private l: LoggingService,
    ) {
        this.init();
    }

    init() {
        const [s, l] = [this.s, this.l];

        const pro = this.pro = new ethers.providers.WebSocketProvider(s.server);
        this.signer = new Wallet(s.accPri, pro);

        const ws = pro._websocket;

        ws.on("open", () => {
            l.info({
                message: "WebSocket connection opened",
                category: LogType.ScriptHealth
            });

            this.internalSubscribe();

            this.keepAliveInterval = setInterval(() => {
                ws.ping();

                this.pingTimeout = setTimeout(() => {
                    l.info({
                        message: "WebSocket Ping-Pong not received",
                        category: LogType.ScriptHealth,
                    });

                    ws.terminate();
                }, EXPECTED_PONG_BACK);
            }, KEEP_ALIVE_CHECK_INTERVAL);
        });

        ws.on("close", () => {
            this.resetConnection();
        });

        ws.on("pong", () => {
            clearTimeout(this.pingTimeout);
        });

        ws.on("error", (ex) => {
            l.error({
                message: "Connection Error",
                error: ex,
                category: LogType.ScriptHealth,
            });

            this.resetConnection();
        })
    }

    resetConnection() {
        this.l.info({
            message: "WebSocket connection closed",
            category: LogType.ScriptHealth
        });

        clearInterval(this.keepAliveInterval);
        clearTimeout(this.pingTimeout);

        this.init();
    }

    public subscribe(callback: BlockCallback): void {
        this.blockSubscriptions.push(callback);
    }

    private internalSubscribe() {
        try {
            this.pro.on("block", (blockNo: number) => {
                for (let sub of this.blockSubscriptions) {
                    sub(blockNo);
                }
            });
        } catch (e) {
            this.l.error({
                message: "Connection Error",
                error: e,
                category: LogType.ScriptHealth,
            });

            this.resetConnection();
        }
    }

}

export type BlockCallback = (blockNo: number) => any;

Environment:

NodeJS v14.16.1 on Windows 10 Pro and Windows Server 2019.

Node Client is latest stable Geth on Windows Server 2019.

Search Terms

WebSocket, reconnect, socket hang up

Please advice on how to fix this, and if possible is there a way that I can "simulate" a WS disconnect to test if my fix works? Thanks

Metadata

Metadata

Assignees

No one assigned

    Labels

    investigateUnder investigation and may be a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions