ResourceRequest timed out

Node: 6.9.1
configuration:

pool: {
    max: 100,
    min: 1,
    idleTimeoutMillis: 15000
}

I am getting this error when there is less traffic or there is gap between requests.

error:  { TimeoutError: ResourceRequest timed out
        at ResourceRequest._fireTimeout (/home/munir/node-workspace/server/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
        at Timeout.bound (/home/munir/node-workspace/server/node_modules/generic-pool/lib/ResourceRequest.js:8:15)
        at ontimeout (timers.js:365:14)
        at tryOnTimeout (timers.js:237:5)
        at Timer.listOnTimeout (timers.js:207:5) name: 'TimeoutError' }

+1, and its random.

Please any help?

If you're using transactions in your calls it seems to be an issue. Improper usage of them makes pool get stacked. Try to get rid off them.

@glebson1

@glebson1 Thanks for your reply. But, I am not getting how it is affecting to it?

@patriksimek

@patriksimek Can you give me any hint that can help?

I got a similar problem with ESOCKETTIMEOUT. It happened when I deploy my app and my docker install the latest version of tedious (v2.3.1, I think) as its dependency. When I ran my batch jobs which used quite a large and complex query, it made a lot of timeout which didn't happen before in the previous version. While I don't know how write reproduce steps without using the same database and query, My current solution is to force its dependency to install tedious (v2.1.1) using npm-shrinkwrap.json that I made from my local repo.

@gentlerainsky Thanks. I will try it.

@gentlerainsky I tried by using tedious (v1.x) But still getting resourceRequest timeout issue.

Sequelize fixed this bug. Please just reference their pull request. This is a huge issue that prevents normal operation!

sequelize/sequelize#8815

@catcher-in-the-try They did same thing which is already done by some developers in pending PR(coopernurse/node-pool#198 & coopernurse/node-pool#221) of generic-pool. So, looks like there is issue in generic-pool

Guys, Any updates in this? My production is heading to a big event today and I am in a big mess because of this issue.

I had a really hard time on my customer with this too. Managed to reduce the damage somehow with the config below, but its urgent to fix this one.
pool: { max: 100000, min: 1, idleTimeoutMillis: 50, evictionRunIntervalMillis: 5, softIdleTimeoutMillis: 5 }

@labs20

@labs20 Are you not facing more issue with this configuration? I think 50ms timeout will create this issue more frequently. Also, It will increase chances of this error.

Well, at least on my customer server it seems to have reduced the problem.

@labs20

@catcher-in-the-try @labs20 It is working fine for me after I used https://github.com/munir131/node-mssql

@munir131

@munir131 I couldn't figure out how to use, it doesn't seem to be documented or published to NPM.

Thanks, though I wasn't looking for an explanation on why poolCreate is called when it's called, i wanted to understand what is it that you fixed by adding that reject in there.

Cool, good luck.

@dsbert

@dsbert That's interesting. I haven't seen an ETIMEOUT error crash our Node services yet but I'll keep an eye out for that. Do you have a stack trace you can share? If there's a defect in my PR then would be keen to fix...

I am also facing same error frequently. So I have updated node-mssql version to v4.2.2. I'll keep an eye out for that.

There is a fix in tedious that would be nice to be taken into node-mssql to fix this issue: tediousjs/tedious#769

Ah sorry, i see it was included in 4.2.2

@dsbert

@dsbert That's interesting. I haven't seen an ETIMEOUT error crash our Node services yet but I'll keep an eye out for that. Do you have a stack trace you can share? If there's a defect in my PR then would be keen to fix...

I am also facing same error frequently. So I have updated node-mssql version to v4.2.2. I'll keep an eye out for that.

I updated node-mssql version to v4.2.2. But I am facing same error too. Connection timeout.

@LoveMeWithoutAll

@LoveMeWithoutAll Can you provide some code that reproduces the error, the config and also details of the environment / platform you are running on, please?

@LoveMeWithoutAll

@LoveMeWithoutAll Can you provide some code that reproduces the error, the config and also details of the environment / platform you are running on, please?

@dhensby I have encountered connection timeout error even if upgraded node-mssql to v4.2.2. I couldn't reproduce the error immediately. But after running 1~2 days, the error has occured. Getting connection every 1 second may raise the error? I'm speculating so. At now, I changed timing of getting connection to only once at a process lifetime, and keeping an eye out for that.

environment/platform

Windows Server 2008R2 standard
MSSQL: Microsoft SQL Server 2014 - 12.0.4100.1 (X64)
Apr 20 2015 17:29:27
Copyright (c) Microsoft Corporation
Enterprise Edition (64-bit) on Windows NT 6.3 (Build 9600: ) (Hypervisor)
Node.js v8.12.0
node-mssql v4.2.2

Error rasing code is below.

// ./config/dbConfig.json
{
  "user": "-",
  "password": "-",
  "server": "-",
  "port": 0,
  "database": "-",
  "options": {
    "encrypt": false
  },
  "pool": {
    "acquireTimeoutMillis": 15000
  }
}

// services/db.js
const sql = require("mssql")
const dbConfig = require("../config/dbConfig.json")
const { onDbError } = require("./log")
let connectionPool

sql.on("error", err => {
	sql.close()
	onDbError(err, "mssql error")
})

const getConnectionPool = async () => {
	try {
		connectionPool = await new sql.ConnectionPool(dbConfig).connect()
		connectionPool.on("error", async err => {
			await closeConnectionPool()
			onDbError(err, "getConnectionPool error")
		})
	} catch (err) {
		onDbError(err, "getConnectionPool error in catch")
	}
}

const closeConnectionPool = async () => {
	try {
		await connectionPool.close()
	} catch (err) {
		onDbError(err, "closeConnectionPool error")
	}
}

const getPushList = async () => {
	try {
		let result = await connectionPool.request()
			.input("GUBUN", sql.NVarChar, "ENGINE_PUSH")
			.input("CONDITION1", sql.NVarChar, "Android")
			.input("CONDITION2", sql.NVarChar, engineConfig.runningServer)
			.execute("b.dbo.TU_M_PUSH_XMLLIST")
		return result.recordset
	} catch (err) {
		onDbError(err, "getPushList")
		return []
	}
}

// ./server.js
// ...
const dbService = require("./services/db")
// ...
const sendPush = async () => {
	await dbService.getConnectionPool()
	dbService.getPushList()
		.then((msgList) => {
			fcmService.sendMsgList(msgList)
		})
		.catch(async (error) => {
			await dbService.closeConnectionPool()
			onServerError(error)
		})
		.finally(async () => {
			await  dbService.closeConnectionPool()
			setTimeout(() => {
				sendPush()
			}, 1000)
		})
}
// ...
sendPush()

error log is below

{ message: 'Error on getConnectionPool error in catch in db.js',
  level: 'error',
  timestamp: '2018-10-23 10:27:42',
  [Symbol(level)]: 'error',
  [Symbol(message)]: '{"message":"Error on getConnectionPool error in catch in db.js","level":"error","timestamp":"2018-10-23 10:27:42"}' }
{ ConnectionError: Failed to connect to 165.246.17.29:3361 in 15000ms
    at Connection.tedious.once.err (D:\push_engine\FcmPushEngine\node_modules\mssql\lib\tedious.js:237:17)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:116:13)
    at Connection.emit (events.js:211:7)
    at Connection.connectTimeout (D:\push_engine\FcmPushEngine\node_modules\tedious\lib\connection.js:944:12)
    at Timeout._onTimeout (D:\push_engine\FcmPushEngine\node_modules\tedious\lib\connection.js:913:16)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
  code: 'ETIMEOUT',
  originalError: 
   { ConnectionError: Failed to connect to 165.246.17.29:3361 in 15000ms
    at ConnectionError (D:\push_engine\FcmPushEngine\node_modules\tedious\lib\errors.js:12:12)
    at Connection.connectTimeout (D:\push_engine\FcmPushEngine\node_modules\tedious\lib\connection.js:944:28)
    at Timeout._onTimeout (D:\push_engine\FcmPushEngine\node_modules\tedious\lib\connection.js:913:16)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
     message: 'Failed to connect to 165.246.17.29:3361 in 15000ms',
     code: 'ETIMEOUT' },
  name: 'ConnectionError',
  level: 'error',
  timestamp: '2018-10-23 10:27:42',
  [Symbol(level)]: 'error',
  [Symbol(message)]: '{"code":"ETIMEOUT","originalError":{"message":"Failed to connect to 165.246.17.29:3361 in 15000ms","code":"ETIMEOUT"},"name":"ConnectionError","level":"error","timestamp":"2018-10-23 10:27:42"}' }

@LoveMeWithoutAll

@LoveMeWithoutAll Something about your getConnectionPoolcode feels weird because you're always reassigning connectionPool with a new pool rather than making use of a pool that may have already been created...

Can you try changing getConnectionPool to be something more like:

const getConnectionPool = async () => {
+	if (connectionPool) {
+ 		return connectionPool;
+	}
	try {
		connectionPool = await new sql.ConnectionPool(dbConfig).connect()
		connectionPool.on("error", async err => {
			await closeConnectionPool()
			onDbError(err, "getConnectionPool error")
		})
	} catch (err) {
		onDbError(err, "getConnectionPool error in catch")
	}
}

There's a possibility you're just creating a connection pool every time and eventually hitting the maximum number of connections allowed on the server and so the new pool never connects again.

Can you monitor the number of active connections on the server and see if it is constantly increasing or remains stable?

@dhensby

@dhensby Thank you for your guide! I changed code as you recommended. And now 3 different version apps is running on my each server.

Old one(always reassigning connectionPool)
New one as you recommended
Another one(Getting connection once at starting app)

Until now, the number of active connections on the all servers are stable.

I'm keeping an eye out for that.

P.S.

I added some code on your if statement

const getConnectionPool = async () => {
	try {
+               if (connectionPool && connectionPool.connected) return
		connectionPool = await new sql.ConnectionPool(dbConfig).connect()
		connectionPool.on("error", async err => {
			await closeConnectionPool()
			onDbError(err, "getConnectionPool error")
		})
	} catch (err) {
		onDbError(err, "getConnectionPool error in catch")
	}
}

I used below query to check the number of active connections

SELECT PROGRAM_NAME, HOSTNAME, STATUS, COUNT(DBID) AS NumberOfConnections
FROM sys.sysprocesses
WHERE PROGRAM_NAME = 'node-mssql'
GROUP BY PROGRAM_NAME, HOSTNAME, STATUS

I added some code on your if statement

It shouldn't matter if the connectionPool is connected or not because it could be in the process of connecting when it's called and you'll still be creating many pools that you lose reference to and thus can't close...

perhaps you should just un-assign the variable in the catch block?

@dhensby

I added some code on your if statement

It shouldn't matter if the connectionPool is connected or not because it could be in the process of connecting when it's called and you'll still be creating many pools that you lose reference to and thus can't close...

perhaps you should just un-assign the variable in the catch block?

@dhensby Thank you for your advice! I have agreed with you. I changed my code.

const getConnectionPool = async () => {
	try {
+		if (connectionPool) return
		connectionPool = await new sql.ConnectionPool(dbConfig).connect()
		connectionPool.on("error", async err => {
			await closeConnectionPool()
			onDbError(err, "getConnectionPool error")
		})
	} catch (err) {
+		connectionPool = null
		onDbError(err, "getConnectionPool error in catch")
	}
}

const closeConnectionPool = async () => {
	try {
		await connectionPool.close()
	} catch (err) {
+		connectionPool = null
		onDbError(err, "closeConnectionPool error")
	}
}

And so far, there is no connection problem in my app I am keeping an eye for 4 days!

Fantastic

@dhensby

@dhensby Finally I got a same error Failed to connect to 165.246.17.29:3361 in 15000ms after 6 days running. No too many connections were in there. So I edited my code. When error occured, the app would be restarted by process.exit() without any trying to reconnect on DB.

@LoveMeWithoutAll

@LoveMeWithoutAll ok, so is it now completely resolved for you by fixing your code?

@dhensby

@dhensby All problems is resolved by restarting app when error occured. Thank you for your kindness!!

Navigation Menu

Repository navigation

ResourceRequest timed out #610

51 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ResourceRequest timed out #610

Description

Activity

labs20 commented on Feb 27, 2018

glebson1 commented on Mar 5, 2018

munir131 commented on Mar 5, 2018

munir131 commented on Mar 7, 2018

tkunanuntakij commented on Mar 7, 2018

munir131 commented on Mar 8, 2018

munir131 commented on Mar 14, 2018

anton-bot commented on Mar 16, 2018

munir131 commented on Mar 16, 2018

JankiGadhiya commented on Mar 16, 2018

labs20 commented on Mar 16, 2018

munir131 commented on Mar 16, 2018

labs20 commented on Mar 16, 2018

munir131 commented on Mar 20, 2018

anton-bot commented on Mar 20, 2018

51 remaining items

willmorgan commented on Sep 9, 2018

LoveMeWithoutAll commented on Oct 22, 2018

madal3x commented on Oct 22, 2018

madal3x commented on Oct 22, 2018

LoveMeWithoutAll commented on Oct 23, 2018

dhensby commented on Oct 23, 2018

LoveMeWithoutAll commented on Oct 25, 2018

dhensby commented on Oct 25, 2018

LoveMeWithoutAll commented on Oct 26, 2018

dhensby commented on Oct 26, 2018

LoveMeWithoutAll commented on Oct 30, 2018

dhensby commented on Oct 31, 2018

LoveMeWithoutAll commented on Nov 14, 2018

dhensby commented on Nov 22, 2018

LoveMeWithoutAll commented on Nov 23, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions