Project

General

Profile

Actions

Bug #14465

closed

Generation fails with "Task FutureTask rejected from ThreadPoolExecutor" due to timeout in JS computation

Added by François ARMAND about 5 years ago. Updated over 4 years ago.

Status:
Released
Priority:
N/A
Category:
Web - Config management
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

We saw that exception during policy generation of a rather big :

⇨ Policy update error for process '754' at 2019-03-13 16:35:10
⇨ Exception caught during policy update process: Task java.util.concurrent.FutureTask@4cc89444 rejected from java.util.concurrent.ThreadPoolExecutor@7a2cae84[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24]

https://stackoverflow.com/questions/42843427/getting-error-futuretask2c7b84de-rejected-when-using-threadpool
https://stackoverflow.com/questions/8183205/what-could-be-the-cause-of-rejectedexecutionexception

Update:
to correct that problem, we need to be able to control the max parallelism in generation globally, and for desesperate cases, we need to also be able to control JS timeout.

These two parameters are quite advanced, and we will only expose them through settings API, but not in the UI.

You can manage them with:

# for max parallelism, either use '1x' to mean "1 time the number of CPU / 2" or '3' to mean '3 threads'
curl -k -H "X-API-Token: xxx"  -X POST 'https://.../rudder/api/latest/settings/rudder_generation_max_parallelism' -d "value=1x" 
# value is in seconds
curl -k -H "X-API-Token: xxx"  -X POST 'https://.../rudder/api/latest/settings/rudder_generation_js_timeout' -d "value=10" 

Subtasks 3 (0 open3 closed)

Bug #15078: Add an option to fail or not during policy generation if NodeConfigurations cannot be createdReleasedNicolas CHARLESActions
Bug #15083: Use human-readable durations in policy generation summaryReleasedFrançois ARMANDActions
Bug #15085: When a policy generation is triggered but nothing needs to be updated a log erroneously states that all nodes failedReleasedNicolas CHARLESActions

Related issues 4 (1 open3 closed)

Related to Rudder - Bug #11594: JVM crash due to thread leak in CachedFindRuleNodeStatusReportsResolvedActions
Related to Rudder - Bug #14462: when javascript engine timeout, we shutdown the thread before having a chance to report about itRejectedActions
Related to Rudder - User story #10551: Make policy generation be node by nodeNewActions
Related to Rudder - Bug #14945: Missing index in ldap directory ResolvedFrançois ARMANDActions
Actions

Also available in: Atom PDF