Bug #20614
openGroup criteria for VMs not working correctly
Added by Lars Koenen over 2 years ago. Updated 3 months ago.
Description
When I want to create a group that contains all nodes with VMs, where the VMs have less than x MB of Memory, rudder says there are none.
Even if I set the condition to
"Node summary"-> ("Virtual machines" "Memory" ">" "0") the group remains empty.
If I change the condition to "=" and specify an exact value, it works.
I have also tried to build a workaround using regular expressions, but no matter what regex I use, rudder always prints the error "Inconsistency: Invalid long"
Updated by Alexis Mousset over 2 years ago
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 52
Updated by François ARMAND over 2 years ago
Hello, I'm not able to reproduce it in unit tests. Would you mind:
- share a screenshot ? It may be not possible to add it here, can you copy/paste it to chat.rudder.io?
- give me an example of the memory value that works with =
The inconsistency error message is a bit strange, I wonder if there is not a problem with saved data.
If you don't mind, could you also share the result of the following ldap query executed from rudder root server?
% ldapsearch -o ldif-wrap=no -H "ldap://localhost:389" -x -D "cn=Manager,cn=rudder-configuration" -w XXXX -b "ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration" -s sub '(objectClass=virtualMachineLogicalElement)' vmName vmMemory
Where XXXX is the LDAP password that you can find in /opt/rudder/etc/rudder-passwords.conf
for RUDDER_OPENLDAP_BIND_PASSWORD
(or use -W
) for interactively providing it)
You should get result like:
# extended LDIF # # LDAPv3 # base <ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration> with scope subtree # filter: (objectClass=virtualMachineLogicalElement) # requesting: vmName vmMemory # # aaaaaaaa-6f1d-4bc0-976d-7ebcfada4a0e, 000000002-55a2-4b97-8529-5154cbb63a18, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=aaaaaaaa-6f1d-4bc0-976d-7ebcfada4a0e,nodeId=000000002-55a2-4b97-8529-5154cbb63a18,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 100000 vmName: vm1 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1
Updated by Nicolas CHARLES over 2 years ago
Search query from from log is
[2022-01-20 15:44:03] DEBUG com.normation.rudder.services.queries.AcceptedNodesLDAPQueryProcessor - LDAP result: 0 entries obtained in 191ms for query { returnType:'NodeReturnType' with 'And' criteria [virtualMachineLogicalElement.vmMemory gt 0] } [2022-01-20 15:44:03] DEBUG com.normation.rudder.services.queries.AcceptedNodesLDAPQueryProcessor - [post-filter:rudderNode] Found 0 nodes when filtering for info service existence and properties (4 ms)
and doesn't return anything. However, if I search with "isDefined", it does work
Updated by Nicolas CHARLES over 2 years ago
ldap query returns
# e2fbebb1-6b9b-403c-ae0e-629f91920b56, e9dfe54a-2925-45f0-a174-455b1696d986, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=e2fbebb1-6b9b-403c-ae0e-629f91920b56,nodeId=e9dfe54a-2925-45f0-a174-455b1696d986,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 8388 vmName: TEST4 # 0b3e57bf-9017-4030-ae14-606caf1ae930, e9dfe54a-2925-45f0-a174-455b1696d986, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=0b3e57bf-9017-4030-ae14-606caf1ae930,nodeId=e9dfe54a-2925-45f0-a174-455b1696d986,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 16777 vmName: TEST22 # e2fbebb1-6b9b-403c-ae0e-629f91920b56, 0a287ab1-f190-4d29-aca0-3c7a99540e0a, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=e2fbebb1-6b9b-403c-ae0e-629f91920b56,nodeId=0a287ab1-f190-4d29-aca0-3c7a99540e0a,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 8388 vmName: TEST3 # 0b3e57bf-9017-4030-ae14-606caf1ae930, 0a287ab1-f190-4d29-aca0-3c7a99540e0a, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=0b3e57bf-9017-4030-ae14-606caf1ae930,nodeId=0a287ab1-f190-4d29-aca0-3c7a99540e0a,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 16777 vmName: TEST22
Updated by Lars Koenen over 2 years ago
# extended LDIF # # LDAPv3 # base <ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration> with scope subtree # filter: (objectClass=virtualMachineLogicalElement) # requesting: vmName vmMemory # # 486c63ef-225d-4f3e-bfa4-31e226cf10a3, bdbab835-ef6f-4d81-8560-170eb67bf3b4, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=486c63ef-225d-4f3e-bfa4-31e226cf10a3,nodeId=bdbab835-ef6f-4d81-8560-170eb67bf3b4,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 4080 vmName: windows-hvm [...] # 9c4a0ca8-3ff7-4d96-aff7-b4953ed58098, 0ad68f11-540b-46cd-a617-6848fe98f238, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=9c4a0ca8-3ff7-4d96-aff7-b4953ed58098,nodeId=0ad68f11-540b-46cd-a617-6848fe98f238,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 3039 vmName: windows-hvm # ec7d505e-ea88-43a5-9948-24817313fd01, 3d60095e-8816-4ac8-9cc4-674aad1804da, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=ec7d505e-ea88-43a5-9948-24817313fd01,nodeId=3d60095e-8816-4ac8-9cc4-674aad1804da,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 4063 vmName: windows-hvm # 4928aa9c-356a-40d1-835f-40cf6886b188, ecf4c3c5-3d58-4bf0-b984-a038b09eeb17, Nodes, Accepted Inventories, Inventories, rudder-configuration dn: virtualMachineUuid=4928aa9c-356a-40d1-835f-40cf6886b188,nodeId=ecf4c3c5-3d58-4bf0-b984-a038b09eeb17,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration vmMemory: 2048 vmName: ubuntu-pvm # search result search: 2 result: 0 Success # numResponses: 725 # numEntries: 724
Updated by Lars Koenen over 2 years ago
See pictures in
https://gitter.im/normation/rudder?at=61e985889a33545406316e63
as you can see it works with a fixed value... like on the pictures for example "=" "3039"
Updated by Nicolas CHARLES over 2 years ago
detailed query generated from rudder
[2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] Start search for { returnType:'NodeReturnType' with 'And' criteria [virtualMachineLogicalElement.vmMemory gteq 0] } [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |-- LDAPObjectType(ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration,Sub,LDAPObjectTypeFilter((objectClass=virtualMachineLogicalElement)),Some((vmMemory>=0)),ParentDNJoin,Set()) [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |--- SearchRequest(baseDN='ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=0, filter='(&(objectClass=virtualMachineLogicalElement)(vmMemory>=0))', attrs={1.1}) [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |---- after ldap search request 0 result(s) [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |---- after post-processing: 0 result(s) [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |--- results are: [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |-- 0 sub-results (merged) [2022-01-20 16:04:09] TRACE com.normation.rudder.services.queries.InternalLDAPQueryProcessor - [331905074922] |-- ObjectType: ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration; ids: [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.AcceptedNodesLDAPQueryProcessor - LDAP result: 0 entries obtained in 8ms for query { returnType:'NodeReturnType' with 'And' criteria [virtualMachineLogicalElement.vmMemory gteq 0] } [2022-01-20 16:04:09] DEBUG com.normation.rudder.services.queries.AcceptedNodesLDAPQueryProcessor - [post-filter:rudderNode] Found 0 nodes when filtering for info service existence and properties (1 ms)
Updated by Nicolas CHARLES over 2 years ago
This is because we use in the ldap schema
attributetype ( InventoryAttributes:400.16 NAME 'vmMemory' DESC 'process virtual memory' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{256} )
which is a string, and not a number
Updated by Nicolas CHARLES over 2 years ago
- XenCitrixServer.pm: MEMORY is a number
- Virtuozzo.pm: MEMORY is a number (painfully converted from output with M, G, K)
- Parralels.pm: MEMORY is a number
- Xen.pm: MEMORY is a number
- HyperV.pm: MEMORY is a number
- Qemu.pm: MEMORY is a number
- VmWareESX.pm: depends on the output of vmware-cmd -l, for line memsize; which seems to always be a number (if command still exists)
- Vserver.pm: no MEMORY field findable
- Jails.pm: no MEMORY field findable
- Lxd.pm: depends on the output of lxc config show, key limits.memory, which can be a string like 256MB
- VmWareDesktop.pm: depends on the output of vmrun list, entry memsize, which doesn't seem there...
- Docker.pm: no MEMORY field
- VirtualBox.pm: depends on the output of VBoxManage -nologo list --long vms, entry Memory Size, which can be a string like 256MB
- SolarisZones.pm: MEMORY is a number
- Lxc.pm: MEMORY is a number
- Libvirt.pm: MEMORY is a number
- Hpvm.pm: MEMORY is a number
So we have Lxd.pm and VirtualBox.pm which can return non number
Updated by Nicolas CHARLES over 2 years ago
I tried to switch to a MemoryComparator in Rudder - it improves for the regex, but still no < nor >
I'm afraid we ought to sanitize this field, change the attribute, and change the comparator
Updated by François ARMAND over 2 years ago
After testing, we have a simpler migration path: it appears that openldap is lenient with already existing stored value, and changing the schema to "integer" will not cause damage for values that are not integer if the case happens.
So we can:
In 6.2.x:
- 1/ change the schema to integer for vmMemory
- 2/ change the vmMemory object to "Memory" type
- 3/ change the parsing of new inventory to check for integer or string and parse accordingly.
These three step ensure new inventories are leading to correct data in ldap.
For existing one:
- 4/ write a special parsing rule when mapping from LDAP vmMemory with a catch in case the value was not an integer, and parsing the string in that case (it will be corrected at next inventory)
- 4bis/ alternatively, we can had a boot check option that sanitize data if needed
Both option can be removed in next minor version without a direct migration path from 6.2 (so, 7.1).
I think I prefer the special parsing rule (4), it didn't change perf in the happy path and it is very simple.
Lars Koenen : in your case, there is a simple workaround, since you have only integer values in your ldap for vmMemory:
- in rudder root server, edit /opt/rudder/etc/openldap/schema/inventory
.
- replace:
attributetype ( InventoryAttributes:400.16 NAME 'vmMemory' DESC 'virtual machine memory' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{256} )
By (on the last three lines change - be careful of spaces at the begining of line after the first, they are mandatory):
attributetype ( InventoryAttributes:400.16 NAME 'vmMemory' DESC 'virtual machine memory' SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 EQUALITY integerMatch ORDERING integerOrderingMatch )
- restart rudder-slapd: systemctl restart rudder-slapd
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.2.13 to 6.2.14
- Priority changed from 52 to 50
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.2.14 to 6.2.15
- Priority changed from 50 to 49
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.2.15 to 6.2.16
- Priority changed from 49 to 48
Updated by Alexis Mousset about 2 years ago
- Target version changed from 6.2.16 to 6.2.17
Updated by Vincent MEMBRÉ about 2 years ago
- Target version changed from 6.2.17 to 997
- Priority changed from 48 to 0
Updated by Vincent MEMBRÉ about 2 years ago
- Target version changed from 997 to 6.2.18
Updated by Vincent MEMBRÉ about 2 years ago
- Target version changed from 6.2.18 to 6.2.19
Updated by Vincent MEMBRÉ about 2 years ago
- Target version changed from 6.2.19 to 6.2.20
Updated by Vincent MEMBRÉ about 2 years ago
- Target version changed from 6.2.20 to old 6.2 issues to relocate
Updated by Alexis Mousset about 1 year ago
- Target version changed from old 6.2 issues to relocate to 7.2.11
- Regression set to No
Updated by Vincent MEMBRÉ about 1 year ago
- Target version changed from 7.2.11 to 1046
Updated by Alexis Mousset almost 1 year ago
- Target version changed from 1046 to 7.3.8
Updated by Vincent MEMBRÉ 12 months ago
- Target version changed from 7.3.8 to 7.3.9
Updated by Vincent MEMBRÉ 11 months ago
- Target version changed from 7.3.9 to 7.3.10
Updated by Vincent MEMBRÉ 10 months ago
- Target version changed from 7.3.10 to 7.3.11
Updated by Vincent MEMBRÉ 8 months ago
- Target version changed from 7.3.11 to 7.3.12
Updated by Vincent MEMBRÉ 8 months ago
- Target version changed from 7.3.12 to 7.3.13
Updated by Vincent MEMBRÉ 7 months ago
- Target version changed from 7.3.13 to 7.3.14
Updated by Vincent MEMBRÉ 6 months ago
- Target version changed from 7.3.14 to 7.3.15
Updated by Vincent MEMBRÉ 4 months ago
- Target version changed from 7.3.15 to 7.3.16
Updated by Vincent MEMBRÉ 3 months ago
- Target version changed from 7.3.16 to 7.3.17