Project

General

Profile

User story #4670

Allows inventories to be augmented by the user with custom properties

Added by d i over 4 years ago. Updated 9 months ago.

Status:
Released
Priority:
3
Category:
Web - Nodes & inventories
Target version:
Suggestion strength:
User visibility:
Effort required:
Tags:

Description

Original question

I had raised a question on the mailing list about injecting arbitrary items into the inventory, and was asked by Francois Armand to open a ticket. The subject of that email was "Class definition". What I was wondering is how much work it would be to add the ability to hook into something like the "addInformationsToInventory" bundle to add user defined information.

What I was struggling with was how to define classes like I would in plain cfengine. There, I can define a class based on pretty much anything, and then use that to conditionally run different promises. Also, with cf-hub, you also gain the ability to query classes that have been defined. I like the both the inventory aspect of rudder and the ability to query it, and thought that I could mimic much of what I was doing with plain cfengine classes with your inventory if I could add arbitrary information.

I'm not sure what would be required to make the web interface handle this, and it looks like a bit of thought would have to go into the LDAP schema. Either way, it seemed like it would be a nice feature.

Proposed solution

When doing an inventory, the agent will run executables (script, binaries, whatever) located in `/var/rudder/hooks.d` sequentially. Each scripts is expected to procuce a JSON object output (not a JSON array).

These output are join together in a JSON array and are put into a new "CUSTOMPROPERTIES" element:

<CUSTOMPROPERTIES>[{"hook1": json1},{"hook2_key1":json21, "hook2_key2":json22}]</CUSTOMPROPERTIES>

Each key is added as the name of a node property with the "inventory" provider, so with the example, the node will get 3 properties: `("hook1", json1)`, `("hook2_key1", json21)`, `("hook2_key2", json22)`.

Remarques about property name unicity:

- if two inventory custom properties have the same name, only one is kept (not specified which one, you should really avoid that)
- if a node property and an inventory custom property share the same key name, the inventory property overrides the node one

Of course, you can use naming convention so that it does not happen, but you also have to possibility to use that mecanism for default value overriding when the inventory knows something more about the node. And you can also use convention + ${node.properties[key1] | default = ${node.properties[inventory-key1]} to get override the other way around.

Other possible choices for inventory custom properties merge in node properties

Make node properties overrides custom inventory properties

With that choice, we keep most of above spec but we reverse the overriding order: if a custom inventory property and a node property have the same name, then the node property is chosen.

I prefer the other one because it follows the same semantic as for datasources: the non-user-updatable property is the one used (it has a kind of "state of fact" semantic).

In all case, you use you own namespacing convention to avoid overriding.

Make what overrides what configurable

"When there is no provable good choice in all case, make it configurable".
That can be good, but I fear the added complexity: do we want a global parameter? More likelly a by key-name one, because the output of that hook states the reallity, but on that other case, it's just better to keep what the user want. Oh, but in fact, that configuration should be made on a node by node basis...

And I'm not even sure of the gain, because I believe all (?) cases can be covered tody with convention, node properties default value syntax, and for exceptionnal cases JS scripting.

Put all custom properties under the same "inventry properties" key

For now, we took as a given that each key:value pair from inventory custom properties get a corresponding node property. But we could merge all inventory custom properties under only one node property like "inventory property". Then the user will just need to go one layer deeper to get the previous key:value pair.

It makes getting to the actual value a little less direct, but at least there is no imposed overriding, and the user choose what he want in all cases.
The main problem is that for now, we don't have a nice search UI for JSON (see #9299)

((( further thinking on that point:
And in fact, we could even go one step further: have an "inventory" key, with a subkey "properties" with CUSTOMPROPERTIES content, and why not in the future an "os" key, a "networkInterfaces" one, etc.

But that would make for some nice duplication in node details information (via REST api espcially) and it would make search etc more cumbersome.
)))

Implementation details

Inventory custom properties are stored in the the node inventory entry, on a "customProperty" multivalued attribute where as node properties are store in the rudder node entry, under a serializedNodeProperty attribute. The merge/overriding logic is done in Rudder logic, when we gather "node information".


Subtasks

User story #11726: Allow inventories to be augmented by script on the agent sideReleasedAlexis MOUSSET
User story #11808: Implement the security checking for Windows script executionNew
User story #12031: Parse inventory custom properties and save then in LDAPReleasedFrançois ARMAND
User story #12107: Update test inventory-schemaReleasedFrançois ARMAND
Bug #12110: Compilation errorReleasedFrançois ARMAND

Related issues

Related to Rudder - User story #5449: Make environment variables accessible as parameter in all TechniquesReleased2014-10-14
Related to Rudder - User story #5506: Customize Nodes by adding attribute on themReleased2014-11-27
Related to Rudder - User story #5821: Proposal: have the possibility to add additional custom information to a nodeRejected2014-11-26
Related to Rudder - Bug #8022: Node's FQDN-Resolution is sometimes invalidNew
Related to Rudder - User story #3112: Allow to get informations from the node inventory to use them in DirectivesNew
Related to Rudder - User story #8335: Document the behavior of inventory hooksReleased
Related to Rudder - Bug #12572: Custom node properties from inventory are not found in group searchReleased
Related to Rudder - Bug #9299: Extend groups search capabilities for complex JSON Properties / JsonPathPending release
Has duplicate Rudder - User story #5968: Have a default group at setupRejected2014-12-11
Has duplicate Rudder - Bug #9738: [Feature Request] - Node user-propertiesRejected

Associated revisions

Revision d21f0127 (diff)
Added by François ARMAND 10 months ago

Fixes #4670: Allows inventories to be augmented by the user

Revision ab460013 (diff)
Added by François ARMAND 10 months ago

Fixes #4670: Allows inventories to be augmented by the user

History

#1 Updated by Jonathan CLARKE over 4 years ago

  • Project changed from ldap-inventory to Rudder
  • Category set to 26
  • Assignee changed from Jonathan CLARKE to d i
  • Target version set to 2.11.0~beta1

d i wrote:

I had raised a question on the mailing list about injecting arbitrary items into the inventory, and was asked by Francois Armand to open a ticket.

Thanks for doing so!

The subject of that email was "Class definition".

And a link to it is here: http://www.rudder-project.org/pipermail/rudder-users/2014-March/000490.html

What I was wondering is how much work it would be to add the ability to hook into something like the "addInformationsToInventory" bundle to add user defined information.

What I was struggling with was how to define classes like I would in plain cfengine. There, I can define a class based on pretty much anything, and then use that to conditionally run different promises. Also, with cf-hub, you also gain the ability to query classes that have been defined. I like the both the inventory aspect of rudder and the ability to query it, and thought that I could mimic much of what I was doing with plain cfengine classes with your inventory if I could add arbitrary information.

I'm not sure what would be required to make the web interface handle this, and it looks like a bit of thought would have to go into the LDAP schema. Either way, it seemed like it would be a nice feature.

I like this idea a lot! It comes up pretty frequently that it would be useful to be able to collect some "extra" data from nodes.

What I could imagine at this stage:
  • We define a directory in which you can add scripts (shell, perl, python, whatever, so long as they are executable). These scripts get copied to all nodes and run at the same time as the inventory is collected.
  • Each script can output one or several key=value pairs. Maybe even key=[list, of, values]. These keys and their associated values would then be inserted into the inventory, and send to the Rudder server.
  • The Rudder web interface would display these key value pairs in an extra tab in the "Node details" screen, and make them searchable via Node search and for dynamic Groups.

Does this sound like it would fit for you?

#2 Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 2.11.0~beta1 to 2.11.0~beta2

#3 Updated by Matthieu CERDA over 4 years ago

  • Target version changed from 2.11.0~beta2 to 2.11.0~rc1

#4 Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 2.11.0~rc1 to 2.11.0~rc2

#5 Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 2.11.0~rc2 to 2.11.0

#6 Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 2.11.0 to 2.11.1

#7 Updated by Nicolas PERRON over 4 years ago

  • Target version changed from 2.11.1 to 2.11.2

#8 Updated by Nicolas PERRON over 4 years ago

  • Target version changed from 2.11.2 to 2.11.3

#9 Updated by Matthieu CERDA about 4 years ago

  • Target version changed from 2.11.3 to 2.11.4

#10 Updated by Jason Maloney about 4 years ago

Hi, I was wondering if there was any movement on this feature. My company is looking to implement configuration management across 100-200 nodes, and if I understand correctly, we would need something like this to properly implement node groups arbitrarily (e.g. production vs development, application vs database vs combined, etc.).

#11 Updated by François ARMAND about 4 years ago

Hello Jason,

The feature is not implemented yet, but we are trying to see how to add it in Rudder 3.0 - so yes, it's in the near roadmap.

We have also a sister feature planned which is to be able to define properties (key/value) in Rudder UI for a node (with possibility to mass update, for example for all nodes on a group) and be able to use these key=value to make group, or as parameter of directive (or even in ncf) with somethink alike ${rudder.node.tag.KEY_NAME}.

The two features are changing the point of view from which the values are defined: either from the node, based on some of its local knowledge, and the information is pull to Rudder, or from Rudder, and the information is pushed to the node.

Alos, note that you can already cover some part of the use case described in that ticket by adding environment variable. Environement variables can be used to build groups, and in Rudder 3.0, they will also be usable with the ${node.env[VAR_NAME]} syntax in Directive or ncf generic methods parameters.

#12 Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 2.11.4 to 2.11.5

#13 Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 2.11.5 to 2.11.6

#14 Updated by François ARMAND almost 4 years ago

I just noticed that I didn't linked to the feature I was refering to in comment #11: http://www.rudder-project.org/redmine/issues/5506

With that, in Rudder 3.0 you can add arbitrary key=value pairs to node (not in the inventory, just in Rudder) and use them to build groups.

#15 Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 2.11.6 to 2.11.7

#16 Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 2.11.7 to 2.11.8

#17 Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 2.11.8 to 2.11.9

#18 Updated by Benoît PECCATTE almost 4 years ago

  • Category changed from 26 to Web - Nodes & inventories

#19 Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 2.11.9 to 2.11.10

#20 Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 2.11.10 to 2.11.11

#21 Updated by Benoît PECCATTE over 3 years ago

#22 Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 2.11.11 to 2.11.12

#23 Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 2.11.12 to 2.11.13

#24 Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 2.11.13 to 2.11.14

#25 Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 2.11.14 to 2.11.15

#26 Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 2.11.15 to 2.11.16

#27 Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 2.11.16 to 2.11.17

#28 Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 2.11.17 to 2.11.18

#29 Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 2.11.18 to 2.11.19

#30 Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 2.11.19 to 2.11.20

#31 Updated by François ARMAND over 2 years ago

  • Related to Bug #8022: Node's FQDN-Resolution is sometimes invalid added

#32 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 2.11.20 to 2.11.21

#33 Updated by François ARMAND over 2 years ago

  • Tracker changed from Question to User story
  • Subject changed from Could inventory be augmented by the user? to Allows inventories to be augmented by the user
  • Assignee deleted (d i)
  • Priority changed from N/A to 3

For future implementation: Fusion allows to extend what go in the report like that: http://fusioninventory.org/documentation/agent/additional_content.html

So we could reuse or plug our own script / convention to match that, the simplests way being that each of our extension script produce an XML fragment which is latter added in the inventory.

#34 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 2.11.21 to 2.11.22

#35 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 2.11.22 to 2.11.23

#36 Updated by Nicolas CHARLES over 2 years ago

  • Tags set to Sponsored

#37 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 2.11.23 to 2.11.24

#38 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 2.11.24 to 308

#39 Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 308 to 3.1.14

#40 Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 3.1.14 to 3.1.15

#41 Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 3.1.15 to 3.1.16

#42 Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 3.1.16 to 3.1.17

#43 Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 3.1.17 to 3.1.18

#44 Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 3.1.18 to 3.1.19

#45 Updated by Benoît PECCATTE almost 2 years ago

  • Has duplicate Bug #9738: [Feature Request] - Node user-properties added

#46 Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 3.1.19 to 3.1.20

#47 Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 3.1.20 to 3.1.21

#48 Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 3.1.21 to 3.1.22

#49 Updated by François ARMAND over 1 year ago

  • Related to User story #3112: Allow to get informations from the node inventory to use them in Directives added

#50 Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 3.1.22 to 3.1.23

#51 Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 3.1.23 to 3.1.24

#52 Updated by Vincent MEMBRÉ about 1 year ago

  • Target version changed from 3.1.24 to 3.1.25

#53 Updated by Benoît PECCATTE about 1 year ago

  • Target version changed from 3.1.25 to 4.1.9

#54 Updated by Nicolas CHARLES about 1 year ago

  • Target version changed from 4.1.9 to 4.3.0~beta1
Inventories will be augmented via scripts in /var/rudder/hooks.d generating json
Questions:
  • do we allow values to be overriden (last script overriding previous value)
  • if so, full key override, or merge ?

we cannot add CDATA in the generated inventory (if is escaped), so we have the followinf result

<CUSTOM_PROPERTIES>{{ &quot;bla&quot; : &quot;blu&quot;},{ &quot;never&quot; : &quot;ever&quot;}}</CUSTOM_PROPERTIES>

#55 Updated by François ARMAND 11 months ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND

#56 Updated by François ARMAND 11 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Nicolas CHARLES
  • Pull Request set to https://github.com/Normation/rudder/pull/1844

#57 Updated by François ARMAND 11 months ago

  • Subject changed from Allows inventories to be augmented by the user to Allows inventories to be augmented by the user with custom properties
  • Description updated (diff)

Update description with the actual chosen solution specifications.

#58 Updated by François ARMAND 11 months ago

  • Description updated (diff)

Any feedback about how inventory custom properies should appear in node properties is welcome!

#59 Updated by Janos Mattyasovszky 11 months ago

Not sure that the merging on the key-value level should take place, it's hard to imagine an usecase where you want to use hooks from different sources that provide overriding values from each other.

On the other hand, one of my biggest issue is that the FQDN of a node is not always what Rudder believes to be the FQDN for the node, and also apparently there is no proper way to change this other than to patch the FusionInventory with a custom patch...

#60 Updated by Vincent MEMBRÉ 11 months ago

  • Target version changed from 4.3.0~beta1 to 4.3.0~rc1

#61 Updated by François ARMAND 10 months ago

  • Assignee changed from Nicolas CHARLES to d i

#62 Updated by Normation Quality Assistant 10 months ago

  • Assignee changed from d i to François ARMAND

#63 Updated by François ARMAND 10 months ago

  • Status changed from Pending technical review to Pending release

#64 Updated by Vincent MEMBRÉ 9 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.3.0~rc1 which was released today.

#65 Updated by Benoît PECCATTE 9 months ago

#66 Updated by François ARMAND 7 months ago

  • Related to Bug #12572: Custom node properties from inventory are not found in group search added

#67 Updated by François ARMAND 6 months ago

  • Related to Bug #9299: Extend groups search capabilities for complex JSON Properties / JsonPath added

Also available in: Atom PDF