Rudder

As this issue only concerns a migration or an update, we can use postinst of rudder-server-root to use cf-agent and force the inventory to be sent only if this is an update.
Actually, we check that the LDAP server is set up to define if this is a first installation or not and it still a good test.

A message occurs when we try to send inventories after a migration and with generated promises:

[root@centos-6-64 ~]# /var/rudder/cfengine-community/bin/cf-agent -KI -b doInventory -D force_inventory
[...]
 -> Updated /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs from source /var/rudder/tmp/inventory/centos-6-64-2012-07-30-18-44-21.ocs on localhost
Transforming: /usr/bin/curl -f -s --user ${rudder.webdav.user}:${rudder.webdav.password} -T /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs  http://server-1.labo.com/inventory-updates/
 !! Finished command related to promiser "/var/rudder/inventories" -- an error occurred (returned 22)
-> Transformer /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs => /usr/bin/curl -f -s --user ${rudder.webdav.user}:${rudder.webdav.password} -T /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs  http://server-1.labo.com/inventory-updates/ returned error
R: @@Inventory@@result_error@@inventory-all@@inventory-all@@8@@inventory@@None@@2012-07-31 18:54:45+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Could not send the inventory

It seems that some properties in rudder-webapp.properties has not been added with the migration: rudder.webdav.user and rudder.webdav.password

Only trying to force the server to send its inventory at rudder-server-root postinst stage can't work since no promises are set on the server to create or send inventory: the folder /var/rudder/cfengine-community/inventory/ is missing.

• Change common technique to add a condition of sending inventory which permit to know that the server has never sent its inventory
• Change rudder-server-root postinst in order to let the webapp be started, loading the latest promises and then force to send inventory as /var/rudder/cfengine-community/inventory/ will be there.

Well, I'm pretty sure that is "Keep It Simple, Stupid !", but I might be mistaken. :D

Nicolas PERRON wrote:

Only trying to force the server to send its inventory at rudder-server-root postinst stage can't work since no promises are set on the server to create or send inventory: the folder /var/rudder/cfengine-community/inventory/ is missing.

I see two ways:

• Change common technique to add a condition of sending inventory which permit to know that the server has never sent its inventory
• Change rudder-server-root postinst in order to let the webapp be started, loading the latest promises and then force to send inventory as /var/rudder/cfengine-community/inventory/ will be there.

I meant to say that /var/rudder/cfengine-community/inputs/inventory/ was missing on the server before any generation of updated promises.

Jonathan CLARKE wrote:

We must avoid postinst scripts doing too much - clearly, waiting for a webapp and running cf-agent is too much, too complex, and too dependant on other parts of the system.

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

The use of the file will need to remove it after sending the inventory.

A class which check that the inventory hasn't be sent for more than 24 hours could be the best solution. I will work on this.
Thanks

Jonathan CLARKE wrote:

We must avoid postinst scripts doing too much - clearly, waiting for a webapp and running cf-agent is too much, too complex, and too dependant on other parts of the system.

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

I wonder, should CFEngine or a script check LDAP ? I don't know how to start

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

I wonder, should CFEngine or a script check LDAP ? I don't know how to start

CFEngine has got some special functions for LDAP queries but they are not available for the community edition.

I've started to work on a script to retrieve the date of the inventory in LDAP in order to compare with a 24 hours ago date but I don't know how to use a script like this. Where should it be located ? How should it be launched ?

Another question I wonder is: should this script used only for the server inventory date during an update or should it be used for all the nodes and everytime ?

Right now, Matthieu and me have worked together on a way to convert the LDAP date to a timestamp in order to compare the date instead of the Rudder's LDAP format which is not standard (date +%Y%m%d%H%M%S%z).

• how to retrieve information about the inventory date of the server:
  

  /opt/rudder/sbin/slapcat | perl -p0e 's/\n //g' | perl -p0e 's/\n([^\n])/%%%%\1/g' | grep -i "^%%%%dn: nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration" | grep -iE --color "%%%%(inventoryDate)::? " | sed 's/.*inventoryDate: \([0-9]*+[0-9]*\).*/\1/g'
  

• how to convert to a timestamp date:
  

  perl -e 'use Time::Piece; my $date = shift; my $time = Time::Piece->strptime($date, "%Y%m%d%H%M%S%z"); $time += $time->localtime->tzoffset; print $time->strftime("%s");' 
  

We have to way to do so:

• Use new initial promises:
  • Copy /opt/rudder/share/initial-promises in a temporary folder
  • Use informations from LDAP to populate a temporary initial-promises
  • Use cf-agent with these promises and use bundle for inventory

• Use shell
  • Use informations from LDAP to check that the inventoryDate is lesser than 24 hours ago
  • Use command line which will create inventory
  • Move the inventory file into /var/rudder/inventories/incoming in order to be processed next time Rudder will be up.

Both of these solutions must be used in the postinst script of rudder-server-root.

I will try to use the first solution which looks like to be the easiest.

Finally, I will do the second option.

The reason is simple, CFEngine will try to send the inventory to the server by WebDAV which will not be enabled during the postinst stage.
The best solution here is then to use shel commands.

The last commit was 19cea206c138613ea3d805541537c6b44259221a and linked to the wrong issue #2637

I've just looked at all of the commits up to now, and I think this approach is not something we want to have in Rudder. The implemented solution basically reproduces functionality implemented in CFEngine (in the system Technique "inventory") in shell, in postinst scripts. This is not DRY, makes maintenance hard, and is very error-prone.

A much simpler, and therefore more robust solution is to use the approach I suggested in comment #6 above: the postinst sets a flag after each upgrade, and the CFEngine promises check for this flag, and if it's present force an inventory and remove the flag if the inventory was successful.

Nicolas and I have worked on this new approach by pair-programming. He is testing it now, and will commit soon.

Note: commit:2c36f903 does not seem to be related, and will not be reverted.

Another question:

• Set the base.url which permit to access Rudder WebUI
• Set the nodeHostname in LDAP (indeed, replace %POLICY_SERVER_HOSTNAME%)

The first action is not important and we can't stop to do it, but it's not the same for the second one. Should we let rudder-init.sh to insert into LDAP %POLICY_SERVER_HOSTNAME% ? At least, the nodeHostname will be fixed with the new inventory send to Rudder.

Nicolas PERRON wrote:

Another question:

In rudder-init.sh script, a part is made for set the server hostname. This hostname initially is used for:

• Set the base.url which permit to access Rudder WebUI
• Set the nodeHostname in LDAP (indeed, replace %POLICY_SERVER_HOSTNAME%)

The first action is not important and we can't stop to do it, but it's not the same for the second one. Should we let rudder-init.sh to insert into LDAP %POLICY_SERVER_HOSTNAME% ? At least, the nodeHostname will be fixed with the new inventory send to Rudder.

Wrong ticket. It was about #2282