Project

General

Profile

Actions

Bug #2745

closed

After a migration from 2.3.8 to 2.4.0~beta3, or an update, the root server doesn't send inventory

Added by Nicolas PERRON over 12 years ago. Updated over 12 years ago.

Status:
Released
Priority:
1 (highest)
Assignee:
Nicolas PERRON
Category:
System techniques
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

No server's inventory has been sent to the endpoint application but CFEngine consider that it should be done the night:

root@server-1:~# /var/rudder/cfengine-community/bin/cf-agent -KI -b doInventory
 !! Duplicate selection of value for variable "execRun" in scope g
 !! Rule from /var/rudder/cfengine-community/inputs/common/1.0/site.cf at/before line 58
 !! Duplicate selection of value for variable "execRun" in scope g
 !! Rule from /var/rudder/cfengine-community/inputs/common/1.0/site.cf at/before line 58
 !! Duplicate selection of value for variable "execRun" in scope g
 !! Rule from /var/rudder/cfengine-community/inputs/common/1.0/site.cf at/before line 58
 >> Using command line specified bundlesequence
 -> Executing '/usr/bin/curl -s -f -o "/var/rudder/tmp/uuid.txt" http://server-1.labo.com/uuid' ...(timeout=-678,owner=-1,group=-1)
 -> Completed execution of /usr/bin/curl -s -f -o "/var/rudder/tmp/uuid.txt" http://server-1.labo.com/uuid
R: @@Inventory@@result_success@@inventory-all@@inventory-all@@8@@inventory@@None@@2012-07-31 17:27:14+02:00##root@#Next inventory scheduled between 00:00 and 06:00

The initial promises permit to send inventory if the uuid is valid but once the promises are updated by the webapp, the inventory is sent only during the night. If a migration or an upgrade is made, the initial promises are not set so the conditions to send inventory are not met.

Actions #1

Updated by Nicolas PERRON over 12 years ago

As this issue only concerns a migration or an update, we can use postinst of rudder-server-root to use cf-agent and force the inventory to be sent only if this is an update.
Actually, we check that the LDAP server is set up to define if this is a first installation or not and it still a good test.

Actions #2

Updated by Nicolas PERRON over 12 years ago

  • Status changed from New to Pending technical review
  • % Done changed from 0 to 100

Applied in changeset commit:50fd790fa5778839f2703fc17b9bbd9528aa828a.

Actions #3

Updated by Nicolas PERRON over 12 years ago

A message occurs when we try to send inventories after a migration and with generated promises:

[root@centos-6-64 ~]# /var/rudder/cfengine-community/bin/cf-agent -KI -b doInventory -D force_inventory
[...]
 -> Updated /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs from source /var/rudder/tmp/inventory/centos-6-64-2012-07-30-18-44-21.ocs on localhost
Transforming: /usr/bin/curl -f -s --user ${rudder.webdav.user}:${rudder.webdav.password} -T /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs  http://server-1.labo.com/inventory-updates/
 !! Finished command related to promiser "/var/rudder/inventories" -- an error occurred (returned 22)
-> Transformer /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs => /usr/bin/curl -f -s --user ${rudder.webdav.user}:${rudder.webdav.password} -T /var/rudder/inventories/centos-6-64-2012-07-30-18-44-21.ocs  http://server-1.labo.com/inventory-updates/ returned error
R: @@Inventory@@result_error@@inventory-all@@inventory-all@@8@@inventory@@None@@2012-07-31 18:54:45+02:00##36076ea3-281c-480e-be25-1392bfc39b71@#Could not send the inventory

It seems that some properties in rudder-webapp.properties has not been added with the migration: rudder.webdav.user and rudder.webdav.password

Actions #4

Updated by Nicolas PERRON over 12 years ago

Only trying to force the server to send its inventory at rudder-server-root postinst stage can't work since no promises are set on the server to create or send inventory: the folder /var/rudder/cfengine-community/inventory/ is missing.

I see two ways:
  • Change common technique to add a condition of sending inventory which permit to know that the server has never sent its inventory
  • Change rudder-server-root postinst in order to let the webapp be started, loading the latest promises and then force to send inventory as /var/rudder/cfengine-community/inventory/ will be there.
Actions #5

Updated by Nicolas PERRON over 12 years ago

  • Status changed from Pending technical review to In progress
  • % Done changed from 100 to 80
Actions #6

Updated by Jonathan CLARKE over 12 years ago

  • Status changed from In progress to Discussion

We must avoid postinst scripts doing too much - clearly, waiting for a webapp and running cf-agent is too much, too complex, and too dependant on other parts of the system.

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

Actions #7

Updated by Matthieu CERDA over 12 years ago

Well, I'm pretty sure that is "Keep It Simple, Stupid !", but I might be mistaken. :D

Actions #8

Updated by Nicolas PERRON over 12 years ago

Nicolas PERRON wrote:

Only trying to force the server to send its inventory at rudder-server-root postinst stage can't work since no promises are set on the server to create or send inventory: the folder /var/rudder/cfengine-community/inventory/ is missing.

I see two ways:
  • Change common technique to add a condition of sending inventory which permit to know that the server has never sent its inventory
  • Change rudder-server-root postinst in order to let the webapp be started, loading the latest promises and then force to send inventory as /var/rudder/cfengine-community/inventory/ will be there.

I meant to say that /var/rudder/cfengine-community/inputs/inventory/ was missing on the server before any generation of updated promises.

Actions #9

Updated by Nicolas PERRON over 12 years ago

Jonathan CLARKE wrote:

We must avoid postinst scripts doing too much - clearly, waiting for a webapp and running cf-agent is too much, too complex, and too dependant on other parts of the system.

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

The use of the file will need to remove it after sending the inventory.

A class which check that the inventory hasn't be sent for more than 24 hours could be the best solution. I will work on this.
Thanks

Actions #10

Updated by Nicolas PERRON over 12 years ago

Jonathan CLARKE wrote:

We must avoid postinst scripts doing too much - clearly, waiting for a webapp and running cf-agent is too much, too complex, and too dependant on other parts of the system.

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

I wonder, should CFEngine or a script check LDAP ? I don't know how to start

Actions #11

Updated by Nicolas PERRON over 12 years ago

This should be KISS - Keep It Simple and Sweet.

If you need an extra condition to run the inventory, then that's fine, we can add one. Maybe just touch a file, like /var/rudder/cfengine-community/force_inventory, and have a CFEngine promise check that filexists("<that path>")?

What would the condition be for doing this? I think we can check that the date of the last inventory in the LDAP tree is older than 24 hours, for example, or I think the default value is 1970/01/01 00:00 (the epoch), so we could check against that, and force the inventory is that is still the value of the inventoryDate attribute in nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration.

I wonder, should CFEngine or a script check LDAP ? I don't know how to start

CFEngine has got some special functions for LDAP queries but they are not available for the community edition.

I've started to work on a script to retrieve the date of the inventory in LDAP in order to compare with a 24 hours ago date but I don't know how to use a script like this. Where should it be located ? How should it be launched ?

Another question I wonder is: should this script used only for the server inventory date during an update or should it be used for all the nodes and everytime ?

Right now, Matthieu and me have worked together on a way to convert the LDAP date to a timestamp in order to compare the date instead of the Rudder's LDAP format which is not standard (date +%Y%m%d%H%M%S%z).

For the moment, I know
  • how to retrieve information about the inventory date of the server:
    /opt/rudder/sbin/slapcat | perl -p0e 's/\n //g' | perl -p0e 's/\n([^\n])/%%%%\1/g' | grep -i "^%%%%dn: nodeId=root,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration" | grep -iE --color "%%%%(inventoryDate)::? " | sed 's/.*inventoryDate: \([0-9]*+[0-9]*\).*/\1/g'
    
  • how to convert to a timestamp date:
    perl -e 'use Time::Piece; my $date = shift; my $time = Time::Piece->strptime($date, "%Y%m%d%H%M%S%z"); $time += $time->localtime->tzoffset; print $time->strftime("%s");' 
    
Actions #12

Updated by Nicolas PERRON over 12 years ago

We have to way to do so:

  • Use new initial promises:
    • Copy /opt/rudder/share/initial-promises in a temporary folder
    • Use informations from LDAP to populate a temporary initial-promises
    • Use cf-agent with these promises and use bundle for inventory
  • Use shell
    • Use informations from LDAP to check that the inventoryDate is lesser than 24 hours ago
    • Use command line which will create inventory
    • Move the inventory file into /var/rudder/inventories/incoming in order to be processed next time Rudder will be up.

Both of these solutions must be used in the postinst script of rudder-server-root.

I will try to use the first solution which looks like to be the easiest.

Actions #13

Updated by Nicolas PERRON over 12 years ago

Finally, I will do the second option.

The reason is simple, CFEngine will try to send the inventory to the server by WebDAV which will not be enabled during the postinst stage.
The best solution here is then to use shel commands.

Actions #14

Updated by Nicolas PERRON over 12 years ago

  • Status changed from Discussion to Pending technical review
  • % Done changed from 80 to 100

After some adjustment, it seems to work now !
The inventory is stored in /var/rudder/inventories/incoming and is send to endpoint the next CFEngine run.

Actions #15

Updated by Nicolas PERRON over 12 years ago

  • Status changed from Pending technical review to In progress
  • % Done changed from 100 to 90

It doesn't work on SuSE. It seems to be a typo on rudder-server-root postinst.

Actions #16

Updated by Nicolas PERRON over 12 years ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 90 to 100

Applied in changeset commit:957b47be7344f45e6c20462aaf652d6d9a2e3e6d.

Actions #17

Updated by Nicolas PERRON over 12 years ago

The last commit was 19cea206c138613ea3d805541537c6b44259221a and linked to the wrong issue #2637

Actions #18

Updated by Matthieu CERDA over 12 years ago

Well, that was a tough one. But migrations are going fine in my tests and I do not see any blatant typos or errors. Validated ! (we still need to address a tiny "manual jetty restart required" though.)

Actions #19

Updated by Nicolas PERRON over 12 years ago

  • Status changed from Pending technical review to In progress
  • Target version changed from 2.4.0~beta3 to 2.4.0~beta5
This bug has not been properly fixed and some revert will be made. The list of the commits is:
  • 19cea206c138613ea3d805541537c6b44259221a
  • dae54393675c97f905fe783c0d75bae6750f5493
  • 957b47be7344f45e6c20462aaf652d6d9a2e3e6d
  • 2808891a6cae5f80043f7459ef846b5f872e0c17
  • 9f5c31ffcd02aa1520fcd018abcecea4d29daef9
  • 21849c7b347377906ad86e248df56b69a021102c
  • 648a077bc19e158dd9153ce78217b020ed89b0bc
  • 50fd790fa5778839f2703fc17b9bbd9528aa828a
Actions #20

Updated by Jonathan CLARKE over 12 years ago

I've just looked at all of the commits up to now, and I think this approach is not something we want to have in Rudder. The implemented solution basically reproduces functionality implemented in CFEngine (in the system Technique "inventory") in shell, in postinst scripts. This is not DRY, makes maintenance hard, and is very error-prone.

A much simpler, and therefore more robust solution is to use the approach I suggested in comment #6 above: the postinst sets a flag after each upgrade, and the CFEngine promises check for this flag, and if it's present force an inventory and remove the flag if the inventory was successful.

Nicolas and I have worked on this new approach by pair-programming. He is testing it now, and will commit soon.

Actions #21

Updated by Jonathan CLARKE over 12 years ago

Note: commit:2c36f903 does not seem to be related, and will not be reverted.

Actions #22

Updated by Nicolas PERRON over 12 years ago

Another question:

In rudder-init.sh script, a part is made for set the server hostname. This hostname initially is used for:
  • Set the base.url which permit to access Rudder WebUI
  • Set the nodeHostname in LDAP (indeed, replace %POLICY_SERVER_HOSTNAME%)

The first action is not important and we can't stop to do it, but it's not the same for the second one. Should we let rudder-init.sh to insert into LDAP %POLICY_SERVER_HOSTNAME% ? At least, the nodeHostname will be fixed with the new inventory send to Rudder.

Actions #23

Updated by Nicolas PERRON over 12 years ago

Nicolas PERRON wrote:

Another question:

In rudder-init.sh script, a part is made for set the server hostname. This hostname initially is used for:
  • Set the base.url which permit to access Rudder WebUI
  • Set the nodeHostname in LDAP (indeed, replace %POLICY_SERVER_HOSTNAME%)

The first action is not important and we can't stop to do it, but it's not the same for the second one. Should we let rudder-init.sh to insert into LDAP %POLICY_SERVER_HOSTNAME% ? At least, the nodeHostname will be fixed with the new inventory send to Rudder.

Wrong ticket. It was about #2282

Actions #24

Updated by Nicolas PERRON over 12 years ago

  • Status changed from In progress to Pending technical review

The last commit from Jon (6d6d6fdcee9738fff7bda2e851b68777cc8555d1) as fixed this issue.

Actions #25

Updated by Jonathan CLARKE over 12 years ago

  • Status changed from Pending technical review to Released

3 pairs of eyes have checked the latest commits and agree they're OK. Technical review validated.

Actions

Also available in: Atom PDF