Project

General

Profile

Actions
Copy link

Bug #20390

closed

webapp doesn't start after upgrade from 6.2 to 7.0

Added by Nicolas CHARLES almost 3 years ago. Updated almost 3 years ago.

Status:
Released
Priority:
N/A
Assignee:
François ARMAND
Category:
Web - Compliance & node report
Target version:
7.0.0~rc3
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
To do
Regression:

Description

It thinks that technique "common" is deleted while it's not

# ls common/1.0/
cf-serverd.st             hooks.cf              properties.cf                rudder-groups.st              rudder_promises_generated.st
common.st                 internal-security.cf  reporting-http.cf            rudder-parameters.st          run_interval.st
cron-setup.cf             metadata.xml          restart-services.cf          rudder-promises-generated.st  site.cf
environment-variables.cf  monitoring.cf         rudder-agent-community-cron  rudder-system-directives.st   update.st
failsafe.st               promises.st           rudder-directives.st         rudder-vars.st
root@server:/var/rudder/configuration-repository/techniques/system# git log common
commit b71a5cffdbe541641331367f16789960ed3ee542
Author: root user (CLI) <root@localhost>
Date:   Wed Dec 8 17:12:18 2021 +0100

    Upgrade system Techniques from /opt/rudder/share/techniques/system/common/ - automatically done by rudder-upgrade script

commit 01b9dcfc8dbfb11250fe0743cd45da2d01c63dfd
Author: root user (CLI) <root@localhost>
Date:   Tue Dec 7 21:24:18 2021 +0100

    initial commit

webapp says

[2021-12-08 17:12:36+0100] INFO  application - Using 'move' behavior when a node is deleted
[2021-12-08 17:12:36+0100] INFO  application - Starting Rudder 7.0.0~rc2-SNAPSHOT web application [build timestamp: 2021-12-08T01:57:55Z]
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable AUTHORIZED_NETWORKS is not defined: perhaps the metadata.xml for
 technique 'common/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable NODEROLE is not defined: perhaps the metadata.xml for technique 
'common/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable SYSLOGPORT is not defined: perhaps the metadata.xml for techniqu
e 'common/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable RUDDER_SYSLOG_PROTOCOL is not defined: perhaps the metadata.xml 
for technique 'common/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable SYSLOG_PROTOCOL_DISABLED is not defined: perhaps the metadata.xm
l for technique 'common/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable SYSLOGPORT is not defined: perhaps the metadata.xml for technique 'distributePolicy/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable STORE_ALL_CENTRALIZED_LOGS_IN_FILE is not defined: perhaps the metadata.xml for technique 'distributePolicy/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable AUTHORIZED_NETWORKS is not defined: perhaps the metadata.xml for technique 'distributePolicy/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable RUDDER_REPORTS_DB_USER is not defined: perhaps the metadata.xml for technique 'distributePolicy/1.0' is not up to date
[2021-12-08 17:12:40+0100] WARN  technique-parser - Parsing: The system variable RUDDER_SERVER_ROLES is not defined: perhaps the metadata.xml for technique 'server-roles/1.0' is not up to date
[2021-12-08 17:12:42+0100] INFO  git-repository - Git directory was not initialised: create a new git repository into folder '/var/rudder/fact-repository' and add all its content as initial release
[2021-12-08 17:12:43+0100] INFO  report - Disable automatic database archive sinces property rudder.batch.reportsCleaner.archive.TTL is 0 or negative
[2021-12-08 17:12:43+0100] WARN  healthcheck - CPU cores: Only one core, recommended value is at least 2
:
(snip)
[2021-12-08 17:12:45+0100] INFO  migration - Check if all Rudder 7.0 system technique are correctly loaded in technique library
[2021-12-08 17:12:47+0100] INFO  techniques.reader - Reloading technique library, found modified technique(s): ['rudder-service-postgresql': updated (1.0: added)], ['firewall': updated (1.0: updated)], ['manageKeyValueFile': updated (1.3: updated)], ['server-roles': deleted (1.0)], ['sudoParameters': updated (3.2: updated)], ['snmp': updated (1.0: updated)], ['partitionSizeMonitoring': updated (3.1: updated)], ['hostsConfiguration': updated (2.0: updated)], ['dnsConfiguration': updated (3.0: updated)], ['packageManagement': updated (1.2: updated, 1.3: updated)], ['rudder-service-relayd': updated (1.0: added)], ['setVars': updated (1.1: updated)], ['distributePolicy': deleted (1.0)], ['servicesManagement': updated (3.0: updated)], ['inventory': updated (1.0: updated)], ['services': updated (1.1: updated)], ['checkGenericFileContent': updated (8.0: updated)], ['aptPackageManagerSettings': updated (3.1: updated, 3.2: updated)], ['genericCommandVariableDefinition': updated (3.0: updated)], ['repoGpgKeyManagement': updated (1.1: updated)], ['variableFromJsonFile': updated (3.0: updated)], ['fileAlterationMonitoring': updated (2.2: updated)], ['filesPermissions': updated (2.2: updated)], ['server-common': updated (1.0: added)], ['userManagement': updated (9.0: updated)], ['zypperPackageManagerRepositories': updated (1.0: updated)], ['cronManagement': updated (3.2: updated)], ['common': deleted (1.0)], ['sshKeyDistribution': updated (4.0: updated)], ['rudder-service-webapp': updated (1.0: added)], ['rudderSNMP': updated (1.0: updated)], ['zmdPackageManagerSettings': updated (3.0: updated)], ['sshConfiguration': updated (5.0: updated)], ['copyGitFile': updated (2.3: updated)], ['clockConfiguration': updated (3.2: updated)], ['downloadFile': updated (4.0: updated)], ['jobScheduler': updated (3.0: updated)], ['fileTemplate': updated (1.0: updated)], ['fstabConfiguration': updated (4.0: updated)], ['rudder-service-apache': updated (1.0: added)], ['userCronManagement': updated (1.0: updated)], ['rudder-service-slapd': updated (1.0: added)], ['genericVariableDefinition': updated (2.0: updated)], ['groupManagement': updated (5.1: updated)], ['zypperPackageManagerSetup': updated (1.0: updated)], ['motdConfiguration': updated (3.3: updated)], ['fileManagement': updated (4.1: updated)]
[2021-12-08 17:12:47+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-postgresql' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:48+0100] ERROR application - System technique 'server-roles' (1.0)' is deleted in git base. This will likely cause grave problem. You should investigate.
[2021-12-08 17:12:48+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-relayd' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:48+0100] ERROR application - System technique 'distributePolicy' (1.0)' is deleted in git base. This will likely cause grave problem. You should investigate.
[2021-12-08 17:12:49+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'server-common' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:49+0100] ERROR application - System technique 'common' (1.0)' is deleted in git base. This will likely cause grave problem. You should investigate.
[2021-12-08 17:12:49+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-webapp' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:50+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-apache' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:50+0100] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-slapd' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-08 17:12:51+0100] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Reload Technique library to allows migration to 7.x system techniques
[2021-12-08 17:12:51+0100] ERROR migration - Error during migration of policy server system configuration to Rudder V7.0 data. It is likely that Rudder won't work as expected. Please check the log above. Error was: Inconsistency: Error when looking for technique with ID 'server-roles/1.0'. Check technique name and version; cause was: Unexpected: Inconsistency: Error when looking for technique with ID 'distributePolicy/1.0'. Check technique name and version
[2021-12-08 17:12:51+0100] INFO  bootchecks - Check mandatory DIT entries
[2021-12-08 17:12:51+0100] ERROR bootchecks - Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up.
[2021-12-08T17:12:51.141+01:00] ERROR FATAL An error happen during Rudder boot. Rudder will stop now. Error: SystemError: An error occured; cause was: javax.servlet.UnavailableException: Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up. 
 -> 
javax.servlet.UnavailableException: Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up.
        at bootstrap.liftweb.checks.consistency.CheckDIT.FAIL$1(CheckDIT.scala:80)
        at bootstrap.liftweb.checks.consistency.CheckDIT.checks(CheckDIT.scala:165)
        at bootstrap.liftweb.SequentialImmediateBootStrapChecks.$anonfun$checks$1(BootstrapChecks.scala:99)
        at bootstrap.liftweb.SequentialImmediateBootStrapChecks.$anonfun$checks$1$adapted(BootstrapChecks.scala:91)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)

webapp restarts, and says

[2021-12-08 17:15:20+0100] INFO  bootchecks - Check if migration of system configuration object and allowed networks to Rudder 7.0 format is need
ed
[2021-12-08 17:15:21+0100] INFO  migration - Check if all Rudder 7.0 system technique are correctly loaded in technique library
[2021-12-08 17:15:23+0100] INFO  techniques.reader - Reloading technique library, found modified technique(s): ['rudder-service-postgresql': upda
ted (1.0: updated)], ['server-common': updated (1.0: updated)], ['rudder-service-relayd': updated (1.0: updated)], ['rudder-service-webapp': upda
ted (1.0: updated)], ['rudder-service-apache': updated (1.0: updated)], ['rudder-service-slapd': updated (1.0: updated)]
[2021-12-08 17:15:23+0100] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Reload Technique library to allows migration 
to 7.x system techniques
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'rudder-service-postgresql'
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'rudder-service-slapd'
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'rudder-service-webapp'
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'server-common'
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'rudder-service-apache'
[2021-12-08 17:15:23+0100] INFO  migration - System technique v7.0 present: 'rudder-service-relayd'
[2021-12-08 17:15:23+0100] INFO  migration - Creating new system directives for policy server 'root'
[2021-12-08 17:15:23+0100] INFO  policy.generation - Start policy generation, checking updated rules
[2021-12-08 17:15:24+0100] ERROR rudder-ldap-entity-mapper - Invalid attribute 'special:all_servers_with_role' for entry ruleTarget.
[2021-12-08 17:15:24+0100] INFO  migration - Creating new system rules for policy server 'root'
[2021-12-08 17:15:24+0100] WARN  com.normation.rudder.repository.ldap.RoLDAPNodeGroupRepository - Error when mapping entry with DN 'ruleTarget=special:all_servers_with_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration' from node groups library, that entry will be ignored; cause was: UnexpectedObject: Can not unserialize target, 'special:all_servers_with_role' does not match any known target format
[2021-12-08 17:15:24+0100] ERROR com.normation.rudder.domain.policies.RuleTarget - Error when trying to read the following serialized Rule target as a composite target: 'special:all_nodes_without_role'. Reported parsing error cause was: unknown token s
Near: sp
[2021-12-08 17:15:24+0100] WARN  com.normation.rudder.repository.ldap.RoLDAPNodeGroupRepository - Error when mapping entry with DN 'ruleTarget=special:all_nodes_without_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration' from node groups library, that entry will be ignored; cause was: UnexpectedObject: Can not unserialize target, 'special:all_nodes_without_role' does not match any known target format
[2021-12-08 17:15:24+0100] INFO  migration - Starting migration for system rules for policy server 'root''
[2021-12-08 17:15:24+0100] INFO  migration - [root] move old 6.2 rule 'hasPolicyServer-root' to 'old_hasPolicyServer-root'
[2021-12-08 17:15:24+0100] INFO  migration - [root] move old 6.2 rule 'root-DP' to 'old_root-DP'
[2021-12-08 17:15:24+0100] INFO  migration - [root] move new 7.0 rule 'new_hasPolicyServer-root' to 'hasPolicyServer-root'
[2021-12-08 17:15:24+0100] INFO  policy.generation - [metrics] Xmx:1015 MB nodes:2 (cached:0) rules:6 (enabled:6) techniques:50 (enabled:50) directives:7 (enabled:7) groups:2 (dynamic:2) parameters:2
[2021-12-08 17:15:24+0100] INFO  migration - [root techniques] delete old rule 'hasPolicyServer-root'
[2021-12-08 17:15:24+0100] INFO  migration - [root techniques] delete old rule 'root-DP'
[2021-12-08 17:15:24+0100] INFO  migration - [root techniques] delete old directive 'common-root'
[2021-12-08 17:15:24+0100] INFO  migration - Migration of system techniques, directives and rules done for 'root'
[2021-12-08 17:15:24+0100] INFO  migration - Deleting old technique 'distributePolicy' information
[2021-12-08 17:15:24+0100] INFO  migration - Deleting old system object for deprecated server-roles: activeTechniqueId=server-roles,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration
[2021-12-08 17:15:24+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleId=server-roles,ou=Rules,ou=Rudder,cn=rudder-configuration
[2021-12-08 17:15:24+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleTarget=special:all_nodes_without_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration
[2021-12-08 17:15:24+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleTarget=special:all_servers_with_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration
[2021-12-08 17:15:24+0100] INFO  migration - Migration of all system configuration to Rudder 7.0: done
[2021-12-08 17:15:24+0100] INFO  bootchecks - Check mandatory DIT entries
[2021-12-08 17:15:24+0100] INFO  policy.generation.timing - Policy generation failed after:           
[2021-12-08 17:15:24+0100] ERROR policy.generation.manager - Error when updating policy, reason was: Cannot build Rule vals <- Could not find configuration vals <- Version '1.0' of technique 'distributePolicy' is not available for directive 'Distribute Policy' [root-distributePolicy]
[2021-12-08 17:15:24+0100] ERROR policy.generation.manager - Policy update error for process '6' at 2021-12-08 17:15:24: Cannot build Rule vals
[2021-12-08 17:15:24+0100] INFO  bootchecks - All the required DIT entries are present in the LDAP directory
[2021-12-08 17:15:24+0100] INFO  bootchecks - Check initialization of User Technique Library
[2021-12-08 17:15:24+0100] INFO  bootchecks - Check that `rudder` global parameter matches default value
[2021-12-08 17:15:24+0100] INFO  bootchecks - Reseting global parameter 'rudder' from system provider to value: {"packages":{"installed_cache_expire":60,"updates_cache_expire":240},"server":{"cf_serverd_bind_address":"::"}}
[2021-12-08 17:15:25+0100] INFO  bootchecks - Check event log migration format 5 -> 6
[2021-12-08 17:15:25+0100] INFO  bootchecks - Check existence of at least one archive of the configuration
[2021-12-08 17:15:25+0100] INFO  migration - No migration detected by migration script (table 'migrationeventlog' is empty or does not exist)
[2021-12-08 17:15:25+0100] INFO  bootchecks - First full archive of configuration-repository items done
[2021-12-08 17:15:25+0100] INFO  bootchecks - Check for force reload of Techniques library

Related issues 3 (0 open — 3 closed)

Related to Rudder - Bug #20359: techniques are seens as deleted even if there are there in Rudder 7.0ReleasedFrançois ARMANDActions
Related to Rudder - User story #20462: Delay policy generation until rudder app is fully bootReleasedNicolas CHARLESActions
Related to Rudder - Bug #20469: error when upgrading from 6.2 to 7.0 in webapp logResolvedActions
Actions
Copy link
 #1

Updated by Nicolas CHARLES almost 3 years ago

  • Translation missing: en.field_tag_list set to Blocks 7.0.0

it fixes itself after 2 restarts

Actions
Copy link
 #2

Updated by Nicolas CHARLES almost 3 years ago

  • Category set to Web - Compliance & node report
  • Target version set to 7.0.0~rc2

It is because it looks for the techniques in the git history: it did not reload the techniques before trying to read them or migrate them

Actions
Copy link
 #3

Updated by Nicolas CHARLES almost 3 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions
Copy link
 #4

Updated by Nicolas CHARLES almost 3 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/4045
Actions
Copy link
 #5

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 7.0.0~rc2 to 7.0.0~rc3
Actions
Copy link
 #6

Updated by François ARMAND almost 3 years ago

  • Related to Bug #20359: techniques are seens as deleted even if there are there in Rudder 7.0 added
Actions
Copy link
 #7

Updated by Nicolas CHARLES almost 3 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #8

Updated by François ARMAND almost 3 years ago

  • Related to User story #20462: Delay policy generation until rudder app is fully boot added
Actions
Copy link
 #9

Updated by Nicolas CHARLES almost 3 years ago

This solved the issue
there still is the weird error message

[2021-12-23 07:57:15+0000] ERROR application - System technique 'server-roles' (1.0)' is deleted in git base. This will likely cause grave problem. You should investigate.
[2021-12-23 07:57:15+0000] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'server-common' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-23 07:57:15+0000] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-relayd' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-23 07:57:16+0000] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-webapp' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-23 07:57:16+0000] ERROR application - System technique 'distributePolicy' (1.0)' is deleted in git base. This will likely cause grave problem. You should investigate.
[2021-12-23 07:57:16+0000] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-apache' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-23 07:57:16+0000] INFO  com.normation.rudder.services.policies.TechniqueAcceptationUpdater - Automatically adding technique 'rudder-service-slapd' in category 'Active techniques used by Rudder (Rudder Internal)' of active techniques library
[2021-12-23 07:57:17+0000] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Reload Technique library at start up
[2021-12-23 07:57:17+0000] ERROR bootchecks - Flag file '/opt/rudder/etc/force_technique_reload' but Techniques library reload failed, cause is: An error occured while updating <- Inconsistency: Error when looking for technique with ID 'server-roles/1.0'. Check technique name and version <- Inconsistency: Error when looking for technique with ID 'distributePolicy/1.0'. Check technique name and version

Actions
Copy link
 #10

Updated by Nicolas CHARLES almost 3 years ago

  • Related to Bug #20469: error when upgrading from 6.2 to 7.0 in webapp log added
Actions
Copy link
 #11

Updated by Vincent MEMBRÉ almost 3 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 7.0.0~rc3 which was released today.

Actions
Copy link

Also available in: Atom PDF