https://issues.rudder.io/https://issues.rudder.io/themes/rudder7/favicon/favicon.ico?17096450182017-02-17T09:56:13ZIssue TrackerRudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=707602017-02-17T09:56:13ZVincent MEMBRÉvme@rudder.io
<ul><li><strong>Category</strong> set to <i>System integration</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>Discussion</i></li><li><strong>Assignee</strong> set to <i>Benoît PECCATTE</i></li></ul><p>send-clean.sh is stateless and do not keep track of what happened, so it may be quite complicated to change it directly.</p>
<p>Maybe inventory should be moved to a "retry" folder that would either: try every 30 minutes to send them / Try one more time then send to fail inventories ?</p>
<p>Benoit do you have more ideas ?</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=707822017-02-17T11:04:00ZJanos Mattyasovszky
<ul></ul><p>Well, it does have some minimal state, because it checks the inventory to be older than X minutes, to ensure the signature was also uploaded, it one was sent on it's way. You could say it is only retried if it is not older than X minutes/hours, if yes, fail it. The downside would be, that if the inventory endpoint is saturated, you'd throw away inventories, so there should be a better check than the is it 5xx or not, so you can handle a plain 500 different from a 503, since 500 could mean the inventory breaks the parsing and should not be tried more than X times, and a 503 means that it has nothing to do with the inventory, but the saturation of the endpoint.</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=735722017-03-21T16:41:27ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Severity</strong> set to <i>Minor - inconvenience | misleading | easy workaround</i></li><li><strong>User visibility</strong> set to <i>Infrequent - complex configurations | third party integrations</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=738432017-03-27T14:07:05ZJonathan CLARKEjonathan.clarke@normation.com
<ul><li><strong>Effort required</strong> set to <i>Small</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=739752017-03-28T14:31:23ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Priority</strong> set to <i>14</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=753022017-04-18T10:19:40ZJanos Mattyasovszky
<ul></ul><p>Well, it's not completely stateless, since it only sends the inventories, if (IIRC) 2mins have elapsed, so any possible delayed signatures can also be uploaded if the sender supports signing inventories.</p>
<p>The same idea could also be used to not try to send inventories that are older than (pi/random) 4 hours.</p>
<p>This could ensure that you don't saturate your sending queue of (default) 50 with broken inventories and never process anything never, and they just keep piling up.</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=756742017-04-25T08:18:36ZJonathan CLARKEjonathan.clarke@normation.com
<ul><li><strong>Status</strong> changed from <i>Discussion</i> to <i>New</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=757562017-04-25T08:19:25ZJonathan CLARKEjonathan.clarke@normation.com
<ul><li><strong>Assignee</strong> deleted (<del><i>Benoît PECCATTE</i></del>)</li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=787602017-06-26T10:39:32ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Priority</strong> changed from <i>14</i> to <i>24</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=839102017-11-06T13:06:32ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Priority</strong> changed from <i>24</i> to <i>32</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=860582017-12-18T14:02:46ZJanos Mattyasovszky
<ul><li><strong>Priority</strong> changed from <i>32</i> to <i>31</i></li></ul><p>Florian found this as well: As soon you have 50 failed inventories, you get no inventory uploaded, because they get retried until the queue is saturated and send-clean exists with 1...</p>
<p>A valid idea would be to put all failed to a <code>failed</code> folder, and always process those after all the "fresh" ones, so you make sure you always try to process those which have not failed before</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=860592017-12-18T15:02:07ZFlorian Heigl
<ul><li><strong>Severity</strong> changed from <i>Minor - inconvenience | misleading | easy workaround</i> to <i>Major - prevents use of part of Rudder | no simple workaround</i></li><li><strong>Priority</strong> changed from <i>31</i> to <i>48</i></li></ul><p>I found something, not exactly the same.</p>
<p>The inventory processing aborts once an exit 1 is propagated from send_clean.<br />This happens for any bad inventory that is moved to "failed"</p>
<p>The bad inventories are normally distributed among all inventories.<br />That means each inventory run can hit one.<br />At that point this run aborts, so it will process not 50 inventories, but some number between 1 and 50.<br />Without doing too much math there is a point where the remaining runs are not sufficient to process the non-broken inventories.<br />This, of course keeps repeating forever.</p>
<p>Simply put, if you inject 200 bad inventories over the name space of all inventories you will NEVER AGAIN reach a point where you'd have working inventories.</p>
<p>Please note to fix that the caller (policy that aborts the loop) should be fixed, not send_clean itself. Alternatively, both could be fixed to have a special return code for this case.<br />There are surely some cases where it's valid to just abort.</p>
<p>It is not valid to abort "upload inventories" because of this though, since uploading DOES work.<br />(unless all failed, which would be a good reason to abort and be verbosely sad)</p>
<p>I'm changing this to SEV: major, and would suggest to consider turning it into critical. It could even cause data loss if there's some very very bad coincidence. (Accepted nodes not getting updates, not for new nodes since they will probably not even get in)<br />Visibility is infrequent, can only happen on setups > 288 hosts though.</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=860602017-12-18T15:31:44ZJanos Matya
<ul></ul><p>The problem is that the api returns 5xx on a Java issue (the ugly way heapdumping) and also on queue full.</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=880312018-02-09T14:27:27ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Target version</strong> set to <i>4.1.10</i></li><li><strong>Priority</strong> changed from <i>48</i> to <i>47</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=880322018-02-09T14:27:31ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In progress</i></li><li><strong>Assignee</strong> set to <i>Benoît PECCATTE</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=880342018-02-09T14:42:30ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Status</strong> changed from <i>In progress</i> to <i>Pending technical review</i></li><li><strong>Assignee</strong> changed from <i>Benoît PECCATTE</i> to <i>Alexis Mousset</i></li><li><strong>Pull Request</strong> set to <i>https://github.com/Normation/rudder-techniques/pull/1249</i></li></ul><p>PR <a class="external" href="https://github.com/Normation/rudder-techniques/pull/1249">https://github.com/Normation/rudder-techniques/pull/1249</a></p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=881132018-02-13T14:09:45ZRudder Quality Assistant
<ul><li><strong>Assignee</strong> changed from <i>Alexis Mousset</i> to <i>Benoît PECCATTE</i></li></ul> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=881232018-02-13T14:55:58ZBenoît PECCATTEbenoit.peccatte@rudder.io
<ul><li><strong>Status</strong> changed from <i>Pending technical review</i> to <i>Pending release</i></li></ul><p>Applied in changeset <a class="changeset" title="Fixes #10019: send-clean.sh blocks on inventory parsing error" href="https://issues.rudder.io/projects/rudder/repository/rudder-techniques/revisions/508a8a66244a0fac4c54e018509f8be9e39d1b67">rudder-techniques|508a8a66244a0fac4c54e018509f8be9e39d1b67</a>.</p> Rudder - Bug #10019: send-clean.sh blocks on inventory parsing errorhttps://issues.rudder.io/issues/10019?journal_id=888412018-03-01T11:13:03ZVincent MEMBRÉvme@rudder.io
<ul><li><strong>Status</strong> changed from <i>Pending release</i> to <i>Released</i></li></ul><p>This bug has been fixed in Rudder 4.1.10 and 4.2.4 which were released today.</p>
<ul>
<li>4.1.10: <a href="http://www.rudder-project.org/pipermail/rudder-announce/2018-March/000262.html" class="external">Announce</a> <a href="http://www.rudder-project.org/changelog-4.1#4.1.10" class="external">Changelog</a></li>
<li>4.2.4: <a href="http://www.rudder-project.org/pipermail/rudder-announce/2018-March/000263.html" class="external">Announce</a> <a href="http://www.rudder-project.org/changelog-4.2#4.2.4" class="external">Changelog</a></li>
<li>Download: <a class="external" href="https://www.rudder-project.org/site/get-rudder/downloads/">https://www.rudder-project.org/site/get-rudder/downloads/</a></li>
</ul>