Bug #13619: Document that resource file of Techniques are shared to nodes with UTF-8 Encoding *only* (breaks for other encoding) - Rudder - Issue Tracker

Actions

Copy link

Bug #13619

closed

Document that resource file of Techniques are shared to nodes with UTF-8 Encoding only (breaks for other encoding)

Added by Nicolas CHARLES about 6 years ago. Updated over 5 years ago.

Status:

Released

Priority:

N/A

Assignee:

Alexis Mousset

Category:

Documentation

Target version:

5.0.10

Pull Request:

https://github.com/Normation/rudder-d...

Severity:

Major - prevents use of part of Rudder | no simple workaround

UX impact:

User visibility:

Getting started - demo | first install | Technique editor and level 1 Techniques

Effort required:

Very Small

Priority:

Name check:

Fix check:

Regression:

Description

somehow, the Rudder-Agent.xml in the generated promises starts with ef bf bd ef bf bd BOM, while the original file starts with ff fe BOM
result is: file is corrupted.

BOM of original file was removed, as a workaround, in #13616, but we need to understand WHY it happened

Hypothesis: scala lib interpreting and adding the BOM, some side effect of #12471, or something else.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Vincent MEMBRÉ about 6 years ago

Target version changed from 4.3-1.11 to 4.3-1.12

Actions

Copy link

Updated by Vincent MEMBRÉ about 6 years ago

Project changed from 78 to Rudder
Subject changed from UTF-8 encoding BOM added to Rudder-Agent.xml file by rudder dsc plugin to Resource file of Techniques are shared to nodes with UTF-8 Encoding
Category changed from System techniques to System techniques
Target version changed from 4.3-1.12 to 4.3.5

If we use a Resource file in a Techniques (using a FILE tag in the metadata.xml) the encoding of the file is enforced to UTF-8, even if the file is in an another encoding (ie an xml we want to use on windows ... )

This is due to this function which reads the content of the file and save it with UTF-8 encoding
https://github.com/Normation/rudder/blob/a5ce997aaf34c64fa1919979e87cfb7c56be60cf/rudder-core/src/main/scala/com/normation/rudder/services/policies/write/PolicyWriterService.scala#L717

Actions

Copy link

Updated by Vincent MEMBRÉ about 6 years ago

Category changed from System techniques to Web - Config management

Actions

Copy link

Updated by Nicolas CHARLES about 6 years ago

EF BF BD is a replacement character for invalid utf8 character
https://apps.timwhitlock.info/unicode/inspect?s=%EF%BF%BD

Actions

Copy link

Updated by Vincent MEMBRÉ about 6 years ago

Target version changed from 4.3.5 to 4.3.6

Actions

Copy link

Updated by Vincent MEMBRÉ about 6 years ago

Target version changed from 4.3.6 to 4.3.7

Actions

Copy link

Updated by François ARMAND about 6 years ago

Here, we need to either document that fact or add an "input encoding" and "output encoding" parameter for resource files.

Actions

Copy link

Updated by François ARMAND about 6 years ago

Subject changed from Resource file of Techniques are shared to nodes with UTF-8 Encoding to Resource file of Techniques are shared to nodes with UTF-8 Encoding *only* (breaks for other encoding)

Actions

Copy link

#10

Updated by Vincent MEMBRÉ about 6 years ago

Target version changed from 4.3.7 to 4.3.8

Actions

Copy link

#11

Updated by Vincent MEMBRÉ almost 6 years ago

Target version changed from 4.3.8 to 4.3.9

Actions

Copy link

#12

Updated by Alexis Mousset almost 6 years ago

Target version changed from 4.3.9 to 4.3.10

Actions

Copy link

#13

Updated by François ARMAND almost 6 years ago

Target version changed from 4.3.10 to 4.3.11

Actions

Copy link

#14

Updated by Vincent MEMBRÉ over 5 years ago

Target version changed from 4.3.11 to 4.3.12

Actions

Copy link

#15

Updated by François ARMAND over 5 years ago

Subject changed from Resource file of Techniques are shared to nodes with UTF-8 Encoding *only* (breaks for other encoding) to Document that resource file of Techniques are shared to nodes with UTF-8 Encoding *only* (breaks for other encoding)
Category changed from Web - Config management to Documentation
Target version changed from 4.3.12 to 5.0.10
Severity changed from Critical - prevents main use of Rudder | no workaround | data loss | security to Major - prevents use of part of Rudder | no simple workaround
User visibility set to Getting started - demo | first install | Technique editor and level 1 Techniques
Effort required set to Very Small
Priority changed from 0 to 90

So, the possibility to add file encoding for all templates is a long term new feature. In the meantime, we need to just clearly document the limitation (it should be ok for most use case, and workaroundable for all even if it supposes a very tedious step of transcoding on the node).

Actions

Copy link

#16