Project

General

Profile

Actions

User story #2738

closed

Handle big files payloads in the Enforce a file content Technique

Added by Matthieu CERDA over 11 years ago. Updated about 5 years ago.

Status:
Rejected
Priority:
2
Category:
Techniques
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

CFengine is unable to handle too big files payloads in the Enforce a file content Technique.

In Rudder 3.0, the limit is at 16ko and the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Before, the limit is at 4ko ou 8ko, and for example, when trying to apply a backup-manager configuration using it, there is a deployment error, outputting this:

Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add " 

# Should we purge only archives built with BM_ARCHIVE_PREFIX
export BM_ARCHIVE_STRICTPURGE="true" 

# You may want to nice the commands run for archive-creation
# (Recommanded for desktop users.)
# Choose a nice level from -20 (most favorable scheduling) to 19 (least favorable).
export BM_ARCHIVE_NICE_LEVEL="10" 

# The backup method to use.
# Available methods are:
# - tarball
# - tarball-incremental
# - mysql
# - svn
# - pipe
# - none
# If you don't want to use any backup method (you don't want to
# build archives) then choose "none" 
export BM_ARCHIVE_METHOD="tarball-incremental" 

##############################################################
# Encryption - because you cannot trust the place your 
#              archives are
##############################################################

# If you want to encrypt your archives locally, Backup Manager 
# can use GPG while building the archive (so the archive is never
# written to the disk without being encrypted.

# Note: this feature is only possible with the following archive types:
# tar, tar.gz, tar.bz2

# Uncomment the following line if you want to enable encryption
# available method: gpg
export BM_ENCRYPTION_METHOD="gpg" 

# The encryption will be made using a GPG ID
# Examples:
# export BM_ENCRYPTION_RECIPIENT="" 
# export BM_ENCRYPTION_RECIPIENT="" 
export BM_ENCRYPTION_RECIPIENT="0C60B2BA" 

##############################################################
# Section "TARBALL" 
# - Backup method: tarball
#############################################################

# Archive filename format
#     long  : host-full-path-to-folder.tar.gz
#     short : parentfolder.tar.gz
export BM_TARBALL_NAMEFORMAT="long" 

# Type of archives
# Available types are:
#     tar, tar.gz, tar.bz2, tar.lz, dar, zip.
# Make sure to satisfy the appropriate dependencies 
# (bzip2, dar, lzma, ...).
export BM_TARBALL_FILETYPE="tar.bz2" 

# You can choose to build archives remotely over SSH.
# You will then need to fill the BM_UPLOAD_SSH variables 
# (BM_UPLOAD_SSH_HOSTS, BM_UPLOAD_SSH_USER, BM_UPLOAD_SSH_KEY).
# If this boolean is set to true, archive will be saved locally (in 
# BM_REPOSITORY_ROOT but will be built by the remote host).
# Thus, BM_TARBALL_DIRECTORIES will be used to backup remote directories.
# Those archive will be prefixed with the remote host name.
export BM_TARBALL_OVER_SSH="false" 

# Do you want to dereference the files pointed by symlinks ? 
# enter true or false (true can lead to huge archives, be careful).
export BM_TARBALL_DUMPSYMLINKS="false" 

# Targets to backup

# You can use two different variables for defining the targets of 
# your backups, either a simple space-separated list (BM_TARBALL_DIRECTORIES)
# or an array (BM_TARBALL_TARGETS[]).
# Use the first one for simple path that doesn't contain spaces in their name.
# Use the former if you want to specify paths to backups with spaces.

# It's recommanded to use BM_TARBALL_TARGETS[] though.
# Warning! You *must not* use both variables at the same time.
# NOTE: The Debian package will only update BM_TARBALL_DIRECTORIES 

# Paths without spaces in their name:
export BM_TARBALL_DIRECTORIES="/etc /home/* /root /var/log" 

# If one or more of the targets contain a space, use the array:
# declare -a BM_TARBALL_TARGETS
# BM_TARBALL_TARGETS[0]="/etc" 
# BM_TARBALL_TARGETS[1]="/boot" 
# export BM_TARBALL_TARGETS

# Files to exclude when generating tarballs, you can put absolute 
# or relative paths, Bash wildcards are possible.
export BM_TARBALL_BLACKLIST="/var/archives" 

# With the "dar" filetype, you can choose a maximum slice limit.
export BM_TARBALL_SLICESIZE="1000M" 

# Extra options to append to the tarball generation 
# (take care to what you do; this will be silently added to the 
# command line.)
export BM_TARBALL_EXTRA_OPTIONS="" 

##############################################################
# The tarball-incremental method uses the same keys as the 
# tarball method, plus two others.
########################################################
Fatal cfengine error: Can't expand varstring
[2012-07-31 12:15:21] ERROR com.normation.rudder.services.policies.RudderCf3PromisesFileWriterServiceImpl - The generated promises at /var/rudder/share/713c52d2-78d8-4dbe-87be-21457496de32.new/rules/cfengine-community are invalid

I do think this is because CFengine is unable to handle such big variables unless compiled with a larger CF_EXPANDSIZE. But in that case, it is better to use a file template, I think.


Related issues 1 (0 open1 closed)

Related to Rudder - Bug #3574: Using checkgenericfilecontent Technique with 16Ko content leads to errorRejectedActions
Actions #1

Updated by Jonathan CLARKE over 11 years ago

  • Target version changed from 2.4.0~beta4 to 2.4.0~rc1
Actions #2

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.4.0~rc1 to 2.4.0~rc2
Actions #3

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.4.0~rc2 to 2.4.0
Actions #4

Updated by Jonathan CLARKE over 11 years ago

  • Assignee deleted (Jonathan CLARKE)
Actions #5

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.4.0 to 61
Actions #6

Updated by François ARMAND over 11 years ago

  • Target version changed from 61 to 2.4.2
Actions #7

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.4.2 to 2.4.3
Actions #8

Updated by François ARMAND over 11 years ago

  • Assignee set to Nicolas CHARLES
Actions #9

Updated by Matthieu CERDA about 11 years ago

  • Target version changed from 2.4.3 to 2.4.4
Actions #10

Updated by Jonathan CLARKE about 11 years ago

  • Project changed from Rudder to 24
  • Category deleted (Techniques)
Actions #11

Updated by Nicolas CHARLES about 11 years ago

  • Target version changed from 2.4.4 to 2.6.0~beta1

This has been fixed in later version of CFEngine; i can't really find out when, but with a CFEngine 3.4.1, it doesn't happen
I suspect that https://github.com/cfengine/core/pull/200/ is part of the solution

So by using 3.4.2, this should be automatically fixed

Actions #12

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.0~beta1 to 2.6.0~rc1

Now that we use CFEngine 3.4.4 on Rudder 2.6, shouldn't we consider this bug as resolved (unless this is a 2.3, 2.4 or 2.5 bug) ?

Actions #13

Updated by Matthieu CERDA almost 11 years ago

  • Target version changed from 2.6.0~rc1 to 2.6.0
Actions #14

Updated by Jonathan CLARKE almost 11 years ago

  • Target version changed from 2.6.0 to 2.6.1
Actions #15

Updated by Nicolas PERRON almost 11 years ago

  • Target version changed from 2.6.1 to 2.6.2
Actions #16

Updated by Nicolas PERRON almost 11 years ago

  • Target version changed from 2.6.2 to 2.6.3
Actions #17

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.3 to 2.6.4
Actions #18

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.4 to 2.6.5
Actions #19

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.5 to 2.6.6
Actions #20

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.6 to 2.6.7
Actions #21

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.7 to 2.6.8
Actions #22

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.8 to 2.6.9
Actions #23

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.9 to 2.6.10
Actions #24

Updated by Nicolas PERRON over 10 years ago

It seems to be a duplicate of #3574

Actions #25

Updated by Vincent MEMBRÉ over 10 years ago

  • Target version changed from 2.6.10 to 2.6.11
Actions #26

Updated by Vincent MEMBRÉ about 10 years ago

  • Target version changed from 2.6.11 to 2.6.12
Actions #27

Updated by Vincent MEMBRÉ about 10 years ago

  • Target version changed from 2.6.12 to 2.6.13
Actions #28

Updated by Vincent MEMBRÉ almost 10 years ago

  • Target version changed from 2.6.13 to 2.6.14
Actions #29

Updated by Jonathan CLARKE almost 10 years ago

  • Target version changed from 2.6.14 to 2.6.16
Actions #30

Updated by Jonathan CLARKE almost 10 years ago

  • Target version changed from 2.6.16 to 2.6.17
Actions #31

Updated by Nicolas PERRON over 9 years ago

  • Target version changed from 2.6.17 to 2.6.18
Actions #32

Updated by Matthieu CERDA over 9 years ago

  • Target version changed from 2.6.18 to 2.6.19
Actions #33

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.6.19 to 2.6.20
Actions #34

Updated by Nicolas CHARLES about 9 years ago

  • Target version changed from 2.6.20 to 3.0.0

Just so that we keep track on this; CFEngine is tackling the issue of removing the 4K barrier, one step at a time.
This should work correctly on Rudder 3.0 (see ticket https://dev.cfengine.com/issues/2912)

Actions #35

Updated by François ARMAND about 9 years ago

Could someone test it in Rudder 3.0 so that we know if it works (or update the bug on CFEngine side ?) Nicolas ?

Actions #36

Updated by François ARMAND about 9 years ago

  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
Actions #37

Updated by François ARMAND about 9 years ago

  • Assignee changed from Benoît PECCATTE to Nicolas CHARLES
Actions #38

Updated by Nicolas CHARLES about 9 years ago

In 3.0, the limit is at 16ko, the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Actions #39

Updated by François ARMAND about 9 years ago

  • Description updated (diff)

OK, I'm updating the description accordingly.
Thanks

Actions #40

Updated by François ARMAND about 9 years ago

  • Target version changed from 3.0.0 to 2.10.11
Actions #41

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.11 to 2.10.12
Actions #42

Updated by Benoît PECCATTE about 9 years ago

  • Project changed from 24 to Rudder
  • Category set to Techniques
Actions #43

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.12 to 2.10.13
Actions #44

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.13 to 2.10.14
Actions #45

Updated by Vincent MEMBRÉ almost 9 years ago

  • Target version changed from 2.10.14 to 2.10.15
Actions #46

Updated by Vincent MEMBRÉ almost 9 years ago

  • Target version changed from 2.10.15 to 2.10.16
Actions #47

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.10.16 to 2.10.17
Actions #48

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.10.17 to 2.10.18
Actions #49

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.10.18 to 2.10.19
Actions #50

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.10.19 to 2.10.20
Actions #51

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.10.20 to 2.11.18
Actions #52

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 2.11.18 to 2.11.19
Actions #53

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 2.11.19 to 2.11.20
Actions #54

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 2.11.20 to 2.11.21
Actions #55

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 2.11.21 to 2.11.22
Actions #56

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 2.11.22 to 2.11.23
Actions #57

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 2.11.23 to 2.11.24
Actions #58

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 2.11.24 to 308
Actions #59

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 308 to 3.1.14
Actions #60

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.14 to 3.1.15
Actions #61

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.15 to 3.1.16
Actions #62

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.16 to 3.1.17
Actions #63

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.17 to 3.1.18
Actions #64

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 3.1.18 to 3.1.19
Actions #65

Updated by Benoît PECCATTE about 7 years ago

  • Tracker changed from Bug to User story
  • Subject changed from CFengine is unable to handle too big files payloads in the Enforce a file content Technique to Handle big files payloads in the Enforce a file content Technique
Actions #66

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 3.1.19 to 3.1.20
Actions #67

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 3.1.20 to 3.1.21
Actions #68

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 3.1.21 to 3.1.22
Actions #69

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 3.1.22 to 3.1.23
Actions #70

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 3.1.23 to 3.1.24
Actions #71

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 3.1.24 to 3.1.25
Actions #72

Updated by Benoît PECCATTE over 6 years ago

  • Target version changed from 3.1.25 to 4.1.9
Actions #73

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.9 to 4.1.10
Actions #74

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.10 to 4.1.11
Actions #75

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.11 to 4.1.12
Actions #76

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.12 to 4.1.13
Actions #77

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.13 to 4.1.14
Actions #78

Updated by Benoît PECCATTE over 5 years ago

  • Target version changed from 4.1.14 to 4.1.15
Actions #79

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.15 to 4.1.16
Actions #80

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.16 to 4.1.17
Actions #81

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.17 to 4.1.18
Actions #82

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.18 to 4.1.19
Actions #83

Updated by Alexis Mousset about 5 years ago

  • Target version changed from 4.1.19 to 4.1.20
Actions #84

Updated by Nicolas CHARLES about 5 years ago

  • Status changed from Discussion to Rejected

This is corrected now, large payload are supported

Actions

Also available in: Atom PDF