Project

General

Profile

Actions

User story #2738

closed

Handle big files payloads in the Enforce a file content Technique

User story #2738: Handle big files payloads in the Enforce a file content Technique

Added by Matthieu CERDA over 13 years ago. Updated almost 7 years ago.

Status:
Rejected
Priority:
2
Category:
Techniques
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

CFengine is unable to handle too big files payloads in the Enforce a file content Technique.

In Rudder 3.0, the limit is at 16ko and the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Before, the limit is at 4ko ou 8ko, and for example, when trying to apply a backup-manager configuration using it, there is a deployment error, outputting this:

Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add " 

# Should we purge only archives built with BM_ARCHIVE_PREFIX
export BM_ARCHIVE_STRICTPURGE="true" 

# You may want to nice the commands run for archive-creation
# (Recommanded for desktop users.)
# Choose a nice level from -20 (most favorable scheduling) to 19 (least favorable).
export BM_ARCHIVE_NICE_LEVEL="10" 

# The backup method to use.
# Available methods are:
# - tarball
# - tarball-incremental
# - mysql
# - svn
# - pipe
# - none
# If you don't want to use any backup method (you don't want to
# build archives) then choose "none" 
export BM_ARCHIVE_METHOD="tarball-incremental" 

##############################################################
# Encryption - because you cannot trust the place your 
#              archives are
##############################################################

# If you want to encrypt your archives locally, Backup Manager 
# can use GPG while building the archive (so the archive is never
# written to the disk without being encrypted.

# Note: this feature is only possible with the following archive types:
# tar, tar.gz, tar.bz2

# Uncomment the following line if you want to enable encryption
# available method: gpg
export BM_ENCRYPTION_METHOD="gpg" 

# The encryption will be made using a GPG ID
# Examples:
# export BM_ENCRYPTION_RECIPIENT="" 
# export BM_ENCRYPTION_RECIPIENT="" 
export BM_ENCRYPTION_RECIPIENT="0C60B2BA" 

##############################################################
# Section "TARBALL" 
# - Backup method: tarball
#############################################################

# Archive filename format
#     long  : host-full-path-to-folder.tar.gz
#     short : parentfolder.tar.gz
export BM_TARBALL_NAMEFORMAT="long" 

# Type of archives
# Available types are:
#     tar, tar.gz, tar.bz2, tar.lz, dar, zip.
# Make sure to satisfy the appropriate dependencies 
# (bzip2, dar, lzma, ...).
export BM_TARBALL_FILETYPE="tar.bz2" 

# You can choose to build archives remotely over SSH.
# You will then need to fill the BM_UPLOAD_SSH variables 
# (BM_UPLOAD_SSH_HOSTS, BM_UPLOAD_SSH_USER, BM_UPLOAD_SSH_KEY).
# If this boolean is set to true, archive will be saved locally (in 
# BM_REPOSITORY_ROOT but will be built by the remote host).
# Thus, BM_TARBALL_DIRECTORIES will be used to backup remote directories.
# Those archive will be prefixed with the remote host name.
export BM_TARBALL_OVER_SSH="false" 

# Do you want to dereference the files pointed by symlinks ? 
# enter true or false (true can lead to huge archives, be careful).
export BM_TARBALL_DUMPSYMLINKS="false" 

# Targets to backup

# You can use two different variables for defining the targets of 
# your backups, either a simple space-separated list (BM_TARBALL_DIRECTORIES)
# or an array (BM_TARBALL_TARGETS[]).
# Use the first one for simple path that doesn't contain spaces in their name.
# Use the former if you want to specify paths to backups with spaces.

# It's recommanded to use BM_TARBALL_TARGETS[] though.
# Warning! You *must not* use both variables at the same time.
# NOTE: The Debian package will only update BM_TARBALL_DIRECTORIES 

# Paths without spaces in their name:
export BM_TARBALL_DIRECTORIES="/etc /home/* /root /var/log" 

# If one or more of the targets contain a space, use the array:
# declare -a BM_TARBALL_TARGETS
# BM_TARBALL_TARGETS[0]="/etc" 
# BM_TARBALL_TARGETS[1]="/boot" 
# export BM_TARBALL_TARGETS

# Files to exclude when generating tarballs, you can put absolute 
# or relative paths, Bash wildcards are possible.
export BM_TARBALL_BLACKLIST="/var/archives" 

# With the "dar" filetype, you can choose a maximum slice limit.
export BM_TARBALL_SLICESIZE="1000M" 

# Extra options to append to the tarball generation 
# (take care to what you do; this will be silently added to the 
# command line.)
export BM_TARBALL_EXTRA_OPTIONS="" 

##############################################################
# The tarball-incremental method uses the same keys as the 
# tarball method, plus two others.
########################################################
Fatal cfengine error: Can't expand varstring
[2012-07-31 12:15:21] ERROR com.normation.rudder.services.policies.RudderCf3PromisesFileWriterServiceImpl - The generated promises at /var/rudder/share/713c52d2-78d8-4dbe-87be-21457496de32.new/rules/cfengine-community are invalid

I do think this is because CFengine is unable to handle such big variables unless compiled with a larger CF_EXPANDSIZE. But in that case, it is better to use a file template, I think.


Related issues 1 (0 open1 closed)

Related to Rudder - Bug #3574: Using checkgenericfilecontent Technique with 16Ko content leads to errorRejectedActions

Updated by Jonathan CLARKE over 13 years ago Actions #1

  • Target version changed from 2.4.0~beta4 to 2.4.0~rc1

Updated by Nicolas PERRON about 13 years ago Actions #2

  • Target version changed from 2.4.0~rc1 to 2.4.0~rc2

Updated by Nicolas PERRON about 13 years ago Actions #3

  • Target version changed from 2.4.0~rc2 to 2.4.0

Updated by Jonathan CLARKE about 13 years ago Actions #4

  • Assignee deleted (Jonathan CLARKE)

Updated by Nicolas PERRON about 13 years ago Actions #5

  • Target version changed from 2.4.0 to 61

Updated by François ARMAND about 13 years ago Actions #6

  • Target version changed from 61 to 2.4.2

Updated by Nicolas PERRON about 13 years ago Actions #7

  • Target version changed from 2.4.2 to 2.4.3

Updated by François ARMAND about 13 years ago Actions #8

  • Assignee set to Nicolas CHARLES

Updated by Matthieu CERDA almost 13 years ago Actions #9

  • Target version changed from 2.4.3 to 2.4.4

Updated by Jonathan CLARKE almost 13 years ago Actions #10

  • Project changed from Rudder to 24
  • Category deleted (Techniques)

Updated by Nicolas CHARLES almost 13 years ago Actions #11

  • Target version changed from 2.4.4 to 2.6.0~beta1

This has been fixed in later version of CFEngine; i can't really find out when, but with a CFEngine 3.4.1, it doesn't happen
I suspect that https://github.com/cfengine/core/pull/200/ is part of the solution

So by using 3.4.2, this should be automatically fixed

Updated by Nicolas PERRON almost 13 years ago Actions #12

  • Target version changed from 2.6.0~beta1 to 2.6.0~rc1

Now that we use CFEngine 3.4.4 on Rudder 2.6, shouldn't we consider this bug as resolved (unless this is a 2.3, 2.4 or 2.5 bug) ?

Updated by Matthieu CERDA over 12 years ago Actions #13

  • Target version changed from 2.6.0~rc1 to 2.6.0

Updated by Jonathan CLARKE over 12 years ago Actions #14

  • Target version changed from 2.6.0 to 2.6.1

Updated by Nicolas PERRON over 12 years ago Actions #15

  • Target version changed from 2.6.1 to 2.6.2

Updated by Nicolas PERRON over 12 years ago Actions #16

  • Target version changed from 2.6.2 to 2.6.3

Updated by Nicolas PERRON over 12 years ago Actions #17

  • Target version changed from 2.6.3 to 2.6.4

Updated by Nicolas PERRON over 12 years ago Actions #18

  • Target version changed from 2.6.4 to 2.6.5

Updated by Nicolas PERRON over 12 years ago Actions #19

  • Target version changed from 2.6.5 to 2.6.6

Updated by Nicolas PERRON over 12 years ago Actions #20

  • Target version changed from 2.6.6 to 2.6.7

Updated by Nicolas PERRON over 12 years ago Actions #21

  • Target version changed from 2.6.7 to 2.6.8

Updated by Nicolas PERRON over 12 years ago Actions #22

  • Target version changed from 2.6.8 to 2.6.9

Updated by Nicolas PERRON about 12 years ago Actions #23

  • Target version changed from 2.6.9 to 2.6.10

Updated by Nicolas PERRON about 12 years ago Actions #24

It seems to be a duplicate of #3574

Updated by Vincent MEMBRÉ about 12 years ago Actions #25

  • Target version changed from 2.6.10 to 2.6.11

Updated by Vincent MEMBRÉ almost 12 years ago Actions #26

  • Target version changed from 2.6.11 to 2.6.12

Updated by Vincent MEMBRÉ almost 12 years ago Actions #27

  • Target version changed from 2.6.12 to 2.6.13

Updated by Vincent MEMBRÉ over 11 years ago Actions #28

  • Target version changed from 2.6.13 to 2.6.14

Updated by Jonathan CLARKE over 11 years ago Actions #29

  • Target version changed from 2.6.14 to 2.6.16

Updated by Jonathan CLARKE over 11 years ago Actions #30

  • Target version changed from 2.6.16 to 2.6.17

Updated by Nicolas PERRON over 11 years ago Actions #31

  • Target version changed from 2.6.17 to 2.6.18

Updated by Matthieu CERDA over 11 years ago Actions #32

  • Target version changed from 2.6.18 to 2.6.19

Updated by Vincent MEMBRÉ about 11 years ago Actions #33

  • Target version changed from 2.6.19 to 2.6.20

Updated by Nicolas CHARLES almost 11 years ago Actions #34

  • Target version changed from 2.6.20 to 3.0.0

Just so that we keep track on this; CFEngine is tackling the issue of removing the 4K barrier, one step at a time.
This should work correctly on Rudder 3.0 (see ticket https://dev.cfengine.com/issues/2912)

Updated by François ARMAND almost 11 years ago Actions #35

Could someone test it in Rudder 3.0 so that we know if it works (or update the bug on CFEngine side ?) Nicolas ?

Updated by François ARMAND almost 11 years ago Actions #36

  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE

Updated by François ARMAND almost 11 years ago Actions #37

  • Assignee changed from Benoît PECCATTE to Nicolas CHARLES

Updated by Nicolas CHARLES almost 11 years ago Actions #38

In 3.0, the limit is at 16ko, the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Updated by François ARMAND almost 11 years ago Actions #39

  • Description updated (diff)

OK, I'm updating the description accordingly.
Thanks

Updated by François ARMAND almost 11 years ago Actions #40

  • Target version changed from 3.0.0 to 2.10.11

Updated by Vincent MEMBRÉ almost 11 years ago Actions #41

  • Target version changed from 2.10.11 to 2.10.12

Updated by Benoît PECCATTE almost 11 years ago Actions #42

  • Project changed from 24 to Rudder
  • Category set to Techniques

Updated by Vincent MEMBRÉ almost 11 years ago Actions #43

  • Target version changed from 2.10.12 to 2.10.13

Updated by Vincent MEMBRÉ almost 11 years ago Actions #44

  • Target version changed from 2.10.13 to 2.10.14

Updated by Vincent MEMBRÉ over 10 years ago Actions #45

  • Target version changed from 2.10.14 to 2.10.15

Updated by Vincent MEMBRÉ over 10 years ago Actions #46

  • Target version changed from 2.10.15 to 2.10.16

Updated by Vincent MEMBRÉ over 10 years ago Actions #47

  • Target version changed from 2.10.16 to 2.10.17

Updated by Vincent MEMBRÉ over 10 years ago Actions #48

  • Target version changed from 2.10.17 to 2.10.18

Updated by Vincent MEMBRÉ over 10 years ago Actions #49

  • Target version changed from 2.10.18 to 2.10.19

Updated by Vincent MEMBRÉ about 10 years ago Actions #50

  • Target version changed from 2.10.19 to 2.10.20

Updated by Vincent MEMBRÉ about 10 years ago Actions #51

  • Target version changed from 2.10.20 to 2.11.18

Updated by Vincent MEMBRÉ almost 10 years ago Actions #52

  • Target version changed from 2.11.18 to 2.11.19

Updated by Vincent MEMBRÉ almost 10 years ago Actions #53

  • Target version changed from 2.11.19 to 2.11.20

Updated by Vincent MEMBRÉ almost 10 years ago Actions #54

  • Target version changed from 2.11.20 to 2.11.21

Updated by Vincent MEMBRÉ over 9 years ago Actions #55

  • Target version changed from 2.11.21 to 2.11.22

Updated by Vincent MEMBRÉ over 9 years ago Actions #56

  • Target version changed from 2.11.22 to 2.11.23

Updated by Vincent MEMBRÉ over 9 years ago Actions #57

  • Target version changed from 2.11.23 to 2.11.24

Updated by Vincent MEMBRÉ over 9 years ago Actions #58

  • Target version changed from 2.11.24 to 308

Updated by Vincent MEMBRÉ over 9 years ago Actions #59

  • Target version changed from 308 to 3.1.14

Updated by Vincent MEMBRÉ over 9 years ago Actions #60

  • Target version changed from 3.1.14 to 3.1.15

Updated by Vincent MEMBRÉ over 9 years ago Actions #61

  • Target version changed from 3.1.15 to 3.1.16

Updated by Vincent MEMBRÉ over 9 years ago Actions #62

  • Target version changed from 3.1.16 to 3.1.17

Updated by Vincent MEMBRÉ about 9 years ago Actions #63

  • Target version changed from 3.1.17 to 3.1.18

Updated by Vincent MEMBRÉ almost 9 years ago Actions #64

  • Target version changed from 3.1.18 to 3.1.19

Updated by Benoît PECCATTE almost 9 years ago Actions #65

  • Tracker changed from Bug to User story
  • Subject changed from CFengine is unable to handle too big files payloads in the Enforce a file content Technique to Handle big files payloads in the Enforce a file content Technique

Updated by Vincent MEMBRÉ almost 9 years ago Actions #66

  • Target version changed from 3.1.19 to 3.1.20

Updated by Vincent MEMBRÉ over 8 years ago Actions #67

  • Target version changed from 3.1.20 to 3.1.21

Updated by Vincent MEMBRÉ over 8 years ago Actions #68

  • Target version changed from 3.1.21 to 3.1.22

Updated by Vincent MEMBRÉ over 8 years ago Actions #69

  • Target version changed from 3.1.22 to 3.1.23

Updated by Vincent MEMBRÉ over 8 years ago Actions #70

  • Target version changed from 3.1.23 to 3.1.24

Updated by Vincent MEMBRÉ about 8 years ago Actions #71

  • Target version changed from 3.1.24 to 3.1.25

Updated by Benoît PECCATTE about 8 years ago Actions #72

  • Target version changed from 3.1.25 to 4.1.9

Updated by Vincent MEMBRÉ about 8 years ago Actions #73

  • Target version changed from 4.1.9 to 4.1.10

Updated by Vincent MEMBRÉ almost 8 years ago Actions #74

  • Target version changed from 4.1.10 to 4.1.11

Updated by Vincent MEMBRÉ almost 8 years ago Actions #75

  • Target version changed from 4.1.11 to 4.1.12

Updated by Vincent MEMBRÉ over 7 years ago Actions #76

  • Target version changed from 4.1.12 to 4.1.13

Updated by Vincent MEMBRÉ over 7 years ago Actions #77

  • Target version changed from 4.1.13 to 4.1.14

Updated by Benoît PECCATTE over 7 years ago Actions #78

  • Target version changed from 4.1.14 to 4.1.15

Updated by Vincent MEMBRÉ over 7 years ago Actions #79

  • Target version changed from 4.1.15 to 4.1.16

Updated by Vincent MEMBRÉ about 7 years ago Actions #80

  • Target version changed from 4.1.16 to 4.1.17

Updated by Vincent MEMBRÉ about 7 years ago Actions #81

  • Target version changed from 4.1.17 to 4.1.18

Updated by Vincent MEMBRÉ about 7 years ago Actions #82

  • Target version changed from 4.1.18 to 4.1.19

Updated by Alexis Mousset almost 7 years ago Actions #83

  • Target version changed from 4.1.19 to 4.1.20

Updated by Nicolas CHARLES almost 7 years ago Actions #84

  • Status changed from Discussion to Rejected

This is corrected now, large payload are supported

Actions

Also available in: PDF Atom