Project

General

Profile

Actions

User story #2738

closed

Handle big files payloads in the Enforce a file content Technique

Added by Matthieu CERDA over 12 years ago. Updated over 5 years ago.

Status:
Rejected
Priority:
2
Category:
Techniques
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

CFengine is unable to handle too big files payloads in the Enforce a file content Technique.

In Rudder 3.0, the limit is at 16ko and the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Before, the limit is at 4ko ou 8ko, and for example, when trying to apply a backup-manager configuration using it, there is a deployment error, outputting this:

Expansion overflow constructing string. Increase CF_EXPANDSIZE macro. Tried to add " 

# Should we purge only archives built with BM_ARCHIVE_PREFIX
export BM_ARCHIVE_STRICTPURGE="true" 

# You may want to nice the commands run for archive-creation
# (Recommanded for desktop users.)
# Choose a nice level from -20 (most favorable scheduling) to 19 (least favorable).
export BM_ARCHIVE_NICE_LEVEL="10" 

# The backup method to use.
# Available methods are:
# - tarball
# - tarball-incremental
# - mysql
# - svn
# - pipe
# - none
# If you don't want to use any backup method (you don't want to
# build archives) then choose "none" 
export BM_ARCHIVE_METHOD="tarball-incremental" 

##############################################################
# Encryption - because you cannot trust the place your 
#              archives are
##############################################################

# If you want to encrypt your archives locally, Backup Manager 
# can use GPG while building the archive (so the archive is never
# written to the disk without being encrypted.

# Note: this feature is only possible with the following archive types:
# tar, tar.gz, tar.bz2

# Uncomment the following line if you want to enable encryption
# available method: gpg
export BM_ENCRYPTION_METHOD="gpg" 

# The encryption will be made using a GPG ID
# Examples:
# export BM_ENCRYPTION_RECIPIENT="" 
# export BM_ENCRYPTION_RECIPIENT="" 
export BM_ENCRYPTION_RECIPIENT="0C60B2BA" 

##############################################################
# Section "TARBALL" 
# - Backup method: tarball
#############################################################

# Archive filename format
#     long  : host-full-path-to-folder.tar.gz
#     short : parentfolder.tar.gz
export BM_TARBALL_NAMEFORMAT="long" 

# Type of archives
# Available types are:
#     tar, tar.gz, tar.bz2, tar.lz, dar, zip.
# Make sure to satisfy the appropriate dependencies 
# (bzip2, dar, lzma, ...).
export BM_TARBALL_FILETYPE="tar.bz2" 

# You can choose to build archives remotely over SSH.
# You will then need to fill the BM_UPLOAD_SSH variables 
# (BM_UPLOAD_SSH_HOSTS, BM_UPLOAD_SSH_USER, BM_UPLOAD_SSH_KEY).
# If this boolean is set to true, archive will be saved locally (in 
# BM_REPOSITORY_ROOT but will be built by the remote host).
# Thus, BM_TARBALL_DIRECTORIES will be used to backup remote directories.
# Those archive will be prefixed with the remote host name.
export BM_TARBALL_OVER_SSH="false" 

# Do you want to dereference the files pointed by symlinks ? 
# enter true or false (true can lead to huge archives, be careful).
export BM_TARBALL_DUMPSYMLINKS="false" 

# Targets to backup

# You can use two different variables for defining the targets of 
# your backups, either a simple space-separated list (BM_TARBALL_DIRECTORIES)
# or an array (BM_TARBALL_TARGETS[]).
# Use the first one for simple path that doesn't contain spaces in their name.
# Use the former if you want to specify paths to backups with spaces.

# It's recommanded to use BM_TARBALL_TARGETS[] though.
# Warning! You *must not* use both variables at the same time.
# NOTE: The Debian package will only update BM_TARBALL_DIRECTORIES 

# Paths without spaces in their name:
export BM_TARBALL_DIRECTORIES="/etc /home/* /root /var/log" 

# If one or more of the targets contain a space, use the array:
# declare -a BM_TARBALL_TARGETS
# BM_TARBALL_TARGETS[0]="/etc" 
# BM_TARBALL_TARGETS[1]="/boot" 
# export BM_TARBALL_TARGETS

# Files to exclude when generating tarballs, you can put absolute 
# or relative paths, Bash wildcards are possible.
export BM_TARBALL_BLACKLIST="/var/archives" 

# With the "dar" filetype, you can choose a maximum slice limit.
export BM_TARBALL_SLICESIZE="1000M" 

# Extra options to append to the tarball generation 
# (take care to what you do; this will be silently added to the 
# command line.)
export BM_TARBALL_EXTRA_OPTIONS="" 

##############################################################
# The tarball-incremental method uses the same keys as the 
# tarball method, plus two others.
########################################################
Fatal cfengine error: Can't expand varstring
[2012-07-31 12:15:21] ERROR com.normation.rudder.services.policies.RudderCf3PromisesFileWriterServiceImpl - The generated promises at /var/rudder/share/713c52d2-78d8-4dbe-87be-21457496de32.new/rules/cfengine-community are invalid

I do think this is because CFengine is unable to handle such big variables unless compiled with a larger CF_EXPANDSIZE. But in that case, it is better to use a file template, I think.


Related issues 1 (0 open1 closed)

Related to Rudder - Bug #3574: Using checkgenericfilecontent Technique with 16Ko content leads to errorRejectedActions
Actions #1

Updated by Jonathan CLARKE about 12 years ago

  • Target version changed from 2.4.0~beta4 to 2.4.0~rc1
Actions #2

Updated by Nicolas PERRON about 12 years ago

  • Target version changed from 2.4.0~rc1 to 2.4.0~rc2
Actions #3

Updated by Nicolas PERRON almost 12 years ago

  • Target version changed from 2.4.0~rc2 to 2.4.0
Actions #4

Updated by Jonathan CLARKE almost 12 years ago

  • Assignee deleted (Jonathan CLARKE)
Actions #5

Updated by Nicolas PERRON almost 12 years ago

  • Target version changed from 2.4.0 to 61
Actions #6

Updated by François ARMAND almost 12 years ago

  • Target version changed from 61 to 2.4.2
Actions #7

Updated by Nicolas PERRON almost 12 years ago

  • Target version changed from 2.4.2 to 2.4.3
Actions #8

Updated by François ARMAND almost 12 years ago

  • Assignee set to Nicolas CHARLES
Actions #9

Updated by Matthieu CERDA almost 12 years ago

  • Target version changed from 2.4.3 to 2.4.4
Actions #10

Updated by Jonathan CLARKE almost 12 years ago

  • Project changed from Rudder to 24
  • Category deleted (Techniques)
Actions #11

Updated by Nicolas CHARLES almost 12 years ago

  • Target version changed from 2.4.4 to 2.6.0~beta1

This has been fixed in later version of CFEngine; i can't really find out when, but with a CFEngine 3.4.1, it doesn't happen
I suspect that https://github.com/cfengine/core/pull/200/ is part of the solution

So by using 3.4.2, this should be automatically fixed

Actions #12

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.6.0~beta1 to 2.6.0~rc1

Now that we use CFEngine 3.4.4 on Rudder 2.6, shouldn't we consider this bug as resolved (unless this is a 2.3, 2.4 or 2.5 bug) ?

Actions #13

Updated by Matthieu CERDA over 11 years ago

  • Target version changed from 2.6.0~rc1 to 2.6.0
Actions #14

Updated by Jonathan CLARKE over 11 years ago

  • Target version changed from 2.6.0 to 2.6.1
Actions #15

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.6.1 to 2.6.2
Actions #16

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.6.2 to 2.6.3
Actions #17

Updated by Nicolas PERRON over 11 years ago

  • Target version changed from 2.6.3 to 2.6.4
Actions #18

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.4 to 2.6.5
Actions #19

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.5 to 2.6.6
Actions #20

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.6 to 2.6.7
Actions #21

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.7 to 2.6.8
Actions #22

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.8 to 2.6.9
Actions #23

Updated by Nicolas PERRON about 11 years ago

  • Target version changed from 2.6.9 to 2.6.10
Actions #24

Updated by Nicolas PERRON about 11 years ago

It seems to be a duplicate of #3574

Actions #25

Updated by Vincent MEMBRÉ almost 11 years ago

  • Target version changed from 2.6.10 to 2.6.11
Actions #26

Updated by Vincent MEMBRÉ over 10 years ago

  • Target version changed from 2.6.11 to 2.6.12
Actions #27

Updated by Vincent MEMBRÉ over 10 years ago

  • Target version changed from 2.6.12 to 2.6.13
Actions #28

Updated by Vincent MEMBRÉ over 10 years ago

  • Target version changed from 2.6.13 to 2.6.14
Actions #29

Updated by Jonathan CLARKE over 10 years ago

  • Target version changed from 2.6.14 to 2.6.16
Actions #30

Updated by Jonathan CLARKE over 10 years ago

  • Target version changed from 2.6.16 to 2.6.17
Actions #31

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.6.17 to 2.6.18
Actions #32

Updated by Matthieu CERDA about 10 years ago

  • Target version changed from 2.6.18 to 2.6.19
Actions #33

Updated by Vincent MEMBRÉ about 10 years ago

  • Target version changed from 2.6.19 to 2.6.20
Actions #34

Updated by Nicolas CHARLES almost 10 years ago

  • Target version changed from 2.6.20 to 3.0.0

Just so that we keep track on this; CFEngine is tackling the issue of removing the 4K barrier, one step at a time.
This should work correctly on Rudder 3.0 (see ticket https://dev.cfengine.com/issues/2912)

Actions #35

Updated by François ARMAND almost 10 years ago

Could someone test it in Rudder 3.0 so that we know if it works (or update the bug on CFEngine side ?) Nicolas ?

Actions #36

Updated by François ARMAND almost 10 years ago

  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
Actions #37

Updated by François ARMAND almost 10 years ago

  • Assignee changed from Benoît PECCATTE to Nicolas CHARLES
Actions #38

Updated by Nicolas CHARLES almost 10 years ago

In 3.0, the limit is at 16ko, the error is at the promise validation time
The error message is:

input buffer overflow, can't enlarge buffer because scanner uses REJECT

Actions #39

Updated by François ARMAND almost 10 years ago

  • Description updated (diff)

OK, I'm updating the description accordingly.
Thanks

Actions #40

Updated by François ARMAND almost 10 years ago

  • Target version changed from 3.0.0 to 2.10.11
Actions #41

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.11 to 2.10.12
Actions #42

Updated by Benoît PECCATTE over 9 years ago

  • Project changed from 24 to Rudder
  • Category set to Techniques
Actions #43

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.12 to 2.10.13
Actions #44

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.13 to 2.10.14
Actions #45

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.14 to 2.10.15
Actions #46

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.15 to 2.10.16
Actions #47

Updated by Vincent MEMBRÉ over 9 years ago

  • Target version changed from 2.10.16 to 2.10.17
Actions #48

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.17 to 2.10.18
Actions #49

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.18 to 2.10.19
Actions #50

Updated by Vincent MEMBRÉ about 9 years ago

  • Target version changed from 2.10.19 to 2.10.20
Actions #51

Updated by Vincent MEMBRÉ almost 9 years ago

  • Target version changed from 2.10.20 to 2.11.18
Actions #52

Updated by Vincent MEMBRÉ almost 9 years ago

  • Target version changed from 2.11.18 to 2.11.19
Actions #53

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.11.19 to 2.11.20
Actions #54

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.11.20 to 2.11.21
Actions #55

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.11.21 to 2.11.22
Actions #56

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.11.22 to 2.11.23
Actions #57

Updated by Vincent MEMBRÉ over 8 years ago

  • Target version changed from 2.11.23 to 2.11.24
Actions #58

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 2.11.24 to 308
Actions #59

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 308 to 3.1.14
Actions #60

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 3.1.14 to 3.1.15
Actions #61

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 3.1.15 to 3.1.16
Actions #62

Updated by Vincent MEMBRÉ about 8 years ago

  • Target version changed from 3.1.16 to 3.1.17
Actions #63

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 3.1.17 to 3.1.18
Actions #64

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 3.1.18 to 3.1.19
Actions #65

Updated by Benoît PECCATTE over 7 years ago

  • Tracker changed from Bug to User story
  • Subject changed from CFengine is unable to handle too big files payloads in the Enforce a file content Technique to Handle big files payloads in the Enforce a file content Technique
Actions #66

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.19 to 3.1.20
Actions #67

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.20 to 3.1.21
Actions #68

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.21 to 3.1.22
Actions #69

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 3.1.22 to 3.1.23
Actions #70

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 3.1.23 to 3.1.24
Actions #71

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 3.1.24 to 3.1.25
Actions #72

Updated by Benoît PECCATTE about 7 years ago

  • Target version changed from 3.1.25 to 4.1.9
Actions #73

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.9 to 4.1.10
Actions #74

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.10 to 4.1.11
Actions #75

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.11 to 4.1.12
Actions #76

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.12 to 4.1.13
Actions #77

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.13 to 4.1.14
Actions #78

Updated by Benoît PECCATTE over 6 years ago

  • Target version changed from 4.1.14 to 4.1.15
Actions #79

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.15 to 4.1.16
Actions #80

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.16 to 4.1.17
Actions #81

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.17 to 4.1.18
Actions #82

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.18 to 4.1.19
Actions #83

Updated by Alexis Mousset almost 6 years ago

  • Target version changed from 4.1.19 to 4.1.20
Actions #84

Updated by Nicolas CHARLES over 5 years ago

  • Status changed from Discussion to Rejected

This is corrected now, large payload are supported

Actions

Also available in: Atom PDF