Project

General

Profile

Actions

User story #9707

closed

Add a Relay API: share files between nodes, launch run on remote run behind relay

Added by Alexis Mousset over 7 years ago. Updated about 7 years ago.

Status:
Released
Priority:
N/A
Assignee:
-
Category:
API
Target version:
UX impact:
Suggestion strength:
User visibility:
Effort required:
Name check:
Fix check:
Regression:

Description

One of Rudder 4.1 new features will be relay APIs. This is the first attempt to describe it.

There are currently 2 API entries we want to add: remote-run and share-files.
They will both be under /rudder/relay-api itself under https://<server>:<port>/ like the current api is.
/rudder is the common root for all rudder service
/relay-api is different from existing api to avoid conflicts with them when it will be installed on the server

1. Remote-run:
The goal is to make a given relay call "rudder remote run" on one of its attached node
The API will be under /rudder/relay-api/remote-run
GET remote-run/node/<node-uuid>
GET remote-run/all
GET remote-run/nodes

Parameters:
- output = keep / discard : to keep the output of the remote-run call or discard its content
- async = yes / no : yes to ignore the return code of the call and return immediately, no to wait until the end of the call and get the return code
- classes = XXX : list of cfengine classes to set during the remote call
- nodes = uuid,... : list of uuid to call in the "/nodes" case

Behavior:
- Loop on all nodes
- Find its hostname from its uuid in a matching file created by promise generation on the server
- The call is descending, so we don't care about host that do not exist
- The call is descending, so we will only accept calls from the policy server
- Call rudder remote
- prefix the remote output lines with <uuid>: to make sure the caller can parse output during async call on multiple nodes
- surround the output with json format lines and include return code, duration and stderr (-> we should escape the output for use within a json string)

2. Share Files

The goal is to share files between an agent and another one via their policy server.
The API will be under /rudder/relay-api/shared-files
POST shared-files/node/<target_uuid>/<file_id> (share a file content with the target uuid using a given file id)
HEAD shared-files/node/<target_uuid>/<file_id> (ask if a file with the given file id is already shared with the given target id)

Parameters:
- source-uuid: the uuid of the node sharing a file with the target uuid
- signature: the file's signature in case of a POST, to authenticate the source node
- hash: the file's hash in case of a HEAD to know is the version already present matches
POST Behavior:
- if the target is known by the local relay
> validate the signature
> store the file in /var/rudder/shared-files-nodes/[..<relay-uuid>..]/<target_uuid>/<source_uuid>/<file_id>
> store the metadata (including, date, hash and signature) in <the same path>.medata
- if the target in not known and we are not the root server
> store the file in a temporary directory
> try to send the file to the relay server
> do nt remove the file and try again as long as there is a fatal error (code >= 500, network error)
- if the target in not known and we are the root server
> ignore the file and return 404

HEAD behavior:
- If the file exists in /var/rudder/shared-files-nodes... with the same hash, return 200
- If it doesn't, return 404

3. Using this API from ncf
We will create 2 new generic methods in ncf:
- sharedfile_to_node(target_uuid, file_id, file_path, ttl) where ttl is infinite by default
- sharedfile_from_node(source_uuid, file_id, file_path)

The first one will call HEAD on the shared-files API and if is gets a 404, call POST to send the content.

The second one will just download the file using regular cfengine protocol.


Subtasks 55 (0 open55 closed)

Architecture #9708: Create a relay API for shared-filesReleasedAlexis Mousset2016-11-22Actions
User story #9709: Add the path of files shared between nodes to cf-serverd configurationReleasedAlexis Mousset2016-11-22Actions
User story #9710: Generate a file in relays promises that contain the map of nodes and relays below itReleasedAlexis Mousset2016-11-22Actions
User story #9715: Install rudder-relay on rudder root policy serverRejected2016-11-22Actions
Architecture #9729: New system variablesReleasedVincent MEMBRÉ2016-12-22Actions
Bug #9847: StackOverflowError on policy generationReleasedAlexis Mousset2016-12-22Actions
Architecture #9730: Package the new relay-apiReleasedAlexis Mousset2016-11-24Actions
User story #9732: Create a relay API for remote runReleasedBenoît PECCATTE2016-11-24Actions
Bug #9825: Wrong path in relay-api packagingReleasedBenoît PECCATTE2016-12-20Actions
Bug #9828: The relay package must depend on python dev package to be able to buildReleasedAlexis Mousset2016-12-20Actions
Bug #9830: The relay package doesn build on 4.1ReleasedAlexis Mousset2016-12-21Actions
Bug #9838: Old debian call libpython-dev python-devReleasedAlexis Mousset2016-12-21Actions
Bug #9840: Old debian call libpython-dev python-devReleasedAlexis Mousset2016-12-21Actions
Bug #9852: the relay tries to packages configuration in /etc/httpdReleasedNicolas CHARLES2016-12-23Actions
Bug #9889: Remove common conf between relay and webapp from webapp packageReleasedBenoît PECCATTE2017-01-05Actions
Bug #9907: Wrong file name in relay packageReleasedBenoît PECCATTE2017-01-05Actions
Bug #9908: Apache conf of webapps were movedReleasedBenoît PECCATTE2017-01-06Actions
Bug #9909: Wrong filename in deb packagingReleasedBenoît PECCATTE2017-01-06Actions
Bug #9911: Wrong filename in rpm packagingReleasedBenoît PECCATTE2017-01-06Actions
Bug #9912: Wrong filename in apache conf packagingReleasedBenoît PECCATTE2017-01-06Actions
Bug #9915: Dependency issue in 4.1 relay packageReleasedBenoît PECCATTE2017-01-06Actions
User story #9890: Remove relay specific conf from rudder apache config filesReleasedBenoît PECCATTE2017-01-03Actions
Bug #9895: relay package fail to build on sles 11ReleasedAlexis Mousset2017-01-05Actions
Bug #9896: relay package fail to build on sles 11ReleasedAlexis Mousset2017-01-05Actions
Bug #9902: relay package fail to build on sles 11ReleasedAlexis Mousset2017-01-05Actions
User story #9904: Add apache configuration for relay-apiReleasedBenoît PECCATTE2017-01-05Actions
User story #9906: Generate apache conf authorizing connection from policy serverReleasedBenoît PECCATTE2017-01-05Actions
Bug #9923: flask doesn't build on sles11ReleasedAlexis Mousset2017-01-09Actions
Bug #9925: Broken postinst in relay packageReleasedBenoît PECCATTE2017-01-09Actions
Bug #9941: Cannot install rudder-server-relay on CentOSReleasedBenoît PECCATTE2017-01-10Actions
Bug #9951: rpm build fail on centos7ReleasedAlexis Mousset2017-01-11Actions
Bug #9954: rpm build fail on centos7ReleasedAlexis Mousset2017-01-11Actions
Bug #9959: rpm build fail on centos7ReleasedAlexis Mousset2017-01-11Actions
Bug #9952: Missing relay-api.wsgi on relaysRejected2017-01-11Actions
Bug #9956: Relay apache conf does not allow accessing the apiReleasedBenoît PECCATTE2017-01-11Actions
User story #9957: Copy nodeslist.json file outside of inputs dirReleasedBenoît PECCATTE2017-01-11Actions
User story #9961: Use nodeslist from /opt/rudder/etcReleasedBenoît PECCATTE2017-01-11Actions
User story #9965: Set the right permissions on nodeslist.jsonReleasedBenoît PECCATTE2017-01-11Actions
User story #9958: Nodeslist file is empty on a root server with one nodeReleasedNicolas CHARLES2017-01-11Actions
User story #9968: Add a CRON task to call cleanup.shReleasedBenoît PECCATTE2017-01-12Actions
Bug #9969: cron.d is not created in .specReleasedBenoît PECCATTE2017-01-12Actions
Bug #9970: old a2ensite does not append .conf automaticallyReleasedBenoît PECCATTE2017-01-12Actions
Bug #9971: Missing permission for wsgiReleasedBenoît PECCATTE2017-01-12Actions
Bug #9973: Missing fix for certificates in relay packageReleasedBenoît PECCATTE2017-01-12Actions
Bug #9975: We sometimes write a wrong require statement in apache 2.4 confReleasedBenoît PECCATTE2017-01-12Actions
Bug #9977: Wrong place for wsgi conf in deb packagesReleasedBenoît PECCATTE2017-01-12Actions
User story #9983: The rudder user should be in the rudder groupReleasedBenoît PECCATTE2017-01-13Actions
Bug #9985: Error when executiong remote runReleasedBenoît PECCATTE2017-01-13Actions
Bug #9986: Relay api conf is not enabled after installReleasedBenoît PECCATTE2017-01-13Actions
Bug #9988: file /opt/rudder/share/relay-api/cleanup.sh is not executable, lots of cron mail about itReleasedAlexis Mousset2017-01-13Actions
Bug #9989: Relay api runs with www-data user instead of rudderReleasedAlexis Mousset2017-01-13Actions
Bug #9990: Broken remote run in 4.1ReleasedBenoît PECCATTEActions
Bug #9991: Syntax error in parent ticketReleasedBenoît PECCATTEActions
User story #9992: Use sudo to execute rudder remote run in the APIReleasedBenoît PECCATTE2017-01-13Actions
Bug #10002: Broken RPM packaging of sudo confReleasedBenoît PECCATTE2017-01-13Actions

Related issues 3 (1 open2 closed)

Related to Rudder - User story #9711: Add generic methods to share files between nodes using rudder relay APIReleasedAlexis MoussetActions
Related to Rudder - Bug #9913: Add SELinux config to relay packageReleasedBenoît PECCATTEActions
Related to Rudder - User story #9914: Apache configuration files status is inconsistent in packagesNewActions
Actions

Also available in: Atom PDF