Project

General

Profile

Actions

User story #9698

closed

Import node properties from external data sources

Added by Vincent MEMBRÉ over 4 years ago. Updated over 4 years ago.

Status:
Released
Priority:
N/A
Assignee:
-
Category:
Web - Config management
Target version:
Suggestion strength:
User visibility:
Effort required:

Description

Some data may be available outside of Rudder to be used to configure your nodes.

To be able to use them in Rudder you need to add them into Rudder as Node properties.

That task may be complicated to achieve and even more to automate (so data stays up to date), we want to add a way to do this easily in Rudder.

We will have to define those external data sources in Rudder. Rudder will then fetch data from them and add them as properties for each nodes.

Data structure:

  • name
  • url
  • request type (one by node, one for all nodes)
  • headers
  • A way to define data access into received data
  • Update frequency

We need:

  • Backend storage for datasources
  • An api Rest to define datasource
  • An ui to define datasource
  • System to fetch data sources and add them as properties
  • Scheduler to launch it asynchronously
  • Use these data in policy generation

Subtasks 21 (0 open21 closed)

User story #9699: UI to define datasources ReleasedVincent MEMBRÉ2016-11-21Actions
User story #9700: API Rest to define Data sourcesReleasedFrançois ARMAND2016-11-23Actions
Bug #9725: Data source Scala modelReleasedFrançois ARMAND2016-11-23Actions
User story #9897: Move datasource parameters into a specific entry in rest jsonReleasedFrançois ARMAND2017-01-04Actions
User story #9899: Change format of time duration in datasource Rest APIReleasedNicolas CHARLES2017-01-05Actions
User story #9701: Store data sources in backendReleasedFrançois ARMAND2016-11-21Actions
User story #9724: Query HTTP datasourceReleasedFrançois ARMAND2017-01-10Actions
User story #9931: Test UpdateHttpDatasetTest does not run anymoreReleasedFrançois ARMAND2017-01-10Actions
User story #9898: 404 on request should empty the properties on NodeReleasedFrançois ARMAND2017-01-04Actions
Bug #9910: Bugs on datasource initilization and updateReleasedNicolas CHARLES2017-01-06Actions
User story #9942: Migration script to add missing table in PostgresReleasedFrançois ARMAND2017-01-10Actions
User story #9944: Migration script to add missing table in PostgresReleasedFrançois ARMAND2017-01-10Actions
Bug #9947: Build broken (missed refactoring)ReleasedVincent MEMBRÉ2017-01-10Actions
User story #9978: Document datasource APIReleasedVincent MEMBRÉActions
Bug #10191: Update datasource API now that it is a module with new endpointsReleasedVincent MEMBRÉActions
Bug #9987: Cannot use " in the result of a DatasourceRejectedFrançois ARMAND2017-01-13Actions
Bug #9995: Forbid modification of properties set by datasourceReleasedVincent MEMBRÉ2017-01-13Actions
Bug #10026: Datasource config: missing interpolation of rudder params in request headerReleasedVincent MEMBRÉ2017-01-20Actions
Bug #10043: Missing refresh API for one data source onlyReleasedVincent MEMBRÉ2017-01-20Actions
User story #10027: Add a POST option for datasource queriesReleasedVincent MEMBRÉ2017-01-18Actions
User story #10039: Document data sources featureReleasedAlexis MOUSSET2017-01-19Actions

Related issues

Related to Rudder - User story #6264: Get directive parameter value from external source (kv store, rest, etc)ResolvedActions
Actions #1

Updated by Janos Mattyasovszky over 4 years ago

Also don't forget to have somekind of failsafe to handle the case when no data is returned (no error but empty dataset) => Would you delete all existing properties?

Actions #2

Updated by François ARMAND over 4 years ago

Janos Matya: in all case, the behavior is limited to the key "data-source-name".

We have thought to have an option in the data source configuration with the following option, when nothing is found for a node:
- ignore (i.e: don't modify the existing state)
- raise an error;
- delete value for "data-source-name" key
- use a default value = ....

Actions #3

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 4.1.0~beta1 to 4.1.0~beta2
Actions #4

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from New to Pending technical review
  • Target version changed from 4.1.0~beta2 to 4.1.0~beta1
Actions #5

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from Pending technical review to 12
Actions #6

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from 12 to Pending release
Actions #7

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.0 which was released today.

Actions #8

Updated by Benoît PECCATTE about 4 years ago

  • Related to User story #6264: Get directive parameter value from external source (kv store, rest, etc) added
Actions

Also available in: Atom PDF