Project

General

Profile

User story #9698

Import node properties from external data sources

Added by Vincent MEMBRÉ almost 3 years ago. Updated over 2 years ago.

Status:
Released
Priority:
N/A
Assignee:
-
Category:
Web - Config management
Target version:
Pull Request:
Suggestion strength:
User visibility:
Effort required:

Description

Some data may be available outside of Rudder to be used to configure your nodes.

To be able to use them in Rudder you need to add them into Rudder as Node properties.

That task may be complicated to achieve and even more to automate (so data stays up to date), we want to add a way to do this easily in Rudder.

We will have to define those external data sources in Rudder. Rudder will then fetch data from them and add them as properties for each nodes.

Data structure:

  • name
  • url
  • request type (one by node, one for all nodes)
  • headers
  • A way to define data access into received data
  • Update frequency

We need:

  • Backend storage for datasources
  • An api Rest to define datasource
  • An ui to define datasource
  • System to fetch data sources and add them as properties
  • Scheduler to launch it asynchronously
  • Use these data in policy generation

Subtasks

User story #9699: UI to define datasources ReleasedVincent MEMBRÉActions
User story #9700: API Rest to define Data sourcesReleasedFrançois ARMANDActions
Bug #9725: Data source Scala modelReleasedFrançois ARMANDActions
User story #9897: Move datasource parameters into a specific entry in rest jsonReleasedFrançois ARMANDActions
User story #9899: Change format of time duration in datasource Rest APIReleasedNicolas CHARLESActions
User story #9701: Store data sources in backendReleasedFrançois ARMANDActions
User story #9724: Query HTTP datasourceReleasedFrançois ARMANDActions
User story #9931: Test UpdateHttpDatasetTest does not run anymoreReleasedFrançois ARMANDActions
User story #9898: 404 on request should empty the properties on NodeReleasedFrançois ARMANDActions
Bug #9910: Bugs on datasource initilization and updateReleasedNicolas CHARLESActions
User story #9942: Migration script to add missing table in PostgresReleasedFrançois ARMANDActions
User story #9944: Migration script to add missing table in PostgresReleasedFrançois ARMANDActions
Bug #9947: Build broken (missed refactoring)ReleasedVincent MEMBRÉActions
User story #9978: Document datasource APIReleasedVincent MEMBRÉActions
Bug #10191: Update datasource API now that it is a module with new endpointsReleasedVincent MEMBRÉActions
Bug #9987: Cannot use " in the result of a DatasourceRejectedFrançois ARMANDActions
Bug #9995: Forbid modification of properties set by datasourceReleasedVincent MEMBRÉActions
Bug #10026: Datasource config: missing interpolation of rudder params in request headerReleasedVincent MEMBRÉActions
Bug #10043: Missing refresh API for one data source onlyReleasedVincent MEMBRÉActions
User story #10027: Add a POST option for datasource queriesReleasedVincent MEMBRÉActions
User story #10039: Document data sources featureReleasedAlexis MOUSSETActions

Related issues

Related to Rudder - User story #6264: Get directive parameter value from external source (kv store, rest, etc)New2015-02-13Actions

History

#1

Updated by Janos Mattyasovszky almost 3 years ago

Also don't forget to have somekind of failsafe to handle the case when no data is returned (no error but empty dataset) => Would you delete all existing properties?

#2

Updated by François ARMAND almost 3 years ago

Janos Matya: in all case, the behavior is limited to the key "data-source-name".

We have thought to have an option in the data source configuration with the following option, when nothing is found for a node:
- ignore (i.e: don't modify the existing state)
- raise an error;
- delete value for "data-source-name" key
- use a default value = ....

#3

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 4.1.0~beta1 to 4.1.0~beta2
#4

Updated by Vincent MEMBRÉ over 2 years ago

  • Status changed from New to Pending technical review
  • Target version changed from 4.1.0~beta2 to 4.1.0~beta1
#5

Updated by Vincent MEMBRÉ over 2 years ago

  • Status changed from Pending technical review to 12
#6

Updated by Vincent MEMBRÉ over 2 years ago

  • Status changed from 12 to Pending release
#7

Updated by Vincent MEMBRÉ over 2 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.0 which was released today.

#8

Updated by Benoît PECCATTE over 2 years ago

  • Related to User story #6264: Get directive parameter value from external source (kv store, rest, etc) added

Also available in: Atom PDF