Architecture #124
closedLimit request size in LDAP
Description
When one use the complexe search UI, it is easy to build request with thousands of attribute comparison in filters.
Limit size to a maximum number.
Of course, it will be interesting to see how that is possible with and/or composition.
Updated by François ARMAND almost 15 years ago
- % Done changed from 0 to 20
First step in commit:dce1d57c : limit the last request size/time
Updated by François ARMAND almost 15 years ago
Since commit:b8362c25 , I limit the time and size of request (to 10s and 1000 entries).
The solution is not ideal because for now, if a limit is reached, the user just don't know anything about it.
So, we have to make a big red alarm to explain it to the user, and to say him that he should be more accurate in its search.
That leads to two other questions:
- what should be the behaviour with dyn group on such case (at least the same) ?
- what to do for policies that should be applied to ALL the park (and say the park is 100 000 servers) ? Just forbid it and explain to the admin that he doesn't really have 100 000 identical servers ?
Updated by François ARMAND almost 15 years ago
- Status changed from New to 2
- Assignee set to François ARMAND
- % Done changed from 20 to 60
Updated by Jonathan CLARKE almost 15 years ago
François ARMAND a écrit:
Since commit:b8362c25 , I limit the time and size of request (to 10s and 1000 entries).
The solution is not ideal because for now, if a limit is reached, the user just don't know anything about it.
So, we have to make a big red alarm to explain it to the user, and to say him that he should be more accurate in its search.
That leads to two other questions:
- what should be the behaviour with dyn group on such case (at least the same) ?
I think we can have dyn groups of > 1000 servers, but only display the first 1000 to the user in the GUI. Even over 100, it seems unlikely the user will actually look at all of them on the web page. In this case, we could restrict getting the full list to specific operations like : exporting the policies to promises-writer, and a special link to download the full list (in CSV format, for example). What do you think?
- what to do for policies that should be applied to ALL the park (and say the park is 100 000 servers) ? Just forbid it and explain to the admin that he doesn't really have 100 000 identical servers ?
If a client has 100 000 servers, I'm sure he can pay for 2 more powerful servers to be dedicated LDAP CMDB servers, and we'll get much better throughput/performance on search from our CMDB. Don't forget our current server is really undersized for the size of it's DB : it's a small VM with 1 GB of RAM.
Also, getting the full list of servers should be fast. I think it's just searching on various criterion that may be slow.
Updated by François ARMAND almost 15 years ago
Jonathan Clarke a écrit:
I think we can have dyn groups of > 1000 servers, but only display the first 1000 to the user in the GUI. Even over 100, it seems unlikely the user will actually look at all of them on the web page. In this case, we could restrict getting the full list to specific operations like : exporting the policies to promises-writer, and a special link to download the full list (in CSV format, for example). What do you think?
Yep, seems the adequate things to do. All in all, it's the path I'm walking on.
If a client has 100 000 servers, I'm sure he can pay for 2 more powerful servers to be dedicated LDAP CMDB servers, and we'll get much better throughput/performance on search from our CMDB. Don't forget our current server is really undersized for the size of it's DB : it's a small VM with 1 GB of RAM.
I've that in mind, no problem :)
Also, getting the full list of servers should be fast. I think it's just searching on various criterion that may be slow.
Yes, that is fast. Each query on it's own is rather fast on the server.
But what becomes non negligible is all the data encoding, network processing, etc. For 10 000 entries, we deal with something like 1Mo of data, and so a delay of several seconds just for the client processing (getting set of server 1, quering ldap with it, getting set of server 2, etc) - clearly not compatible with UI synchronous need.
I'm going to see how to do that as soon as the first policy will be link in the UI.
Updated by Jonathan CLARKE almost 15 years ago
François ARMAND a écrit:
But what becomes non negligible is all the data encoding, network processing, etc. For 10 000 entries, we deal with something like 1Mo of data, and so a delay of several seconds just for the client processing (getting set of server 1, quering ldap with it, getting set of server 2, etc) - clearly not compatible with UI synchronous need.
It would be interesting to run the web server on the same machine as the LDAP server, to see how performance is impacted.
A lot less network latency, of course. I participated in some tests, which showed that using a WUI above a LDAP server was a good application to have the two programs on the same machine: basically because when the WUI asks the LDAP server for something, to WUI doesn't do any processing, it just waits while the LDAP server processes it's request. So they are not parallel applications.
How could we do this? Do you have a WAR we could deploy on the server in the lab?
Updated by François ARMAND almost 15 years ago
Jonathan Clarke a écrit:
....
How could we do this? Do you have a WAR we could deploy on the server in the lab?
We will :)
For now, there is no stable version (not even built snapshot), so the only solution is to build from sources.
Updated by François ARMAND almost 15 years ago
- Status changed from 2 to New
- Priority changed from 4 to 5 (lowest)
Change priority/assignement
Updated by François ARMAND almost 15 years ago
- Assignee deleted (
François ARMAND)
Remove assignement
Updated by François ARMAND almost 14 years ago
- Tracker changed from User story to Bug
- Project changed from 9 to Rudder
Updated by Jonathan CLARKE over 13 years ago
- Tracker changed from Bug to User story
Updated by Jonathan CLARKE over 12 years ago
- Target version changed from 24 to 18
Updated by Jonathan CLARKE over 12 years ago
- Target version changed from 18 to Ideas (not version specific)
Updated by Benoît PECCATTE over 9 years ago
- Category set to Architecture - Internal libs
Updated by Benoît PECCATTE over 9 years ago
- Related to Architecture #123: Add asynchron request/result in LDAP search added
Updated by Benoît PECCATTE over 9 years ago
This doesn't match how we want to do this and how we can do filtering.
So I'm closing this ticket. But there is still a problem somewhere to be solved for 5k hosts.
Updated by François ARMAND almost 7 years ago
- Tracker changed from User story to Architecture
Updated by François ARMAND over 4 years ago
- Status changed from New to Rejected
There is too many cases where we need to have all results with current architecture to be able to limit that.