Skip to content

Commit 51a5810

Browse files
aaannzaaannz
authored
Merge pull request #51 from uyuni-project/json_pillars
DB based JSON pillar storage
2 parents ef71069 + c7a74d6 commit 51a5810

File tree

2 files changed

+314
-0
lines changed

2 files changed

+314
-0
lines changed

‎accepted/00080-jsonb-pillar-db-storage.md

+314
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
- Feature Name: Database backed salt pillar storage
2+
- Start Date: 2021-01-13
3+
- RFC PR: https://github.com/uyuni-project/uyuni-rfc/pull/51
4+
5+
# Summary
6+
[summary]: #summary
7+
8+
Use database as a storage for Uyuni salt pillars instead of plain files.
9+
10+
# Motivation
11+
[motivation]: #motivation
12+
13+
Pillars are used for passing reliable and trustworthy data to salt clients. Salt integration into Uyuni generates pillar files which are then loaded to the salt ecosystem by Uyuni external pillar.
14+
Advantage of this solution is rather easy implementation and quick visibility of the data when inspecting in filesystem, however there are disadvantages to consider. Data consistency [issue#5678](https://github.com/SUSE/spacewalk/issues/5678) and [issue#10679](https://github.com/SUSE/spacewalk/issues/10679), [filesystem performance](https://github.com/SUSE/spacewalk/pull/10391#discussion_r375147529), backup procedure must consider these files as well.
15+
16+
This RFC proposes to instead of plain files, database based storage should be used for all pillars produced by Uyuni.
17+
18+
This is expected to help with maintaining consistency, help with backup procedure and with interserver sync as well, potential performance improvement (TBD - filesystem access vs. database indexes).
19+
20+
# Detailed design
21+
[design]: #detailed-design
22+
23+
## Database storage
24+
25+
In this RFC I propose to use native JSON type of PostgreSQL ([see](https://www.postgresql.org/docs/12/datatype-json.html)) to use as a storage of the pillar data, particularly JSONB type:
26+
27+
28+
```sql
29+
CREATE TABLE suseSaltPillars
30+
(
31+
id integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
32+
server_id NUMERIC
33+
CONSTRAINT suse_salt_pillar_sid_fk
34+
REFERENCES rhnServer (id)
35+
ON DELETE CASCADE,
36+
group_id NUMERIC
37+
CONSTRAINT suse_salt_pillar_gid_fk
38+
REFERENCES rhnServerGroup (id)
39+
ON DELETE CASCADE,
40+
org_id NUMERIC
41+
CONSTRAINT suse_salt_pillar_oid_fk
42+
REFERENCES web_customer (id)
43+
ON DELETE CASCADE,
44+
category VARCHAR NOT NULL,
45+
pillar JSONB NOT NULL,
46+
UNIQUE (server_id, category),
47+
UNIQUE (group_id, category),
48+
UNIQUE (org_id, category),
49+
CONSTRAINT suse_salt_pillar_only_one_target CHECK (
50+
( server_id is not null and group_id is null and org_id is null ) or
51+
( server_id is null and group_id is not null and org_id is null ) or
52+
( server_id is null and group_id is null and org_id is not null ) or
53+
( server_id is null and group_id is null and org_id is null)
54+
)
55+
);
56+
57+
CREATE INDEX suse_salt_pillar_server_id_idx ON suseSaltPillars (server_id);
58+
59+
CREATE INDEX suse_salt_pillar_group_id_idx ON suseSaltPillars (group_id);
60+
61+
CREATE INDEX suse_salt_pillar_org_id_idx ON suseSaltPillars (org_id);
62+
63+
CREATE INDEX suse_salt_pillar_category ON suseSaltPillars (category);
64+
```
65+
66+
To see differences between TEXT, JSON and JSONB datatype I quote PostgreSQL documentation:
67+
68+
> JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in RFC 7159.Such data can also be stored as text, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules.
69+
>
70+
> The json and jsonb data types accept almost identical sets of values as input. The major practical difference is one of efficiency. The json data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb also supports indexing, which can be a significant advantage.
71+
72+
Pillar can be minion pillar, group pillar, organizational pillar and global pillar. Although `salt` itself does not have this divisions, Uyuni does and is indeed in use e.g. in image building which creates organizational pillars and formulas with group pillars:
73+
74+
* Minion pillars have `server_id` FK referencing `rhnServer` row.
75+
This is not native to `salt`, in Uyuni world however the server id is the main identifier and Uyuni supports changing salt minion ids. To prevent errors in case of forgotten minion rename, in this RFC I suggest to use foreign key to `rhnServer` table to address minion pillars.
76+
* Group pillars have `group_id` FK referencing `rhnServerGoup` row.
77+
* Organizational pillar have `org_id` FK referencing `web_customer` row which is used to id organizations.
78+
* Global pillars have all three of these FKs NULL.
79+
80+
Pillar can have only one target to avoid confusions. Therefore `CONSTRAINT suse_salt_pillar_only_one_target` is added to ensure each row can have only one not null foreign key.
81+
82+
Added `category` VARCHAR is there to provide easer manipulation and understanding of stored data. Currently all Uyuni pillar data are written using `com.suse.manager.webui.services.pillar.MinionPillarManager` (assuming [PR#3093](https://github.com/uyuni-project/uyuni/pull/3093) is merged) and each type of pillars have their own respective [generator class](https://github.com/nadvornik/uyuni/tree/custominfo/java/code/src/com/suse/manager/webui/services/pillar). To maintain some form of distinction I propose this `category` column to allow to have multiple pillar entries per salt client.
83+
84+
## Salt access
85+
86+
Salt ecosystem needs a way how to access these pillars. For this salt provides mechanism of external pillar, particularly [salt.pillar.postgres module](https://docs.saltproject.io/en/latest/ref/pillar/all/salt.pillar.postgres.html).
87+
Accessing database can be done in one query, however that would need a change in `sql_base.py` in salt because currently it supports only one `%s` replacement for minion id.
88+
89+
This limitation can be solved by using postgres function and then call just function from the external pillar:
90+
91+
```sql
92+
CREATE OR REPLACE FUNCTION suse_minion_pillars(
93+
mid VARCHAR
94+
) RETURNS TABLE (pillar JSONB) LANGUAGE plpgsql
95+
AS $$
96+
DECLARE
97+
sid NUMERIC;
98+
BEGIN
99+
SELECT server_id INTO sid FROM suseMinionInfo WHERE minion_id = mid;
100+
RETURN QUERY
101+
SELECT p.pillar from suseSaltPillars AS p WHERE (p.server_id is NULL AND p.group_id is NULL AND p.org_id is NULL)
102+
OR (p.org_id = (SELECT s.org_id FROM rhnServer AS s WHERE s.id = sid))
103+
OR (p.group_id IN (SELECT g.server_group_id FROM rhnServerGroupMembers AS g WHERE g.server_id = sid))
104+
OR (p.server_id = sid)
105+
ORDER BY CASE WHEN p.org_id IS NULL AND p.group_id IS NULL and p.server_id is NULL THEN 0 ELSE 1 END, p.org_id NULLS LAST, p.group_id ASC NULLS LAST, p.server_id ASC NULLS FIRST;
106+
END
107+
$$;
108+
```
109+
110+
```yaml
111+
# postgresql configuration for external pillar
112+
postgres:
113+
db: susemanager
114+
host: localhost
115+
pass: spacewalk
116+
port: 5432
117+
user: spacewalk
118+
119+
# Configure external pillar
120+
ext_pillar:
121+
- postgres:
122+
- query: "SELECT suse_minion_pillars(%s)"
123+
as_json: True
124+
```
125+
126+
127+
128+
Alternatively database can be accessed using multiple queries, however with some performance hit:
129+
130+
```yaml
131+
postgres:
132+
db: susemanager
133+
host: localhost
134+
pass: spacewalk
135+
port: 5432
136+
user: spacewalk
137+
138+
ext_pillar:
139+
- postgres:
140+
- query: "SELECT pillar from suseSaltPillars WHERE server_id is NULL AND group_id is NULL AND org_id is NULL"
141+
as_json: True
142+
- query: "SELECT pillar from suseSaltPillars WHERE org_id = (SELECT org_id FROM rhnServer AS S LEFT JOIN suseMinionInfo AS M on S.id = M.server_id WHERE M.minion_id = %s)"
143+
as_json: True
144+
- query: "SELECT pillar from suseSaltPillars WHERE group_id IN (SELECT server_group_id FROM rhnServerGroupMembers AS G LEFT JOIN suseMinionInfo AS M ON G.server_id = M.server_id WHERE M.minion_id = %s)"
145+
as_json: True
146+
- query: "SELECT pillar from suseSaltPillars WHERE server_id = (SELECT server_id FROM suseMinionInfo AS M WHERE M.minion_id = %s)"
147+
as_json: True
148+
```
149+
150+
Salt automatically merge results from these queries, latest with the highest priority.
151+
152+
### Upstream changes needed
153+
154+
salt.pillar.postgres module, resp. salt.pillar.sql_base is not yet designed to work with JSON results directly from database and expects always to get results in *key:value* format. To enable JSON output, small patch need to be negotiated with upstream (patch against pre-blackened version):
155+
156+
```diff
157+
diff --git a/salt/pillar/sql_base.py b/salt/pillar/sql_base.py
158+
index eeb2269f77..dae0b36204 100644
159+
--- a/salt/pillar/sql_base.py
160+
+++ b/salt/pillar/sql_base.py
161+
@@ -352,8 +352,13 @@ class SqlBaseExtPillar(six.with_metaclass(abc.ABCMeta, object)):
162+
# dict, descend.
163+
crd = crd[ret[i]]
164+
165+
+ # We have just one field without any key, assume returned row is already a dict
166+
+ # aka JSON storage
167+
+ if self.num_fields == 1:
168+
+ crd.update(ret[0])
169+
+
170+
# If this test is true, the penultimate field is the key
171+
- if self.depth == self.num_fields - 1:
172+
+ elif self.depth == self.num_fields - 1:
173+
nk = self.num_fields-2 # Aka, self.depth-1
174+
# Should we and will we have a list at the end?
175+
if ((self.as_list and (ret[nk] in crd)) or
176+
```
177+
178+
Tracked in [PR#59777](https://github.com/saltstack/salt/pull/59777).
179+
180+
## Performance impact
181+
182+
I did a small scale E2E test:
183+
* 1001 minions (one real, 1000 evil minions)
184+
* 10 pillar items repetitions
185+
* 4000 rows in pillar database (1000 rows per organization, group, minion and global)
186+
* regular Uyuni pillar files for one registered minion
187+
* default Uyuni salt worker settings (8 worker threads)
188+
189+
Command:
190+
191+
```bash
192+
time bash -c 'for i in `seq 1 10`;do time salt "*" pillar.items >/dev/null;salt "*" test.ping > /dev/null;echo "== Round $i done"; done'
193+
```
194+
195+
Results:
196+
197+
![Comparison of different pillar access methods](images/performance-pillars.png)
198+
199+
200+
### Impact of increased worker threads on database connections
201+
202+
In default settings, using 8 salt worker threads I observed:
203+
Connections to postgres up to 14
204+
Postgres backends up to 6
205+
206+
207+
Given the discrepancy between number of salt clients and opened connections to the database, I increased number of salt worker threads to test hypothesis that indeed number of salt threads is limiting number of connections to the database. Upon increasing salt worker threads to 28 I got following results:
208+
209+
210+
Connections to postgres up to 44
211+
Postgres backends up to 14
212+
213+
214+
In conclusion I do not see performance impact in this isolated scenario. Number of opened connections is correlated with number of salt worker threads, this needs to be taking into account in case of tuning individual installations.
215+
216+
# Drawbacks
217+
[drawbacks]: #drawbacks
218+
219+
Why should we **not** do this?
220+
221+
* what other parts of the product will be affected?
222+
223+
Salt integration and thus basically whole product is affected.
224+
225+
* will the solution be hard to maintain in the future?
226+
227+
I expect easier maintaining then with file generators.
228+
229+
* Hibernate support
230+
231+
Hibernate itself does not support JSONB data type. It is possible to either implement our own user type in hibernate or use (and package) existing project [hibernate-types](https://github.com/vladmihalcea/hibernate-types).
232+
233+
* Affecting users usage patterns
234+
235+
Example is lack of OS Images import capability that lead some users to use their own solution - copy both OS image and pillar files from one Uyuni to another. This works currently, but switching from file based pillars to database will break their unsupported, but working solution.
236+
Mitigation in this scenario is just not to use database backed pillar for images until there is another way to move the images, but there may be other users who depends on how pillars currently work in Uyuni.
237+
238+
# Alternatives
239+
[alternatives]: #alternatives
240+
241+
- Use single pillar row for each minion and without `category` column.
242+
243+
This was the original idea how to implement this, however during prototype implementation having separate rows for separate pillar data made implementation better match how it is done now. I am not opposed using single row per minion, but I also do not see disadvantages for using multiple rows per minion.
244+
245+
- Instead of using upstream `salt.pillar.postgres` module, option is to write our own customized for Uyuni database. This would help for example with integrating formula pillars into database at minimal costs to adapt formula system. As formulas are not storing complete pillar information, rather in format of changes to form.yml. So `suma_minion.py` on top of loading formula pillar JSON needs to parse formula `form.yml` and merge data properly. Our custom db module would be able to read formula JSON pillar from database and do the same parsing of `form.yml` and data merge.
246+
247+
- Use single `target` column instead of three FK columns:
248+
```sql
249+
CREATE TABLE suseSaltPillars
250+
(
251+
target VARCHAR(256) NOT NULL,
252+
category VARCHAR,
253+
pillar JSONB
254+
);
255+
256+
CREATE INDEX suse_salt_pillar_target_idx ON suseSaltPillars (target);
257+
```
258+
259+
This was original idea for this PR, but this had problem with minion id renaming.
260+
261+
`target` column is to contain `minion_id` of salt client, however to support universal and system group pillars I opted to use VARCHAR with limit of 256 (as is the limit of minion id, group name is limited to 64 characters). For universal pillar I propose to use `*` character.
262+
263+
Integration with salt would then use postgresql external pillar configuration:
264+
```yaml
265+
postgres:
266+
db: susemanager
267+
host: localhost
268+
pass: spacewalk
269+
port: 5432
270+
user: spacewalk
271+
272+
ext_pillar:
273+
- postgres:
274+
- query: "SELECT pillar from suseSaltPillars WHERE target = '*'"
275+
- query: "SELECT pillar from suseSaltPillars WHERE target = (SELECT CONCAT('org_id:', org_id::VARCHAR) FROM rhnServer AS S LEFT JOIN suseMinionInfo AS M on S.id = M.server_id WHERE M.minion_id = %s)"
276+
- query: "SELECT pillar from suseSaltPillars WHERE target IN (SELECT CONCAT('group_id:', server_group_id::VARCHAR) FROM rhnServerGroupMembers AS G LEFT JOIN suseMinionInfo AS M ON G.server_id = M.server_id WHERE M.minion_id = %s)"
277+
- query: "SELECT pillar from suseSaltPillars WHERE target = %s"
278+
```
279+
280+
This example contains three different queries:
281+
- Universal pillars looking for target `*`
282+
- Organization pillars looking for target `org_id:<organization id>`
283+
- Group pillars. This is done by looking up salt client `server_id` using its `minion_id` joining server group membership table to obtain `server_group_id` set where salt client is a member. Then looking up target `group_id:<group id>` in this result set.
284+
- Minion pillars looking for `minion_id`
285+
286+
Responsibility to provide up to date data remains, as is the case today, on the component changing the data. That means, when information about minion is changed, then relevant pillar generator must be called to refresh the pillar in database.
287+
However with the pillar in database and loosely (no foreign key relation) connected to minions and groups, database trigger which can remove pillar entries when minion or group is removed:
288+
289+
```sql
290+
CREATE OR REPLACE FUNCTION minion_removed() RETURNS TRIGGER AS $$
291+
BEGIN
292+
DELETE FROM suseSaltPillars WHERE target = OLD.minion_id;
293+
RETURN OLD;
294+
END $$ LANGUAGE PLPGSQL;
295+
296+
CREATE OR REPLACE FUNCTION group_removed() RETURNS TRIGGER AS $$
297+
BEGIN
298+
DELETE FROM suseSaltPillars WHERE target = CONCAT('group_id:',OLD.id::VARCHAR);
299+
RETURN OLD;
300+
END $$ LANGUAGE PLPGSQL;
301+
302+
CREATE TRIGGER minion_removed BEFORE DELETE ON suseMinionInfo FOR EACH ROW EXECUTE PROCEDURE minion_removed();
303+
304+
CREATE TRIGGER group_removed BEFORE DELETE ON rhnServerGroup FOR EACH ROW EXECUTE PROCEDURE group_removed();
305+
```
306+
307+
In case of organization removal, enhance `delete_org` stored procedure to delete organizational pillar as well.
308+
309+
310+
# Unresolved questions
311+
[unresolved]: #unresolved-questions
312+
313+
- Migration from file based pillars to database based pillars need to be part of the implementation. However I did not yet touch this subject how to exactly perform migration.
314+
Needs to take care also about messaging the change to users in clear and visible manner not to surprise users with change of behaviour.

‎accepted/images/performance-pillars.png

182 KB
Loading

0 commit comments

Comments
 (0)