Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in https://stackable-workspace.slack.com/archives/C03DAR3SJLX/p1786648565827699 I would prefer putting this behind a annotation, e.g. restarter.stackable.tech/change-tracking-mechanism with either resource-version or data-hash. We would need to decide on the default, I would suggest to keep the current default for now, test it at customers and than switch the default to the new mechanism and afterwards remove the old

Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ All notable changes to this project will be documented in this file.

- Install and manage the OpenLineageConnection Custom Resource Definition ([#435]).
- Add `/ready` endpoint to the operator Deployment, which reports the CRD installation status ([#461]).
- Add the StatefulSet annotation `restarter.stackable.tech/change-tracking-mechanism`. With the
value `data-hash`, the restarter only restarts Pods when the content of a referenced ConfigMap
or Secret changes, so content-identical writes by third parties no longer trigger a rolling
restart. The default `resource-version` keeps the previous behavior ([#449]).

### Changed

Expand All @@ -17,6 +21,9 @@ All notable changes to this project will be documented in this file.
- Extend Helm deployed RBAC permissions to list and watch customresourcedefinitions required for startup condition ([#461]).
- The operator Deployment runs with a restricted security context, so it can be deployed into namespaces that enforce the `restricted` Pod Security Standard ([#462]).
- The operator Deployment runs with a read-only root filesystem ([#469]).
- Extend Helm deployed RBAC permissions to get configmaps and secrets, required to digest their
content for the `data-hash` change tracking mechanism, and to create secrets in the operator
namespace, required to store the key of these digests ([#449]).

### Fixed

Expand All @@ -28,6 +35,7 @@ All notable changes to this project will be documented in this file.
[#439]: https://github.com/stackabletech/commons-operator/pull/439
[#443]: https://github.com/stackabletech/commons-operator/pull/443
[#444]: https://github.com/stackabletech/commons-operator/pull/444
[#449]: https://github.com/stackabletech/commons-operator/pull/449
[#461]: https://github.com/stackabletech/commons-operator/pull/461
[#462]: https://github.com/stackabletech/commons-operator/pull/462
[#466]: https://github.com/stackabletech/commons-operator/pull/466
Expand Down
17 changes: 10 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 19 additions & 7 deletions Cargo.nix

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,12 @@ chrono = "0.4"
clap = "4.6"
futures = { version = "0.3", features = ["compat"] }
http = "1.5"
hmac = "0.13"
json-patch = "4.2"
rand = "0.10"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
sha2 = "0.11"
snafu = "0.9"
strum = { version = "0.28", features = ["derive"] }
tokio = { version = "1.54", features = ["full"] }
Expand Down
43 changes: 42 additions & 1 deletion deploy/helm/commons-operator/templates/roles.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,23 @@ metadata:
{{- include "commons-operator.labels" . | nindent 4 }}
rules:
# Watch pods to detect expired restart annotations.
# Watch configmaps and secrets to trigger rolling restarts of referencing StatefulSets.
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
# Watch configmaps and secrets to trigger rolling restarts of referencing StatefulSets.
# Get them to digest their content for StatefulSets using the change tracking mechanism
# data-hash (annotation restarter.stackable.tech/change-tracking-mechanism).
- apiGroups:
- ""
resources:
- configmaps
- secrets
verbs:
- get
- list
- watch
# For automatic cluster domain detection.
Expand Down Expand Up @@ -73,3 +82,35 @@ rules:
# Required for startup condition
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "commons-operator.fullname" . }}-role
labels:
{{- include "commons-operator.labels" . | nindent 4 }}
rules:
# Create the Secret holding the key of the content digests on first use of the change tracking
# mechanism data-hash (annotation restarter.stackable.tech/change-tracking-mechanism).
# Reading it is covered by the ClusterRole above.
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ include "commons-operator.fullname" . }}-rolebinding
labels:
{{- include "commons-operator.labels" . | nindent 4 }}
subjects:
- kind: ServiceAccount
name: {{ include "commons-operator.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
roleRef:
kind: Role
name: {{ include "commons-operator.fullname" . }}-role
apiGroup: rbac.authorization.k8s.io
34 changes: 32 additions & 2 deletions docs/modules/commons-operator/pages/restarter.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,43 @@ spec:
...
----

Annotation:: `restarter.stackable.tech/change-tracking-mechanism`

This annotation selects what counts as a change of a referenced ConfigMap or Secret:

`resource-version` (default):: Every write to the object counts as a change, including writes that only touch its metadata (labels, annotations, managed fields) or that write identical content.
`data-hash`:: Only a change of the content (`data` and `binaryData` of a ConfigMap, `data` of a Secret) counts as a change.
Writes by third parties that leave the content as it is do not restart the Pods.
To hash the content, the operator reads a referenced object whenever its resource version changes.
It only keeps the hash, not the content.
The hash is keyed with a random key that only the operator can read and bound to the UID of the object, so the hash in the Pod template annotations reveals nothing about the content beyond whether it changed.
On the first use of `data-hash`, the operator creates this key in the Secret `commons-operator-restarter-digest-key` in its own namespace.
Deleting that Secret, or the namespace, makes the operator create a new key on its next start, which changes every hash and therefore rolling-restarts every StatefulSet using `data-hash` once.

With any other value, the operator logs an error and stops updating the Pod template annotations of the StatefulSet until the value is fixed.
Switching between the mechanisms changes the values of the Pod template annotations described below and therefore rolling-restarts the StatefulSet once.

[source,yaml]
----
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-with-content-based-restarts
labels:
restarter.stackable.tech/enabled: "true"
annotations:
restarter.stackable.tech/change-tracking-mechanism: data-hash
...
----

==== Pod template annotations

To detect changes, the operator adds an annotation for every referenced ConfigMap and Secret to the Pod template of the StatefulSet.
Its key is `configmap.restarter.stackable.tech/\{name\}` or `secret.restarter.stackable.tech/\{name\}`, its value is the UID and resource version of the referenced object.
Its key is `configmap.restarter.stackable.tech/\{name\}` or `secret.restarter.stackable.tech/\{name\}`, its value is the UID of the referenced object, followed by its resource version or, with the `data-hash` mechanism, a hash of its content.
The value is `changes-ignored` instead if the object is excluded, either by the StatefulSet annotations described above or by the `restarter.stackable.tech/ignore` label described below.
The value is also `changes-ignored` if the referenced object does not exist (yet), for example because the StatefulSet was created before a mounted Secret.
This is intended: once the object is created, the value changes to its UID and resource version (unless it is excluded), and the Pods are restarted to pick it up.
This is intended: once the object is created, the value changes as described above (unless it is excluded), and the Pods are restarted to pick it up.
Kubernetes rolling-restarts the Pods whenever one of these values changes.
These annotations are managed by the operator, there is no need to set them yourself.

Expand Down
3 changes: 3 additions & 0 deletions rust/operator-binary/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,12 @@ chrono.workspace = true
clap.workspace = true
http.workspace = true
futures.workspace = true
hmac.workspace = true
json-patch.workspace = true
rand.workspace = true
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true
snafu.workspace = true
strum.workspace = true
tokio.workspace = true
Expand Down
5 changes: 4 additions & 1 deletion rust/operator-binary/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,10 @@ async fn main() -> anyhow::Result<()> {
)
.await?;

let (ctx, cm_store_tx, secret_store_tx) = create_context(client.clone());
let (ctx, cm_store_tx, secret_store_tx) = create_context(
client.clone(),
operator_environment.operator_namespace.clone(),
);

let mut readiness_checks = HealthCheckRegistry::new();
let authentication_class_crd_check = readiness_checks.register(format!(
Expand Down
Loading
Loading