While RealTheory recommends how to right-size your workloads, guardrails let you decide how RealTheory applies those recommendations automatically. Guardrails define which workloads automation can change, how large and how frequent each change can be, and the conditions that a workload must meet first. Together, guardrails keep automation within the limits that suit your environment.
Automation is opt-in. Automation is disabled by default and enabled by you. When you enable automation, RealTheory can make configuration changes within the scope and parameters of your automation profiles, including any default values that you have not changed. See Automation Opt-In Requirements Explained.
How Guardrails Work
Guardrails only shape how RealTheory applies a recommendation. They never change a value further than the recommendation calls for, or in a different direction.
Guardrails never cancel a recommendation. When a guardrail stops a change or lets RealTheory apply only part of the change, the workload is not yet fully optimized, so RealTheory continues to recommend a change, and automation applies the change when the guardrails allow.
Because most automated changes restart the affected pods, three settings help you control any disruption: Resize frequency limits how often a workload can change, Workload stability requires a workload to be healthy before a change, and the profile's maintenance window limits when changes can happen.
Default Values
Some guardrails apply default values to every profile: Workload scope, Resize delta, Resize frequency, Resize direction, Recommendation category, Workload metrics, and Workload stability.
Workload scope is always part of a profile, so you can edit Workload scope directly. The other guardrails apply their defaults even if you do not add those guardrails to the profile. To change the defaults, add the guardrail and change the guardrail's settings. Deleting a guardrail from the profile does not turn the guardrail off. The defaults apply again.
Default Values for All Guardrails
| Guardrail | Default values |
|---|---|
| Workload scope | All workload types and all workloads |
| Resize delta | For every CPU and memory request and limit: 100% for under-provisioned workloads with observed reliability issues, 50% for under-provisioned workloads without observed reliability issues, and 25% for over-provisioned workloads |
| Resize frequency | 30 minutes for under-provisioned workloads with observed reliability issues, 1 hour for under-provisioned workloads without observed reliability issues, and 6 hours for over-provisioned workloads |
| Resize direction | CPU request, Memory request, and Memory limit can increase or decrease. CPU limit is off. |
| Resize boundary | None |
| Automation exclusions | None |
| Recommendation category | Cost, Performance, and Reliability are all on. |
| Workload metrics | 1 day for Deployments and ReplicaSets, 2 days for DaemonSets and Custom Resources, 6 days for StatefulSets, and 12 hours for Pods |
| Workload stability | For over-provisioned workloads: at least 90% of pods running, at least 90% of pods ready, and no more than 5% of pods restarted within the last hour. For under-provisioned workloads: all conditions off, so RealTheory does not check stability. |
Guardrail Details
Workload scope: which workloads automation can change
Purpose
Defines which workloads are eligible for automated changes. Every profile has this guardrail.
Default values
All workload types and all workloads.
What to know
- A workload must match both the workload types and the workloads that you choose.
- If you choose A subset of workloads, configure a label selector or at least one pattern. You can use both, and a workload that matches either one is in scope.
Resize delta: how much of a recommended change applies at once
Purpose
Limits how much of a recommended change RealTheory can apply in one step so that automation eases toward the recommended values instead of making one large change. The limits apply to every CPU and memory request and limit.
Default values
- Under-provisioned workloads with observed reliability issues: 100%
- Under-provisioned workloads without observed reliability issues: 50%
- Over-provisioned workloads: 25%
Example
RealTheory recommends lowering a memory request by 400 MiB, and the limit is 25%. RealTheory applies 100 MiB. The workload is not yet fully optimized, so RealTheory recommends a new, smaller change and applies 25% of that change after the Resize frequency interval.
Resize frequency: how often a workload can change
Purpose
Sets the minimum time between automated changes to the same workload. The wait gives each change time to take effect, so that you can observe the impact before RealTheory applies another change.
Default values
- Under-provisioned workloads with observed reliability issues: 30 minutes
- Under-provisioned workloads without observed reliability issues: 1 hour
- Over-provisioned workloads: 6 hours
What to know
- A new recommendation can appear right after a change, either because Resize delta applied only part of the previous recommendation or because the workload's conditions changed. Automation does not apply the new recommendation until the interval passes.
- Because Resize delta can apply a large recommendation in several steps, this interval also sets the pace of those steps.
- To give some workload types more time between changes, for example StatefulSets, choose Customize by workload type.
Resize direction: which fields can change and in which direction
Purpose
Controls which resource fields automation can change and whether each field can increase, decrease, or both.
Default values
- CPU request: Both
- CPU limit: off
- Memory request: Both
- Memory limit: Both
What to know
- CPU limit is off by default because CPU limits are rarely the constraint in practice.
- Because this default applies even if you do not add the guardrail, automation does not change CPU limits unless you add this guardrail and turn on CPU limit.
- When a limit is off or decrease-only, RealTheory skips a request increase that would exceed the limit.
Resize boundary: the highest and lowest values automation can set
Purpose
Sets upper and lower values for container CPU and memory that automation cannot go beyond.
Default values
None. Set at least one value to save the guardrail.
What to know
- RealTheory applies the change but constrains the change to the specified boundary.
- Because container boundaries apply in order and the first match applies, put the most specific boundaries first.
- Fallback boundaries apply to every workload that no container boundary matches.
Automation exclusions: workloads that automation never changes
Purpose
Removes specific workloads from automation. Exclusions take precedence over every other guardrail.
Default values
None.
What to know
- Automation excludes any workload that matches the label selector or any pattern.
- When you roll back a change, you can also exclude the workload, which adds a pattern to this guardrail. See Rolling Back an Automation Change.
Recommendation category: which kinds of recommendations automation applies
Purpose
Limits automation to recommendations in the categories that you choose. RealTheory leaves recommendations in other categories for manual review.
Default values
Cost, Performance, and Reliability are all on, so automation applies recommendations in every category.
Workload metrics: how much data RealTheory needs first
Purpose
Sets how long RealTheory must collect continuous metric data for a workload before automation can apply a recommendation for the workload.
Default values
- Deployments and ReplicaSets: 1 day
- DaemonSets and Custom Resources: 2 days
- StatefulSets: 6 days
- Pods: 12 hours
What to know
- Because these defaults apply even if you do not add the guardrail, a newly created workload waits at least the period for the workload's type before automation can change the workload.
- When you add a cluster to RealTheory, historical Kubernetes data fills the gap, so the cluster's existing workloads do not wait.
- To learn how the amount of data affects recommendations, see Understanding Recommendation Confidence in RealTheory.
Workload stability: how healthy a workload must be before a change
Purpose
Requires a workload to meet conditions for running pods, ready pods, and recent restarts before automation changes the workload.
Default values
- Over-provisioned workloads: at least 90% of pods running, at least 90% of pods ready, and no more than 5% of pods restarted within the last hour
- Under-provisioned workloads: all conditions off, so RealTheory does not check stability
What to know
- The conditions are strict for over-provisioned workloads to protect workloads that are running well.
- The conditions are off for under-provisioned workloads so that automation can fix workloads that might be unstable because those workloads lack resources.
- When a workload does not meet the conditions, RealTheory does not apply the change and evaluates the recommendation again later.
See Also
Automation Profiles Explained
Creating an Automation Profile
Rolling Back an Automation Change
How To Use Label Selectors to Identify Resources