Skip to main content
Version: v0.22 (EOL)
Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Native sleep mode

warning

Native sleep mode is intended for pre-production use cases only, and comes with some limitations and caveats and should not be enabled on virtual clusters that are using the current sleep mode configuration found in external.platform.

Not all workloads need to run all the time, and scaling them down saves time and money. With vCluster-native sleep mode you can scale down workloads based on a set schedule, or activity from users and ingress.

Enable native sleep mode​

To enable vCluster-native sleep mode, use the following configuration inside your vcluster.yaml:

native sleep mode configuration
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 1h

How it works​

Sleep mode operations​

Sleep mode performs two main operations:

Sleeping Deletes bare pods and scales down the following resources:

  • Deployments
  • ReplicaSets
  • ReplicationControllers
  • DaemonSets

Waking

  • Scales resources back to their original state
  • Cannot restore previously deleted bare pods

Resource exemption​

Resources can be exempted from sleep mode using either:

  • Adding the annotation sleepmode.loft.sh/exclude: true
  • Configuring sleepMode with specific labels
  • Adding configured labels to workloads that should keep running

Detecting activity​

To wake a sleeping cluster or to update the last active time, native sleep mode captures the following:

  • Access of cluster resources through API calls (e.g., kubectl get <resource>)
  • Attempts to contact Ingress endpoints (NGINX only)
note

Ingress activity detection is only available for NGINX ingress controllers, making use of the mirror-target annotation. This has the effect of overwriting any previously set mirror-target annotation.

Ingress configuration​

Sync to host​

If you install your ingress controllers in the vCluster you'll need to exempt the controller from sleeping so it remains active for requests that would wake the vCluster. If you install the Ingress controller in the host cluster, you'll need to enable syncing ingresses to the host.

enable ingress syncing
sync:
  toHost:
    ingresses:
      enabled: true

Reachability​

For proper ingress activity detection:

  • The vCluster pod must be discoverable by the ingress controller
  • DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local must resolve
  • Proper dnsPolicy configuration in the ingress controller Helm installation

Examples​

The below examples demonstrate how to configure sleep mode in a virtual cluster. Please make sure to install the necessary prerequisites before proceeding.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
    brew install loft-sh/tap/vcluster

    The binaries in the tap are signed using the Sigstore framework for enhanced security.

    Confirm that you've installed the correct version of the vCluster CLI.

    vcluster --version
  • docker: Container runtime installation guide.
  • kind: Kubernetes in Docker installation guide.
  • curl: Command-line tool for transferring data (pre-installed on most systems, see installation guide if needed)

Deployment​

Sleep mode with deployment resource.

Configure sleep mode for a deployment resource​

  1. Create the kind cluster.

    create kind cluster
    kind create cluster --name sleep-mode-demo

  2. Deploy a virtual cluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s
        exclude:
          selector:
            labels:
              sleep: no-thanks

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

    Workloads with the label sleep: no-thanks don't enter sleep mode after 30 seconds.

  3. Create demo deployments in your virtual cluster.

    Use the following deployment YAML to create two deployments.

    Example deployments
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleepy-deployment
      labels:
        app: sleepy-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-1
      template:
        metadata:
          labels:
            app: demo-dep-1
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: sleepy-demo

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: no-sleep-deployment
      labels:
        sleep: no-thanks
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-2
      template:
        metadata:
          labels:
            app: demo-dep-2
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: not-sleepy-demo

    The first deployment does not have any special configurations for sleep mode. You can replace it with another deployment if needed. The second deployment includes a special label on the Deployment, preventing it from scaling down after 30 seconds.

    You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

  4. Verify Deployments sleep status.

    deployment sleep check
    > sleep 30; kubectl get deployments
    NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    default       no-sleep-deployment   2/2     2            2           1m
    default       sleepy-deployment     0/2     0            0           1m

    The sleepy-deployment reports 0/2 replicas after 30 seconds. Running kubectl counts as cluster activity, which is why it reports 0/2 instead of 0/0. The kubectl command triggers vCluster to update the replica count back to the original value of 2, but the replicas haven't become ready by the time kubectl get ... returns.

Next steps​

Experiment with the sleep mode feature by trying the following:

  • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
  • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Ingress controller​

Sleep mode with ingress resource.

Configure sleep mode for an ingress controller​

  1. Create the kind cluster.

    Create a kind cluster
    kind create cluster --name ingress-demo --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      apiServerAddress: "0.0.0.0"
    nodes:
      - role: control-plane
        extraPortMappings:
          - containerPort: 80
            hostPort: 80
            protocol: TCP
    EOF

  2. Install the NGINX IngressController.

    install ingress controller
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.dnsPolicy=ClusterFirstWithHostNet \
      --set controller.hostNetwork=true \
      --set controller.service.type=ClusterIP

  3. Create the vCluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sync:
      toHost:
        ingresses:
          enabled: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

  4. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

  5. Create resources for the Ingress such as a Deployment and Service.

    Use the following manifest to create:

    • Namespace named bar
    • Deployment for the pods backing the Service
    • Service for the Ingress
    • Ingress resource
    Example deployments
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bar

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bar-deployment
      namespace: bar
      labels:
        app: bar-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: bar
      template:
        metadata:
          labels:
            app: bar
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: bar-app

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: bar-service
      namespace: bar
    spec:
      selector:
        app: bar
      ports:
        # Default port used by the image
        - port: 8080

    ---

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: example-ingress
      namespace: bar
    spec:
      ingressClassName: nginx
      rules:
        - http:
            paths:
              - pathType: Prefix
                path: /bar
                backend:
                  service:
                    name: bar-service
                    port:
                      number: 8080
          host: backend.local

  6. Verify the ingress is working properly with curl.

    Test the Ingress endpoint within the 30-second activity window by running curl --silent backend.local/bar. The pod name from the Deployment that responds is displayed.

  7. Allow the virtual cluster to go to sleep.

    Wait 30 seconds for the cluster to sleep, then run the curl command again. For convenience, run watch -d curl --silent backend.local/bar to repeatedly test the endpoint. On the first attempt, you’ll see the message Client sent an HTTP request to an HTTPS server. because an HTTP request was sent to the HTTPS wake endpoint. Subsequent requests display new pod names.

Additional examples​

Sleep mode label selectors and schedule.
Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 3h  # Uses Go's Duration with a max unit of hour
      exclude:
        selector:
          labels:
            dont: sleep
Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
experimental:
  sleepMode:
    enabled: true
    timezone: America/Denver
    autoSleep:
      schedule: 30 17 * * 5
      wakeup:
        schedule: 0 7 * * 1

Config reference​

sleepMode required object pro​

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean pro​

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string pro​

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object pro​

AutoSleep holds autoSleep details

afterInactivity required string pro​

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string pro​

Schedule represents a cron schedule for when to sleep workloads

wakeup required object pro​

Wakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string pro​

exclude required object pro​

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object pro​
labels required object pro​

Labels defines what labels should be looked for