For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.
Install kagent
Install kagent 1.0 and Agent Substrate on a Kubernetes cluster.
kagent 1.0 runs every agent on Agent Substrate, so an installation sets up two systems in the same cluster. Agent Substrate provides the sandboxed compute that agents run on, and kagent provides the Harness, AgentTemplate, and AgentInstance API that you author against. Install Agent Substrate first, because the kagent controller connects to it at startup.
Note
These steps install kagent 1.0 fresh. kagent 1.0 has no in-place upgrade from the 0.10.x version line, and installing its custom resource definitions replaces the ones that a 0.10.x installation uses. To move an existing installation, start with Upgrade from 0.x.
Before you begin
Install the following CLI tools.
helm, the Kubernetes package manager. Use Helm 3.kubectl, the Kubernetes command line tool.jq, to read the cluster’s token issuer and the root certificate out of the generated CA pool.openssl, to convert that certificate to PEM format.kubectl-ate, the Agent Substrate command line tool, published as akubectlplugin with each Agent Substrate release.curl -fsSL -o kubectl-ate \ "https://github.com/kagent-dev/substrate/releases/download/v0.2.0-beta5/kubectl-ate-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')" chmod +x kubectl-ate sudo mv kubectl-ate /usr/local/bin/ kubectl ate --help
Set your model provider API key. The examples in this guide use OpenAI. For other providers, see Configure model providers.
export OPENAI_API_KEY="your-api-key-here"Prepare a Kubernetes cluster at 1.37 or later and enable it with the following requirements for Agent Substrate.
For local testing and development, create a kind cluster at Kubernetes 1.37 or later. Use kind v0.32.0 or later. Enable the
certificates.k8s.io/v1beta1API, which Agent Substrate depends on.kind create cluster --image kindest/node:v1.37.0 --config=- <<EOF kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: kagent runtimeConfig: "certificates.k8s.io/v1beta1": "true" EOF
Install Agent Substrate
Deploy the Agent Substrate control plane and data plane into the ate-system namespace, then create the identity material that its components authenticate with. Agent Substrate signs pod identities and service certificates from certificate authority (CA) pools that you generate, and it authenticates callers against a JSON Web Token (JWT) authority pool.
Important
Creating the identity material is required, and no Helm chart performs it for you. Agent Substrate authenticates its components with mutual Transport Layer Security (mTLS), and the identity material that mTLS depends on is created by the kubectl-ate plugin, not by Helm.
Install the Agent Substrate custom resource definitions (CRDs).
helm upgrade --install substrate-crds \ oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \ --version 0.2.0-beta5 \ --namespace ate-system --create-namespaceInstall the Agent Substrate control plane and data plane. Do not add
--waitto this command, because the pods cannot become ready until you create the identity material in the following steps.The
credentialProvider.namespacePoliciesvalue lets agents use the model provider API key. Agents do not hold the key themselves. Instead, the Agent Substrate egress gateway reads the key from a Kubernetes Secret and adds it to each model request. By default, Agent Substrate denies every agent access to every Secret. The following grant lets agents in thekagentatespace read Secrets in thekagentnamespace, where the kagent chart stores the key. Without this grant, all pods report healthy, but every model call fails with a403error.helm upgrade --install substrate \ oci://ghcr.io/kagent-dev/substrate/helm/substrate \ --version 0.2.0-beta5 \ --namespace ate-system \ -f - <<EOF credentialProvider: namespacePolicies: - atespace: kagent allowedNamespaces: [kagent] EOFCreate the CA pools that sign service DNS and pod identity certificates.
kubectl ate admin make-ca-pool --ca-id=1 \ --name=service-dns-ca-pool \ --secret-namespace=podcertificate-controller-system kubectl ate admin make-ca-pool --ca-id=1 \ --name=pod-identity-ca-pool \ --secret-namespace=podcertificate-controller-systemCreate the actor identity pools that Agent Substrate uses to issue and verify actor credentials.
kubectl ate admin make-jwt-pool --key-id=1 \ --name=actor-id-jwt-pool \ --secret-namespace=ate-system kubectl ate admin make-ca-pool --ca-id=1 \ --name=actor-id-ca-pool \ --secret-namespace=ate-systemCreate the CA pool for the egress gateway. The egress gateway intercepts HTTPS requests from agents so that it can inject credentials, and it signs a certificate for each destination from this pool. The pool uses an ECDSA P-256 key rather than the default key type, because the clients inside an agent sandbox might not support Ed25519 certificates. Without this pool, the
atenet-egresspod cannot start, and the rollout in a later step times out.kubectl ate admin make-ca-pool --ca-id=1 \ --name=egress-mitm-ca-pool \ --secret-namespace=ate-system \ --key-type=ECDSAP256Extract the actor identity root certificate and store it in the secret that the Agent Substrate API server reads.
actor_id_ca_root="$(kubectl get secret actor-id-ca-pool -n ate-system \ -o jsonpath='{.data.pool}' | base64 --decode \ | jq -r '.CAs[0].RootCertificateDER' | base64 --decode \ | openssl x509 -inform der -outform pem)" kubectl create secret generic actor-id-ca-certs -n ate-system \ --from-literal=ca.crt="${actor_id_ca_root}"Create the authentication configuration. The
kubernetesprovider accepts Kubernetes ServiceAccount tokens that are issued for the Agent Substrate API server audience. Kubernetes distributions advertise different issuers, so read the issuer from the cluster rather than naming one. An issuer that does not match the cluster’s own is accepted when you create the ConfigMap, and surfaces later astoken issuer ... not trustedon everykubectl atecall.k8s_issuer="$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)" kubectl create configmap ate-api-authentication -n ate-system \ --from-literal=authentication.yaml="actorIdentityJWTProvider: kubernetes jwtProviders: - name: kubernetes issuer: ${k8s_issuer} audiences: [api.ate-system.svc] certificateAuthorityFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt discoveryTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token "Note
An in-cluster issuer, such as
https://kubernetes.default.svcorhttps://kubernetes.default.svc.cluster.local, publishes no discovery document that a public client can reach, socertificateAuthorityFileanddiscoveryTokenFilepoint the API server at its own projected ServiceAccount certificate authority and token. Omit both lines on a cluster that advertises an external issuer, such as a GKE cluster.certificateAuthorityFilereplaces the client’s root certificate authorities with the cluster’s own, so retaining it fails the Transport Layer Security (TLS) handshake against an external issuer.Roll Agent Substrate out again so that its pods mount the identity material, and wait for them to become ready.
helm upgrade substrate \ oci://ghcr.io/kagent-dev/substrate/helm/substrate \ --version 0.2.0-beta5 \ --namespace ate-system --reuse-values --wait --timeout 10mVerify that Agent Substrate is running.
kubectl get pods -n ate-systemExample output:
NAME READY STATUS RESTARTS AGE ate-api-server-7b66644b7d-87gcr 1/1 Running 0 12m ate-api-server-7b66644b7d-hxfnp 1/1 Running 0 12m ate-controller-5dd748fd67-pzg9x 1/1 Running 0 12m atelet-9qbx8 1/1 Running 0 12m atenet-egress-68bd5b8768-9pjsp 2/2 Running 0 12m atenet-router-555b6598fb-vzp6w 2/2 Running 0 12m k8s-credential-provider-8cf8d559f-ntgtq 1/1 Running 0 12m postgres-0 2/2 Running 0 12m rustfs-6c4c677cb6-5c42d 1/1 Running 0 12m rustfs-bucket-init-qbbln 0/1 Completed 0 12m
Install kagent
The kagent chart connects the controller to Agent Substrate and creates a WorkerPool for agents to run on. A WorkerPool is platform capacity that you provision once, and every Harness references it. No Harness can run until a WorkerPool exists. The values in the following steps are evaluation defaults, including a bundled PostgreSQL instance, one controller replica, and one Worker. For a production installation, add the production values from Operational considerations to the same command.
Important
Install kagent 1.0 with Helm. The kagent install command does not yet provision Agent Substrate and cannot produce a working 1.0 installation.
Install the kagent CRDs.
helm upgrade --install kagent-crds \ oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ --version 1.0.0-alpha3 \ --namespace kagent --create-namespace --waitInstall kagent with the Agent Substrate integration enabled.
helm upgrade --install kagent \ oci://ghcr.io/kagent-dev/kagent/helm/kagent \ --version 1.0.0-alpha3 \ --namespace kagent --create-namespace --timeout 10m \ -f - <<EOF providers: default: openAI openAI: apiKey: ${OPENAI_API_KEY} controller: grpc: reflection: true substrate: enabled: true ateApiEndpoint: dns:///api.ate-system.svc:443 atenetRouterURL: http://atenet-router.ate-system.svc:80 defaultWorkerPool: name: kagent-default substrateWorkerPool: create: true replicas: 1 workerImage: "ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.2.0-beta5" EOFThe
controller.grpc.reflectionsetting lets a gRPC client discover the controller’s methods without a local copy of kagent’s proto files. The kagent CLI does not need it, because the CLI ships with generated clients for every kagent API. Leave reflection on to explore the API with a general-purpose client such as grpcurl, and turn it off for a production installation.Wait for the controller to roll out.
kubectl rollout status deployment/kagent-controller -n kagent --timeout=300sThe controller can restart a few times during a first install while it waits for its bundled PostgreSQL database to accept connections. The controller logs
dial tcp ...:5432: connect: connection refusedand then recovers on its own. A restart loop that reports anate-apidial failure instead indicates an incomplete identity bootstrap.
Verify the installation
Confirm that the kagent pods are running.
kubectl get pods -n kagentExample output:
NAME READY STATUS RESTARTS AGE kagent-controller-56c67b6cd7-hqj5z 1/1 Running 3 2m kagent-default-774d4496bd-sfx4b 1/1 Running 0 2m kagent-grafana-mcp-7c67f5697c-4mf78 1/1 Running 0 2m kagent-kmcp-controller-manager-6676b45958-zcvrp 1/1 Running 0 2m kagent-postgresql-856b475f59-n258n 1/1 Running 0 2m kagent-tools-54959b659c-lfkrz 1/1 Running 0 2m kagent-ui-5fdb6fd85c-vzb2z 1/1 Running 0 2mConfirm that the WorkerPool reports a ready replica.
kubectl get workerpools -n kagentExample output:
NAME DESIRED REPLICAS READY AGE kagent-default 1 1 1 2mGet the address to reach the kagent gRPC API, which serves the AgentInstance lifecycle and conversation calls. The guide to create your first agent assumes port-forwarding.
Forward the controller port and leave the command running. The API is then available at
localhost:8083.kubectl port-forward -n kagent svc/kagent-controller 8083:8083