Backup and restore

Back up Graph Platform persistent data before upgrades, migrations, or destructive operations. Three methods are available:

Method	When to use	Extra tooling
A — Manual journal copy	Quick pre-upgrade backups, no CSI support	None
B — Velero with CSI VolumeSnapshots	Production / staging with full namespace coverage	Velero, CSI snapshot CRDs, snapshot controller
C — Standalone CSI VolumeSnapshots	CSI available but Velero not installed	CSI snapshot CRDs, snapshot controller

Before you begin

What to back up

Component	State	Backup required
Anzo system journal (`anzo.jnl`)	User data, graphmarts, dashboards, permissions	Yes
Secondary datasource journals	Additional Bigdata journals	Yes, if present
Elasticsearch indices	Text-search indices	Only if on a PVC; emptyDir rebuilds automatically
Configuration and secrets	Keystores, certificates, licenses	Yes — Velero captures these as K8s objects
Custom resources	Anzo, AnzoGraph CRs	Yes — operators reconcile from these

Pod specs, Deployments, ReplicaSets, container images, and logs do not require backup.

Prerequisites

Method A requires only kubectl access. Methods B and C require:

A CSI driver with snapshot support (GKE: pd.csi.storage.gke.io, AKS: disk.csi.azure.com, AWS: ebs.csi.aws.com)
VolumeSnapshot CRDs and a snapshot controller

Method B additionally requires:

Velero installed with --features=EnableCSI and a backup storage location (GCS bucket, Azure Blob, or S3-compatible store)

Set the namespace

All commands in this document use the NAMESPACE variable:

export NAMESPACE='<namespace>'

Common procedures

Methods A and C reuse the same lock and unlock commands for backup consistency. They are defined here once and referenced by name throughout the document.

Lock writes

The backupLock service requires a DatasourceRequest RDF input specifying which datasource to lock. Without it, the service returns silently without locking anything.

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- bash -c '
PW=$(python3 -c "import json;print(json.load(open(\"/k8s-startup-data/firstbootFiles/init.json\"))[\"password\"])")
cat <<INNEREOF | /opt/graphstudio/Client/gs call \
  http://openanzo.org/semanticServices/datasources#backupLock \
  -h localhost -p 61616 -u sysadmin -w "$PW"
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix system: <http://openanzo.org/ontologies/2008/07/System#> .
@prefix ds: <http://openanzo.org/datasource/> .

ds:request {
    ds:request system:datasource ds:systemDatasource ;
        rdf:type system:DatasourceRequest .
}

ds:systemDatasource {
    ds:systemDatasource rdf:type system:Datasource , system:Component .
}
INNEREOF
'

Unlock writes

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- bash -c '
PW=$(python3 -c "import json;print(json.load(open(\"/k8s-startup-data/firstbootFiles/init.json\"))[\"password\"])")
cat <<INNEREOF | /opt/graphstudio/Client/gs call \
  http://openanzo.org/semanticServices/datasources#backupUnlock \
  -h localhost -p 61616 -u sysadmin -w "$PW"
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix system: <http://openanzo.org/ontologies/2008/07/System#> .
@prefix ds: <http://openanzo.org/datasource/> .

ds:request {
    ds:request system:datasource ds:systemDatasource ;
        rdf:type system:DatasourceRequest .
}

ds:systemDatasource {
    ds:systemDatasource rdf:type system:Datasource , system:Component .
}
INNEREOF
'

Stop Anzo and delete the PVC

This procedure is shared by the restore flows for Methods B and C. It stops the operator (if deployed), scales down the StatefulSet, and deletes the Anzo data PVC so it can be recreated from a snapshot.

Stop the operator to prevent it from reconciling the StatefulSet back to 1 replica:
```
kubectl scale deployment anzo-operator -n "${NAMESPACE}" --replicas=0
```

Scale down the Anzo StatefulSet:

kubectl scale statefulset <anzo-sts> -n "${NAMESPACE}" --replicas=0
kubectl wait --for=delete pod/<anzo-pod> -n "${NAMESPACE}" --timeout=60s

Delete the Anzo data PVC:

kubectl delete pvc data-volume-<anzo-sts>-0 -n "${NAMESPACE}"

Method A — Manual journal copy

Use this method for quick pre-upgrade backups or environments without CSI snapshot support.

1. Identify the pod and PVC

kubectl get pods -n "${NAMESPACE}" -l app=Anzo
kubectl get pvc -n "${NAMESPACE}"

2. Lock writes

Run the Lock writes procedure.

3. Copy data and logs

kubectl cp requires tar inside the container. The Anzo container does not include tar, but it has python3. Use python3 tarfile to stream a gzip archive to stdout:

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  python3 -c "import tarfile,sys; t=tarfile.open(fileobj=sys.stdout.buffer,mode='w:gz'); t.add('/opt/graphstudio/Server/data',arcname='data'); t.close()" \
  > ./anzo-data-backup-"$(date +%Y%m%d-%H%M%S)".tar.gz

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  python3 -c "import tarfile,sys; t=tarfile.open(fileobj=sys.stdout.buffer,mode='w:gz'); t.add('/opt/graphstudio/Server/logs',arcname='logs'); t.close()" \
  > ./anzo-logs-backup-"$(date +%Y%m%d-%H%M%S)".tar.gz

4. Unlock writes

Run the Unlock writes procedure.

5. Store the backup

Copy the journal file to external storage (object storage, NFS, or a secure local location outside the cluster).

warning

Method A backs up the data directory (journal, indices, FLDS) and logs. It does not capture Secrets, ConfigMaps, or custom resources. Use Method B for full namespace coverage.

Method B — Velero with CSI VolumeSnapshots

Use this method for production and staging environments with CSI snapshot support. Velero snapshots all PVCs in the namespace and captures Secrets, ConfigMaps, and custom resources as Kubernetes objects.

1. Annotate the pod for consistency hooks

Apply pre-backup and post-backup hook annotations to the Anzo pod so that Velero locks writes before snapshotting and unlocks them afterward. The hooks extract the admin password at runtime from init.json on the shared startup volume:

metadata:
  annotations:
    pre.hook.backup.velero.io/container: anzo
    pre.hook.backup.velero.io/timeout: "30s"
    pre.hook.backup.velero.io/command: >-
      ["/bin/sh", "-c",
      "PW=$(python3 -c 'import json;print(json.load(open(\"/k8s-startup-data/firstbootFiles/init.json\"))[\"password\"])') &&
      printf '@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix system: <http://openanzo.org/ontologies/2008/07/System#> .\n@prefix ds: <http://openanzo.org/datasource/> .\nds:request { ds:request system:datasource ds:systemDatasource ; rdf:type system:DatasourceRequest . }\nds:systemDatasource { ds:systemDatasource rdf:type system:Datasource , system:Component . }\n' |
      /opt/graphstudio/Client/gs call http://openanzo.org/semanticServices/datasources#backupLock
      -h localhost -p 61616 -u sysadmin -w $PW"]
    post.hook.backup.velero.io/container: anzo
    post.hook.backup.velero.io/timeout: "30s"
    post.hook.backup.velero.io/command: >-
      ["/bin/sh", "-c",
      "PW=$(python3 -c 'import json;print(json.load(open(\"/k8s-startup-data/firstbootFiles/init.json\"))[\"password\"])') &&
      printf '@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .\n@prefix system: <http://openanzo.org/ontologies/2008/07/System#> .\n@prefix ds: <http://openanzo.org/datasource/> .\nds:request { ds:request system:datasource ds:systemDatasource ; rdf:type system:DatasourceRequest . }\nds:systemDatasource { ds:systemDatasource rdf:type system:Datasource , system:Component . }\n' |
      /opt/graphstudio/Client/gs call http://openanzo.org/semanticServices/datasources#backupUnlock
      -h localhost -p 61616 -u sysadmin -w $PW"]

warning

Do not hardcode credentials in annotations. The init.json file is populated by the init-container from the anzo-no-persistence Secret (key admin_password) and mounted on a shared emptyDir volume at /k8s-startup-data/.

2. Create a backup

velero backup create anzo-backup-"$(date +%Y%m%d)" \
  --include-namespaces "${NAMESPACE}" \
  --snapshot-volumes=true

Velero executes: pre-hook (lock) → CSI VolumeSnapshot of all PVCs → post-hook (unlock) → copy to object storage.

3. Verify the backup

velero backup describe anzo-backup-<date> --details

Confirm Phase: Completed with no errors or warnings in the hook execution output.

What Velero backs up

When run with --include-namespaces and --snapshot-volumes=true, Velero captures all Kubernetes resources in the namespace plus a CSI VolumeSnapshot of every PVC.

Kubernetes objects (stored in the backup tarball):

Category	Resources
ConfigMaps	Anzo configuration, trust bundles, extra imports
Secrets	Anzo admin credentials, JVM options, image pull secrets, Keycloak credentials, TLS certificates, Helm release metadata
PersistentVolumeClaims	Anzo data volume (contains journal, indices, FLDS)
ServiceAccounts	Anzo, operator, and Keycloak service accounts
Deployments	`anzo-operator`, `keycloak-operator`
StatefulSets	Anzo, Keycloak, PostgreSQL
Services / Ingresses	All service and ingress definitions
Custom Resources	`Anzo` CR, `Keycloak` CR, `KeycloakRealmImport` CR
RBAC	Roles and RoleBindings for operators
NetworkPolicies	Keycloak network policy
Jobs	Keycloak realm import job

Volume data (CSI VolumeSnapshot):

The Anzo data PVC is snapshotted via the CSI driver. The snapshot contains the full data directory: system journal (anzo.jnl), secondary datasource journals (if present), Elasticsearch indices, FLDS, and any configuration files on the volume.

Excluded by default: Nodes, Events, CSINodes, VolumeAttachments, and Velero's own CRDs.

tip

Run velero backup describe <name> --details to see the exact item count and resource list for a specific backup.

Method C — Standalone CSI VolumeSnapshots

Use this method when CSI snapshot support is available but Velero is not installed. It snapshots the Anzo data PVC directly using the Kubernetes VolumeSnapshot API with manual write-locking for consistency.

1. Create a VolumeSnapshotClass

Define the snapshot class for your CSI driver. Use Retain as the deletion policy for production environments to prevent accidental snapshot deletion:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
driver: ebs.csi.aws.com          # replace with your CSI driver
deletionPolicy: Retain            # use Retain for production

kubectl apply -f volumesnapshotclass.yaml

2. Lock writes before snapshot

Run the Lock writes procedure.

3. Create the VolumeSnapshot

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: anzo-data-snapshot
  namespace: <namespace>
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: data-volume-<anzo-sts>-0

kubectl apply -f volumesnapshot.yaml

4. Verify the snapshot

Wait for the snapshot to become ready before unlocking writes:

kubectl get volumesnapshot anzo-data-snapshot -n "${NAMESPACE}" \
  -o jsonpath='{.status.readyToUse}'

Confirm Status.ReadyToUse: true and a non-null Status.BoundVolumeSnapshotContentName:

kubectl get volumesnapshot anzo-data-snapshot -n "${NAMESPACE}" \
  -o jsonpath='{.status.boundVolumeSnapshotContentName}'

5. Unlock writes

Run the Unlock writes procedure.

warning

Method C snapshots only the Anzo data PVC. It does not capture Secrets, ConfigMaps, or custom resources. Back up those objects separately if needed, or use Method B for full namespace coverage.

Restore

Data replacement

Restoring replaces existing data. Verify that the backup is valid and that you have confirmed the target namespace before proceeding.

Restore from Method A

Stop the graphstudioServer process, overwrite data and logs using python3 tarfile, and start it again.

Stop graphstudioServer:

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  /opt/graphstudio/Server/graphstudioServer stop

Copy data and logs back into the pod:

kubectl exec -i <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  python3 -c "import tarfile,sys; t=tarfile.open(fileobj=sys.stdin.buffer,mode='r:gz'); t.extractall('/opt/graphstudio/Server'); t.close()" \
  < ./anzo-data-backup.tar.gz

kubectl exec -i <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  python3 -c "import tarfile,sys; t=tarfile.open(fileobj=sys.stdin.buffer,mode='r:gz'); t.extractall('/opt/graphstudio/Server'); t.close()" \
  < ./anzo-logs-backup.tar.gz

Start graphstudioServer:

kubectl exec <anzo-pod> -n "${NAMESPACE}" -c anzo -- \
  /opt/graphstudio/Server/graphstudioServer start

Restore from Method B

If the namespace was deleted (disaster recovery)

Ensure the target cluster has the required CRDs, operators, and Velero installed.
Restore the namespace:
```
velero restore create --from-backup anzo-backup-<date> \
  --include-namespaces "${NAMESPACE}"
```
Velero recreates: namespace → PVCs from CSI VolumeSnapshots → Secrets and ConfigMaps → StatefulSets and Deployments → custom resources. Operators reconcile automatically.

Wait for pods:

kubectl wait --for=condition=Ready pod -l app=Anzo \
  -n "${NAMESPACE}" --timeout=120s

If the namespace still exists (data-only restore)

Velero does not overwrite existing resources. If the namespace and PVCs are still present (for example, the journal was deleted or corrupted but the infrastructure is intact), you must delete the PVC first so Velero can recreate it from the VolumeSnapshot.

Run the Stop Anzo and delete the PVC procedure (steps 1–3).

Restore from Velero — the PVC is now missing, so Velero recreates it from the VolumeSnapshot:

velero restore create --from-backup anzo-backup-<date> \
  --include-namespaces "${NAMESPACE}"

Scale the operator and StatefulSet back up:

kubectl scale deployment anzo-operator -n "${NAMESPACE}" --replicas=1
kubectl scale statefulset <anzo-sts> -n "${NAMESPACE}" --replicas=1

Wait for pods:

kubectl wait --for=condition=Ready pod -l app=Anzo \
  -n "${NAMESPACE}" --timeout=120s

warning

Without deleting the PVC first, velero restore create restores Kubernetes objects (Secrets, ConfigMaps, CRs) but skips the existing PVC — the volume data is not replaced and any data loss persists.

Restore from Method C

Restoring from a standalone VolumeSnapshot requires replacing the Anzo data PVC with a new one sourced from the snapshot and letting the operator recreate the StatefulSet.

Run the Stop Anzo and delete the PVC procedure (steps 1–3).
Delete the Anzo StatefulSet so the operator can recreate it with the updated volume configuration:
```
kubectl delete statefulset <anzo-sts> -n "${NAMESPACE}"
```

Edit the Anzo CR to source the volume from the snapshot. Add a dataSource block inside volumeClaimTemplates:

kubectl edit anzo <anzo-cr> -n "${NAMESPACE}"

Add the following under the relevant volumeClaimTemplates entry:

volumeClaimTemplates:
  - spec:
      dataSource:
        name: anzo-data-snapshot
        kind: VolumeSnapshot
        apiGroup: snapshot.storage.k8s.io

Scale the operator back up — it will reconcile the Anzo CR, recreate the StatefulSet with the snapshot-backed PVC, and start the Anzo pod:
```
kubectl scale deployment anzo-operator -n "${NAMESPACE}" --replicas=1
```

Wait for pods:

kubectl wait --for=condition=Ready pod -l app=Anzo \
  -n "${NAMESPACE}" --timeout=120s

tip

Once the restore is verified and Anzo is healthy, remove the dataSource block from the Anzo CR to prevent future StatefulSet recreations from attempting to use a potentially expired snapshot.

Verify

Run these checks after every restore operation.

Check	Command	Pass criteria
Pods running	`kubectl get pods -n "${NAMESPACE}"`	All pods `Running` and `Ready`
PVCs bound	`kubectl get pvc -n "${NAMESPACE}"`	All PVCs `Bound`
Journal exists	`kubectl exec <anzo-pod> -c anzo -- ls -lh /opt/graphstudio/Server/data/journal/anzo.jnl`	File present, size consistent with pre-backup
Health endpoint	`curl -sk https://<anzo-service>:8443/`	HTTP 200 or 302
SPARQL query	Run a `SELECT (COUNT(*) AS ?total) WHERE { ?s ?p ?o }` query via the Anzo CLI or UI	Expected row count
Secrets present	`kubectl get secrets -n "${NAMESPACE}"`	All expected secrets listed

Method comparison

	Method A (manual copy)	Method B (Velero + CSI)	Method C (standalone CSI snapshot)
Coverage	Data directory and logs	Full volume (all PVCs, Secrets, ConfigMaps)	Anzo data PVC only
Consistency	Manual lock and unlock	Automated via hook annotations	Manual lock and unlock
Scheduling	Manual or custom CronJob	Velero Schedule resource	Manual or custom CronJob
Extra tooling	None	Velero, CSI snapshot CRDs, snapshot controller	CSI snapshot CRDs, snapshot controller
Restore mechanism	Stop server, file copy, start server	Velero restore (recreates PVC from snapshot)	Anzo CR edit with `dataSource` referencing the snapshot

Before you begin​

What to back up​

Prerequisites​

Set the namespace​

Common procedures​

Lock writes​

Unlock writes​

Stop Anzo and delete the PVC​

Method A — Manual journal copy​

1. Identify the pod and PVC​

2. Lock writes​

3. Copy data and logs​

4. Unlock writes​

5. Store the backup​

Method B — Velero with CSI VolumeSnapshots​

1. Annotate the pod for consistency hooks​

2. Create a backup​

3. Verify the backup​

What Velero backs up​

Method C — Standalone CSI VolumeSnapshots​

1. Create a VolumeSnapshotClass​

2. Lock writes before snapshot​

3. Create the VolumeSnapshot​

4. Verify the snapshot​

5. Unlock writes​

Restore​

Restore from Method A​

Restore from Method B​

If the namespace was deleted (disaster recovery)​

If the namespace still exists (data-only restore)​

Restore from Method C​

Verify​

Method comparison​

Before you begin

What to back up

Prerequisites

Set the namespace

Common procedures

Lock writes

Unlock writes

Stop Anzo and delete the PVC

Method A — Manual journal copy

1. Identify the pod and PVC

2. Lock writes

3. Copy data and logs

4. Unlock writes

5. Store the backup

Method B — Velero with CSI VolumeSnapshots

1. Annotate the pod for consistency hooks

2. Create a backup

3. Verify the backup

What Velero backs up

Method C — Standalone CSI VolumeSnapshots

1. Create a VolumeSnapshotClass

2. Lock writes before snapshot

3. Create the VolumeSnapshot

4. Verify the snapshot

5. Unlock writes

Restore

Restore from Method A

Restore from Method B

If the namespace was deleted (disaster recovery)

If the namespace still exists (data-only restore)

Restore from Method C

Verify

Method comparison