#Infrastructure Resources for Huawei DCS

#Overview

Before creating clusters on Huawei DCS, you need to configure infrastructure resources including cloud credentials, IP pools, and machine templates.

You can manage infrastructure resources using either the web UI or YAML manifests. The web UI provides a guided interface with validation, while YAML offers automation capabilities.

On Huawei DCS, the IP pool also carries any disks that must survive VM replacement. This includes the platform-required /var/cpaas disk.

#Cloud Credentials

Cloud credentials store the DCS platform access information required for cluster operations.

#Using the Web UI

#Prerequisites

Before creating a cloud credential, verify the following DCS platform requirements:

User Configuration:

• User Type: One of the following:
  • Interface interconnection user — default; existing behaviour. Use this when an interconnect-style account is already provisioned in the DCS portal.
  • Domain user — LDAP/AD-backed user managed in the directory service. Requires that DCS has domain authentication configured by your DCS administrator, and that the directory user is pre-created in the DCS portal with the role mapping below.
• Role: Must be administrator

Password Policy (applies to Interface interconnection user only):

Navigate to System Management → Rights Management → Rights Management Policy and verify:

• Policy: Whether to modify the password of an interface interconnection user upon password resetting and first login
• Value: Must be set to No

If set to Yes, the user's password will be forced to change upon first login, breaking authentication and causing cluster creation failures.

For Domain user, password lifecycle is governed by your LDAP/AD directory service, not by DCS portal policy.

#Creating a Cloud Credential

Navigation: Clusters → Cloud Credentials → Create Cloud Credential → Select Huawei DCS

Form Fields:

Validation Rules:

• Name must be 1-63 characters, containing only lowercase letters, numbers, and hyphens, and must start and end with a letter or number
• DCS Endpoint must be a valid URL format starting with http:// or https://

#Managing Cloud Credentials

Viewing Credentials: Navigate to Clusters → Cloud Credentials to view all configured credentials with their type, creation time, and creator.

Updating Credentials: Click Update on a credential to modify the Display Name. Password updates are not supported in the current version (planned for a future release).

Deleting Credentials: Click Delete to remove a credential. Confirm the deletion in the dialog.

#Using YAML

Create a Secret resource to store DCS authentication information:

apiVersion: v1
data:
  authUser: <base64-encoded-auth-user>
  authKey: <base64-encoded-auth-key>
  endpoint: <base64-encoded-endpoint>
  userType: <base64-encoded-user-type>   # optional; "interconnect" (default) or "domain"
kind: Secret
metadata:
  name: <auth-secret-name>
  namespace: cpaas-system
type: Opaque

apiVersion: v1
stringData:
  authUser: <plain-text-auth-user>
  authKey: <plain-text-auth-key>
  endpoint: <plain-text-endpoint>
  userType: <plain-text-user-type>   # optional; "interconnect" (default) or "domain"
kind: Secret
metadata:
  name: <auth-secret-name>
  namespace: cpaas-system
type: Opaque

Parameter Descriptions:

Example:

# Encode credentials
echo -n "admin" | base64
echo -n "your-password" | base64
echo -n "https://dcs.example.com:7443" | base64
echo -n "domain" | base64   # only when using a domain user; omit for interconnect (default)

# Apply the Secret
kubectl apply -f dcs-secret.yaml -n cpaas-system

#Credential User Types

DCS Provider supports two credential user types, selected by the optional Secret key userType:

Compatibility: Existing Secrets without a userType key continue to authenticate as before; no migration is required to keep using interconnect-style credentials.

UI coverage: The current web UI form does not expose the userType field. To use the domain value, configure the Secret through YAML.

#IP Pools

IP pools define the network configuration (IP addresses, subnet masks, gateways, DNS) for cluster nodes. Each pool can contain multiple node entries, and each node can have multiple network interface configurations.

For DCS persistent disks, each IP entry can also declare persistentDisk items. These disks are bound to the IP slot instead of the VM lifecycle, so they can be detached from an old VM and reattached to a replacement VM during rolling upgrades. Use this mechanism for the platform-required /var/cpaas disk and for any other node-local data that must survive delete-recreate operations. Since DCS provider v1.0.16, this persistentDisk declaration is supported through YAML only.

#Using the Web UI

#Prerequisites

• Cloud Credential has been created

#Creating an IP Pool

Navigation: Clusters → Virtual Machine → IP Pools → Create IP Pool → Select Credential

Form Structure:

The IP Pool form consists of a list of Pools. Each Pool represents one node and contains:

1. Node IP (required, exactly one per Pool)
2. Additional NIC IPs (optional, multiple per Pool)

Node IP Fields:

Additional NIC IPs Fields:

Validation Rules:

• IP addresses must be unique within the same IP Pool
• IP addresses must be valid IPv4 format
• Subnet mask must be valid format
• IP address must be within the configured subnet range
• Gateway must be a valid IPv4 address within the subnet range

Tips:

• At least one node entry is required
• Exactly one Node IP configuration is required per node
• Additional NIC IPs are optional for multi-NIC scenarios (e.g., storage network separation)

#Managing IP Pools

Viewing Pools: Navigate to Clusters → Virtual Machine → IP Pools to view all configured pools with their node IPs and creation time.

Updating Pools: Click Update to add or remove node entries and modify network configurations.

Deleting Pools: Click Delete to remove a pool. Confirm the deletion in the dialog.

#Using YAML

Create a DCSIpHostnamePool resource:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: <iphostname-pool-name>
  namespace: cpaas-system
spec:
  pool:
  - ip: "<ip-1>"
    mask: "<mask>"
    gateway: "<gateway>"
    dns: "<dns>"
    hostname: "<hostname-1>"
    machineName: "<machine-name-1>"
  - ip: "<ip-2>"
    mask: "<mask>"
    gateway: "<gateway>"
    dns: "<dns>"
    hostname: "<hostname-2>"
    machineName: "<machine-name-2>"
  - ip: "<ip-3>"
    mask: "<mask>"
    gateway: "<gateway>"
    dns: "<dns>"
    hostname: "<hostname-3>"
    machineName: "<machine-name-3>"

Parameter Descriptions:

#Persistent Disks in the IP Pool

Declare upgrade-preserved disks in DCSIpHostnamePool.spec.pool[].persistentDisk, not in DCSMachineTemplate.

Since DCS provider v1.0.16, YAML is the only supported way to declare these persistent disks.

• Use the IP entry to bind each persistent disk to a fixed (ip, slot) identity.
• Use this model for the platform-required /var/cpaas disk.
• Keep DCSMachineTemplate focused on the system disk and any template-local disks that may be recreated with the VM.
• When a replacement VM boots, the guest disk setup script checks for an existing filesystem. If one already exists, it skips formatting and mounts the disk directly.
• Pool-managed persistent disks require one-by-one replacement. Keep KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge = 0 and MachineDeployment.spec.strategy.rollingUpdate.maxSurge = 0 when you rely on this feature.

If you plan to use persistent disks, ensure the DCS VM template is 4.2.1 or later because safe shutdown and disk detach depend on guest tools inside the guest OS.

Example:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: <iphostname-pool-name>
  namespace: cpaas-system
spec:
  pool:
  - ip: "<ip-1>"
    mask: "<mask>"
    gateway: "<gateway>"
    dns: "<dns>"
    hostname: "<hostname-1>"
    machineName: "<machine-name-1>"
    persistentDisk:
    - slot: 0
      quantityGB: 40
      datastoreClusterName: <datastore-cluster-name>
      path: /var/cpaas
      format: xfs
      mountOptions:
      - defaults

Persistent disk field descriptions:

Update rules:

• You can append new persistentDisk entries to an existing IP slot. The controller attaches the newly added disk to the running VM on the DCS side, but it does not format or mount the disk inside the guest OS. Guest formatting and mounting take effect only after the VM is replaced and the replacement VM runs the generated disk setup during bootstrap.
• Do not delete existing persistentDisk entries from the spec. The webhook rejects removal.
• Treat format, options, and pciType as immutable after creation.
• Treat quantityGB and datastore changes as rollout-sensitive changes. The webhook performs best-effort platform validation when the pool carries a cluster label.
• Treat path and mountOptions changes as guest-side changes that take effect on replacement machines during rolling updates.

#DCS Platform Capacity and Placement

Before authoring DCSMachineTemplate resources, verify that the target DCS site, datastore, and compute cluster have enough free capacity to host the planned VMs, and that the DCS-side scheduling policy can place new VMs on the available hosts. Skipping these checks is the single most common cause of VMs that appear to hang in creating state with no error returned by the DCS API.

#Running the diagnostic snippets

The capacity and placement checks below use read-only DCS REST calls. Before running any snippet, set the following environment variables in your shell:

Set the three variables once at the start of the diagnostic session:

# Replace the placeholders with values appropriate for your environment.
export DCS_BASE='https://<dcs-api-host>:7443'
export SITE='<site-id>'

# Read the user and key from the cloud-credential Secret without printing them.
DCS_USER="$(kubectl -n cpaas-system get secret <credential-secret-name> -o jsonpath='{.data.authUser}' | base64 -d)"
DCS_KEY="$(kubectl -n cpaas-system get secret <credential-secret-name> -o jsonpath='{.data.authKey}' | base64 -d)"

# Acquire a session token. The response header X-AUTH-TOKEN carries the value used by subsequent calls.
HEADERS="$(mktemp)"
curl -k -sS -D "$HEADERS" -o /dev/null -X POST "${DCS_BASE}/service/session" \
  -H "X-Auth-User: $DCS_USER" \
  -H "X-Auth-Key: $DCS_KEY" \
  -H 'X-ENCRYPT-ALGORITHM: 1' \
  -H 'X-Auth-UserType: 2' \
  -H 'version: 8.1'
export TOKEN="$(grep -i '^X-AUTH-TOKEN:' "$HEADERS" | tail -n1 | cut -d' ' -f2- | tr -d '\r\n ')"
rm -f "$HEADERS"
unset DCS_USER DCS_KEY

Reuse $DCS_BASE, $SITE, and $TOKEN in every subsequent diagnostic snippet on this page. Refresh the token by re-running the POST /service/session command when the API begins returning errorCode: 10000002 (session expired or not logged in).

#Choosing a datastore

Each DCSMachineDiskSpec entry binds a disk to a DCS datastore through one of two mutually exclusive fields:

• datastoreName — pins the disk to a specific datastore. The DCS scheduler does not look at sibling datastores even if they have free space. This is the strictest binding.
• datastoreClusterName — pins the disk to a datastore cluster (a DCS-side group of datastores). The DCS scheduler chooses one specific datastore from the cluster that satisfies capacity. This is the safer default when the platform provides a datastore cluster.

Prefer datastoreClusterName whenever the DCS administrator has configured a datastore cluster, so a single near-full datastore does not block deployments while sibling datastores still have free space.

Capacity preflight:

Before applying a new DCSMachineTemplate, list the existing DCS datastores and confirm that the target has enough free capacity for the worst case of system disk + ∑(spec disk size) × replica count. From an environment with DCS API access, the read-only listing looks like this:

curl -k -sS "${DCS_BASE}/service/sites/${SITE}/datastores" \
  -H "X-Auth-Token: ${TOKEN}" -H 'version: 8.1' \
  | python3 -c "
import json,sys
for ds in json.load(sys.stdin).get('datastores',[]) or []:
  print(f\"{ds.get('name','?'):24} cap={ds.get('capacityGB','?')}GB used={ds.get('usedSizeGB','?')}GB free={ds.get('freeSizeGB','?')}GB state={ds.get('status','?')}\")"

Each DCSMachineTemplate cp / worker pair commonly needs the system disk (template-default, often ~80 GB) plus the explicit data disks declared in the spec, so a single control-plane node with etcd 10G + kubelet 100G + containerd 100G + /var/cpaas 100G consumes roughly 390 GB before the template's system disk. A datastore used above ~70 % is a strong signal to pick a different datastore — even if the absolute free space looks large, fragmentation and reservations can cause new VM placement to stall.

#Host placement and high availability

Host placement on the DCS platform is performed by the DCS scheduler. The DCS Provider does not expose a way to pin a virtual machine to a specific DCS Host through DCSMachineTemplate. Virtual machines cloned from the same template can therefore land on the same physical host.

Two consequences follow from this:

• Capacity hot spots: If the DCS cluster has uneven host load (for example, one host is severely CPU-over-committed while another is mostly idle), the DCS scheduler may refuse new placements on the over-committed host. The deploy task can stall until the load is rebalanced on the DCS side or DRS is enabled. Use the snippet below to confirm whether host load is the cause.
• No automatic spread for control plane: Control-plane virtual machines from a single DCSMachineTemplate are not automatically distributed across multiple physical hosts. For high-availability deployments, configure VM-VM anti-affinity on the DCS platform side as described in Cross-Host High Availability for Control Plane below.

To check current host load before applying a new template:

curl -k -sS "${DCS_BASE}/service/sites/${SITE}/hosts" \
  -H "X-Auth-Token: ${TOKEN}" -H 'version: 8.1' \
  | python3 -c "
import json,sys
for h in json.load(sys.stdin).get('hosts',[]) or []:
  print(f\"{h.get('name','?'):10} status={h.get('status','?'):10} cpuUsed/total={h.get('cpuUsedCores','?')}/{h.get('cpuTotalCores','?')}c memUsed={h.get('memUsedSizeMB','?')}MB cpuMuxRatio={h.get('cpuMuxRatio','?')}\")"

A cpuMuxRatio (CPU allocation / physical cores) above ~3× on every host in the cluster indicates the cluster is already saturated and will likely refuse new placements until the load is rebalanced on the DCS side.

#Cross-Host High Availability for Control Plane

The DCS Provider does not currently implement host pinning or Cluster API FailureDomain spreading. As a result, the three control-plane virtual machines of a Kubernetes cluster created through this provider can be placed on the same physical DCS Host. If that host fails, the entire Kubernetes control plane is unavailable until the host recovers.

To force control-plane virtual machines onto different physical hosts, configure a VM-VM anti-affinity rule on the DCS platform side. The DCS platform calls this feature 互斥虚拟机 (Mutually Exclusive Virtual Machines). The rule is configured under the DCS Cluster's compute resource scheduling settings, lists the control-plane virtual machines by name, and instructs the DCS scheduler to keep those virtual machines on different hosts.

A full operational runbook for this configuration is being prepared and will be published as a separate document. Until it is available, follow the DCS / FusionCompute platform documentation, or contact the DCS platform support team. Two prerequisites apply on the DCS side: the target cluster must have enough healthy hosts (typically at least one host per control-plane virtual machine), and the cluster must have DRS enabled if the rule should also be honoured during runtime rebalancing.

#Troubleshooting: VM stuck in creating

If a DCSMachine stays in Provisioning with status.cdRomFile set (so DCS API auth and ISO upload succeeded) but the DCS-side VM never leaves status=creating and never gets a hostName, the failure is almost always at the DCS placement layer, not in the controller.

Diagnostic ladder, in order:

1. Confirm it is a placement stall, not a slow boot. Query the VM and the matching deploy task on DCS:

   curl -k -sS "${DCS_BASE}/service/sites/${SITE}/vms?name=${VM_NAME}&limit=10" \
     -H "X-Auth-Token: ${TOKEN}" -H 'version: 8.1'
   curl -k -sS "${DCS_BASE}/service/sites/${SITE}/tasks?status=running&limit=50" \
     -H "X-Auth-Token: ${TOKEN}" -H 'version: 8.1'

   Symptoms of a placement stall: status=creating, hostName= (empty / unscheduled), disks array empty or unchanging, and the deploy task sitting at the same progress value with no errors for more than ~5 minutes.

2. Check datastore free space and host load using the snippets in Choosing a datastore and Host placement requirements. Datastore above 70 % used, or all hosts at high cpuMuxRatio, are the leading causes.

3. Rebalance load on the DCS side. If a single host is saturated while others have free capacity, ask the DCS platform team to rebalance the load (manual Migrate or DRS-driven rebalance) before retrying. The DCS Provider does not expose a per-template host pin, so this remediation is owned by the DCS platform team.

4. If the DCS-side deploy task is genuinely deadlocked (cancellable but unresponsive to ACP-driven DeleteVm), escalate to a DCS administrator with portal-admin credentials to cancel the task. The 7443 REST API does not expose a task-cancel endpoint for non-admin users, so the recovery path is owned by the DCS platform team, not by ACP.

5. Watch out for serialization side-effects. In environments with a global DeployVM lock, a single stuck task can cause every new deploy to stall at the same progress percentage. Resolving the original stuck task usually unblocks the queue.

A DCSMachine whose status conditions show InfrastructureReady=False reason=WaitingForInfrastructure for more than 10 minutes, together with the symptoms above, should be treated as a DCS-platform incident and not as a Cluster API or DCS Provider regression.

#Machine Templates

Machine templates define the virtual machine specifications (VM template, CPU, memory, disk, network) for cluster nodes. Each machine template has a Type that determines its usage:

• Control Plane: For control plane nodes
• Worker Node: For worker nodes

#Using the Web UI

#Prerequisites

• IP Pool has been created
• VM Template has been created in the DCS platform using MicroOS image
• ConfigMap YAML has been applied to the global cluster

VM Template and ConfigMap:

Each MicroOS release includes a ConfigMap YAML that maps VM templates to Kubernetes versions. Apply this YAML before creating machine templates:

apiVersion: v1
data:
  corednsTag: 1.12.4-v4.2.3
  etcdTag: v3.5.21-251117
  kubernetesVersion: v1.33.6
  vmImageVersion: MicroOS-5.5-v4.2.0
kind: ConfigMap
metadata:
  labels:
    cpaas.io/dcs-vm-template: microos5.5-4.2.0
    cpaas.io/distribution-version: v4.2.0
    cpaas.io/kubernetes-version: v1.33
  name: 420-dcs-vm-template
  namespace: cpaas-system

Important: The cpaas.io/dcs-vm-template label value must match the VM template name in the DCS platform.

#Creating a Machine Template

Navigation: Clusters → Virtual Machine → Machine Templates → Create Machine Template → Select Credential

Form Fields:

Disk Configuration:

The disk configuration varies by template type.

Control Plane Required Disks:

Worker Node Required Disks:

You may add additional disks, but must retain all mandatory disks listed above.

Disk Field Descriptions:

VM Template Selection Tip:

#Managing Machine Templates

Viewing Templates: Navigate to Clusters → Virtual Machine → Machine Templates to view all templates with their VM Template Name, Location, Specs, and IP Pool.

Updating Templates: Click Update to modify specifications. Note that the Name field cannot be changed after creation.

Deleting Templates: Click Delete to remove a template. Confirm the deletion in the dialog.

#Using YAML

Create a DCSMachineTemplate resource:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: <machine-template-name>
  namespace: cpaas-system
spec:
  template:
    spec:
      vmTemplateName: <vm-template-name>
      location:
        type: folder
        name: <folder-name>
      vmConfig:
        dvSwitchName: <dv-switch-name> # Optional
        portGroupName: <port-group-name> # Optional
        dcsMachineCpuSpec:
          quantity: <cpu-cores>
        dcsMachineMemorySpec: # MB
          quantity: <memory-mb>
        dcsMachineDiskSpec: # GB
        - quantity: 0
          datastoreClusterName: <datastore-cluster-name>
          systemVolume: true
        - quantity: 100
          datastoreClusterName: <datastore-cluster-name>
          path: /var/lib/kubelet
          format: xfs
        - quantity: 100
          datastoreClusterName: <datastore-cluster-name>
          path: /var/lib/containerd
          format: xfs
        - quantity: 10
          datastoreClusterName: <datastore-cluster-name>
          path: /var/lib/etcd
          format: xfs
      ipHostPoolRef:
        name: <iphostname-pool-name>

Parameter Descriptions:

*Required when parent object is specified

#Advanced: Multi-Cluster Deployment with Shared Storage

The default workflow assumes that the DCS VM Template (vmTemplateName) and the cloned virtual machines live in the same DCS Cluster. For most deployments, this assumption is correct and the spec.template.spec.location field is set to a VM Folder for organizational grouping only.

Deployments that span multiple DCS Clusters and want to clone from a single shared DCS VM Template into a different target DCS Cluster require the following conditions:

• The DCS Datastore that holds the DCS VM Template must be visible to the target DCS Cluster. This is the case for SAN, IPSAN, NAS, or FusionStorage datastores configured for cross-cluster access; it is not the case for datastores backed by host-local disk.
• The DCS-side network resources (distributed virtual switches and port groups) referenced by the vmConfig must also be visible to the target DCS Cluster.

When both conditions are met, the cloned virtual machine can be placed under the target DCS Cluster by setting spec.template.spec.location.type: cluster and spec.template.spec.location.name: <target-cluster-name> instead of the default type: folder form. If the datastore is not shared across DCS Clusters, the clone task fails at the DCS platform side. The recommended alternative is to prepare a separate DCS VM Template inside each target DCS Cluster, so each DCSMachineTemplate references a template that already lives in the cluster where the virtual machines will run.

Confirm cross-cluster datastore visibility with the DCS platform administrator before using this configuration.

#Resource Relationships

Infrastructure resources have the following dependency relationships:

Cloud Credential
    ↓
IP Pool
    (network settings + persistent disk declarations)
    ↓
Machine Template → references IP Pool
    ↓
Cluster Creation

Resource Reusability:

• One Cloud Credential can be used for multiple clusters
• Multiple IP Pools can be created for different network segments
• Multiple Machine Templates can be created for different node types and specifications

#Next Steps

After configuring infrastructure resources:

• Create a Cluster
• Manage Node Pools
• Upgrade Clusters