# mountOS Skill: Deploy on a cloud (provision the substrate and bring up the HUB)

> Stand up the cloud substrate and a running, verified HUB using the mountos-io/deployment package.
> Stop when the HUB answers the Admin API. Tenant setup is the provision skill.

This is the layer below provisioning. You create the cloud infrastructure (network, KMS/key-store,
database, secret store wiring, the appserv fleet behind load balancers, DNS) and boot the HUB, then
hand off to the provision skill (https://mountos.io/skills/provision.md), which assumes a running HUB. Use
the `mountos-io/deployment` package. Do not hand-roll Terraform or systemd.

## What you are driving

The `mountos-io/deployment` package is the single source of the deploy, version-pinned by its
`release.yaml`. Get the version that matches the binaries you will run, two equivalent ways.

- Clone the repo at the release tag.
- Run `curl -fsSL https://n.sh | bash -s -- --pkg deploy --version <X.Y.Z>` to drop the package locally.

One version pins the package, the appserv binary, the admin SDK, and the helpers together. Never
mix versions. Each cloud is self-contained under `clouds/<cloud>/terraform` and selected via
`CLOUD=<cloud> make ...` (default `aws`). Three clouds are available: `aws`, `gcp`, `azure` — the
steps below are cloud-agnostic (same `make` targets, same bootstrap scripts) unless a step calls
out a specific cloud.

**Verification levels differ by cloud.** AWS is Floci-plan-verified — a local AWS API emulator lets
`terraform plan` run against the full graph without a real account. GCP and Azure have no
equivalent local emulator, so they are validated at `terraform validate` + tflint (schema/syntax
level) only — graph-correct but PLAN-UNPROVEN and RUNTIME-UNPROVEN until exercised against a real
project/subscription. Flag this to the operator before an unattended GCP/Azure apply.

**Secret store: this package NEVER installs or launches HashiCorp Vault** (HashiCorp's BSL defines
"embedded" as packaging a product so the Licensed Work must be downloaded for it to operate — the
vendor posture is zero reliance on the license's compete-qualifier). Two supported providers per
cloud, selected by `vault_provider`/`region_vault_provider`:
- **Cloud-native (default, RECOMMENDED)** — `aws` | `gcp` | `azure`: the platform's own secret
  store (Secrets Manager / Secret Manager / Key Vault). Platform-managed HA, instance-identity
  auth, no AppRole/token/CA machinery; `make bootstrap`/`make region-bootstrap` seed the
  `mountos` secrets with your operator CLI credentials. The permission matrix is enforced by
  IAM name-scoping (appserv can never read `api-master`; region services can never read the
  appserv secret).
- **`hashicorp` (byo details only)** — the operator brings a Vault (HCP Vault Dedicated or their
  own HA cluster) and supplies `vault_addr` (https required; `vault_ca_pem` when it serves a
  private CA) plus a short-lived bootstrap token to the seed scripts, which create the KVv2
  mount, policies, AppRole, and initial values via plain API calls. Nothing is installed.

**BYO PostgreSQL is the recommended production DB path** (`admin_db_mode`/`region_db_mode = byo`):
bring a managed HA PostgreSQL you already operate. The DSN is NEVER a terraform value — set
`ADMIN_DB_URL`/`REGION_DB_URL` only in the `make bootstrap`/`make region-bootstrap` environment
(the seeds fail closed unless a postgresql DSN carries `sslmode=verify-full` or `sslmode=require`,
and a mysql DSN carries `tls=`/`ssl-mode=`). `provision-rds` remains as the simple mode.

**Provisioned-DB password handoff (no DSN outputs exist on any cloud)**: terraform outputs the DB
host plus a cloud-native password reference, and the seed scripts fetch the password and build the
DSN themselves — AWS `admin_db_secret_arn`/`region_db_secret_arn` (Secrets Manager), GCP
`admin_db_password_secret`/`region_db_password_secret` (Secret Manager via gcloud), Azure
`admin_db_secret_id`/`region_db_secret_id` (Key Vault via az).

**Parity gap on GCP/Azure**: neither Cloud SQL nor Postgres Flexible Server has an equivalent to
AWS RDS's `manage_master_user_password` (a platform-owned, rotated, never-a-Terraform-value
password). On GCP/Azure the provisioned DB master password is a Terraform value
(`random_password`, stored in Secret Manager/Key Vault) — present in tfstate, unlike AWS. byo mode
avoids this entirely.

Azure also has no zero-touch DNS-validated managed-cert primitive like AWS ACM or Google-managed
certs — `hub_certificate_secret_id` must point at an operator-supplied certificate in Key Vault.

## Rules (do not violate)

- Non-destructive only. There is no destroy or down target. `apply` and `bootstrap` converge
  forward and are safe to re-run. Never tear down a production hub. Decommissioning is a separate
  human decision, not part of this flow.
- No license file. The HUB runs on the built-in free tier (currently 10 TiB) until a license is loaded
  through the Admin SDK or admin client. Do not set MOUNTOS_LICENSE_PATH.
- Secrets. `make bootstrap` generates fresh keys into `secrets.local.json`. The `admin_private` key
  is the operator root credential. Keep it offline and never commit or paste it. hashicorp store
  only: deliver the Vault AppRole secret_id by response-wrapping, not as a plaintext variable.
- Never install or launch HashiCorp Vault from this flow, and never suggest doing so.
- The client owns their AWS account, Terraform remote state, keys, and DNS.

## Steps (the order is load-bearing)

1. Answers. Run `make interview` to scaffold `answers.env` and fill it (mode, AWS region,
   hub_domain, secret store provider, admin DB mode, appserv count). Fill
   `clouds/aws/terraform/terraform.tfvars` from the sample and set a remote state backend from
   `backend.tf.sample`.
2. Provision. Run `make plan` to preview, then `make apply`. This creates the VPC, the hub KMS CMK,
   the admin database (RDS or BYO), the secret-store IAM wiring, the appserv ASG, the ALB on 443
   and NLB on 9443, and DNS. No secret store is launched: cloud-native mode uses the platform
   store directly; hashicorp mode points at your byo Vault (`vault_addr`).
3. hashicorp store only: have your byo Vault reachable and a short-lived bootstrap token issued
   (HCP or your own cluster — its operation is entirely yours).
4. Seed. Always AFTER `make apply` (on GCP, Terraform owns the empty `mountos__*` secret
   containers — seeding first makes the later apply fail with AlreadyExists). Cloud-native mode:
   your operator CLI credentials are enough. hashicorp mode: export
   `VAULT_ADDR` and a short-lived `VAULT_TOKEN`. For the admin DB: in provision-rds mode AWS
   manages the master password in Secrets Manager, so export `ADMIN_DB_HOST` and
   `ADMIN_DB_SECRET_ARN` from the Terraform output instead of a DSN — `make bootstrap` fetches
   the password itself and never puts it in tfstate. In byo mode (recommended), export
   `ADMIN_DB_URL` directly. Then run `make bootstrap`. It generates keys and seeds the store with
   the service config and service-verifiers; in hashicorp mode it also creates the appserv AppRole
   (api-master is NOT seeded here — it is region-only, provisioned independently per region by
   `make region-bootstrap`). hashicorp mode only: put the role_id and a wrapped secret_id into
   tfvars and re-run `make apply` so instances can read the Vault.
5. The fleet self-installs. Each appserv instance installs the binary from n.sh, runs
   `appserv db install` (idempotent and migration-locked), and starts under systemd behind the LBs.
6. Verify. Run `make verify`. It checks HTTP up with auth enforced (401), the admin-JWT smoke (200),
   and — hashicorp store only, when VAULT_ADDR is exported — Vault unsealed. A green verify is a
   usable HUB.

## Done and hand-off

When verify is green the HUB answers `https://<hub_domain>/api/v1/*`. Continue with the provision
skill (https://mountos.io/skills/provision.md) to create the account, region, and region services. To add
capacity beyond the free tier, load a license through the Admin SDK with no redeploy.

## Add a region (after the hub is up)

A region runs its own database, its own secret-store scope, and a 3-node dataserv quorum with
co-located gcserv. It registers with the hub over the internal RPC, and its `uno` cluster flips ready once
dataserv registers. The order matters.

1. Create the region record on the hub first. Use the provision skill (https://mountos.io/skills/provision.md)
   to create the account and the region, then read back the region cluster id (the uno UUID).
2. Set the region values in tfvars. Fill region_cluster_id from step 1, plus dataserv_count,
   arena_size, and the region database and secret-store choices (region_vault_provider defaults to
   the cloud-native store; on aws/gcp the hub and region then share one physical namespace,
   isolated by IAM). region_vpc_mode defaults to shared (region resources live in the hub's VPC —
   simplest); set it to dedicated for a separate VPC in the same AWS account and region, peered to
   the hub VPC, when you want network-isolation between hub and region. Run `make apply` to
   provision the region RDS, the secret-store IAM wiring, and the dataserv fleet (plus the region
   VPC and peering connection, in dedicated mode).
3. hashicorp store only: have your byo region Vault reachable with a bootstrap token issued.
4. Seed and fan out. Cloud-native mode: operator CLI credentials are enough. hashicorp mode:
   export REGION_VAULT_ADDR and REGION_VAULT_TOKEN, HUB_VAULT_ADDR and HUB_VAULT_TOKEN. For the
   region DB: in provision-rds mode export REGION_DB_HOST and REGION_DB_SECRET_ARN from the
   Terraform output (AWS manages the master password in Secrets Manager; the script fetches it and
   never puts it in tfstate); in byo mode (recommended) export REGION_DB_URL directly. Then run
   `make region-bootstrap`. It generates the dataserv, gcserv, and blockserv keys (unconditional,
   every region gets all three service keypairs seeded regardless of whether block_enable will
   actually run blockserv), seeds the region store, and fans every service-verifier
   both ways so the hub trusts the region services and the region trusts the hub. hashicorp mode
   only: put the printed region_vault_role_id and a wrapped secret_id in tfvars and re-run
   `make apply` so the dataserv fleet can read its Vault (blockserv reuses this same region
   AppRole).
5. dataserv registers over the internal RPC and the region uno cluster flips ready. Confirm in the admin client
   or the SDK cluster list. Create volumes with the volumes skill (https://mountos.io/skills/volumes.md).

## Add blockserv (optional, local-block backends)

blockserv is the block data plane. It absorbs the byte traffic for a block storage's volume set, in a
group of up to three active-active members across distinct clusters. Skip it for pure object storage.
Its secret-store keys are already seeded by `make region-bootstrap` above — no separate seed step.

1. Provision a block storage on the hub with the Admin SDK. It yields up to three block volume member
   UUIDs across distinct clusters.
2. Set block_enable true and list the members in tfvars, one entry per member with its block_volume_id
   and an az_index, then run `make apply`. Each member runs on a network-enhanced instance with its own
   XFS cache disk, registers over the internal RPC, and peers with the others on 9101. Each member gets a stable
   Elastic IP and advertises it directly (clients reach blockserv by IP, no proxy).

## S3 and WebHDFS (no fleet to deploy)

Both protocols run inside the `mountos` client, bound to one volume, on the host that wants them:

```sh
mountos gateway --gateway s3,hdfs -a <access-key-id> -s
```

There is no gateway ASG, no gateway Vault entry, and nothing to size in tfvars. The client
authenticates with the volume's access key pair and binds loopback unless
`--gateway-no-loopback` is passed with a TLS cert and key.

## Day 2

- Upgrade. Bump `MOUNTOS_VERSION` in the answers and tfvars, then run `make apply`. The ASG rolls the
  fleet through an instance refresh. No data is touched.
- Re-run `make apply`, `make bootstrap`, or `make region-bootstrap` at any time. All are idempotent.