# mountOS Skill: Integrate over S3, WebHDFS, CSI, and the SDK

> Reach a mountOS volume's data plane and control plane from outside a filesystem mount: AWS S3, WebHDFS/Hadoop, Kubernetes CSI, the change-event feed, and the Ed25519-signed Admin SDK.

This skill assumes a volume already exists with an access-key pair. If you still need to
stand up the HUB, a region, a storage, or a volume, do that first (see
https://mountos.io/skills/provision.md) and come back. Every data-plane surface here is reached with
the **same** volume access key pair (`apiKey` / `apiSecret`, a 20-char id + 40-char secret)
returned by `POST /api/v1/volumes/:volumeId/api-keys/generate`. Pick a surface per workload;
they all front the same bytes and metadata.

| Surface | How it runs | Auth | Use when |
| --- | --- | --- | --- |
| S3 REST | `mountos gateway --gateway s3` | AWS SigV4-compatible, service `s3` | Any S3 SDK/tool needs buckets, listings, multipart, lifecycle |
| WebHDFS | `mountos gateway --gateway hdfs` | mountOS SigV4, service `hdfs` | Stock Hadoop tooling: Spark, Hive, Trino, distcp, Flink |
| Kubernetes CSI | `mountos kubernetes` (driver `csi.mountos.io`) | node-stage Secret | Volumes as pod PersistentVolumes |
| Change-event feed | `mountos event` (stdout or local REST) | same access key | Follow a volume's changes without walking the tree |
| Control plane | Admin API `/api/v1/*` | Ed25519 JWT (admin key) | Programmatic admin/automation |

Every one of these runs from the `mountos` binary. There is no separate gateway or CSI service
to deploy. The gateway serves general object volumes only; lake volumes are reached through
`mountos` plus a lake REST catalog, not covered here.

## Running the gateway

```sh
# gateway only, no filesystem mount
mountos gateway --gateway s3,hdfs -a "$MOUNTOS_ACCESS_KEY_ID" -s \
  --discovery-url https://hub.example.com

# or alongside a mount, from the same process
mountos mount -m /mnt/ws -a "$MOUNTOS_ACCESS_KEY_ID" -s \
  --discovery-url https://hub.example.com --gateway s3,hdfs
```

- `--gateway` takes a comma-separated protocol list (`s3`, `hdfs`, or `s3,hdfs`).
- `--gateway-port` accepts `0` (auto-assign per protocol, the default), one number for every
  protocol, or the per-protocol form `s3=9000,hdfs=9870`.
- The listener binds loopback (`127.0.0.1`). To serve other hosts, pass
  `--gateway-no-loopback` **with** `--gateway-cert` and `--gateway-key` (TLS required).
- The assigned endpoint URLs are written to a descriptor JSON under `~/.mountOS/gateway/`,
  with a copy in the system temp directory, so local tooling can discover the ports.
- Iceberg/lake volumes ignore `--gateway` and start their own REST catalog and lake S3
  listener on loopback instead.

## Block storage and S3

`blockserv` is the optional block-storage byte plane: a way to back volumes with your own
block devices instead of a cloud object store. A block storage is up to three active-active
block volumes that form a peer mesh across distinct clusters, reached over the block protocol
(not S3). It can proxy an external object store or be the storage itself on local block
devices. Volumes whose storage points at AWS S3, R2, Azure, or another S3-compatible store
reach that store without it. None of this is the S3 REST surface, which is the client gateway
above and authenticates with **AWS SigV4-compatible** signing using service name `s3` and the
volume access key. mountOS SigV4 (service `hdfs`) is WebHDFS-only.

Point any AWS S3 SDK at the gateway endpoint with the volume key pair. Region for the
signature is a placeholder unless your storage cares; use the storage's region.

```sh
# aws CLI against the gateway endpoint published by `mountos gateway --gateway s3`
aws --endpoint-url http://127.0.0.1:9000 \
    s3api list-objects-v2 --bucket workspace --max-keys 1000
```

```ts
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'

const s3 = new S3Client({
  endpoint: 'http://127.0.0.1:9000',
  region: 'us-east-1',
  forcePathStyle: true,
  credentials: { accessKeyId: apiKey, secretAccessKey: apiSecret },
})
await s3.send(new PutObjectCommand({ Bucket: 'workspace', Key: 'a/b.bin', Body: data }))
```

Limits that matter for SDK config:

- **Multipart**: minimum part size is **8 MiB** (every part except the last). Size your SDK's
  multipart threshold and part size at or above 8 MiB.
- **Listing**: `max-keys` caps at **1000** per page. Paginate with the continuation token.
- Use **path-style** addressing (`forcePathStyle: true` / virtual-host off); the bucket maps
  to the volume.
- TLS: pass `--gateway-cert` and `--gateway-key` to serve HTTPS, or terminate at a proxy in
  front. SigV4 signs the host, so the endpoint the SDK uses must match what the gateway sees.

## WebHDFS (the client gateway and hadoop-mountos)

`mountos gateway --gateway hdfs` accepts WebHDFS HTTP (`/webhdfs/v1`, `?op=` dispatch) from
stock Hadoop clients and translates to the `dataserv` protocol. Pin the port with
`--gateway-port hdfs=9870` to match the HDFS NameNode convention. Auth is **mountOS SigV4 with
service name `hdfs`**, signed with the same volume access key pair. Do not point an S3 SigV4
signer at it; the service name differs.

Two-phase WebHDFS operations (CREATE, APPEND, OPEN) emit a redirect, and the redirect host is
only trusted when `MOUNTOS_HDFS_TRUSTED_HOSTS` lists it. With that variable unset those
operations return 421. Set it to the public host (`host` or `host:port`, comma-separated for
several) before pointing Hadoop tooling at a non-loopback endpoint.

The supported path is the `hadoop-mountos` HCFS jar, which wraps Hadoop's `WebHdfsFileSystem`
and injects the SigV4 signing for you. Its Java SPI registers `io.mountos.hadoop.MountOSFileSystem`
for the `mountos` URI scheme, exposing `mountos://` to Spark, Hive, Trino, distcp, and Flink.

Maven coordinates `io.mountos:hadoop-mountos` (profiles: `hadoop3` against Hadoop 3.4.1, jar
classifier `hadoop3`; `hadoop2` against Hadoop 2.10.2, classifier `hadoop2`). Put the jar on
the classpath and configure `core-site.xml` (or `spark.hadoop.*` overrides):

```xml
<configuration>
  <property><name>fs.mountos.impl</name>
    <value>io.mountos.hadoop.MountOSFileSystem</value></property>
  <property><name>fs.AbstractFileSystem.mountos.impl</name>
    <value>io.mountos.hadoop.MountOSAbstractFileSystem</value></property>
  <property><name>fs.mountos.endpoint</name>
    <value>http://127.0.0.1:9870</value></property>
  <property><name>fs.mountos.access.key</name><value>AKMOS...</value></property>
  <property><name>fs.mountos.secret.key</name><value>...</value></property>
  <!-- optional: fs.mountos.fker = <fork-name> to select a named fork (illustrative value) -->
  <property><name>fs.mountos.fork</name><value>my-fork</value></property>
</configuration>
```

```sh
hadoop fs -ls mountos://workspace/data
hadoop distcp hdfs:///src mountos://workspace/dst
```

For Spark, pass the same keys as `--conf spark.hadoop.fs.mountos.endpoint=...` etc. Note the
gateway returns 501 for `GETFILECHECKSUM` unless `MOUNTOS_HDFS_FILE_CHECKSUM=true`, so distcp
skips checksum verification by default. That is expected, not an error.

## Kubernetes CSI (mountos kubernetes)

`mountos kubernetes` (alias `mountos csi`, Linux only) is the CSI node driver, registered as
**`csi.mountos.io`**. It mounts mountOS volumes into pods as PersistentVolumes (ReadWriteMany,
through FUSE) by launching one `mountos` mount subprocess per volume, which is what lets a
wedged mount be stopped, recovered by pid, or hot upgraded on its own. It runs as the
`mountos-csi-node` DaemonSet in `mountos-system`. The `CSIDriver` object declares
`attachRequired: false`. No image is published; build from `deploy/csi/Dockerfile`, whose
entrypoint is `mountos kubernetes`.

A PersistentVolume references the driver, points `discoveryURL` at the **appserv HTTPS
discovery URL** (the `<HUB>` domain), carries the volume access-key id as `volumeHandle`, and
pulls the secret pair through `nodeStageSecretRef`. The driver launches the `mountos` FUSE
client, which resolves `dataserv` through appserv discovery, so the attribute is the appserv
HTTPS URL, not a raw `dataserv` `host:port`. The driver routes `discoveryURL` to the client's
`MOUNTOS_DISCOVERY_URL`; any other attribute is passed through as a client flag.

```yaml
apiVersion: v1
kind: PersistentVolume
metadata: { name: workspace-pv }
spec:
  accessModes: [ReadWriteMany]
  capacity: { storage: 100Gi }
  csi:
    driver: csi.mountos.io
    volumeHandle: "<access-key-id>"           # 20-char access key ID
    volumeAttributes:
      discoveryURL: "https://<HUB>"           # appserv HTTPS discovery URL
      disk-cache-dir: "/var/cache/mountos"
    nodeStageSecretRef:
      name: workspace-keys
      namespace: mountos-system
---
apiVersion: v1
kind: Secret
metadata: { name: workspace-keys, namespace: mountos-system }
stringData:
  accessKeyID: "<access-key-id>"              # 20 chars
  secretAccessKey: "<secret-access-key>"      # 40 chars
```

- The node-stage Secret keys are exactly `accessKeyID` and `secretAccessKey`.
- Optional `volumeAttributes`: `xattr`, `acl`, `xattr-security`, `mount-opts`, `io-uring`,
  `mem-limit`, `buffer-size`, `disk-cache-size`.
- Bind a static PV by matching the PVC's `volumeName`. The DaemonSet cache volume is an
  `emptyDir` by default (lost on pod restart); switch to a hostPath under `/var/cache/mountos`
  for a persistent disk cache.
- Driver health check is HTTP `/healthz` on port **9810**.

## Change-event feed (mountos event)

A general volume with `eventLogRetentionPeriod` > 0 (volume knob, 0-30 days, default 0 = off)
keeps an ordered feed of changes: `create`, `delete`, `modify`, `rename`. Per fork,
monotonic `seq`, full paths, near real time (capture flushes ~3 s). Same access-key pair and
HUB discovery as a mount; no mount required.

```sh
mountos event -a "$MOUNTOS_ACCESS_KEY_ID" -s --since 7d --follow      # stream / tail to stdout
mountos event -a "$MOUNTOS_ACCESS_KEY_ID" -s --path-prefix /docs --json
mountos event -a "$MOUNTOS_ACCESS_KEY_ID" -s --http --http-port 9876  # local REST API (daemonizes)
```

- Stream flags: `--since` (`7d`, `2h`, `"2025-01-01 14:30"`), `--start-seq`, `--path-prefix`
  (server-side subtree filter), `--follow` (5 s poll; starts at the live end unless a floor is
  given), `--json`, `--fork-name` (global flag, empty = main).
- HTTP mode ignores stream flags and serves the fork the process was started against
  (`--fork-name`). `POST /v1/auth` takes
  `{ access_key_id, timestamp, nonce, signature: hex(HMAC-SHA256(secret, key:timestamp:nonce)) }`
  (5-minute drift window, single-use nonce) and returns a 1-hour bearer.
  `GET /v1/events?since=&start_seq=&path_prefix=&limit=&seek_to_end=` returns
  `{ events, continuation_token, has_more }`; pass only `continuation_token` on follow-ups.
  Limit values under 100 are served as 100. Loopback by default; `--http-no-loopback`
  requires `--http-cert` + `--http-key`.
- Iceberg/lake volumes are rejected (the feed is captured from the metadata arena, which they
  bypass). Full contract, event shape, and consumer semantics:
  https://mountos.io/ai/topics/change-events.md.

## The Admin SDK as an integration surface

The control plane itself is an integration surface: drive `/api/v1/*` programmatically instead
of clicking the dashboard. The Admin SDK is the open, public connector to a running deployment's
HUB (https://github.com/mountos-io/mountos-admin-sdk). Integrate several ways: the TypeScript, Go, or Rust
package, the REST API directly (reference in `api.md`), or a client generated in
any language from the API spec `api.yaml`. The repo also ships `SKILL.md`, a drop-in agent
skill that mirrors this section. Use the SDK rather than hand-signing JWTs. The full
reference (auth model, response envelope, every namespace) is the admin-sdk topic:
https://mountos.io/ai/topics/admin-sdk.md.

- TypeScript: `@mountos-io/admin-sdk`
- Go: `github.com/mountos-io/mountos-admin-sdk/go`
- Rust: `mountos-admin-sdk` (crate)

The TypeScript package exposes `createServerClient({ baseUrl, privateKey })`; Go uses
`sdk.NewClient(sdk.Config{ BaseURL, PrivateKey })`; the Rust crate uses
`Client::new(Config { base_url, private_key, ..Default::default() })`. `baseUrl` is the HUB domain;
`privateKey` is the admin Ed25519 seed (base64). The SDK **mints and caches an Ed25519
JWT** (`sub=mountos:provider`, `aud=mountos/appserv`, 1-hour TTL, refreshed 5 minutes
before expiry) and sends the cached token on every request. Never ship the admin private
key to a browser.

```ts
import { createServerClient, MountOSError } from '@mountos-io/admin-sdk'

const client = createServerClient({
  baseUrl: 'https://hub.example.com',
  privateKey: process.env.MOUNTOS_PRIVATE_KEY,   // Ed25519 seed, base64
})
```

Verified SDK resource namespaces: `accounts`, `users`, `regions`, `regionClusters`,
`storages`, `volumes` (including `generateAPIKeys` / `revokeAPIKey`), `volumeForkTrees`,
`volumeForkEntries`, `volumeForkSearches`, `auditLogs`, `regionAuditLogs`, `serviceNodes`,
`nodes`, `clientSessions`, `alerts`, `regionAlerts`, `discover`, `dashboard`, `license`,
`vault`. The full per-namespace method list is in the admin-sdk topic
(https://mountos.io/ai/topics/admin-sdk.md). Representative verified calls:

```ts
const { items, pagination } = await client.accounts.list({ page: 1, limit: 10 })
const region  = await client.regions.get(regionId)
const nodes   = await client.serviceNodes.list(regionId)   // ServiceNode[]
const { items: logs, nextCursor } = await client.auditLogs.list({ accountId, limit: 5 })
await client.vault.resync()
```

**Response envelope** (`StandardResponse`). The SDK unwraps it; raw REST callers see:

```
{ "status": "success"|"failure", "message": string, "data"?: object, "errorCode"?: int }
```

**Pagination** comes in two shapes, both nested in `data`:

- Page-based (lists like accounts, users, storages, volumes):
  `{ items, pagination: { page, limit, total, totalPages } }`.
- Cursor-based (append-only feeds like audit logs, fork trees, searches):
  `{ items, nextCursor: int64|null }`. Loop until `nextCursor` is null.

**Errors** surface as `MountOSError` with a `.status` and a numeric `errorCode` (AppServ codes
are `1XXXX`, for example `10201 VALIDATION_FAILED`, `10901 SERVICE_UNAVAILABLE`). Catch it to
branch on failure.

```ts
try {
  await client.volumes.updateQuota(volumeId, { quotaLimit: 10 * 1024 ** 3 })
} catch (err) {
  if (err instanceof MountOSError) console.error(err.errorCode, err.message, err.status)
}
```

When a method is not exposed on the SDK, call the REST path directly with the same JWT; the
full surface is in the api reference at the admin-sdk repo (`api.md`). Do not invent method
names. For a fresh language binding, generate from `api.yaml` or contact support@mountos.io.

## Read next

- Skill index: https://mountos.io/skill.md
- Provision the volume and keys you integrate against, and bring up the cluster-scoped gateway
  services (provision.md section 5): https://mountos.io/skills/provision.md
- Day-2 operations, scaling, and cluster moves for the gateway pools: https://mountos.io/skills/operate.md
- Volume data features (forks, time travel, versioning): https://mountos.io/skills/volumes.md
- Full topics: https://mountos.io/ai/topics/components.md, https://mountos.io/ai/topics/architecture.md,
  https://mountos.io/ai/topics/configuration.md, https://mountos.io/ai/topics/deployment.md,
  https://mountos.io/ai/topics/change-events.md
- Admin SDK, the open connector to a deployment's HUB: https://github.com/mountos-io/mountos-admin-sdk
  (`SKILL.md` agent skill, `api.md` REST reference, `api.yaml` API spec for codegen) and the
  full topic at https://mountos.io/ai/topics/admin-sdk.md
- Admin dashboard (open source): https://github.com/mountos-io/mountos-admin-client
- Downloads and install: https://download.mountos.sh
- Support: support@mountos.io