# Administration overview (/docs/administration)
**For:** Administrators
Admins keep the workspace healthy: confirm new instruments, watch the fleet, issue tokens, manage who is an admin, and roll out watcher releases. Personal and workspace notification settings are under [Configure notifications](/docs/configure-notifications).
Roles and scopes are defined in [Security and permissions](/docs/security). Day-to-day operator setup is in [Set up an instrument](/docs/set-up-an-instrument).
## What you manage [#what-you-manage]
# Architecture (/docs/architecture)
**For:** Engineers and operators
Data Hub has four components that coordinate through S3, a REST API, and a shared
Python library. This page is the conceptual map; the deployment mechanics live in
the [first-time deployment guide](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md)
in the data-hub developer docs.
## System overview [#system-overview]
## Components [#components]
| Component | Package | Description |
| -------------- | ------------------ | -------------------------------------------------------------------------------------------- |
| Web app | `data-hub-web` | Next.js web application, REST API, and MCP server. Deployed on Vercel. |
| Lambda | `data-hub-lambda` | AWS Lambda function triggered by S3 uploads. Runs instrument-specific processing. |
| Watcher | `data-hub-watcher` | CLI on lab instrument PCs. Detects new files, uploads them to S3, reports status to the API. |
| Shared library | `data-hub-shared` | Shared Python library: S3 utilities, instrument enums, test infrastructure. |
## Data flow [#data-flow]
### Automatic upload (auto mode) [#automatic-upload-auto-mode]
1. A lab instrument writes output files to a watched directory.
2. The **watcher** detects new stable files via filesystem events.
3. It groups files into runs and reports each run to the **API**.
4. It uploads raw files to **S3** at the key `{instrument_id}/{run_id}/{filename}`.
5. The S3 upload triggers the **Lambda**.
6. Lambda downloads the file and dispatches to the instrument’s processor for preprocessing.
7. Lambda creates/updates the run and files via the **API**, which sends a **Slack** notification once per new run.
8. Users view the run in the **web dashboard**.
### Manual upload (manual mode) [#manual-upload-manual-mode]
Steps 1–3 are the same, but the watcher doesn’t upload immediately:
4. The server adds files to an upload queue.
5. On each heartbeat tick the watcher polls the queue and uploads requested files.
6. Steps 5–8 from auto mode follow.
## Key design decisions [#key-design-decisions]
* **S3 is the integration boundary.** The watcher and Lambda never talk directly; S3 is the durable hand-off: the watcher writes, Lambda reads.
* **API-driven coordination.** The watcher registers, syncs its YAML config, and heartbeats, which is what powers dashboard health and the upload queue.
* **Pre-signed URLs from the API.** The web app mints pre-signed S3 upload/download URLs so bytes move directly to/from S3 without routing through the API. On Vercel the app assumes an AWS Identity and Access Management (IAM) role via OpenID Connect (OIDC), with no long-lived AWS keys.
* **Lambda-built run archives.** “Download all” delegates to the Lambda, which zips files into a separate archives bucket via S3 multipart upload; the web app then 302s the browser to a short-lived presigned URL. Bytes never traverse Vercel.
* **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so Lambda and the watcher stay consistent.
* **MCP for AI access.** The web app serves an MCP endpoint at `/mcp/v1` exposing tools, resources, and prompts to AI clients. Clients authenticate with OAuth against the same Better Auth instance that signs people in to the dashboard, so an agent acts as the person who approved it. See [MCP overview](/docs/mcp).
## Branch and environment strategy [#branch-and-environment-strategy]
| Branch | Purpose |
| ------------ | ----------------------------------------------- |
| `staging` | Pre-production. Pull requests merge here first. |
| `production` | Live. Changes are promoted from `staging`. |
Feature branches target `staging` via pull requests (PRs). Continuous integration
(CI) runs on every PR and on merges to both branches; merges to `staging` and
`production` deploy each component to its environment automatically. Staging and
production are independent deployments with separate databases.
# Browse and analyze runs (/docs/browse-runs)
**For:** Everyone
After instruments are uploading, the web app is where you find runs and act on them: inspect results, claim work, download files, and leave lab notes. The sections below follow the order you’ll use them in.
## Orient yourself in the app [#find-a-run]
The sidebar and the home page are the two starting points for daily work.
**Home** lists recent runs across instruments, most recent first, with counts for today and this week at the top. Filter by date range, who ran it, and instrument-specific metadata when it’s available, then page through results.
**Instruments** in the sidebar opens each instrument’s page. That page’s runs table adds column filters that match the instrument type (for example wavelengths on a plate reader).
**My runs** lists runs you’ve claimed, so you can return to your own work without re-filtering the full dashboard.
### Narrow a long run list [#narrow-a-long-run-list]
Home and the instrument pages share the same four controls:
* **Search runs**: filters the list by run name as you type
* **Date range**: presets from Today through Last 4 weeks, plus a custom range
* **Add filter**: filter by run status, or turn on **Show deleted runs**
* **Column headers**: sort by size, by who ran it, or by any instrument-specific column
### What each run status means [#run-status]
A run’s status tells you how far its files have got from the instrument to the dashboard:
| Status | Meaning |
| -------------- | ---------------------------------------------------------------------------------------------- |
| Pending upload | The watcher found the files but hasn’t sent them yet. |
| Uploaded | The files are in storage. Processing hasn’t started, or this instrument type has no processor. |
| Processing | The server is reading the files, extracting metadata, and building reports. |
| Completed | Processing finished. |
| Failed | Processing stopped on an error. |
| Empty | The run has no files. |
## Search the workspace [#search]
Press **⌘ K** (macOS) or **Ctrl K** (Windows/Linux) to open global search from any page. Results are grouped into runs, files, instruments, users, and comments. That means you can also look up a colleague’s work or the text of a lab note. A file hit opens the parent run with the files table already filtered to that file.
## Open a run and inspect results [#open-run-detail]
Each run has a details page at `/instruments/your_instrument_id/runs/your_run_id`. The arrows beside the run name step you to the next or previous run without going back to the list.
What you see depends on the instrument type and whether a processor ran:
* Metadata badges for extracted fields
* Processed reports such as tables, images, or PDFs, including interactive plate maps for plate readers
* A files table of everything in the run, labeled Raw or Processed, with each file’s size and status
Reports with many images, PDFs, or spectra show one item at a time. Search by filename to jump anywhere in the run, or use the arrows to step through it. The counter beside the arrows shows your position out of the total.
Per-type inputs and outputs are in [Instrument data preprocessing](/docs/instrument-preprocessing).
## Claim who ran it [#claim-who-ran-it]
Use **Claim** on the run details page to attribute yourself. You can claim or unclaim your own user only, so nobody else can attribute a run to you. Claims show on the run and appear in “ran by” filters on Home and elsewhere.
## Download files [#download-files]
From the files table you can:
* Download a single file, which opens a temporary download link
* Choose **Download all** to build a zip of active files in the run, with any active file filters applied
Large zips build in the background. When one isn’t ready yet, Data Hub tells you how long to wait before trying again.
## Act on several runs at once [#bulk-actions]
Select the checkbox on any run row to open an action bar at the bottom of the list:
* **I ran these**: claim every selected run in one step
* **Remove my attribution**: unclaim them
* **Download**: download the files from the selected runs
* **Reprocess**: run server-side processing again for all of them
* **Delete**: mark the selected runs deleted
The checkbox in the header row selects every run on the page, and **Clear** drops the selection.
## Comment on a run [#comment]
The comments thread on run details is for lab notes and follow-ups. Comment activity can notify people who claimed the run or already commented. Set preferences under **Settings → Notifications**; see [Configure notifications](/docs/configure-notifications).
## Reprocess or delete a run [#reprocess-and-delete]
After a failed process, or after an engineer fixes the code that reads a file type, use **Reprocess** on a file or the whole run (where available) to run server-side processing again. The row menu in a run list reprocesses a whole run without opening it.
**Delete** marks the run deleted without removing stored files. Admins and operators can restore it from the same page when that control is shown. To find a deleted run again, turn on **Show deleted runs** under **Add filter**.
For instruments in [manual upload mode](/docs/concepts#upload-mode), detected files may also show upload or dismiss actions while the watcher waits for approval. See [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
## Use the same data outside the UI [#programmatic-access]
You can also list and fetch the same runs and files over the [REST API](/docs/api) or the [Model Context Protocol (MCP) endpoint](/docs/mcp). REST needs a [personal access token](/docs/concepts#personal-access-token); scopes and creation are covered in [Issue and revoke tokens](/docs/manage-tokens). MCP clients sign in through the browser instead.
# Watcher CLI (/docs/cli-reference)
**For:** Lab operators
Flags and subcommands for `data-hub-watcher`, generated from the Click CLI in the watcher package. For task walkthroughs see [Install and update the watcher](/docs/install-the-watcher), [Run as a Windows service](/docs/windows-service), and [Roll out watcher releases](/docs/watcher-releases).
## Command reference [#command-reference]
The catalog below lists every command, argument, and option from the installed CLI help text.
## Global flags
These options apply to every `data-hub-watcher` command (watcher 0.5.0).
**Flags**
- `--config PATH` `path` env: `DATA_HUB_CONFIG_PATH` — Override config file path.
- `--verbose` `boolean` — Enable debug logging.
- `--version` `boolean` — Show the version and exit.
## `data-hub-watcher config`
View and manage the watcher config file.
### `data-hub-watcher config edit`
Re-prompt each config field with current values as defaults.
### `data-hub-watcher config open`
Open the config file in your editor, then re-validate.
**Flags**
- `--editor` `text` — Editor command (e.g. --editor code).
### `data-hub-watcher config path`
Print the resolved config file path.
### `data-hub-watcher config set-environment`
Switch the watcher to a different API environment.
**Arguments**
- `ENVIRONMENT` (required) `choice (staging | production | preview)`
**Flags**
- `--api-base-url` `text` — API base URL for the target environment. Required the first time you switch to an environment; reused from config on later switches.
- `--api-key` `text` — API key for the target environment.
- `--show-key` `boolean` — Echo the API key as typed (useful on Windows terminals).
- `--no-register` `boolean` — Fail instead of registering a new watcher if none is stored for the target env.
### `data-hub-watcher config show`
Pretty-print the current config.
### `data-hub-watcher config validate`
Validate the config file (offline, no network calls).
## `data-hub-watcher init`
Interactive setup wizard + API registration.
**Flags**
- `--show-key` `boolean` — Echo the API key as it is typed/pasted. Useful on Windows terminals where hidden input is unreliable for paste.
## `data-hub-watcher self-update`
Check the server for a newer watcher release and upgrade in place. Designed for unattended use — schedule via Windows Task Scheduler (e.g. weekly) so lab PCs converge on the latest published version without operator intervention.
**Flags**
- `--check` `boolean` — Print the server-reported target version without performing the upgrade.
- `--force` `boolean` — Run the upgrade subprocess even if the local version already matches the target.
## `data-hub-watcher service`
Windows service management.
### `data-hub-watcher service install`
Install the watcher as a Windows service.
**Flags**
- `--env-path FILE` `file` — Path to the .env file. Defaults to ~/.data-hub/.env..
### `data-hub-watcher service reinstall`
Stop, uninstall, install, and start the watcher Windows service. Useful after upgrading the watcher wheel from an Administrator shell so the SCM picks up the new ``data_hub_watcher.service`` entrypoint without needing four separate ``service`` invocations.
**Flags**
- `--env-path FILE` `file` — Path to the .env file. Defaults to ~/.data-hub/.env..
### `data-hub-watcher service start`
Start the watcher Windows service.
### `data-hub-watcher service status`
Show the watcher Windows service status.
### `data-hub-watcher service stop`
Stop the watcher Windows service.
### `data-hub-watcher service uninstall`
Uninstall the watcher Windows service.
## `data-hub-watcher upload`
Upload files to Data Hub (manual trigger). One-shot mode: pass both --file and --run-id to upload a single file. Queue mode: omit both to process the server-side upload queue.
**Flags**
- `--file PATH` `path` — Specific file to upload.
- `--run-id` `text` — Associate upload with a specific run.
- `--dry-run` `boolean` — Log what would be uploaded without uploading.
## `data-hub-watcher watch`
Start watching for new files.
**Flags**
- `--dry-run` `boolean` — Validate and log but don't start monitoring.
## Interactive setup notes [#interactive-setup-notes]
`init` and `config edit` prompt for values in the terminal. Those prompts are not CLI flags, so they do not appear in the catalog.
Keys must start with `dhub_`. The watcher strips zero-width and non-breaking-space characters that some Windows clipboards inject. See [Install and update the watcher → Configure](/docs/install-the-watcher#configure).
## Upload examples [#upload-examples]
One-shot upload outside the watch loop:
```sh
data-hub-watcher upload --file /path/to/file.csv --run-id RUN001
data-hub-watcher upload
data-hub-watcher upload --dry-run
```
Pass both `--file` and `--run-id`, or omit both to process the server-side queue in manual mode.
## Switching environments [#switching-environments]
`config set-environment` (and `config edit` when the environment changes) can take `--api-base-url`, `--api-key`, `--show-key`, and `--no-register`. Preview switches require `--api-base-url` the first time. See [Watcher configuration → Switching environments](/docs/configuration-reference#switching-environments).
## Windows service notes [#windows-service-notes]
`service` commands require Windows and the `windows-service` install extra. Use `--env-path` on `service install` and `service reinstall` when the service should load a non-default `.env` file. Full details are in [Run as a Windows service](/docs/windows-service).
## Self-update examples [#self-update-examples]
```sh
data-hub-watcher self-update
data-hub-watcher self-update --check
data-hub-watcher self-update --force
```
Upgrade behavior depends on how you installed the watcher. See [Install and update the watcher → Manual update](/docs/install-the-watcher#manual-update-self-update).
# Concepts (/docs/concepts)
**For:** Everyone
Short definitions of the vocabulary the rest of these docs use. If you’ve arrived mid-page and a term is unfamiliar, it’s defined here. Terms are grouped by what they describe: the equipment, the data, the rules that turn files into runs, and who can do what.
## Equipment and fleet [#equipment-and-fleet]
### Instrument [#instrument]
A piece of lab equipment registered in Data Hub, identified by a kebab-case **instrument ID** (e.g., `akta-fplc`, `bio-rad-cfx96`). The ID is permanent, because it becomes the storage key prefix and the identifier used across the system. A human-readable display name sits alongside it.
An instrument has three states. The dashboard groups instruments into a tab per state and shows connection health separately:
| State | Shown as | Meaning |
| ---------- | --------------------------------------------- | ---------------------------------------------------------------------------- |
| `pending` | **Pending** tab | Registered but not yet confirmed by an admin. A watcher can’t upload for it. |
| `active` | **Active** tab, with Online or Offline health | Confirmed and accepting uploads. |
| `inactive` | **Retired** tab | Deactivated. No longer accepting uploads, but its runs stay browsable. |
See [Set up an instrument](/docs/set-up-an-instrument), [Manage instruments](/docs/manage-instruments), and [Retire an instrument](/docs/retire-an-instrument).
### Instrument type [#instrument-type]
Separate from the instrument ID. The type tells Data Hub which [preprocessor](#preprocessor) to run on files from this instrument, so it controls what metadata, badges, and reports you get.
Set it from the instrument page: open the three-dot menu, choose **Edit**, and pick a supported type. An instrument with no matching type still stores and serves raw files. It gets no extracted metadata and no processed artifacts. See [Instrument data preprocessing](/docs/instrument-preprocessing).
### Watcher [#watcher]
The `data-hub-watcher` command-line program that runs on an instrument PC. It monitors a directory, groups new files into runs, uploads them to cloud storage, and reports status to the API. Each instrument can have at most one active watcher at a time.
A watcher is registered with the API and carries one registration ID per [environment](#environment). Deregistering a watcher is a [soft delete](#soft-delete): its history stays visible for auditing. See [Manage the watcher fleet](/docs/manage-watchers).
### Watcher status [#watcher-status]
What the **Watchers** page shows for each watcher, derived from its most recent [heartbeat](#heartbeat):
| Status | Meaning |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Online** | A heartbeat arrived within the expected window. |
| **Unresponsive** | No heartbeat for longer than that window. The process stopped, the PC is off, or the network is blocking the API. |
An unresponsive watcher keeps its registration and history, so you can read its last events to work out what happened. See [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
{/* TODO: confirm the exact staleness threshold with an engineer and state it here in place of “the expected window.” */}
### Heartbeat [#heartbeat]
A status ping the watcher sends to the API every 60 seconds. It carries the watcher version, instrument ID, watch directory, upload mode, activity counters, and uptime, and it drives the **Last Heartbeat** column and the online health shown on the dashboard. The same tick runs the auto-updater and, in manual mode, polls the upload queue.
### Events [#events]
A per-watcher log of lifecycle moments and errors: `watcher_started`, `run_reported`, `file_uploaded`, `update_succeeded`, and `error` (with a `details.kind` discriminator), among others. Events are batched and flushed on each heartbeat and are visible on the dashboard, which makes them the primary tool for diagnosing a watcher remotely. See [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
### Environment [#environment]
Which Data Hub deployment a watcher talks to: `staging`, `production`, or `preview` (a Vercel preview deployment, which needs an explicit API base URL). Staging and production are separate deployments with separate databases, so a single PC keeps an independent registration and local state per environment and can switch between them.
## Data [#data]
### Run [#run]
A logical grouping of files produced by one acquisition. The watcher assigns each file a **run ID** using its configured [run detection](#run-detection) method, then reports the run to the API. The first file for a run creates it; later files are added incrementally to the same run’s manifest.
### Run status [#run-status]
How far a run’s files have got from the instrument to the dashboard. These are the values you can filter by under **Add filter** on any run list:
| Status | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------- |
| Pending upload | The watcher found the files but hasn’t sent them yet. |
| Uploaded | The files are in storage. Processing hasn’t started, or this instrument type has no preprocessor. |
| Processing | The preprocessor is reading the files, extracting metadata, and building reports. |
| Completed | Processing finished. |
| Failed | Processing stopped on an error. [Reprocess](/docs/browse-runs#reprocess-and-delete) after a fix. |
| Empty | The run has no files. |
### File and file category [#file-and-file-category]
Every file in a run carries a category, shown as a label in the run’s files table:
* **Raw**: what the instrument wrote, uploaded unchanged
* **Processed**: what a [preprocessor](#preprocessor) generated from a raw file, such as a CSV of well values, a contrast-adjusted PNG, or a preview JPG
Both categories download the same way, and file filters on the run page work on this label. See [Download files](/docs/browse-runs#download-files).
### Preprocessor [#preprocessor]
The server-side code that reads a raw file, extracts metadata, and writes processed files. It runs in an AWS Lambda when a file arrives in raw storage, or when you choose **Reprocess**. Which preprocessor runs depends on the [instrument type](#instrument-type), and not every type has one. Per-type inputs and outputs are in [Instrument data preprocessing](/docs/instrument-preprocessing).
### Run archive [#run-archive]
The zip that **Download all** builds for a run. It covers the run’s active files and respects any file filter you have applied. Large archives build in the background as an archive job, so the first request can return a wait time rather than a file.
### Attribution [#attribution]
The record of who ran a run. Use **Claim** on the run to attribute yourself. Attribution is self-only: you claim or unclaim your own user, and you can’t attribute someone else. Claims drive **My runs** and the “ran by” filters, and claiming a run subscribes you to its comments. See [Claim who ran it](/docs/browse-runs#claim-who-ran-it).
## Turning files into runs [#turning-files-into-runs]
### Run detection [#run-detection]
The rule that maps a file to a run ID. Two broad strategies:
* **Prefix**: a regex extracts the run ID from the filename. The default `^([^_]+)` takes everything before the first underscore, so `RUN001_data.csv` → `RUN001`
* **Directory**: each subdirectory under the watch directory is its own run, so `RUN001/data.csv` → `RUN001`
The full set of presets, and the custom-regex option, is in the [Configuration reference](/docs/configuration-reference#run-detection).
### Stability period [#stability-period]
How many seconds a file must stay unchanged (same size and modification time) before the watcher treats it as fully written and eligible for upload. The default is 5 seconds; raise it for instruments that write large files slowly. A file that keeps changing past `max_stability_wait_seconds` (default 300, or 5 minutes) is abandoned with a `stability_timeout` error event.
### Upload mode [#upload-mode]
How files get from the instrument to cloud storage:
* **auto**: files upload immediately after they’re detected and stabilized
* **manual**: runs are reported to the server, but files aren’t uploaded until approved through the server-side upload queue, which the watcher polls on each heartbeat. Use this when uploads need human approval
### Initial scan [#initial-scan]
What the watcher does with files already sitting in the watch directory the first time an environment is entered. `production` uploads the existing backlog (`full`). `staging` and `preview` record it as a baseline and skip it (`new-only`), so test environments aren’t flooded with history. See the [Configuration reference](/docs/configuration-reference#initial-scan).
### Dismiss [#dismiss]
For instruments in [manual upload mode](#upload-mode), dismissing a detected file tells the watcher not to upload it. Dismissing differs from deleting: the file never reached Data Hub storage, so there’s nothing to restore. See [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
### Soft delete [#soft-delete]
Data Hub marks things deleted rather than erasing them, so history survives for auditing. Deleting a run hides it from run lists without removing its stored files, and **Show deleted runs** brings it back into view. Deregistering a watcher keeps its events and heartbeats. Admins can restore a deleted run from the run page.
## People and access [#people-and-access]
### Role [#role]
Data Hub has two roles, **Member** and **Admin**. Members sign in, browse instruments and runs, and read the token audit list. Admins can also confirm and update instruments, create and revoke tokens, and promote or demote other users. Full permission tables are in [Security and permissions](/docs/security#roles).
### Personal access token [#personal-access-token]
A `dhub_`-prefixed bearer token that authenticates the watcher, and other API clients, with the Data Hub API. Tokens carry permission [scopes](#scope) and are created and revoked by admins. The plaintext is shown once, at creation. See [Issue and revoke tokens](/docs/manage-tokens).
### Scope [#scope]
A permission string carried by a token, such as `runs:read`. A token-authenticated request is refused with `403 FORBIDDEN` when the token’s scopes don’t cover what the route requires. Dashboard sessions hold every scope, so scope checks only apply to token requests. The scope table is in [Security and permissions](/docs/security#token-scopes).
# Watcher configuration (/docs/configuration-reference)
**For:** Lab operators
The complete reference for the watcher’s configuration: the `config.yaml` schema,
where files live, the environment variables that influence it, and the behaviors
(run detection, upload modes, initial scan) those fields drive.
## File locations [#file-locations]
Everything the watcher persists lives under `~/.data-hub/`:
| Path | Contents |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `~/.data-hub/config.yaml` | The main config (below). Override with `--config` or `DATA_HUB_CONFIG_PATH`. |
| `~/.data-hub/.env.` | The API key for an environment (e.g. `.env.staging`, `.env.production`, `.env.preview`). The legacy `~/.data-hub/.env` is still loaded, with the per-environment file taking precedence. |
| `~/.data-hub/watcher-.db` | Per-environment SQLite state (uploaded files, runs, detected files, baseline). |
| `~/.data-hub/watcher.log` | Rotating log (10 MB × 5 backups). On Windows: `C:\ProgramData\DataHubWatcher\watcher.log`. |
| `~/.data-hub/upgrade-worker.log` | Windows uv-tool upgrade transcript. |
## Environment variables [#environment-variables]
| Variable | Purpose |
| ---------------------------- | --------------------------------------------------------------------------------------------- |
| `DATA_HUB_API_KEY` | Supplies the API key, skipping the `init` prompt. Saved into the per-environment `.env` file. |
| `DATA_HUB_CONFIG_PATH` | Override the config file path (same as `--config`). |
| `DATA_HUB_WATCHER_LOG_LEVEL` | Set to `DEBUG` (in the service’s `.env` file) to enable debug logging without a reinstall. |
## `config.yaml` [#configyaml]
```yaml
version: 1
environment: production # "staging", "production", or "preview"
api_base_url: null # required when environment is "preview"
watcher_ids: # one registration id per environment
production:
initial_scan: null # null (default), "full", or "new-only"
instrument:
id: akta-fplc # kebab-case instrument ID
watch_directory: /path/to/data
file_patterns:
- "*.csv"
- "*.xlsx"
enabled: true
upload_mode: auto # "auto" or "manual"
stability_period_seconds: 5 # 1–300
max_stability_wait_seconds: 300 # 1–86400; must be >= stability_period_seconds
run_detection:
pattern: '^([^/]+)/' # regex with one capture group (run ID)
recursive: true
```
### Field reference [#field-reference]
| Field | Type | Description |
| --------------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version` | int | Config schema version. Currently `1`. |
| `environment` | enum | `staging`, `production`, or `preview`. |
| `api_base_url` | string \| null | Required only for `preview` (the deployment’s `/api/v1` URL). |
| `watcher_ids` | map | One watcher registration id per environment, assigned by the API. |
| `initial_scan` | enum \| null | `null` (env default), `full`, or `new-only`. See [Initial scan](#initial-scan). |
| `instrument.id` | string | Kebab-case instrument ID. Permanent; the storage key prefix. |
| `instrument.watch_directory` | string | Absolute path the instrument writes to. |
| `instrument.file_patterns` | string\[] | Glob patterns; only matching files are uploaded. |
| `instrument.enabled` | bool | Whether the watcher processes this instrument. |
| `instrument.upload_mode` | enum | `auto` or `manual`. See [Upload modes](#upload-modes). |
| `instrument.stability_period_seconds` | int (1–300) | Seconds a file must stay unchanged before it’s considered written. Default `5`. |
| `instrument.max_stability_wait_seconds` | int (1–86400) | Seconds a still-changing file may stay pending before the watcher abandons it with a `stability_timeout` event. Must be ≥ `stability_period_seconds`. Default `300` (5 minutes). |
| `instrument.run_detection.pattern` | regex | One-capture-group regex applied to each file’s path relative to `watch_directory`. |
| `instrument.run_detection.recursive` | bool | `true` watches subdirectories; `false` only the top level. |
A config written by an older watcher used a single top-level `watcher_id`. It’s
migrated transparently on load: lifted into `watcher_ids` under the active
environment and dropped on the next save.
## Run detection [#run-detection]
The `pattern` regex is applied with `re.search` to each file’s path relative to
`watch_directory`, with backslashes normalized to `/`. Capture group 1 is the run
ID, and the pattern must have exactly one capture group. In YAML, single-quote
patterns so backslash sequences like `\d` don’t need escaping.
The `init` wizard offers these presets (or supply a custom regex):
| Preset | Pattern | Recursive | Run ID is… |
| ---------------------- | ------------------------------ | --------- | ----------------------------------------------------------------- |
| Filename prefix | `^([^_]+)` | no | everything before the first `_`: `RUN001_data.csv` → `RUN001` |
| Top subdirectory | `^([^/]+)/` | yes | the top-level folder: `RUN001/data.csv` → `RUN001` |
| Deepest subdirectory | `([^/]+)/[^/]+$` | yes | the immediate parent folder: `plate-a/well-b/data.csv` → `well-b` |
| Timestamp subdirectory | `(?:^\|/)(\d{8}_\d{6}_\d{3})/` | yes | a `YYYYMMDD_HHMMSS_fff` folder anywhere in the path |
| Filename stem | `^(?:.+/)?([^/]+?)\.[^/.]+$` | no | the filename without its extension (each file is its own run) |
## Upload modes [#upload-modes]
* **`auto`**: files upload to cloud storage immediately after run detection.
* **`manual`**: runs are reported without uploading; the server decides which files to upload via a queue polled on each heartbeat. Useful when uploads need human approval. See [Manage the watcher fleet → Approving manual uploads](/docs/manage-watchers#approving-manual-uploads).
## Initial scan [#initial-scan]
The first time an environment is entered (via `init` or `config set-environment`),
`initial_scan` decides what happens to files already in the watch directory. It
defaults by environment:
* `production` → `full`: the existing backlog is uploaded (production is the source of truth).
* `staging` / `preview` → `new-only`: the on-disk files are recorded as a baseline and skipped; only files created afterwards are uploaded.
This is why a fresh `init` on `staging`/`preview` doesn’t upload a PC’s history. Set
`initial_scan: full` to opt back in (e.g. to deliberately populate staging), or
`initial_scan: new-only` on production to suppress its backlog. Upgrading an
existing watcher is unaffected: the environment’s local DB already carries history,
so baseline seeding is skipped.
## Switching environments [#switching-environments]
A single PC can move between `staging`, `production`, and `preview`. Because these
are separate deployments with separate databases, each keeps its own registration
in `watcher_ids` and its own credentials in `~/.data-hub/.env.`.
```sh
# Switch to staging (reuses a stored registration, or registers one).
data-hub-watcher config set-environment staging
# Point at a preview deployment (the base URL is required).
data-hub-watcher config set-environment preview \
--api-base-url https://data-hub-git-my-branch.vercel.app/api/v1
```
A switch validates the target API, reuses (or registers) the watcher id for that
environment, pushes the config, and leaves a breadcrumb event in the old
environment. A running `watch` process or service keeps using the old
environment until restarted: on Windows, `service stop && service start`;
elsewhere, restart `watch`.
# Configure notifications (/docs/configure-notifications)
**For:** Administrators, members, and engineers
Data Hub can notify you in the app and via Slack when new runs arrive or when someone comments on a run you care about. Open **Settings → Notifications** while signed in. Personal Slack DMs also need a Slack app and environment variables on the web app.
In-app and Slack preferences are independent: you can enable either channel for a type without forcing the other.
## In-app preferences [#in-app-preferences]
| Setting | Effect |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Mute all instrument notifications** | Suppresses new-run notifications for every instrument. Per-instrument choices are preserved and resume when you unmute. |
| **Comments on runs you ran** | Notify when someone comments on a run you are attributed to. |
| **Replies to your comments** | Notify when someone comments on a run you previously commented on. |
### Per-instrument subscriptions [#per-instrument-subscriptions]
Under **Per-instrument**, toggle the instruments you want new-run notifications for. Rows are disabled while **Mute all** is on. You can also subscribe from the instruments table’s notifications column when that column is shown.
The bell icon in the app chrome lists recent notifications and links to the relevant run.
## Slack direct messages [#slack-direct-messages]
Connect your Slack account on the same settings page to receive matching events as personal DMs. After connecting, choose which types also go to Slack:
* New runs (respects your instrument subscriptions and mute)
* Comments on runs you ran
* Replies to your comments
Disconnect or reconnect from the same card. OAuth must use your organization’s Slack workspace; the wrong workspace is rejected.
## Workspace Slack channel (admins) [#workspace-slack-channel-admins]
Admins see a **Slack channel** section for the shared webhook that posts when a new run is created (lab-wide channel traffic, separate from personal DMs). Configure or rotate the webhook there; the card shows who last updated it. That webhook is set in the UI after deploy; it doesn’t use the bot token variables below.
## Set up the Slack app for personal DMs [#set-up-the-slack-app-for-personal-dms]
Personal Slack DMs use a shared bot token and Sign in with Slack (OpenID Connect). Create the app once in the [Slack API portal](https://api.slack.com/apps), then set the variables on each environment that should send DMs. Without them, **Connect to Slack** doesn’t complete.
You can reuse an existing Slack app, including one already used for the channel webhook.
### Create the app and scopes [#create-the-app-and-scopes]
In the Slack API portal, choose **Create New App → From scratch**, or open an existing app.
Under **OAuth & Permissions**, add these scopes:
**Bot Token Scopes:**
| Scope | Purpose |
| ------------ | -------------------------------- |
| `chat:write` | Send DMs with `chat.postMessage` |
**User Token Scopes:**
| Scope | Purpose |
| --------- | ---------------------------------- |
| `openid` | Sign in with Slack (OIDC) |
| `profile` | Display name in the identity token |
Under **Redirect URLs**, add one callback URL per environment. The path is always `/api/v1/settings/slack/callback`:
| Environment | Example URL |
| ----------- | -------------------------------------------------------------------- |
| Local | `http://localhost:3000/api/v1/settings/slack/callback` |
| Staging | `https://staging.datahub.example.com/api/v1/settings/slack/callback` |
| Production | `https://datahub.example.com/api/v1/settings/slack/callback` |
`SLACK_REDIRECT_URI`, or the request origin plus that path when unset, must match a listed URL exactly.
Under **Install App**, choose **Install to Workspace** (or **Reinstall** after changing scopes). Copy the bot token and app credentials for the variables below.
### Set environment variables [#set-environment-variables]
Set these on the web app for each environment that should offer Slack DMs:
| Variable | Where to find it | Required |
| --------------------- | --------------------------------------------------------- | ------------------------------------------------- |
| `SLACK_BOT_TOKEN` | **Install App** → Bot User OAuth Token (`xoxb-…`) | Yes |
| `SLACK_CLIENT_ID` | **Basic Information** → App Credentials | Yes |
| `SLACK_CLIENT_SECRET` | **Basic Information** → App Credentials | Yes |
| `SLACK_STATE_SECRET` | Generate with `openssl rand -base64 32` | No (falls back to `AUTH_SECRET`) |
| `SLACK_TEAM_ID` | Workspace URL segment `T…` in `app.slack.com/client/T…` | No (recommended so OAuth stays on your workspace) |
| `SLACK_REDIRECT_URI` | Full callback URL when it differs from the request origin | No |
Redeploy or restart the web app after setting them. Members can then connect under **Settings → Notifications**. Deploy runbooks are linked from [Deploy Data Hub](/docs/self-hosting).
# Configure run detection and uploads (/docs/configure-run-detection)
**For:** Lab operators
Tune three behaviors that decide what becomes a run and when bytes leave the instrument PC: run detection, stability period, and upload mode. Field-by-field schema lives in [Watcher configuration](/docs/configuration-reference); this page is the intent guide.
Change these during `data-hub-watcher init`, with `data-hub-watcher config edit`, or by editing `~/.data-hub/config.yaml` then syncing.
## Group files into runs [#group-files-into-runs]
A **run** is one acquisition: a logical group of files the watcher reports together. **Run detection** maps each file’s path (relative to the watch directory) to a run ID.
Common choices:
* **Prefix from filename**: a regex capture, for example everything before the first underscore so `RUN001_data.csv` → `RUN001`
* **One subdirectory per run**: each folder under the watch directory is its own run, so `RUN001/data.csv` → `RUN001`
Presets and custom regex options are listed under [Run detection](/docs/configuration-reference#run-detection). Set `recursive: true` when files live in subfolders; otherwise only the top level is watched.
If files appear on disk but never become runs, check for `pattern_mismatch` events on the watcher detail page and run `data-hub-watcher watch --dry-run`.
## Wait until writes finish [#wait-until-writes-finish]
**Stability period** is how many seconds a file must keep the same size and modification time before the watcher treats it as fully written. Default is 5 seconds (allowed range 1–300). Raise it for instruments that write large files slowly.
A file that keeps changing past `max_stability_wait_seconds` (default 300, or 5 minutes) is abandoned with a `stability_timeout` error event. Raise that cap for instruments whose writes take longer; it must be at least the stability period.
## Choose auto or manual upload [#choose-auto-or-manual-upload]
| Mode | Behavior |
| ---------- | -------------------------------------------------------------------------------------------------------------------------- |
| **auto** | Upload as soon as a file is detected and stable. |
| **manual** | Report the run, but wait for server approval before uploading bytes. The watcher polls the upload queue on each heartbeat. |
Use manual mode when someone must approve which files leave the instrument PC. Admins manage the queue from the dashboard; see [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
## Existing files on first setup [#existing-files-on-first-setup]
`initial_scan` controls the backlog already in the watch directory when an environment is first entered:
* **`production` default**: upload the backlog (`full`)
* **`staging` / `preview` default**: record a baseline and skip existing files (`new-only`)
Override in config when you need the other behavior. Details: [Initial scan](/docs/configuration-reference#initial-scan).
# Install and update the watcher (/docs/install-the-watcher)
**For:** Lab operators
Install the watcher on an instrument PC, verify it, and apply manual updates. Fleet auto-update (what admins advertise and how PCs upgrade in the background) is covered in [Roll out watcher releases](/docs/watcher-releases).
## Prerequisites [#prerequisites]
* **[uv](https://docs.astral.sh/uv/getting-started/installation/)**: recommended. It installs Python for you; you do not need a separate Python install.
* **A personal access token** starting with `dhub_`. Admins create these at **Settings → Access Tokens**; members cannot mint their own. See [Issue and revoke tokens](/docs/manage-tokens).
* **Watch directory** and **file patterns** for the instrument output folder.
## Install [#install]
Most lab instrument PCs run Windows, so install the `windows-service` extra by default. It adds `pywin32`, which the watcher needs for the [Run as a Windows service](/docs/windows-service) setup:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
This puts the `data-hub-watcher` CLI on PATH in an isolated venv managed by `uv`.
On macOS or Linux, or on a PC you will only run in the foreground, omit the extra:
```sh
uv tool install data-hub-watcher
```
## Configure [#configure]
```sh
data-hub-watcher init
```
The wizard walks through environment, API key, instrument, watch directory, file patterns, run detection, stability period, and upload mode. New instruments start as `pending` until an admin confirms them; see [Set up an instrument](/docs/set-up-an-instrument).
Config saves to `~/.data-hub/config.yaml` and syncs to the server. Intent-level guidance for detection and uploads is in [Configure run detection and uploads](/docs/configure-run-detection); every field is in [Watcher configuration](/docs/configuration-reference).
## Verify [#verify]
```sh
data-hub-watcher watch --dry-run
```
Dry-run validates config, checks API reachability and instrument status, and previews matching files without starting the monitor.
## Run [#run]
On a lab PC, run the watcher as a Windows service so it starts on boot, survives logout, and auto-updates in the background. From an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
Lifecycle commands, log locations, and recovery settings are in [Run as a Windows service](/docs/windows-service).
For a one-off check, or on macOS or Linux, run in the foreground instead:
```sh
data-hub-watcher watch
```
The watcher monitors the directory, waits for files to stabilize, groups them into runs, uploads (or queues in manual mode), and heartbeats every 60 seconds. Stop a foreground watcher with `Ctrl+C`; stop the service with `data-hub-watcher service stop`.
## Reinstall [#reinstall]
Reinstall the same package (for example after a broken venv or missing service extra):
```sh
uv tool install --reinstall data-hub-watcher
# or with the Windows service extra:
uv tool install --reinstall "data-hub-watcher[windows-service]"
```
On Windows services, follow with `data-hub-watcher service reinstall` from an Administrator shell so the service and upgrade Scheduled Task point at the new install.
## Change configuration later [#change-configuration-later]
```sh
data-hub-watcher config edit # re-prompt each field
data-hub-watcher config open # open the YAML in your editor
data-hub-watcher config show # print the current config
```
Changes sync to the server automatically.
## Manual update (`self-update`) [#manual-update-self-update]
Use this when you want an upgrade immediately, or on a non-service install. Background fleet upgrades are described in [Roll out watcher releases](/docs/watcher-releases).
```sh
data-hub-watcher self-update # check + upgrade if needed
data-hub-watcher self-update --check # report status only
data-hub-watcher self-update --force # re-run even if versions match
```
The command asks the API for the latest published version and upgrades according to how the watcher was installed:
| Install method | Upgrade flow |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Windows + `uv tool install` | Routes through the `DataHubWatcherUpgrade` Scheduled Task. The CLI returns when the task accepts; the service restarts automatically. Tail `~/.data-hub/upgrade-worker.log` for progress. |
| POSIX + `uv tool install` | `uv tool install --reinstall data-hub-watcher==` inline. **Restart the watcher** afterward. |
| Plain venv `pip install` | `pip install -U data-hub-watcher==` inline. **Restart the watcher.** |
| Editable / `uv sync` checkout | Refused; upgrade with `git pull && uv sync`. |
On Windows uv-tool installs, `data-hub-watcher service install` (or `service reinstall`) must have run once from an Administrator shell so the upgrade Scheduled Task exists.
### Pin a specific version [#pin-a-specific-version]
```sh
uv tool install data-hub-watcher==0.3.0
```
Run `self-update --check` to confirm the server’s advertised target. When the server’s latest moves past your pin, the next auto-update tick tries to upgrade again. Fleet-wide pins and rollbacks are admin actions under [Roll out watcher releases](/docs/watcher-releases).
# Instrument data preprocessing (/docs/instrument-preprocessing)
**For:** Lab operators, admins, and engineers
After a [watcher](/docs/concepts#watcher) uploads a raw file, Data Hub can run a **preprocessor** on it: code that reads the vendor’s file, pulls out settings like wavelength or imaging mode, and builds something you can look at in the browser. Seven instrument types have one today. This page covers what each produces, why a file sometimes gets no processing, and what adding a new instrument involves.
Support for a new instrument isn’t a settings change. It takes a new Python module, a database migration, an edit in two repositories, and a Lambda redeploy. See [Add support for a new instrument](#add-support-for-a-new-instrument).
## How a file reaches a processor [#how-preprocessing-works]
1. A file arrives in raw storage, which triggers the Lambda. The bucket notifies on every upload, with no per-instrument filtering.
2. The Lambda checks the filename against a **filename gate**, a cheap test that skips files no processor wants before making any API call.
3. It reads the instrument, then looks up a processor by the instrument’s [type](#set-the-instrument-type). Unmapped types stop here.
4. The processor downloads the file, extracts metadata, writes processed files when it has any, and updates the run through the API.
5. The file’s status moves from `processing` to `completed` or `failed`.
### Why an upload sometimes produces nothing [#why-an-upload-produces-nothing]
Two silent no-ops explain a file that uploaded and then produced nothing. Neither marks the file failed, because nothing went wrong:
* **The filename missed the gate.** Each type accepts a narrow pattern, listed in the table below. An Azure Cielo run only triggers on files ending `_cq values.csv`, so the other CSVs a qPCR run writes are stored and ignored.
* **The instrument has no type set, or a type with no processor.** Files upload and stay browsable; you get no metadata and no reports.
Filename gates apply to uploads only. Choosing **Reprocess** in the web app skips the gate, because clicking it states your intent. That’s why a file that produced nothing on upload can still process when you reprocess it by hand.
### Set the instrument type [#set-the-instrument-type]
Processing is chosen by the instrument’s **type**, not by its instrument ID. The ID (`azure-cielo-qpcr`) names one physical machine. The type (`qpcr`) names a vendor’s file format, and several instruments can share one.
To set it, open the instrument page, choose **Edit** from the three-dot menu, and pick a type.
One type means one vendor’s output format. Names like `qpcr` and `fplc` read as generic, but each is bound to a single vendor’s parser: `fplc` expects ÄKTA PDFs, `qpcr` expects Azure Cielo CSVs. Typing a different vendor’s instrument into an existing type feeds files to a parser that can’t read them. Supporting a second vendor means splitting the type, which is a code change.
## Supported instrument types [#supported-instruments-at-a-glance]
| Type | Vendor and model | Gate for uploads | You get |
| -------------------- | --------------------------- | -------------------------- | --------------------------------------------- |
| `tape_station` | Agilent 4150 TapeStation | `.pdf` | Tape-type badge, PDF reports |
| `fplc` | Cytiva ÄKTA FPLC | `.pdf` | PDF you can read on the run page, no metadata |
| `gel_doc` | Azure 600 Gel Doc | `.tif`, `.tiff` | Contrast-enhanced PNG, imaging badges |
| `qpcr` | Azure Cielo qPCR | ends with `_cq values.csv` | Dye-channel badges |
| `epson_v700_scanner` | Epson V700 Scanner | `.tif`, `.tiff` | Plate preview JPG, colony measurements |
| `hina_microscope` | Hina Microscope (Nikon ND2) | `.nd2` | Channel overlay JPG, channel badges |
| `plate_reader` | SpectraMax iD3 and iD5 | `.xls` | Well-data CSV, interactive plate maps |
## Agilent 4150 TapeStation [#agilent-4150-tapestation]
Reads the tape type out of peak-table CSV filenames and shows TapeStation reports on the run page.
* **Expected files**: CSVs named like `YYYY-MM-DD - HH-MM-SS-_peakTable.csv`, plus matching PDFs
* **Metadata**: `Tape Type`, for example `gDNA` or `HSD1000`, when the filename matches that pattern
* **Processed files**: none
* **On the run page**: tape-type badge, a PDF viewer, and peak-table reports
Uploads trigger on the PDF, not the CSV. A run of CSVs alone stays unprocessed until you reprocess it by hand.
## Cytiva ÄKTA FPLC [#akta-fplc]
Stores ÄKTA chromatography PDFs so you can read them in the browser. The processor reads nothing out of the PDF itself yet, so this type gets no badges or filters.
* **Expected files**: `.pdf` reports from the ÄKTA
* **Metadata**: none
* **Processed files**: none
* **On the run page**: files table, plus each PDF embedded under Report Data
A newly seeded FPLC stays `generic` on purpose until an operator has checked its output format.
## Azure 600 Gel Doc [#azure-600-gel-doc]
Rebuilds each gel TIFF as a viewable image and reads the imaging settings the instrument recorded.
The processor stretches intensities to the 1st to 99th percentile, per RGB channel or as grayscale, then lays every page of the TIFF side by side in one 300 dpi PNG. The stretch is cosmetic: it makes faint bands visible on screen and doesn’t change the raw file.
* **Expected files**: TIFF from the Azure 600
* **Metadata**: `capture_type`, `imaging_mode` (true color, fluorescence, or chemiluminescence), `wavelengths`, `colors`, read from the TIFF `XPComment` tag
* **Processed files**: contrast-enhanced PNG
* **On the run page**: imaging badges, files table, image viewer for the PNGs
Those metadata fields also drive the column filters on the instrument page.
## Azure Cielo qPCR [#azure-cielo-qpcr]
Reads dye channel names so you can filter and badge runs by channel.
* **Expected files**: Cq Values CSV. Other files in the run upload normally but contribute no metadata
* **Metadata**: `dye_channels`, the unique names in the `Fluorescence` column
* **Processed files**: none
* **On the run page**: dye-channel badges and report tables
## Epson V700 Scanner [#epson-v700-scanner]
Finds agar plates in a flatbed scan and measures the colonies on them.
The processor looks for plates sitting in gold 3D-printed frames. For each plate it finds, it crops the plate, then detects colonies: crop the edge, compare against the plate background, apply a difference-of-Gaussians filter, threshold, and measure whatever survives. Measurements come out in physical units, so colony areas are comparable between scans.
* **Expected files**: TIFF plate scans
* **Metadata**: `dpi`, `color_mode`, `plate_count`, `plate_boxes`, plus `colony_detection` summaries when colonies were measured
* **Processed files**: a web-sized JPG with plate outlines and colony boxes drawn on, and a `_colonies.csv` of per-colony area, centroid, eccentricity, and mean RGB
* **On the run page**: scanner badges, files table, and reports for the processed CSV
When no plates are found, you still get a resized JPG of the whole scan, and colony detection is skipped. An empty `plate_count` usually means the frames weren’t recognized rather than that the plates were blank.
## Hina Microscope [#hina-microscope]
Turns each Nikon ND2 into a single composite image you can view in the browser.
Every channel is reduced to one 2D frame (a maximum projection over Z, the first index over time or position), stretched to the 1st to 99th percentile, then composited onto a brightfield background in each channel’s fluorophore color.
* **Expected files**: ND2
* **Metadata**: `sizes`, `channels` (name, excitation and emission wavelength, color), and `dimensions`
* **Processed files**: an RGB overlay JPG for every file in the run
* **On the run page**: channel and size badges, files table, image viewer for the JPGs
Channel layout is identical across a Hina run, so metadata is written once by whichever file finishes processing first. Later files still get their JPG and skip the metadata write. Reprocessing one file therefore won’t correct run metadata if the layout changed mid-run.
## SpectraMax iD3 and iD5 plate readers [#spectramax-id3-and-id5-plate-readers]
Both models share one processor, which converts SoftMax Pro exports into the well data behind the interactive plate maps.
* **Expected files**: SoftMax `.xls`, which is UTF-16 text with a `Plate:` header rather than a real Excel workbook
* **Metadata**: `measurement_mode` (for example Absorbance), `measurement_type` (for example Endpoint or Kinetic), and `wavelengths` as bare numbers with no unit suffix
* **Processed files**: `{run_id}_raw_well_data.csv`
* **On the run page**: wavelength, mode, and type badges, plus plate maps built from that CSV
## Types with no processor [#instruments-without-a-matching-processor]
`generic` and `instant_raman` have no processor, and neither does an instrument whose type is unset. Those uploads still work: files are stored, and you can browse, search, claim, comment on, and download them. What you don’t get is extracted metadata, badges, filters, or processed artifacts.
Leaving a new instrument as `generic` is the safe default. It keeps data flowing while someone confirms which processor, if any, can read its files.
## When processing fails [#when-processing-fails]
A file with status `failed` carries an error message on the run page. Open the run, then choose **Reprocess** on that file, or on the whole run where that action is available. See [Reprocess or delete a run](/docs/browse-runs#reprocess-and-delete).
Reprocessing is also the fix after an engineer corrects a parser: the code changed, but the stored result didn’t, until you ask for it again.
## Add support for a new instrument [#add-support-for-a-new-instrument]
**For:** Engineers
Adding an instrument type is a development task spanning the web app and the Lambda, not a configuration change an admin can make in the dashboard. Budget for a database migration and a Lambda image deploy.
**Add the type.** Extend `instrumentTypeEnum` in `web/lib/db/schema.ts` and generate an `ALTER TYPE … ADD VALUE` migration. Reuse an existing type only when the new instrument writes the same vendor’s format.
**Write the processor.** Add a module under `lambda/src/data_hub_lambda/` exposing `process_file(instrument_id, run_id, filename)`, which reports progress through the Data Hub API.
**Register it twice.** Add the type to `PROCESSORS` in `lambda/src/data_hub_lambda/processors.py`, with its filename gate, and to `PROCESSABLE_INSTRUMENT_TYPES` in `web/lib/instruments/processable-types.ts`. These two lists have to agree: the Lambda uses the first to dispatch, and the web app uses the second to decide whether to offer **Reprocess**.
**Add tests** for the processor and for the registry gate.
**Deploy the Lambda image.** No new S3 trigger is needed, since the raw bucket already notifies on every upload.
You can exercise a processor locally without S3 or API access using the `data-hub-process` CLI, and drive the whole handler against a local S3 mirror with its `handler` subcommand.
Full detail, including the archive builder and the Docker build, is in [the Lambda developer docs](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/lambda.md#adding-a-new-instrument--processor).
# Manage instruments (/docs/manage-instruments)
**For:** Administrators
When a lab operator registers an instrument from the watcher, it appears as `pending` and cannot upload until you confirm it. Operator-side setup is in [Set up an instrument](/docs/set-up-an-instrument). Retiring and reactivating are in [Retire an instrument](/docs/retire-an-instrument).
## Instrument states [#instrument-states]
The **Instruments** page has tabs for active, pending, and retired instruments.
| Status | UI label | Meaning |
| ---------- | -------- | ------------------------------------------------------------ |
| `pending` | Pending | Registered but not confirmed. Watchers cannot upload. |
| `active` | Active | Confirmed and accepting uploads. |
| `inactive` | Retired | Deactivated; no new uploads. Historical runs stay browsable. |
## Confirm a pending instrument [#confirm-a-pending-instrument]
Open the Data Hub web app and go to **Instruments**.
Open the **Pending** tab (or find the yellow `pending` badge).
Click **Confirm**. Status becomes `active` and the operator’s watcher can upload.
Confirming is admin-only for session users. Under the hood it is `PATCH /api/v1/instruments/:id`. Bearer automation uses the `instruments:write` scope instead. See the [generated API docs](/docs/api).
## View instrument health [#view-instrument-health]
The instrument detail page shows run count and watcher online/offline counts. Watcher health comes from heartbeats; see [Manage the watcher fleet](/docs/manage-watchers).
You can also rename the display name and set instrument type from the admin actions on the instrument header. The instrument ID never changes: it is the storage key prefix.
# Manage members (/docs/manage-members)
**For:** Administrators
Workspace members sign in with Google OAuth. Admins decide who else has the admin role. Member vs admin capabilities are summarized in [Security and permissions](/docs/security#roles).
## Open the members list [#open-the-members-list]
Go to **Settings → Members**. Non-admins who open the URL see an “Admins only” message; the settings sidebar hides the entry for them.
The table lists every signed-in user: name, email, role badge, and an **Admin** toggle.
## Promote or demote [#promote-or-demote]
Toggle **Admin** on a row to grant or revoke admin access. Every privileged action re-checks the database, so the change applies to that user’s next one. Their view of the dashboard can lag by up to 5 minutes, because admin-only buttons and sidebar entries come from their [cached session](/docs/security#session-lifetime).
You cannot demote yourself. That guard prevents locking the workspace out of its last admin. Ask another admin to demote you if needed.
## First admin bootstrap [#first-admin-bootstrap]
The first admins come from the `ADMIN_EMAILS` environment variable (comma-separated, case-insensitive). Listed users are promoted on every sign-in. After that, any admin can promote others from this page.
There is no invite flow in the product: people appear here after they sign in successfully to your deployment. Every Google account your OAuth client accepts can sign in, so that client, not this page, controls who appears here. See [Who can sign in](/docs/security#who-can-sign-in).
# Issue and revoke tokens (/docs/manage-tokens)
**For:** Administrators
Personal access tokens authenticate the watcher and other API clients. Creating and revoking tokens requires the workspace admin role. Members can view the audit list at **Settings → Access Tokens** but cannot mint or delete tokens. Roles and scopes: [Security and permissions](/docs/security).
## Create a token [#create-a-token]
### In the web app [#in-the-web-app]
Sign in as an admin.
Go to **Settings → Access Tokens**.
Click **Create token**.
Enter a **Name** that identifies the client (for example, `FPLC watcher, Lab 201`).
Set **User** to whoever the token acts as. Data Hub attributes write actions, such as claiming a run, to this user. Defaults to you.
Pick an **Expiration**: 30 days, 90 days (the default), 1 year, or no expiry.
Under **Scopes**, click a preset, or click **Customize scopes** to pick individual scopes.
Click **Create token**.
The plaintext token is shown once: copy it immediately. It starts with `dhub_` followed by a 64-character hex string.
### Scope presets [#scope-presets]
A preset selects the minimum scopes a common client needs:
* **Read-only**: view instruments, runs, files, and watchers; change nothing
* **Watcher**: what a watcher needs to register, report runs, and upload files
* **Lambda**: what the processing Lambda needs to write run and file results
You cannot change a token’s scopes after creation. To change them, revoke the token and create a new one.
### Via the API [#via-the-api]
Token create/delete are admin-only and session-only (cookie auth). Bearer tokens cannot manage other tokens.
```sh
curl -X POST https://datahub.example.com/api/v1/tokens \
-H "Cookie: session_cookie_here" \
-H "Content-Type: application/json" \
-d '{
"name": "FPLC watcher",
"scopes": ["instruments:read", "instruments:write", "watchers:read", "watchers:report", "runs:create", "runs:update", "runs:upload", "files:update"],
"expires_at": "2027-01-01T00:00:00Z"
}'
```
`scopes` is required and must be a non-empty list of explicit scopes. The wildcard `*` and the older coarse scopes `runs:write`, `files:write`, and `watchers:write` are rejected. `expires_at` is optional; if omitted, the token never expires. The response includes the plaintext once in the `token` field.
## Use a token [#use-a-token]
### With the watcher [#with-the-watcher]
During `data-hub-watcher init`, paste the token when prompted. Or set it first:
```sh
export DATA_HUB_API_KEY=dhub_your_token_here
data-hub-watcher init
```
### With the REST API [#with-the-rest-api]
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
More in the [API overview](/docs/api).
Tokens don’t authenticate MCP clients, which sign in with OAuth instead. See [MCP overview](/docs/mcp).
## View tokens [#view-tokens]
**Settings → Access Tokens** shows name, prefix only (full token is never stored), last used, expiration, and created time.
## Revoke a token [#revoke-a-token]
Go to **Settings → Access Tokens**.
Delete the token and confirm.
The token is invalidated immediately. Clients using it start receiving `401 Unauthorized`.
```sh
curl -X DELETE https://datahub.example.com/api/v1/tokens/ \
-H "Cookie: session_cookie_here"
```
## After revoking a token [#after-revoking-a-token]
If a running watcher used that token:
Create a new token.
On the instrument PC, re-run `data-hub-watcher init` and enter the new token.
Restart the watcher (`data-hub-watcher watch` or the Windows service).
## Token hygiene [#token-hygiene]
* Name tokens so you can tell which PC or client each belongs to
* Set expirations for temporary setups
* Scope to least privilege; see [Token scopes](/docs/security#token-scopes)
* Revoke immediately when a watcher is decommissioned or a token may have been exposed
Tokens are hashed with SHA-256 before storage; plaintext is never persisted.
# Manage the watcher fleet (/docs/manage-watchers)
**For:** Administrators
The **Watchers** area is the fleet view: every registered watcher, its health, deregistration, and (for manual upload mode) the upload queue. Cutting releases and advertising versions is separate: [Roll out watcher releases](/docs/watcher-releases).
## Monitor watcher health [#monitor-watcher-health]
Each watcher sends a [heartbeat](/docs/concepts#heartbeat) every 60 seconds with version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:
* **Effective status**: online vs offline from the last heartbeat
* **Last heartbeat**: a stale value is the first sign of a stopped, disconnected, or crashed watcher
* **Hostname and instrument assignment**
* **Events**: per-watcher log (`watcher_started`, `run_reported`, `file_uploaded`, `update_*`, and `error` with a `details.kind` discriminator)
Events are the primary remote-diagnosis tool. For operator-side symptoms, see [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
## Deregister a watcher [#deregister-a-watcher]
Each instrument can have at most one active watcher. Deregister the old one when a PC is reimaged or replaced, or to clear “instrument already has an active watcher” on a new install.
* **Web app**: open the watcher under **Watchers** and click **Deregister**
* **API**: `DELETE /api/v1/watchers/:watcherId` (requires `watchers:admin`)
Deregistration is a soft-delete: heartbeats, events, and runs stay under **Watchers → Deregistered**.
Retiring an instrument also deregisters its watchers automatically; see [Retire an instrument](/docs/retire-an-instrument).
## Approving manual uploads [#approving-manual-uploads]
For instruments in [manual upload mode](/docs/concepts#upload-mode), the watcher reports runs but waits for server approval before uploading file bytes. It polls the upload queue on each heartbeat and uploads files the server has marked for upload.
Safeguards that keep the queue healthy:
* Changing an instrument’s `watch_directory` reverts pending upload requests to `detected` so stale paths drain instead of erroring
* A queued file that keeps failing (missing on disk or failing to upload) is retried on at most three heartbeat polls, then cancelled server-side
# Overview (/docs/overview)
**For:** Everyone
Instrument data tends to stay where it was made. Files sit on the PC wired to the plate reader, named by whoever was at the bench, backed up if someone remembers. Finding a result from three months ago means knowing which machine made it and who was running it.
Data Hub moves that data off the instrument PC automatically. A small program called a watcher runs on each instrument PC, monitors its output folder, uploads new files as they appear, and groups them into runs. From then on the data is searchable in a browser, tied to the person who ran it, and reachable from a script or an AI agent without anyone copying files to a shared drive.
## What Data Hub isn’t [#what-data-hub-isnt]
Worth setting expectations early, because the name invites a few wrong guesses:
* It doesn’t change your raw files. Uploads are byte-for-byte, and processing only ever adds new files alongside them
* It isn’t an electronic lab notebook. There’s a comment thread per run for notes, and that’s the extent of it
* It isn’t analysis software. It parses vendor formats into something viewable and downloadable; the analysis is still yours
* It doesn’t run on the instrument. Processing happens server-side after upload, so nothing competes with acquisition software for resources
## How a file becomes a run [#how-a-file-becomes-a-run]
1. A [watcher](/docs/concepts#watcher) on the instrument PC notices a new file in the folder it’s watching, waits for the file to stop changing, and groups it with the other files from the same acquisition into a [run](/docs/concepts#run).
2. The watcher uploads those files to cloud storage. For [supported instrument types](/docs/instrument-preprocessing), server-side code then reads the vendor’s format, pulls out settings like wavelength or imaging mode, and builds viewable versions such as plate maps or contrast-adjusted images.
3. The run appears in the web app, the REST API, and the Model Context Protocol (MCP) endpoint at the same time. They’re three views of one record.
The full picture, including where each piece runs, is in [Architecture](/docs/architecture).
## Where your data shows up [#where-your-data-shows-up]
Four places in the web app, in the order most people use them:
* **Home**: every instrument’s recent runs together, with counts for today and this week
* **Instruments**: one instrument’s runs, with columns matching what that instrument records
* **My runs**: runs you’ve claimed, so you can find your own work without filtering
* **Watchers**: which instrument PCs are reporting in, for whoever maintains them
If a term on those pages is unfamiliar, [Concepts](/docs/concepts) defines the vocabulary in one page. Run, watcher, run status, and instrument type are the four worth reading first.
## Get at the same data from code [#get-at-the-same-data-from-code]
The web app has no private back door. Everything it shows comes from a REST API that you can call yourself, and an MCP endpoint that AI clients can call on your behalf. Both read the same records, so a script sees exactly what the dashboard sees.
* **REST**: served at `/api/v1/` on your own deployment host. Each deployment publishes its own OpenAPI 3.1 document at `/api/v1/openapi.json`, so you can generate a client rather than hand-write one. See the [API overview](/docs/api)
* **MCP**: served at `/mcp/v1` over Streamable HTTP, and added to Claude Code, Claude Desktop, or Cursor by pasting the URL. Agents can then search runs, read metadata, and fetch files during a conversation. See the [MCP overview](/docs/mcp)
The two differ in how you authenticate. REST clients carry a [personal access token](/docs/concepts#personal-access-token) that you create in the web app, scoped and revocable. MCP clients send you through a browser sign-in and a consent screen instead, then act with your own permissions, so nobody has to paste a credential into an AI client.
Typical uses look like pulling every run from an instrument for a batch reanalysis, writing the latest run’s archive to disk from a script, or asking an agent which plates from last week failed processing.
## What’s supported today [#whats-supported-today]
Seven instrument types have a processor, covering plate readers, gel docs, qPCR, TapeStation, FPLC, flatbed plate scanners, and ND2 microscopy. [Instrument data preprocessing](/docs/instrument-preprocessing) lists what each one extracts.
An instrument without a matching processor still works. Its files upload, and you can search, claim, comment on, and download them. What you don’t get is extracted metadata, badges, or rendered previews. Adding a new type is a development task, not a settings change.
## Self-hosted by design [#self-hosted-by-design]
Data Hub isn’t a service you sign up for. Your team runs it, which is the point: raw instrument data and everything derived from it stay on infrastructure you control, and the source is open under the MIT license.
Running it means four pieces, deployed once by an engineer:
* The web app, which also serves the API and the MCP endpoint, on Vercel
* A PostgreSQL database for instruments, runs, files, and comments
* AWS S3 for the files themselves, split into raw and processed buckets
* An AWS Lambda that does the per-instrument processing and builds download archives
A Google OAuth client, created during that deploy, is the fifth piece and the one worth knowing about even if you’ll never touch the stack. It decides who can sign in at all: only your Google Workspace organization, or any Google account. Nothing inside Data Hub overrides that choice, so it’s a deployment decision rather than a setting an admin can change later.
Until the stack exists, nobody can sign in and no watcher can upload. The deployment runbooks live in the repository rather than on this site; [Deploy Data Hub](/docs/self-hosting) points to them.
## Getting access [#getting-access]
Sign in with your Google account, assuming it’s one the deployment allows. Everyone who signs in can browse instruments, runs, and files. Admins can additionally confirm instruments, issue tokens, and promote other users, and only an existing admin can grant you that. See [Security and permissions](/docs/security#roles) for what each role can do.
## Start here [#start-here]
## Reading these docs with AI [#reading-these-docs-with-ai]
Every page has a **Copy Markdown** button and a **View Options** menu. Use them to:
* Copy a page as Markdown into an assistant
* Open raw Markdown (`/docs/.md`) or the dumps at [`/docs/llms.txt`](/docs/llms.txt) and [`/docs/llms-full.txt`](/docs/llms-full.txt)
# Retire an instrument (/docs/retire-an-instrument)
**For:** Administrators
Retire an instrument when it is decommissioned or out of service for a long stretch. Retirement sets status to `inactive` (shown as **Retired**), removes it from the sidebar and dashboard lists of live instruments, and stops new uploads. Nothing is deleted from storage; you can reactivate later.
## What retirement does [#what-retirement-does]
When you retire an instrument:
* Status becomes `inactive` (UI: **Retired**), with `retiredAt` / `retiredBy` audit fields
* It stops accepting new runs and watchers
* All existing runs and files stay browsable
* Every active watcher on that instrument is deregistered in the same request so they stop heartbeating
Instrument IDs stay permanent. Retire rather than trying to rename or delete an instrument that has already produced data.
## Retire from the web app [#retire-from-the-web-app]
Open the instrument’s detail page.
Open the admin actions menu and choose **Retire**.
Confirm in the dialog. The dialog lists run count and how many watchers will be deregistered.
The instrument moves to the **Retired** tab on **Instruments**.
## Reactivate [#reactivate]
From a retired instrument’s detail page, choose **Reactivate**. Status returns to `active` and the instrument reappears in the sidebar and dashboard.
Watchers deregistered at retirement do **not** reconnect automatically. An operator must re-register (usually `data-hub-watcher init` on the instrument PC) after reactivation.
## Via the API [#via-the-api]
```sh
# Retire
curl -X PATCH https://datahub.example.com/api/v1/instruments/bio-rad-cfx96 \
-H "Authorization: Bearer dhub_your_token_here" \
-H "Content-Type: application/json" \
-d '{"status":"inactive"}'
# Reactivate
curl -X PATCH https://datahub.example.com/api/v1/instruments/bio-rad-cfx96 \
-H "Authorization: Bearer dhub_your_token_here" \
-H "Content-Type: application/json" \
-d '{"status":"active"}'
```
Session callers need the admin role; bearer tokens need `instruments:write`.
# Security and permissions (/docs/security)
**For:** Administrators and developers
This page explains who can do what in Data Hub: the two roles, the scope system
that gates token-authenticated requests, and how to handle tokens safely. For the
mechanics of creating and revoking tokens, see [Issue and revoke tokens](/docs/manage-tokens).
## Authentication [#authentication]
The API accepts two kinds of credentials:
* **Session cookies**: used by the web dashboard, which signs people in with Google OAuth via [Better Auth](https://www.better-auth.com). Session callers implicitly hold every scope; scope checks only apply to token requests.
* **Bearer tokens**: personal access tokens (`dhub_…`) used by the watcher, the Lambda, and scripts, sent as `Authorization: Bearer `.
* **OAuth access tokens**: issued to MCP clients after you sign in and approve a consent screen. Covered in [MCP access](#mcp-access).
Tokens are hashed with SHA-256 before storage; the plaintext is shown exactly once, at creation.
### Who can sign in [#who-can-sign-in]
Any Google account that finishes the sign-in flow becomes a member of your deployment. Data Hub itself checks nothing: there is no invite flow, no email allowlist, and no email-domain check, so a first successful sign-in creates the member record.
Restrictions live on the Google side instead, in the OAuth client your team creates at deploy time. A consent screen set to **Internal** accepts only accounts in your Google Workspace organization. Set to **External**, it accepts any Google account, including personal ones. Engineers make that choice when they [create the OAuth client](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md#create-a-google-oauth-client).
Data Hub accepts 10 Google sign-in attempts per minute from one IP address and rejects the rest.
Every new member starts in the member role, never admin. See [First admin bootstrap](/docs/manage-members#first-admin-bootstrap) for how the first admin gets the role.
### Session lifetime [#session-lifetime]
Signing in stores a session in the database and sets a signed cookie. The session expires 7 days after its last refresh, and the dashboard refreshes it at most once a day, so a session in daily use doesn’t lapse on its own.
That cookie also carries a signed copy of the session for 5 minutes, which keeps most page loads off the database. Requests read the copy instead of the row. A change made in the database directly, such as deleting a session, therefore applies when the copy expires rather than on the next request.
Revoking someone’s access on the Google side, by removing them from your Workspace organization for example, blocks their next sign-in but leaves any session already open working until it expires. Delete that user’s rows from the `session` table to cut them off sooner. Signing out from the sidebar ends the session immediately.
## Roles [#roles]
| Role | Can do |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Member** | Sign in to the dashboard; view instruments, runs, files, and the token audit list. Cannot mint or delete tokens, confirm/update instruments, or manage other users. |
| **Admin** | Everything members can, plus: confirm and update instruments, create and revoke any user’s tokens, and promote/demote other users. |
The first admin is bootstrapped from the `ADMIN_EMAILS` environment variable
(comma-separated, case-insensitive); listed users are promoted on every sign-in.
After that, any admin can promote others from **Settings → Members**. Admins
cannot demote themselves: that guards against locking the workspace out of its
last admin.
### Admin-gated operations [#admin-gated-operations]
A subset of mutations require the admin role in addition to (or instead of) a scope
check:
* `PATCH /api/v1/instruments/:id`: session callers must be admin (bearer-token automation still uses `instruments:write`, so watcher/Lambda flows are unaffected).
* `POST /api/v1/tokens` and `DELETE /api/v1/tokens/:id`: admin-only, session-only. Bearer tokens cannot manage other tokens.
* `GET /api/v1/users`, `PATCH /api/v1/users/:userId`: admin-only, session-only. Used by **Settings → Members**.
## Token scopes [#token-scopes]
Every personal access token carries an array of permission **scopes**. A token-authenticated request is rejected with `403 FORBIDDEN` when the token’s scopes don’t include the scope a route requires. Issue tokens with the least privilege they need. New tokens must list fine-grained scopes; `POST /api/v1/tokens` rejects the wildcard `*` and the older coarse scopes `runs:write`, `files:write`, and `watchers:write`. Tokens that already carry those still work until you rotate them.
| Scope | Grants |
| -------------------- | ---------------------------------------------------------------------------------- |
| `instruments:read` | Read instruments, dashboard status, and file patterns |
| `instruments:write` | Create instruments and edit their configuration |
| `runs:read` | Read and search runs, their files, comments, and attributions |
| `runs:create` | Create run records |
| `runs:update` | Update run metadata |
| `runs:delete` | Soft-delete and restore runs |
| `runs:reprocess` | Re-run the processing workflow for a run |
| `runs:upload` | Request presigned S3 URLs to upload files to a run |
| `runs:attribute` | Claim or unclaim runs |
| `runs:comment` | Add, edit, and delete run comments |
| `files:read` | Read file metadata and download files and run archives |
| `files:create` | Register file records against a run |
| `files:update` | Update file metadata and upload state |
| `files:delete` | Delete files |
| `files:reprocess` | Re-run the processing workflow for a file |
| `watchers:read` | Read watcher status, heartbeats, and upload queues |
| `watchers:report` | Register, heartbeat, send events, and push watcher config |
| `watchers:admin` | Delete watchers |
| `archive-jobs:read` | Read run-archive job status |
| `archive-jobs:write` | Update run-archive job status (Lambda callback) |
| `*` | Matches every scope. Reserved for legacy backfilled tokens; rejected on new tokens |
A watcher token typically needs `instruments:read`, `instruments:write`, `watchers:read`, `watchers:report`, `runs:create`, `runs:update`, `runs:upload`, and `files:update`. That is what the **Watcher** preset selects. A read-only analysis token needs the `:read` scopes for the data you want to query.
## MCP access [#mcp-access]
The MCP server at `/mcp/v1` doesn’t accept personal access tokens. An AI client runs an OAuth flow instead: you sign in to Data Hub, approve a consent screen that names the client and the access it wants, and the client receives an access token valid for that deployment’s MCP endpoint alone. A token issued to a client with no person behind it is rejected, so every MCP action attributes to a user.
MCP recognizes two scopes rather than the `:` pairs above:
* `read`: required to connect, and covers every read-only tool, resource, and prompt
* `write`: additionally required by any tool that changes data, such as claiming a run or deleting one
A grant belongs to one client and one person, and it carries that person’s own permissions. Clients register themselves at connect time, which creates an OAuth client record but grants nothing on its own: Google sign-in still decides who gets through, and the consent screen still decides what they hand over.
The dashboard has no screen for reviewing or revoking MCP grants yet. Removing someone from your Google Workspace organization blocks new grants; cutting off a client that already holds a token means deleting the matching rows from the `oauthAccessToken` and `oauthRefreshToken` tables.
Client setup is in [MCP overview](/docs/mcp).
## Token hygiene [#token-hygiene]
* **Name tokens descriptively** (e.g. “FPLC watcher, Lab 201”) so you can tell what each one is for.
* **Set expirations** for temporary setups.
* **Scope to least privilege**: don’t hand out broad tokens for a single read-only integration.
* **Revoke immediately** when a watcher is decommissioned or a token may have been exposed. Revocation is instant; clients using the token start getting `401 Unauthorized`. See [After revoking a token](/docs/manage-tokens#after-revoking-a-token).
## Public pages, gated bodies [#public-pages-gated-bodies]
Some web pages (dashboard, instruments, run detail, settings) are reachable without
a session so link unfurlers can read page metadata. The page body renders a
sign-in prompt instead of real data when there’s no session, and three independent
layers (`robots` metadata, `robots.txt`, and an `X-Robots-Tag` header) keep the
product app out of search indexes. The `/api/v1/*` surface always requires a
session cookie or bearer token.
# Deploy Data Hub (/docs/self-hosting)
**For:** Engineers
Data Hub is self-hosted, so nobody can sign in or install a watcher until your team has stood up the backend. This page is orientation only: what gets deployed, what access you need, and in what order. It doesn’t walk through the commands.
For the actual deploy instructions, open [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md) in the data-hub repository. That guide has every command, env var, and checklist. Read this page first if you want the map before you start.
## What you’ll deploy [#what-youll-deploy]
Three pieces, each on infrastructure you choose:
| Piece | What it does | Where it runs |
| ---------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------- |
| PostgreSQL database | Stores instruments, runs, files, watchers, and tokens | Any PostgreSQL host, managed or self-run |
| Web app | Serves the dashboard, the REST API, and the MCP endpoint | Vercel |
| Storage and processing | S3 buckets for raw, processed, and archived files, plus the Lambda that processes uploads | AWS, deployed with AWS SAM |
You’ll also create a Google OAuth client, which is what actually controls who can reach your deployment. Data Hub has no invite flow and no email allowlist of its own, so that client’s setting is the whole access policy. Decide it deliberately and read [Who can sign in](/docs/security#who-can-sign-in) first.
Give `staging` and `production` separate databases and separate AWS stacks. They’re independent deployments holding independent data.
## Access you need before you start [#access-you-need-before-you-start]
The deploy stalls without all four, so confirm them up front:
* **Admin-level AWS credentials.** The first deploy creates the stack, which the automated deploy role can’t do. It can only update a stack that already exists
* **Vercel project access**, including permission to set environment variables for the environment
* **Google Cloud Console access** to create the OAuth client
* **GitHub repository settings access**, to store the secrets that let continuous integration handle every deploy after this one
You’ll also need the AWS CLI and the AWS SAM CLI installed locally.
## How the deploy is sequenced [#how-the-deploy-is-sequenced]
The web app and the AWS stack each need something the other produces, so the order matters. This is the sequence only; [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md) has the commands for each step:
**Create the database** and keep its connection string. Make sure the host accepts connections from Vercel and from your continuous integration runners.
**Deploy the web app**, set its environment variables, apply the database migrations, then sign in and create the access token the Lambda will use.
**Bootstrap AWS resources**, once per AWS account rather than once per environment. This creates the shared resources both environments draw on.
**Deploy the AWS stack** for this environment, using the web app’s URL and that token.
**Wire the AWS outputs back into the web app**, then redeploy so it picks them up.
Expect the web app to deploy cleanly at step 2 and still not work. Anything touching files, uploads, reprocessing, and run archives stays broken until step 5 connects it to AWS. A green deployment is not the finish line.
## Things that catch people out [#things-that-catch-people-out]
Six failures account for most lost time on a first deploy:
* **Set `ADMIN_EMAILS` before you sign in for the first time.** It’s what promotes the first admin. Sign in before setting it and you get an account with no permission to create tokens or confirm instruments
* **Point `BETTER_AUTH_URL` at this deployment’s own public origin.** Data Hub builds the Google callback URL from it, so a value copied from another environment sends people to that other host and sign-in fails. AI clients read the same value when they sign in over MCP, so a wrong one breaks them too
* **Google sign-in on Vercel preview deployments needs `OAUTH_PROXY_URL` and `OAUTH_PROXY_SECRET`.** Google matches each redirect URI exactly and every preview gets a new URL, so previews route the callback through one stable deployment whose URI is registered. Production registers its own and leaves both variables unset
* **Give the Lambda’s token the `instruments:read` scope.** The Lambda checks an instrument’s type before choosing a processor, so without that scope every single upload fails on a permission error rather than an obvious misconfiguration
* **Bootstrapping fails if the AWS account already has a GitHub identity provider** from an unrelated project. Reuse the existing one instead of creating a second
* **Slack notifications aren’t an environment variable.** A workspace admin pastes the webhook URL into Settings after the deploy, so don’t hunt for a variable that doesn’t exist
## What finished looks like [#what-finished-looks-like]
Operators can install watchers and instruments start uploading. Hand them [Set up an instrument](/docs/set-up-an-instrument), and create their first tokens through [Issue and revoke tokens](/docs/manage-tokens).
From then on, deploys are automatic. Merges deploy the web app, apply migrations, and ship the Lambda, so this manual sequence runs once per environment.
## The runbooks [#the-runbooks]
The deploy commands live next to the code in the data-hub repository. Start with [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md):
* [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/first-time-deployment.md): the full sequence above, with every command
* [Continuous integration and deployment](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/ci-and-deployment.md): how later deploys, migrations, and watcher releases happen
* [Local development](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/local-development.md): run the app on a laptop, including a setup that needs no AWS or Google credentials
* [Architecture for contributors](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/architecture.md): internals
For a product-level map of the components and how data moves between them, rather than deploy steps, see [Architecture](/docs/architecture).
# Set up an instrument (/docs/set-up-an-instrument)
**For:** Lab operators and admins
This walkthrough takes you from nothing to an uploading instrument: create an access token, install the watcher, register the instrument, get an admin to confirm it, then start watching. CLI flags, the full config schema, and Windows service setup each have their own page; this guide links them once.
Data Hub is self-hosted. The web app, PostgreSQL database, and AWS stack (S3 plus the processing Lambda) must already be running. If they are not, an engineer needs the [Deploy Data Hub](/docs/self-hosting) guide first.
## Before you start [#before-you-start]
You need:
* **The watch directory**: the folder your instrument writes output into.
* **File patterns**: globs to upload, for example `*.csv`, `*.xlsx`, `*.tiff`.
* **[uv](https://docs.astral.sh/uv/getting-started/installation/)** installed on the instrument PC (it manages Python for you).
To make setup and maintenance easier, we recommend setting up remote access to the instrument PC via [Windows Remote Desktop](https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/remotepc/remote-desktop-allow-access).
## 1. Create an access token [#1-create-an-access-token]
The watcher authenticates to Data Hub with a personal access token. Only workspace admins can create tokens, so ask an admin if you are not one.
In the Data Hub web app, go to **Settings → Access Tokens** and click **Create token**.
Give the token a **Name** that identifies the instrument PC, for example `Plate Reader PC`.
Set **Expiration** to “No expiry”. (The token can always be revoked if necessary.)
Under **Scopes**, click the **Watcher** preset. It grants only what a watcher needs: registering, reporting runs, and uploading files.
Click **Create token**, then copy the token. It starts with `dhub_` and is shown only once.
Keep the token handy: you paste it into the setup wizard in step 3. For rotation, revocation, and the full scope list, see [Issue and revoke tokens](/docs/manage-tokens).
## 2. Install the watcher [#2-install-the-watcher]
On the instrument PC, install the watcher with `uv`. The `windows-service` extra adds `pywin32` so you can follow [Run as a Windows service](/docs/windows-service), which is the default on a lab PC:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
On macOS or Linux, omit the extra:
```sh
uv tool install data-hub-watcher
```
Full install, verify, and update steps live in [Install and update the watcher](/docs/install-the-watcher).
## 3. Run the setup wizard [#3-run-the-setup-wizard]
Start the wizard with `data-hub-watcher init`. It prompts you for one thing at a time:
Choose the environment. For a live lab, this is production.
Enter your Data Hub API URL e.g. `https://datahub.your-domain.com/api/v1`.
Paste the token from step 1 when it asks for the API key. The wizard checks the key, then lists the instruments already registered.
Pick your instrument from the list, or choose **Register a new instrument** if it is not there.
For a new instrument, enter an ID and a display name. The ID is a short, lowercase, hyphenated name like `bio-rad-cfx96`. **It cannot be changed later.**
Enter the full path to the folder your instrument writes into.
List which files to upload, such as `*.pcrd,*.csv`. The watcher ignores files that do not match.
Choose how files get grouped into runs. Pick the preset that matches how your instrument names its output, or write your own. The wizard previews a few matches so you can check the run IDs. For example, the default preset uses the filename prefix before the first underscore, so `RUN001_quant.csv` and `RUN001_melt.csv` both group into run `RUN001`.
Set the stability period. This is how much time (in seconds) a file must stay unchanged before the watcher counts it as finished. The default is 5 seconds; raise it for instruments that write large files slowly.
Choose the upload mode. “Auto” uploads each file as soon as it is stable; “Manual” waits for your approval first.
For run detection options, see [Configure run detection and uploads](/docs/configure-run-detection); [Watcher configuration](/docs/configuration-reference#run-detection) lists every preset.
The wizard then registers the watcher, saves the config to `~/.data-hub/config.yaml` (`C:\Users\\data-hub\config.yaml` on Windows), and syncs it to Data Hub. When it reports that setup is complete, start watching.
## 4. Confirm the instrument’s registration [#4-confirm-the-instruments-registration]
A pending instrument cannot upload to Data Hub until an admin confirms it from the web app.
Open the Data Hub web app and navigate to **Instruments > View all** from the app sidebar.
Select the “Pending” tab and find your instrument. It should show a yellow `pending` badge.
Click **Confirm**. The instrument should move to the “Active” tab and appear as online shortly.
Some instruments also get automated preprocessing (metadata extraction, image conversion, and related outputs). To choose a processor, open the three-dot menu, select **Edit**, and pick a supported instrument type. What each type produces is in [Instrument data preprocessing](/docs/instrument-preprocessing).
Adding a new processor requires code changes; see [Adding a new instrument](https://github.com/Arcadia-Science/data-hub/blob/staging/developer-docs/lambda.md#adding-a-new-instrument) in the developer docs.
See [Manage instruments](/docs/manage-instruments) for more info.
## 5. Start the watcher service [#5-start-the-watcher-service]
First, preview what the watcher will do without starting it:
```sh
data-hub-watcher watch --dry-run
```
This validates config, checks that the API is reachable and the instrument is active, and previews which files would be picked up.
When it looks right, install and start the watcher as a Windows service. Run these from an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
The service starts on boot, keeps running after you log out, and auto-updates in the background. Lifecycle and log details are in [Run as a Windows service](/docs/windows-service).
On macOS or Linux, run the watcher in the foreground instead:
```sh
data-hub-watcher watch
```
The watcher detects new files, groups them into runs, uploads them (in auto mode), and sends a heartbeat every 60 seconds. Runs appear in the dashboard after upload.
## What you get once it is running [#what-you-get-once-it-is-running]
* Raw files upload to cloud storage
* Instrument runs and files appear in the web dashboard
* Watcher service automatically restarts on PC restart
* Watcher health (heartbeats, events) appears under **Watchers**
* Subscribed users can get in-app and Slack notifications for new runs
# Troubleshoot a watcher (/docs/troubleshoot-a-watcher)
**For:** Lab operators and admins
Find your symptom below. When you are stuck, the [logs](#where-are-the-logs) are the fastest way in. Admins can also inspect heartbeats and events under **Watchers** in the web app.
## Setup and registration [#setup-and-registration]
### “Instrument is still pending activation” [#instrument-is-still-pending-activation]
The instrument was registered but an admin has not confirmed it. Ask an admin to open **Instruments** and click **Confirm**. See [Manage instruments](/docs/manage-instruments).
### “Instrument already has an active watcher” [#instrument-already-has-an-active-watcher]
Each instrument can have at most one active watcher. An earlier install (often on another PC, or before a reimage) is still registered. The CLI prints the existing watcher id; deregister it before re-running `init`:
* **Web app**: **Watchers**, open the existing watcher, click **Deregister**
* **API**: `DELETE /api/v1/watchers/` with a bearer token that has `watchers:admin`
Deregistration is a soft-delete: history stays under **Watchers → Deregistered**.
### `init` rejects my API key [#init-rejects-my-api-key]
The wizard rejects values that do not start with `dhub_`, and strips zero-width and non-breaking-space characters that Windows clipboards inject. If a paste keeps failing, pass `--show-key` so you can see what the prompt received.
## The watcher will not start [#the-watcher-will-not-start]
* Run `data-hub-watcher config show` to confirm config and environment
* Run `data-hub-watcher watch --dry-run`: it prints the first failing check
* Confirm the instrument is `active`, not `pending`
* On Windows, if the **service** will not start, see [Windows service](#windows-service)
## “Connection error” or “Request timed out” [#connection-error-or-request-timed-out]
The watcher cannot reach the Data Hub API. Check:
* Network connectivity from the instrument PC
* Environment in config (`staging`, `production`, or `preview`): `data-hub-watcher config show`
* That the API base URL for that environment is reachable
## Files are not being detected [#files-are-not-being-detected]
* Confirm the watch directory: `data-hub-watcher config show`
* Check file patterns. Globs are case-sensitive on some systems (`*.csv` does not match `data.CSV`)
* If files live in subfolders, set run detection to recursive; see [Configure run detection and uploads](/docs/configure-run-detection)
* Run `data-hub-watcher watch --dry-run` to preview matches
* Look for `pattern_mismatch` events on the dashboard
## Files are detected but not uploading [#files-are-detected-but-not-uploading]
* In **manual** mode, files wait for approval in the upload queue; see [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads)
* Check the log for `upload_failed`
* A revoked or expired token surfaces as `401 Unauthorized`. Create a new token and re-run `init`; see [Issue and revoke tokens](/docs/manage-tokens#after-revoking-a-token)
## Auto-update never fires [#auto-update-never-fires]
Check, in order:
* Prefer a long-running service (or scheduled) install; a one-off console session may not stay up for the hourly tick
* Environment is not `preview`: auto-update is hard-disabled there
* Activity-window guard is not holding it up: run `data-hub-watcher self-update`, or ask an admin to flag the release **mandatory**
* **Last Heartbeat** on the dashboard is recent; a stale watcher is not ticking
Upgrade failure modes (`update_started` → `update_failed`, editable installs, missing Scheduled Task) are covered in [Install and update the watcher](/docs/install-the-watcher#manual-update-self-update) and [Roll out watcher releases](/docs/watcher-releases).
## Windows service [#windows-service]
### The service exits immediately [#the-service-exits-immediately]
If it stops before writing to `watcher.log`:
* Check `C:\ProgramData\DataHubWatcher\service-bootstrap.log` for a pre-dispatcher crash (missing `pywin32`, moved venv, corrupt install)
* Query the Windows Application event log:
```powershell
Get-WinEvent -FilterHashtable @{LogName='Application'; StartTime=(Get-Date).AddHours(-1)} |
Where-Object { $_.ProviderName -match 'Python|DataHubWatcher' -or $_.Message -match 'DataHubWatcher' } |
Format-List TimeCreated, ProviderName, Id, LevelDisplayName, Message
```
* Run the service in the foreground to see the traceback:
```powershell
& "C:\path\to\venv\Scripts\python.exe" -m win32serviceutil debug DataHubWatcher
```
### `self-update` says “Upgrade dispatched” but nothing happens [#self-update-says-upgrade-dispatched-but-nothing-happens]
On Windows uv-tool installs, the upgrade runs out of process under SYSTEM and takes 30–60s. Track it:
```powershell
Get-Service data-hub-watcher
Get-Content ~/.data-hub/upgrade-worker.log -Tail 20 -Wait
```
### Turn on debug logging for the service [#turn-on-debug-logging-for-the-service]
Add `DATA_HUB_WATCHER_LOG_LEVEL=DEBUG` to the env file the service loads (typically `~/.data-hub/.env.`) and restart the service. No reinstall needed.
## Where are the logs? [#where-are-the-logs]
Rotating logs (10 MB each, 5 backups). CLI `watch` and the Windows service share one file:
| Platform | Log file |
| ------------- | ------------------------------------------- |
| Windows | `C:\ProgramData\DataHubWatcher\watcher.log` |
| macOS / Linux | `~/.data-hub/watcher.log` |
Add `--verbose` for debug console output: `data-hub-watcher --verbose watch`. Windows uv-tool upgrade output lands in `~/.data-hub/upgrade-worker.log`.
## API clients get `401 Unauthorized` [#api-clients-get-401-unauthorized]
The bearer token is missing, mistyped, revoked, or expired. Check **Settings → Access Tokens** and re-issue if needed. See [Issue and revoke tokens](/docs/manage-tokens).
## MCP clients get `401 Unauthorized` [#mcp-clients-get-401-unauthorized]
MCP doesn’t accept personal access tokens, so a client configured with `Authorization: Bearer dhub_…` gets a 401 on every call. Remove the header, leave only the URL, and let the client run its sign-in flow. Setup per client is in [MCP overview](/docs/mcp#connect-a-client).
An already-connected client that starts failing has usually lost its grant: sign in again from the client. If reads work and one tool fails with a missing-scope message, the client holds `read` but not `write`; reconnect and approve both.
## Ask an agent to diagnose it [#ask-an-agent-to-diagnose-it]
Use the `troubleshoot_instrument` MCP prompt under [MCP prompts](/docs/mcp/prompts).
# Roll out watcher releases (/docs/watcher-releases)
**For:** Administrators
Lab PCs upgrade themselves toward the version you advertise per environment. You control that from **Settings → Watchers**. How an individual PC installs or runs `self-update` is in [Install and update the watcher](/docs/install-the-watcher).
## How fleet upgrades reach a PC [#how-fleet-upgrades-reach-a-pc]
The in-process updater runs from the heartbeat tick on every platform (not only Windows services). Roughly once an hour the watcher:
1. Asks the server for the latest version and compares it to its own
2. Upgrades only if a newer version is available, no files have uploaded for several heartbeats, and no run was reported recently (the activity-window guard). Releases flagged **mandatory** skip this guard
3. Installs the new wheel and restarts into it (Windows services restart via failure-actions policy)
4. Emits `update_started`, then `update_succeeded` or `update_failed`, visible on the dashboard
Auto-update is disabled in the `preview` environment so pull-request preview deployments cannot push code to lab PCs.
## Cut a new release [#cut-a-new-release]
Releases are tag-driven and publish from a `production` commit.
Bump the version in `watcher/pyproject.toml` and merge through `staging` to `production`.
Tag and push from `production`:
```sh
git checkout production && git pull
git tag watcher-v0.3.0 && git push origin watcher-v0.3.0
```
Approve the `pypi` deployment under **Actions → Publish watcher** in GitHub.
In **Settings → Watchers**, set **Latest version** to the new tag. Each environment has its own database, so bump `staging` and `production` separately to roll the fleet gradually.
Always tag → publish → verify on PyPI → save. Saving **Latest version** before the wheel is live triggers a wave of `update_failed` events across the fleet.
## Release-config fields [#release-config-fields]
| Field | Purpose |
| ----------------------------- | ------------------------------------------------------- |
| **Latest version** | Required. Blank means “no update info available”. |
| **Minimum supported version** | Reserved; not enforced server-side yet. |
| **Release channel** | Defaults to `stable`. Surfaced in `self-update` output. |
| **Mandatory update** | Skips the activity-window guard; see below. |
## Mandatory updates [#mandatory-updates]
Toggling **Mandatory update** skips the activity-window guard and fires the upgrade on the next hourly check. The server still compares the running version to **Latest version**, so a correctly pinned PC is not forced past its pin.
Use sparingly: a forced upgrade mid-acquisition can lose data. Reserve it for security fixes, wire-protocol breaks, or cases where leaving the bad version running is worse than restarting in flight.
## Roll back [#roll-back]
Rollback is another release: set **Latest version** back to the older tag for the affected environment(s), turn **Mandatory update** on if you need to bypass the activity-window guard, and wait for the next hourly tick. Turn **Mandatory update** off once the fleet converges. The older wheel remains on PyPI; it is no longer advertised.
## Operator-side failures [#operator-side-failures]
If `update_started` is followed by `update_failed`, or auto-update never fires, point operators at [Troubleshoot a watcher](/docs/troubleshoot-a-watcher#auto-update-never-fires) and [Install and update the watcher](/docs/install-the-watcher#manual-update-self-update).
# Run as a Windows service (/docs/windows-service)
**For:** Lab operators
On a lab instrument PC, run the watcher as a Windows service. This is the default setup: the service starts on boot, keeps running after you log out, restarts on failure, and participates in [fleet auto-update](/docs/watcher-releases). Foreground `data-hub-watcher watch` is for quick checks and non-Windows hosts.
## Install the service extra [#install-the-service-extra]
The service depends on `pywin32`, which ships in the `windows-service` extra. The [default install](/docs/install-the-watcher#install) already includes it. If you installed without it, add it now:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
Run [`data-hub-watcher init`](/docs/install-the-watcher#configure) first if you have not configured the watcher yet. The service uses the same `~/.data-hub/config.yaml` and per-environment `.env` file.
## Install and start [#install-and-start]
From an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
`service install` registers delayed auto-start (so it does not race the network stack at boot), configures restart-on-failure recovery, and registers the `DataHubWatcherUpgrade` Scheduled Task used by the auto-updater.
The Windows auto-update path requires that `service install` (or `service reinstall`) has been run from an Administrator shell at least once. Fleet PCs upgrading into an auto-update-capable build for the first time need a one-time `service reinstall` to pick up the task.
## Lifecycle commands [#lifecycle-commands]
| Command | What it does |
| ------------------------------------ | --------------------------------------------- |
| `data-hub-watcher service install` | Register the service |
| `data-hub-watcher service start` | Start the service |
| `data-hub-watcher service stop` | Stop the service |
| `data-hub-watcher service status` | Show whether the service is running |
| `data-hub-watcher service uninstall` | Remove the service |
| `data-hub-watcher service reinstall` | Stop, uninstall, install, and start in one go |
`service reinstall` is the right command after an out-of-band wheel swap (for example a manual `uv tool install -U` from an Administrator shell). Stop and uninstall are best-effort, so it works even if the service is already gone.
`service install` and `service reinstall` accept `--env-path PATH` to override which `.env` file the service loads (defaults to `~/.data-hub/.env.`).
## Do not run `watch` alongside the service [#do-not-run-watch-alongside-the-service]
Running `data-hub-watcher watch` interactively while the service is also running is not supported: both processes would race on the same log file and watch directory. Stop the service first if you need the CLI for debugging:
```powershell
data-hub-watcher service stop
data-hub-watcher --verbose watch
data-hub-watcher service start
```
## Where the service writes logs [#where-the-service-writes-logs]
The service and the CLI share one rotating log:
* `C:\ProgramData\DataHubWatcher\watcher.log`
A separate `service-bootstrap.log` next to it captures crashes before the service control dispatcher takes over (missing `pywin32`, moved venv, corrupt install). To turn on debug logging without a reinstall, add `DATA_HUB_WATCHER_LOG_LEVEL=DEBUG` to the env file the service loads and restart the service.
# Overview (/docs/api)
**For:** Developers and integrators
The Data Hub API is served by the web application at `/api/v1/`. The watcher, Lambda, and dashboard all call it. Data Hub is self-hosted, so the base URL is your deployment host plus `/api/v1`. Examples use `https://datahub.example.com`; substitute your host.
Per-endpoint request and response shapes live in the generated pages under this section (same sidebar). Each deployment also publishes OpenAPI 3.1 at `/api/v1/openapi.json`.
## Authentication [#authentication]
Two methods (full model in [Security and permissions](/docs/security)):
* **Session cookies**: web dashboard (Google OAuth). Session callers implicitly hold every scope.
* **Bearer tokens**: `Authorization: Bearer dhub_…` for the watcher, the Lambda, and scripts
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
Every token request needs a [scope](/docs/security#token-scopes). A missing scope returns `403 FORBIDDEN`. Some mutations also require the admin role; see [Admin-gated operations](/docs/security#admin-gated-operations).
Mint tokens in the UI: [Issue and revoke tokens](/docs/manage-tokens).
## First request [#first-request]
List instruments:
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
Fetch the OpenAPI document (no auth required):
```sh
curl https://datahub.example.com/api/v1/openapi.json
```
The document covers the integrator surface (instruments, runs, files, watchers, archive jobs, search). Session-only admin routes (`/tokens`, `/users`, `/notifications`, `/settings/*`) and the MCP transport are documented elsewhere.
## Pagination and filtering [#pagination-and-filtering]
List endpoints that return pages (notably run search) accept:
* `page`: 1-based page index
* `per_page`: page size, 1 to 100
Responses include pagination metadata such as `page`, `per_page`, `total`, and `total_pages`.
Run report items page by index instead, with `offset` and `limit` (max 200). Their responses return `offset`, `limit`, and `total`.
Run search also supports filters (instrument, date range, attribution via `ranBy`, and instrument-specific metadata). Exact query parameters are on each generated operation page in the sidebar under this **API** section.
## Error responses [#error-responses]
Errors use a nested envelope:
```json
{
"error": {
"code": "NOT_FOUND",
"message": "Instrument not found.",
"details": null
}
}
```
Common codes: `VALIDATION_ERROR` (400), `UNAUTHORIZED` (401), `FORBIDDEN` (403), `NOT_FOUND` (404), `CONFLICT` (409). Scope failures use `code: "FORBIDDEN"` with a message naming the missing scope.
## MCP [#mcp]
`GET` and `POST` `/mcp/v1` expose the MCP server over Streamable HTTP, outside the `/api/v1` tree. Clients authenticate with OAuth rather than a personal access token, and hold `read` to connect plus `write` to call any tool that changes data. See [MCP overview](/docs/mcp).
# Overview (/docs/mcp)
**For:** Developers and integrators
Data Hub exposes a Model Context Protocol (MCP) server at `/mcp/v1` so AI clients can query instruments, runs, files, and watcher health. Clients authenticate by sending you through a browser sign-in.
Catalogs for [tools](/docs/mcp/tools), [prompts](/docs/mcp/prompts), and [resources](/docs/mcp/resources) are generated from the live server schema, so they match what your client discovers when it connects.
## Authentication [#authentication]
The MCP server authenticates with OAuth. All you need to provide is the endpoint URL; for example, `https://datahub.example.com/mcp/v1`.
The first connection opens your browser:
Sign in to Data Hub with Google, unless you already have a session open.
Read the **Authorize application** screen. It names the client, the host it redirects to, and the access it’s asking for.
Click **Allow**.
Your client stores the access token it receives and renews it without asking again. **Allow** grants everything on the screen: there are no per-permission checkboxes, so click **Deny** if any line looks wrong.
### Scopes [#scopes]
MCP uses two scopes rather than the `:` scopes that gate REST requests:
* **`read`**: required to connect at all, and covers every read-only tool, resource, and prompt
* **`write`**: additionally required by tools that change something, including claiming runs, commenting, requesting uploads, reprocessing, and deleting
Most clients ask for both, because the server advertises both. A client granted only `read` still connects and still reads; its first mutating tool call fails with a message naming the missing scope.
Each entry in the [tools catalog](/docs/mcp/tools) lists the scope its REST equivalent requires, which tells you what data the tool touches. Over MCP, only `read` and `write` are enforced.
## Connect a client [#connect-a-client]
Examples use `https://datahub.example.com`; substitute your deployment host.
### Claude Code [#claude-code]
Register the server with [`claude mcp add`](https://code.claude.com/docs/en/mcp-quickstart). Run this in your terminal, not inside a `claude` session:
```bash
claude mcp add --transport http --scope user data-hub https://datahub.example.com/mcp/v1
```
Confirm that the server was added by running `claude mcp list`. Then, start a session with `claude` and run `/mcp` to authenticate.
### Claude Desktop [#claude-desktop]
Remote servers go through **Settings → Connectors → Add custom connector**. The `claude_desktop_config.json` file handles local servers only. Paste `https://datahub.example.com/mcp/v1` as the URL, click **Add**, then click **Connect** to sign in. Leave the OAuth client fields under **Advanced settings** empty, since Data Hub registers clients on demand.
Claude reaches your server from Anthropic’s infrastructure rather than from your own machine, so the deployment has to be reachable from the public internet. A deployment running on `localhost` can’t be added this way; use Claude Code or Cursor for local work.
### Cursor [#cursor]
Add the server to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for every project:
```json
{
"mcpServers": {
"data-hub": {
"url": "https://datahub.example.com/mcp/v1"
}
}
}
```
Open **Settings → Tools & MCP**. The server appears with a **Needs authentication** label; click it to start the browser flow. The tools show up once you approve the grant.
### Other clients [#other-clients]
Any client that speaks Streamable HTTP and the MCP authorization spec needs one setting, the URL `https://datahub.example.com/mcp/v1`. For local development, point at `http://localhost:3000/mcp/v1` instead.
Clients that speak MCP but not OAuth can bridge through [`mcp-remote`](https://www.npmjs.com/package/mcp-remote), which runs the browser flow on their behalf.
## Machine-readable catalog [#machine-readable-catalog]
Fetch the same catalog these pages render from:
```bash
curl https://datahub.example.com/mcp/v1/schema.json
```
It lists every tool, prompt, and resource, with JSON Schema for arguments and each tool's response. The schema document needs no authentication.
# Prompts (/docs/mcp/prompts)
**For:** Developers and integrators
Prompts are scripted workflows the client surfaces to the user. Each prompt assembles a multi-step instruction that the model then executes using MCP tools. Argument schemas below match what the server advertises over the protocol.
See [MCP overview](/docs/mcp) for authentication and client setup.
### `daily_summary`
Summarize all instrument activity for a given day, including run counts, failures, and system health.
**Arguments**
- `date` (optional, `string`) — Date to summarize (YYYY-MM-DD, UTC calendar day). Defaults to the current UTC day.
### `troubleshoot_instrument`
Diagnose connectivity or processing issues for an instrument by inspecting its status and watcher health.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier to troubleshoot
### `compare_runs`
Compare the results of two runs side by side, highlighting differences in experimental outcomes.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier (both runs must be on the same instrument)
- `runId1` (required, `string`) — First run identifier
- `runId2` (required, `string`) — Second run identifier
### `find_my_runs`
List runs claimed by the authenticated user, optionally scoped to an instrument and date range.
**Arguments**
- `instrumentId` (optional, `string`) — Optional instrument to narrow results
- `dateFrom` (optional, `string`) — Start date (YYYY-MM-DD, UTC), inclusive
- `dateTo` (optional, `string`) — End date (YYYY-MM-DD, UTC), inclusive
### `explain_failed_run`
Diagnose why a run failed and suggest reprocess or upload fixes.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `runId` (required, `string`) — Run identifier within the instrument
### `claim_unattributed_runs`
Find unattributed runs on an instrument and claim them for the authenticated user after confirmation.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `dateFrom` (optional, `string`) — Optional start date (YYYY-MM-DD, UTC)
- `dateTo` (optional, `string`) — Optional end date (YYYY-MM-DD, UTC)
### `summarize_instrument_week`
Summarize one instrument's activity over the last seven days (or a provided window).
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `dateFrom` (optional, `string`) — Start date (YYYY-MM-DD, UTC). Defaults to 7 days ago (UTC).
- `dateTo` (optional, `string`) — End date (YYYY-MM-DD, UTC). Defaults to today (UTC).
# Resources (/docs/mcp/resources)
**For:** Developers and integrators
Resources are reference context that clients can attach to prompts without an explicit tool call. URI templates (for example filter-options) are listed alongside static URIs.
See [MCP overview](/docs/mcp) for authentication and client setup.
### `instruments`
List of all instrument IDs, display names, and types. Use as reference context when constructing tool calls.
*application/json*
**URI**
`datahub://instruments`
### `me`
Authenticated user's identity (id, name, email, image, isAdmin). Same payload as the get_me tool.
*application/json*
**URI**
`datahub://me`
### `glossary`
Static reference: run status derivation, instrument types, ranBy literals, UTC date semantics, and archive polling.
*application/json*
**URI**
`datahub://glossary`
### `instrument-filter-options`
Available filter values for an instrument. Values map directly to search_runs metadata filter arguments (wavelength/measurementMode/measurementType for plate readers; captureType/imagingMode/gelWavelength/gelColor for gel-doc; dyeChannel for qPCR; hinaChannel/hinaDimension/hinaSize for Hina; dpi/colorMode for Epson).
*application/json*
**URI**
`datahub://instruments/{instrumentId}/filter-options`
# Tools (/docs/mcp/tools)
**For:** Developers and integrators
All tools return JSON encoded as a single text content block. Error cases set `isError: true` and return a plain-text message. Each tool below carries its annotations (`readOnly`, `idempotent`, `destructive`) and the schema of its JSON response.
Connecting requires the `read` scope. Every tool without a `readOnly` annotation also requires `write`. See [MCP overview](/docs/mcp) for authentication and client setup.
## Instruments
### `list_instruments`
List all registered lab instruments with run counts, watcher status, and file patterns. Optionally filter by status.
*readOnly*
**Parameters**
- `status` (optional, `"pending" | "active" | "inactive"`) — Filter instruments by status
**Response**
- `instruments` (`array