# Deploying to another machine

How to get Extraction Studio running on a different computer — a colleague's
laptop, a second dev box, or a fresh Linux server — reachable on that machine's
own `localhost`.

There is one route: **Node processes against a local MySQL server.** No Docker,
no Redis, no message broker. The job queue is a table (`QueueJob`) in the same
database as everything else, so MySQL is the only service to install.

Three processes, none of which needs the others to be started first:

| Process | Command             | Port | What it does                                   |
| ------- | ------------------- | ---- | ---------------------------------------------- |
| API     | `pnpm start:api`    | 3000 | accepts uploads, serves results, enqueues jobs |
| Worker  | `pnpm start:worker` | —    | polls `QueueJob` once a second and extracts    |
| Web     | `pnpm start:web`    | 3001 | optional Next.js UI                            |

The API only queues. Without the worker running, uploads succeed and nothing
ever extracts.

---

## Before you start

| Requirement | Version                      | Check                               |
| ----------- | ---------------------------- | ----------------------------------- |
| Node.js     | ≥ 20.11 (developed on 22/24) | `node --version`                    |
| pnpm        | ≥ 9 (pinned 11.21 in-repo)   | `corepack enable && pnpm --version` |
| MySQL       | 8.x                          | `mysql --version`                   |
| Python      | 3.9–3.12, **only for OCR**   | `python3 --version`                 |

Ports used — make sure nothing else holds them, or override as shown in
[Changing ports](#changing-ports):

| Port | Service           |
| ---- | ----------------- |
| 3000 | API               |
| 3001 | Web UI (optional) |
| 3306 | MySQL             |

### Get the code

```bash
git clone <your-remote> extraction_studio
cd extraction_studio
corepack enable            # provides the pinned pnpm from package.json
```

If you're copying rather than cloning, **do not copy** `node_modules`,
`dist`, `.next`, `*.tsbuildinfo` or `.storage/` — they contain machine-specific
paths, compiled output, and in the last case other people's statements. A clean
copy:

```bash
git clone --depth 1 file:///path/to/extraction_studio ./extraction_studio
```

---

## 1. MySQL

Install the server if the machine has none:

```bash
# macOS — the official package installs to /usr/local/mysql
brew install mysql && brew services start mysql

# Debian/Ubuntu
sudo apt-get install -y mysql-server && sudo systemctl enable --now mysql
```

The schema is generated by Prisma with `provider = "mysql"` and is developed
against **MySQL 8**. A 5.7 server is not tested — see the MAMP note in
[Troubleshooting](#troubleshooting) if that is what answers on your machine.

Create the database and a user for it:

```sql
CREATE DATABASE extraction_studio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'extraction'@'localhost' IDENTIFIED BY 'extraction';
GRANT ALL PRIVILEGES ON extraction_studio.* TO 'extraction'@'localhost';
FLUSH PRIVILEGES;
```

Prisma needs no more than that on an existing database. It does want
`CREATE`/`ALTER` on it, which the grant above covers; migrations are DDL.

Confirm it answers:

```bash
mysql -u extraction -p -h 127.0.0.1 -P 3306 -e "select version();"
```

> Use `127.0.0.1`, not `localhost`, if you hit a socket error: on macOS
> `localhost` sends the client down a Unix socket that may belong to a
> different server than the one Prisma reaches over TCP.

---

## 2. Configuration

```bash
cp .env.example .env
```

Everything is read from `.env` at startup and validated once — a bad value
stops the process rather than surfacing later inside a worker. The lines to
read before the first run:

```bash
# The only required variable. No default; nothing starts without it.
DATABASE_URL=mysql://extraction:extraction@127.0.0.1:3306/extraction_studio

# Development only. Serves requests with no API key, and says so in the log on
# every request. Leave it false on anything reachable by other people.
ALLOW_ANONYMOUS=false

# Where uploaded PDFs are written. Relative to the repository root, and
# git-ignored — it holds real documents.
STORAGE_ROOT=.storage
```

> Uploaded statements are confidential. `STORAGE_ROOT` is excluded from git, but
> nothing stops it being picked up by a backup or a `scp -r` of the working
> tree. Point it outside the repository on any shared machine.

There is no `REDIS_URL`. If you are carrying an old `.env` across, the line is
now ignored rather than honoured — the queue is in MySQL.

---

## 3. Install and build

```bash
pnpm install       # also runs `prisma generate`
pnpm build         # tsc -b across the workspace
```

`pnpm start:api` and `pnpm start:worker` run `node dist/main.js` — they need the
build. `pnpm dev:api` / `pnpm dev:worker` run the TypeScript directly under
`tsx watch` and reload on change.

---

## 4. Create the schema

```bash
pnpm db:deploy     # applies committed migrations — use this on any real machine
```

Use `pnpm db:push` **only** on a throwaway development database: it forces the
schema to match without recording a migration, so the two drift apart silently.

Verify:

```bash
mysql -u extraction -p extraction_studio -e "SHOW TABLES;"
```

Expect `ApiKey`, `AuditLog`, `BankProfile`, `CandidateRule`, `Document`,
`ExtractionCorrection`, `ExtractionJob`, `FeedbackExample`, `QueueJob`,
`RegionSelection`, `WebhookDelivery` and Prisma's `_prisma_migrations`.
`QueueJob` is the queue itself — if it is missing, you are on a migration older
than `20260818081752_queue_jobs` and the worker has nothing to poll.

---

## 5. Run it

Two terminals:

```bash
pnpm start:api      # http://localhost:3000
```

```bash
pnpm start:worker   # claims jobs from QueueJob
```

Check it:

```bash
curl http://localhost:3000/health     # {"status":"ok"} — no dependencies touched
curl http://localhost:3000/ready
```

`/ready` reports both backing services and answers 503 while either is down:

```json
{
  "status": "ok",
  "uptimeSeconds": 368,
  "version": "0.1.0",
  "dependencies": {
    "database": { "status": "up", "latencyMs": 89 },
    "queue": { "status": "up", "latencyMs": 158 }
  }
}
```

`/health` is deliberately dependency-free — an orchestrator must not restart a
healthy process because MySQL blipped. `/ready` is the one a load balancer
should read.

---

## First extraction

With `ALLOW_ANONYMOUS=true`:

```bash
curl -X POST http://localhost:3000/v1/extract \
  -H 'Content-Type: application/pdf' \
  --data-binary @tests/fixtures/safra/investment-report-zero-balance/input.pdf
```

A small document comes back `200` with the result inline. A large one comes back
`202` and a `Location` header; poll it:

```bash
curl http://localhost:3000/v1/extract/<jobId>
```

The split is `SYNC_MAX_BYTES` / `SYNC_MAX_PAGES` (2 MB / 20 pages): above either,
the request is queued rather than run on the API's event loop.

Add the audit trail for any field:

```bash
curl -X POST 'http://localhost:3000/v1/extract?include_evidence=true' \
  -H 'Content-Type: application/pdf' --data-binary @statement.pdf
```

---

## API keys

There is **no external service** to get these from. It is your API, so you mint
your own keys — no signup, no dashboard, no vendor. They are rows in your
`ApiKey` table.

```bash
pnpm key:create --name "laptop"
```

```
  esk_WRGddmaCG_xzYmPvD1bvzB7OSOrBHlo4hn9slKHCrS0

  This is the only time the key is shown. It is stored as a SHA-256
  hash and cannot be recovered — issue a new one if it is lost.
```

Only the hash is stored, so a leaked database dump is not a leaked set of
credentials — and nothing, including this CLI, can recover a key you lose.
Issue a replacement and revoke the old one.

```bash
pnpm key:list                    # never shows the key itself
pnpm key:revoke esk_WRGddmaC     # by prefix or id
```

Optional limits, each overriding the deployment default for that key alone:

```bash
pnpm key:create --name "ci" --rate-limit 600 --max-bytes 10485760 --expires-in 90
```

Then set `ALLOW_ANONYMOUS=false`, restart the API, and send it:

```bash
curl -X POST http://localhost:3000/v1/extract \
  -H 'Authorization: Bearer esk_…' \
  -H 'Content-Type: application/pdf' --data-binary @statement.pdf
```

`X-API-Key: esk_…` works too, if a client cannot set `Authorization`.

### Why there is no `/v1/keys` endpoint

Deliberate. An endpoint that mints credentials is the one you least want
reachable by accident, and whoever issues keys already has shell access to the
deployment. The CLI is also the only place a key can be _shown_ at all.

### While developing

`ALLOW_ANONYMOUS=true` skips all of this and serves requests with no key. It
logs a warning on every request, because an API that silently stops checking
credentials is worse than one that refuses to start. Use it on a machine only
you can reach, and nowhere else.

---

## The web UI (optional)

A Next.js frontend lives in `apps/web` and runs on port 3001.

```bash
cp apps/web/.env.example apps/web/.env.local
pnpm dev:web                  # or: pnpm build:web && pnpm start:web
```

Two variables matter:

```bash
# Inlined into the browser bundle at build time. Never put a secret here.
NEXT_PUBLIC_API_URL=http://localhost:3000

# Server-only. The proxy route attaches it as `Authorization: Bearer`, so the
# browser never sees it.
EXTRACTION_API_KEY=esk_…
```

Browser requests go through this app's own `/api/upstream/*` proxy
(`NEXT_PUBLIC_API_PROXY`, on by default) rather than straight to Fastify. That
is required today, not a preference: the API sends no CORS headers and answers
preflight with 404, so a direct call from the browser fails before it reaches a
route. It is also what keeps `EXTRACTION_API_KEY` server-side.

---

## OCR (optional)

Scanned statements need PaddleOCR. Documents with a text layer — most of them —
extract without it, so skip this until a scan fails. With OCR off, a fully
scanned document fails with `OCR_FAILED` rather than producing wrong output.

```bash
python3 -m venv .ocr-venv
.ocr-venv/bin/pip install paddlepaddle paddleocr
.ocr-venv/bin/python packages/ocr/python/paddle_bridge.py --selftest
```

The self-test exits non-zero if OCR does not come back, so a silent
half-install fails loudly. Roughly 800 MB installed, plus ~180 MB of model
weights downloaded to `~/.paddlex` on first use.

Then point the worker at that interpreter and restart it:

```bash
OCR_ENABLED=true
OCR_PYTHON=/path/to/extraction_studio/.ocr-venv/bin/python
```

The worker constructs the engine at startup when `OCR_ENABLED=true`; the API
does the same for reading reviewer-drawn regions. Full notes:
[packages/ocr/README.md](packages/ocr/README.md).

---

## Configuration reference

Full list with comments: [.env.example](.env.example). The ones that matter on
a new machine:

| Variable                            | Default             | Notes                                                |
| ----------------------------------- | ------------------- | ---------------------------------------------------- |
| `DATABASE_URL`                      | —                   | required; `mysql://…`                                |
| `API_HOST` / `API_PORT`             | `0.0.0.0` / `3000`  |                                                      |
| `API_BODY_LIMIT`                    | `52428800`          | 50 MB upload cap                                     |
| `PDF_MAX_PAGES`                     | `1000`              | rejected before parsing starts                       |
| `SYNC_MAX_BYTES` / `SYNC_MAX_PAGES` | `2097152` / `20`    | above either, a job is queued instead of run inline  |
| `WORKER_CONCURRENCY`                | `4`                 | jobs claimed at once by one worker                   |
| `JOB_ATTEMPTS` / `JOB_BACKOFF_MS`   | `3` / `5000`        | retries and their backoff                            |
| `QUEUE_PREFIX`                      | `extraction-studio` | namespaces queue names in the table                  |
| `STORAGE_DRIVER`                    | `filesystem`        | or `s3`                                              |
| `STORAGE_ROOT`                      | `.storage`          | filesystem driver only                               |
| `STORAGE_RETENTION_DAYS`            | `30`                | see the sweeper caveat at the end                    |
| `RATE_LIMIT_PER_MINUTE`             | `60`                | per key, counted in-process                          |
| `ALLOW_ANONYMOUS`                   | `false`             | development only                                     |
| `CONFIGS_DIR`                       | `configs`           | bank profiles                                        |
| `PROFILE_WRITE_THROUGH`             | `true`              | mirrors saved profiles back to `CONFIGS_DIR` as JSON |
| `OCR_ENABLED` / `OCR_PYTHON`        | `false` / `python3` |                                                      |
| `CONFIDENCE_AUTOMATIC` / `_WARNING` | `0.90` / `0.70`     | review bands; a bank profile may override            |
| `MODELS_DIR`                        | `models`            | trained classifiers — use an absolute path           |
| `EVALUATION_DIR`                    | `evaluation`        | datasets and reports, append-only                    |
| `CLASSIFIER_MODE`                   | `off`               | `off` / `shadow` / `enforce`                         |

The rate limit is a per-process counter, so N API processes means N times the
limit. One process per deployment is the assumption; the `RedisRateLimitStore`
in `apps/api/src/security/rateLimit.ts` exists for the day that stops being
true, and nothing wires it up.

### Object storage instead of the filesystem

For S3, Cloudflare R2 or MinIO:

```bash
STORAGE_DRIVER=s3
S3_BUCKET=extraction-studio
S3_REGION=auto
S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com   # omit for AWS
S3_ACCESS_KEY_ID=…
S3_SECRET_ACCESS_KEY=…
```

Incomplete S3 settings fail at **boot**, not on the first upload — a deployment
that starts and then cannot store anything is the worst of both.

> The S3 adapter's request signing is verified against AWS's published test
> vectors, but the adapter has **not** been exercised against a live bucket. Do
> one upload and one download before trusting it in production.

### Changing ports

If 3306 or 3000 are taken on the target machine, everything follows from `.env`:

```bash
API_PORT=3100
DATABASE_URL=mysql://extraction:extraction@127.0.0.1:3307/extraction_studio
```

The web app's port is fixed in `apps/web/package.json` (`next dev --port 3001`);
change it there, and set `NEXT_PUBLIC_API_URL` to match a moved API.

---

## Keeping it running

Nothing restarts these processes for you. On a server, give each a unit:

```ini
# /etc/systemd/system/extraction-api.service
[Unit]
Description=Extraction Studio API
After=network.target mysql.service

[Service]
WorkingDirectory=/srv/extraction_studio
ExecStart=/usr/bin/node apps/api/dist/main.js
EnvironmentFile=/srv/extraction_studio/.env
Restart=always
User=extraction

[Install]
WantedBy=multi-user.target
```

The worker is the same file with `apps/worker/dist/main.js`. Both must run from
the repository root: config walks up from the working directory to find `.env`,
and `CONFIGS_DIR`, `MODELS_DIR` and `STORAGE_ROOT` are relative to it unless you
made them absolute.

On macOS, `launchd` or `pm2 start apps/api/dist/main.js` does the same job.

---

## Reaching it from another machine

By default the API binds `0.0.0.0` and is reachable at
`http://<host-ip>:3000` — but with `ALLOW_ANONYMOUS=true` that is an open
endpoint that will accept and store confidential documents from anyone who can
route to it. Before exposing it beyond loopback:

1. Set `ALLOW_ANONYMOUS=false` and issue keys.
2. Put it behind TLS. The service speaks plain HTTP by design; terminate TLS at
   a reverse proxy and set `NODE_ENV=production` so `trustProxy` is on.
3. Keep MySQL on loopback — `bind-address = 127.0.0.1` in `my.cnf`. Nothing but
   the API and worker on the same host needs to reach it.
4. Bind the API itself to loopback (`API_HOST=127.0.0.1`) if a reverse proxy on
   the same machine is the only thing that should talk to it.

---

## Verifying the install

```bash
pnpm check        # format, lint, build, tests
pnpm evaluate     # regression scores against the fixtures
```

`pnpm check` needs no MySQL and no bucket: the integration tests that do are
opt-in and skipped by default. If it passes, the code is sound on this machine
and anything still wrong is configuration.

To include the integration tests, which start the API against the live database:

```bash
RUN_INTEGRATION_TESTS=1 DATABASE_URL=mysql://… pnpm test:integration
```

---

## Troubleshooting

**`DATABASE_URL is required`** — no `.env`, or you started the process from a
directory outside the repository. Config walks up from the working directory to
find `.env`; `cp .env.example .env` at the repository root.

**`P1001: Can't reach database server at localhost:3306`** — MySQL is not
running, or `localhost` resolved to a socket rather than TCP. Use `127.0.0.1` in
`DATABASE_URL`, and check `brew services list` / `systemctl status mysql`.

**`P1000: Authentication failed`** — the user exists but not for the host you
connected from. A user granted at `'extraction'@'localhost'` is a different
principal from `'extraction'@'127.0.0.1'` as far as MySQL is concerned; grant
both, or connect the way you granted.

**MySQL 5.7 answers instead of 8** — a MAMP or XAMPP install can own the client
on `PATH` and a second server can hold another port. Check with
`mysql --version` and `mysql -h 127.0.0.1 -P 3306 -e "select version()"`; use
the absolute path (`/usr/local/mysql/bin/mysql`) if the wrong client is first.
Point `DATABASE_URL` at the 8.x server — the migrations are only tested there.

**`Table 'extraction_studio.QueueJob' doesn't exist`** — migrations have not
been applied, or the database was built with `db push` and has no migration
history. `pnpm db:deploy`; if it reports nothing to apply while the table is
missing, drop the database and recreate it, then deploy again.

**`Port 3000 is already in use`** — something else holds it. `lsof -i :3000`, or
set `API_PORT`.

**A 401 on every request** — expected: `ALLOW_ANONYMOUS=false` and no valid key.
Run `pnpm key:create --name "laptop"`, or set `ALLOW_ANONYMOUS=true` on a
machine only you can reach. (`/health` and `/ready` never need a key.) The
response is identical for a missing, unknown and revoked key, deliberately —
telling them apart would make the endpoint an oracle for which keys exist. The
_reason_ is in the API log.

**429 with `Retry-After`** — the rate limit. Raise `RATE_LIMIT_PER_MINUTE`, or
set `rateLimitPerMinute` on the key's row.

**Uploads work but nothing extracts** — the worker is not running. It is a
separate process (`pnpm start:worker`); the API only queues. Look at
`QueueJob`: rows sitting in `PENDING` mean nobody is polling, rows in `FAILED`
carry the reason in `lastError`.

**A job retries forever, or never** — `JOB_ATTEMPTS` and `JOB_BACKOFF_MS`. A
processor that raises `UnrecoverableError` stops immediately and the row is
dead-lettered as `dl-<id>` for a human to read rather than retried.

**The web UI shows "rejected our key"** — `EXTRACTION_API_KEY` in
`apps/web/.env.local` is missing, wrong or revoked. Next.js reads it at server
start; restart `pnpm dev:web` after editing.

**`INVALID_PDF` on a file that opens fine** — the upload arrived without
`Content-Type: application/pdf`, or the bytes were mangled in transit. Use
`--data-binary`, never `-d`, which strips newlines.

**Build fails on `sharp` or `@napi-rs/canvas`** — native modules. On Linux:
`apt-get install -y build-essential python3` and reinstall.

---

## What this deployment does not include

Stated plainly so nobody discovers it in production:

- **No key-management endpoint.** Keys are minted with the CLI, as above.
- **No TLS.** Terminate it at a reverse proxy.
- **No CORS on the API.** The browser reaches it through the web app's proxy;
  a direct cross-origin call fails at preflight.
- **The rate limiter counts per process.** Two API processes, twice the limit.
- **The S3 adapter is unverified against a live bucket** — the signing is proven
  against AWS's vectors, the request handling is not.
- **The storage sweeper is implemented but not scheduled.** `sweepExpired` is
  exported from `@extraction-studio/storage` and nothing calls it on a timer, so
  uploaded documents are not actually deleted at `STORAGE_RETENTION_DAYS` — run
  it from cron, or expect the directory to grow.
- **No database backups.** MySQL holds the extractions, the corrections, the
  trained-rule candidates and the API keys. Nothing in this repository backs
  them up.

### The Docker files

`Dockerfile` and `docker-compose.yml` are still in the tree and have been kept
pointing at MySQL, and `pnpm docker:*` still drives them. They are not the
supported path and nothing above uses them; treat them as unmaintained until
someone re-verifies a container build end to end.