Deploying research-kg's daily KG extraction to a Linux server (systemd)

This is the Linux/systemd counterpart to deploy/launchd/README.md (which covers running the same pipeline locally on macOS via launchd). Use this runbook to run the daily knowledge-graph extraction on an always-on Linux box (for example the team's mgc-hub droplet), independent of anyone's laptop.

Read the SCOPE NOTE at the bottom before you start: this gets the pipeline running on its own schedule with its own local SQLite store. It does not by itself connect the graph to the live site or the warehouse Postgres.

(a) Provision the box

Requirements:

• Linux (any modern distro with systemd; the team's droplet is fine).
• At least 8GB RAM. The pipeline loads GLiNER-multi plus sentence-transformer embedding models, peaking at roughly 5GB RSS; 8GB leaves headroom for the OS and anything else running on the box.
• At least 25GB free disk: the repo + Python dependencies (torch is large), plus roughly 13GB for the HuggingFace model cache that downloads on first run.

Steps:

1. Create a dedicated service user (do not run this as root): sudo useradd -r -m -s /usr/sbin/nologin scraper # EDIT: reuse an existing service user if this box already has one

2. Clone research-kg to the deploy path: sudo mkdir -p /opt/research-kg # EDIT: change this path everywhere below if you deploy elsewhere sudo chown scraper:scraper /opt/research-kg sudo -u scraper git clone <repo-url> /opt/research-kg

3. Create a virtualenv and install dependencies: sudo -u scraper python3 -m venv /opt/research-kg/.venv sudo -u scraper /opt/research-kg/.venv/bin/pip install --upgrade pip sudo -u scraper /opt/research-kg/.venv/bin/pip install -r /opt/research-kg/requirements.txt torch is a large download; this step can take a while on a slow link.

4. First-time model pre-download (optional but recommended). The first real run downloads roughly 13GB of HuggingFace models (GLiNER, embeddings, NLI) into the HuggingFace cache. Warm the cache ahead of the first scheduled run by invoking the runner once by hand, as the service user: sudo -u scraper env PYTHON_BIN=/opt/research-kg/.venv/bin/python3 \ bash /opt/research-kg/scripts/kg_server_run.sh Do this only once you also have secrets in place (step (b)); it makes real network + LLM calls.

(b) Secrets

1. Copy the template: sudo cp /opt/research-kg/deploy/systemd/mgc-kg.env.example /etc/mgc-kg.env

2. Fill in /etc/mgc-kg.env: DEEPSEEK_API_KEY, MGC_API_USERNAME, MGC_API_PASSWORD, and set PYTHON_BIN to /opt/research-kg/.venv/bin/python3 (or your actual venv path). Leave MGC_API_BASE_URL at the current working domain (https://api.mapglobalchina.com) unless it has changed again.

3. Lock the file down: sudo chmod 600 /etc/mgc-kg.env sudo chown scraper:scraper /etc/mgc-kg.env # EDIT: match the service user from (a)

No secrets belong in the .service/.timer files or anywhere in the repo.

(c) Install the systemd units

1. Copy the unit files and edit every # EDIT: marker inside them (service user, WorkingDirectory, ExecStart path, EnvironmentFile path if you used something other than /etc/mgc-kg.env): sudo cp /opt/research-kg/deploy/systemd/mgc-kg-daily.service /etc/systemd/system/ sudo cp /opt/research-kg/deploy/systemd/mgc-kg-daily.timer /etc/systemd/system/ sudo $EDITOR /etc/systemd/system/mgc-kg-daily.service

2. Reload systemd and enable the timer: sudo systemctl daemon-reload sudo systemctl enable --now mgc-kg-daily.timer enable --now both starts the timer immediately and makes it survive reboots. It does NOT run the service immediately (the timer's OnCalendar decides when the first run fires); use systemctl start mgc-kg-daily.service (see below) to trigger a run right away.

(d) Operate

• Check the schedule and next run time: systemctl list-timers mgc-kg-daily
• Tail live output while a run is in progress or has just finished: journalctl -u mgc-kg-daily -f
• Trigger a run immediately, without waiting for the timer: sudo systemctl start mgc-kg-daily.service
• Log and state locations (all under the repo's data/ directory on the server disk, i.e. /opt/research-kg/data/ with the default WorkingDirectory):
• data/kg_server_run.log -- the wrapper's own timestamped log (also visible live through journalctl since the wrapper streams to stdout).
• data/mgc_kg.sqlite -- the working graph, rebuilt every run.
• data/mgc_kg.demo.sqlite -- the served snapshot, refreshed only when the health guard passes (see SCOPE/COST section below).
• data/checkpoint.sqlite -- the durable LLM extraction cache.
• Change the cumulative budget cap: edit BUDGET_USD in /etc/mgc-kg.env, no restart of the timer needed (it is read fresh on the next run).
• Change the schedule: edit OnCalendar in /etc/systemd/system/mgc-kg-daily.timer, then: sudo systemctl daemon-reload sudo systemctl restart mgc-kg-daily.timer

(e) State and cost

The graph (data/mgc_kg.sqlite) and the LLM checkpoint (data/checkpoint.sqlite) both persist under data/ on the server's own disk between runs. Every run resets and rebuilds the graph from the current corpus, but the checkpoint cache replays already-extracted documents at zero cost; only brand-new or previously-failed documents make a paid LLM call, and total spend across every run (not per day) is capped by the cumulative BUDGET_USD in /etc/mgc-kg.env. This is what keeps daily reruns cheap; see the header comment in scripts/kg_server_run.sh (and its macOS sibling scripts/kg_daily.sh) for the full rationale.

(f) Scope note -- what this deployment does and does not do

This setup runs the research-kg extraction pipeline standalone, on its own schedule, writing its graph to a local SQLite file on the server disk. That achieves the goal of always-on, laptop-independent daily collection.

It does NOT, by itself:

• write into the warehouse kg.* Postgres schema,
• feed the live Mapping Global China site,
• run inside the mgc-hub enrichment-module cadence (NER / entity bridge / entity resolver) that the rest of the warehouse pipeline uses.

Wiring this pipeline's output into the warehouse and the live site is a separate, later production-integration step. See kg-handoff/INTEGRATION_GUIDE.md and design/system/SYSTEM_DESIGN.md sections 2-4 for the full design of that step (the kg.* schema, the enrichment-module wrapper, the /kg API routes, and the Option A/B decision on how research-kg's core gets embedded into mgc-hub's cadence). Treat this deployment as "the extraction job is now running independently on a server," not as "the KG is now live on the site."