IT Administrator Manual

CivicRecords AI — AI-Powered Open Records Support for American Cities
v1.4.1 April 2026 CAIA: Not High-Risk Apache 2.0

v1.2.0 — what changed since this manual was authored

This administrator manual was first written against v1.1.0. The v1.2.0 release (2026-04-23) adds five operator-facing capabilities that are not individually documented in every section below. Where this manual and the CHANGELOG disagree, the CHANGELOG wins. Always treat CHANGELOG.md §[1.2.0] as the authoritative delta.

Test coverage at v1.2.0: 617 backend pytest + 36 frontend vitest, CI-verified (GitHub Actions run 24859606170 on commit cc06abc). See the v1.2.0 release page for the curated release notes.

1. System Overview

Architecture Summary

CivicRecords AI is a locally-hosted, Docker Compose-based application that assists municipal clerks in processing open records (public records / FOIA) requests. The system runs entirely on-premises with no cloud dependencies, no telemetry, and no outbound data transmission. All AI inference is performed locally via Ollama using open-weight models.

The architecture follows a standard three-tier pattern: a React single-page application serves as the administrative front end, communicating with a FastAPI REST backend, which in turn interfaces with PostgreSQL (with pgvector for semantic search), Redis (for task queuing), and Ollama (for local LLM inference). Asynchronous document ingestion is handled by Celery workers with Redis as the message broker.

For the full architectural diagram and data flow, refer to docs/architecture/system-architecture.html.

Docker Services

Service Image Port Purpose Health Check
postgres pgvector/pgvector:pg17 5432 (internal) Primary database with vector search extensions pg_isready -U civicrecords
redis redis:7.2-alpine 6379 (internal) Task queue broker and result backend redis-cli ping
ollama ollama/ollama:latest 11434 (internal) Local LLM inference engine ollama list
api Custom (Dockerfile.backend) 8000 FastAPI REST API server curl -sf http://localhost:8000/health
worker Custom (Dockerfile.backend) Celery async task worker (ingestion, embeddings) celery inspect ping
beat Custom (Dockerfile.backend) Celery periodic task scheduler
frontend Custom (Dockerfile.frontend) 8080 React admin panel (served via nginx) curl -sf http://localhost:80/

Technology Stack

LayerTechnologyVersion
RuntimePython3.12
API FrameworkFastAPI0.115+
ORMSQLAlchemy2.0
MigrationsAlembic1.13+
AuthFastAPI-Users + JWT
Task QueueCelery5.x
DatabasePostgreSQL + pgvector17
Cache / BrokerRedis7.2 (BSD licensed)
AI InferenceOllamalatest
FrontendReact + shadcn/ui + Tailwind CSS18.x
ContainerizationDocker Composev2

License Compliance

CivicRecords AI is released under the Apache License 2.0. All runtime dependencies use permissive or weak-copyleft licenses (MIT, Apache 2.0, BSD, LGPL, MPL). Redis is pinned to version 7.2 (BSD licensed); version 8.x and later changed to a non-permissive license and must not be used.

Warning
Do not upgrade the Redis image beyond redis:7.2-alpine. Redis 8.x uses the Server Side Public License (SSPL), which is incompatible with the project's permissive licensing requirements.

2. Hardware Requirements

Minimum Specification

ComponentMinimumNotes
CPU8 cores (x86_64 or ARM64)Modern Intel Core i7 / AMD Ryzen 7 or better
RAM32 GBRequired for 12B parameter model inference
Storage50 GB SSDOS + Docker images + models + small document corpus
Network1 Gbps EthernetInternal network only; no internet required at runtime

Recommended Specification

ComponentRecommendedNotes
CPU12–16 coresIntel Xeon / AMD EPYC for production workloads
RAM64 GBRecommended tier; headroom for workstation Gemma 4 tags (26b/31b) plus a large document corpus
Storage2 TB NVMe SSDRoom for extensive document archives and database growth
GPU (optional)Dedicated GPU with 8+ GB VRAMDramatically accelerates inference; see GPU support below

GPU Acceleration Support

PlatformGPU FrameworkDeployment Method
Linux (AMD)ROCmdocker-compose.gpu.yml overlay with GPU device passthrough
Windows (AMD/other)DirectMLHost-native Ollama via docker-compose.host-ollama.yml
Linux/Windows (NVIDIA)CUDANVIDIA Container Toolkit + Ollama GPU support

The installer automatically detects GPU hardware and selects the appropriate Docker Compose overlay. GPU acceleration is optional; the system functions fully on CPU-only hardware.

Performance Target

Performance Benchmark
On the baseline target profile (Windows 11 Pro 23H2+, 8 cores, 32 GB RAM, CPU inference with gemma4:e4b), the system targets completion of a single records query — including document retrieval, exemption scanning, and LLM analysis — in under 30 seconds.

3. Installation

Prerequisites

Windows Installation

Open PowerShell as Administrator in the project directory and run:

.\install.ps1

The script performs the following steps:

  1. Verifies Docker Desktop is installed and the daemon is running
  2. Verifies Docker Compose v2 is available
  3. Creates .env from .env.example and generates a cryptographically random JWT secret
  4. Pauses for you to configure admin credentials in .env
  5. Runs hardware detection (scripts/detect_hardware.ps1) and writes .env.hardware
  6. Selects the appropriate Docker Compose overlay (GPU or host Ollama if applicable)
  7. Pulls base Docker images and builds application images
  8. Starts PostgreSQL and Redis; waits for database readiness (up to 60 seconds)
  9. Runs Alembic database migrations
  10. Starts all seven services
  11. Waits for the API health endpoint to respond
  12. Pulls the embedding model (nomic-embed-text)
  13. Prints the recommended language model command and access URLs

Linux Installation (Ubuntu/Debian)

Run from the project directory:

chmod +x install.sh
./install.sh

The script will automatically install Docker and Docker Compose if they are not present on the system. On Linux with AMD GPUs, the script detects ROCm support and applies the docker-compose.gpu.yml overlay for GPU device passthrough.

macOS Installation

Docker Desktop for Mac is required. Install it from docker.com, then run:

chmod +x install.sh
./install.sh

macOS does not support GPU passthrough to Docker containers. All inference runs on CPU.

Post-Installation Verification

After installation completes, verify that all services are healthy:

# Check all container health statuses
docker compose ps
NAME                STATUS              PORTS
civicrecords-api    Up (healthy)        0.0.0.0:8000->8000/tcp
civicrecords-beat   Up
civicrecords-frontend Up (healthy)      0.0.0.0:8080->80/tcp
civicrecords-ollama Up (healthy)
civicrecords-postgres Up (healthy)      5432/tcp
civicrecords-redis  Up (healthy)        6379/tcp
civicrecords-worker Up (healthy)

# Test API health endpoint
curl http://localhost:8000/health
{"status":"ok","version":"1.4.1"}

# Verify OpenAPI documentation is accessible
curl -s http://localhost:8000/docs | head -1

# Verify frontend is serving
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080
200

# Run data sovereignty verification
bash scripts/verify-sovereignty.sh    # Linux/macOS
.\scripts\verify-sovereignty.ps1      # Windows

First Admin Account Setup

The system automatically creates the first administrator account on startup using the credentials specified in the .env file:

Security
The application refuses to start if JWT_SECRET is left at its default placeholder value. The installer generates a secure secret automatically, but if you are configuring .env manually, generate one with: openssl rand -hex 32

After installation, navigate to http://<server-ip>:8080 to access the admin panel and log in with the configured credentials. Change the admin password immediately after first login.

4. Configuration Reference

Environment Variables (.env)

Variable Description Default Required
DATABASE_URL PostgreSQL connection string (asyncpg driver) postgresql+asyncpg://civicrecords:civicrecords@postgres:5432/civicrecords Yes
JWT_SECRET Secret key for signing JWT tokens. Minimum 32 characters. Application refuses to start with insecure defaults. None (must be set) Yes
JWT_LIFETIME_SECONDS JWT token expiration time in seconds 3600 (1 hour) No
FIRST_ADMIN_EMAIL Email address for the auto-created initial admin account admin@example.gov Yes
FIRST_ADMIN_PASSWORD Password for the initial admin account None (must be set) Yes
OLLAMA_BASE_URL URL for the Ollama inference server http://ollama:11434 Yes
REDIS_URL Redis connection string for Celery task queue redis://redis:6379/0 Yes
AUDIT_RETENTION_DAYS Number of days to retain audit log entries 1095 (3 years) No
CORS_ORIGINS Allowed CORS origins (JSON list format) ["http://localhost:8080"] No
EMBEDDING_MODEL Ollama model name for document embeddings nomic-embed-text No
CHAT_MODEL Ollama model name for query analysis and summarization gemma4:e4b No
VISION_MODEL Ollama model name for document image/PDF analysis gemma4:e4b No

Hardware Detection (.env.hardware)

The installer generates a .env.hardware file with auto-detected values. These variables are consumed by the Docker Compose overlays and are not typically edited manually.

VariableDescriptionExample
CIVICRECORDS_GPU_ENABLEDWhether a compatible GPU was detectedtrue
CIVICRECORDS_PLATFORMDetected GPU platform identifierAMD
CIVICRECORDS_GFX_VERSIONGPU microarchitecture versiongfx1100
CIVICRECORDS_TOTAL_RAM_GBTotal system RAM in gigabytes64
CIVICRECORDS_RECOMMENDED_MODELDefault pre-filled into the installer picker (operator can change)gemma4:e4b
CIVICRECORDS_USE_HOST_OLLAMAUse native host Ollama instead of container (Windows GPU)true

GPU Configuration

GPU support is configured through Docker Compose overlay files:

Docker Compose Overrides

The system supports standard Docker Compose override patterns. To customize resource limits, port mappings, or volume mounts without modifying the base file, create a docker-compose.override.yml:

# docker-compose.override.yml
services:
  api:
    ports:
      - "443:8000"
    deploy:
      resources:
        limits:
          memory: 4G

5. User & Department Administration

Creating Users

User accounts are created exclusively by administrators through the admin API or admin panel. There is no self-registration. This is a deliberate security design: only authorized municipal staff should have access to the system.

# Create a new user via the API
curl -X POST http://localhost:8000/admin/users \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "clerk@city.gov",
    "password": "SecurePassword123!",
    "full_name": "Jane Doe",
    "role": "staff"
  }'

Roles and Permissions

Role Code Create Requests Search Documents Review / Approve Manage Users System Config Audit Logs
Admin admin Yes Yes (all depts) Yes Yes Yes Yes
Reviewer reviewer Yes Yes (all depts) Yes No No View only
Staff staff Yes Own dept only No No No No
Read Only read_only No Own dept only No No No No

Creating Departments

curl -X POST http://localhost:8000/departments/ \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "Public Works", "description": "Roads, water, utilities"}'

Assigning Users to Departments

Use the PATCH /departments/{dept_id} endpoint to update department membership, or assign a department when creating users via the admin panel. A user can belong to one department at a time.

Department Scoping Behavior

Service Accounts for Federation

Service accounts enable machine-to-machine API access for integrations with other municipal systems (e.g., a records management system pushing documents). Service accounts authenticate with API keys rather than JWT tokens.

# Create a service account
curl -X POST http://localhost:8000/service-accounts/ \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "RMS Integration", "description": "Automated document feed from Records Management System"}'
Security
The API key is returned only once at creation time. It is stored as a bcrypt hash in the database and cannot be retrieved. Record it securely immediately. If lost, create a new service account.

6. Data Source Management

Adding File Directory Sources

A directory data source points to a file system path accessible to the API container. This path must be mounted as a Docker volume.

curl -X POST http://localhost:8000/datasources/ \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "City Council Minutes",
    "source_type": "directory",
    "config": {"path": "/data/council-minutes"}
  }'

Uploading Files

Files can be uploaded directly through the API or the admin panel. Uploaded files are stored in the uploads Docker volume at /tmp/civicrecords-uploads inside the container.

curl -X POST http://localhost:8000/datasources/upload \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -F "file=@meeting-minutes-2025.pdf"

Ingestion Pipeline Overview

  1. Discovery — Files in a data source directory are scanned and new/changed files are identified by hash comparison.
  2. Parsing — Each file is processed by the appropriate parser based on file extension.
  3. Chunking — Parsed text is split into overlapping chunks suitable for embedding and retrieval.
  4. Embedding — Each chunk is embedded using the configured embedding model (nomic-embed-text by default) and stored as a vector in pgvector.
  5. Indexing — Document metadata and chunk vectors are written to PostgreSQL for both keyword and semantic search.

Ingestion is performed asynchronously by the Celery worker service.

Triggering Ingestion

# Trigger ingestion for a specific data source
curl -X POST http://localhost:8000/datasources/{source_id}/ingest \
  -H "Authorization: Bearer <admin-jwt-token>"

Monitoring Ingestion Status

# View data source statistics including ingestion progress
curl http://localhost:8000/datasources/stats \
  -H "Authorization: Bearer <admin-jwt-token>"

Each document tracks its ingestion status: pending, processing, completed, or failed.

Supported File Formats

FormatExtensionsParserTrack
PDF.pdfPyMuPDF / Vision model fallbackText + OCR
Microsoft Word.docxpython-docxText
Plain Text.txtDirect readText
CSV.csvCSV parserTabular
Markdown.mdDirect readText
HTML.html, .htmBeautifulSoupText
JSON.jsonJSON parserStructured
Email.emlEmail parserText
Images.png, .jpg, .tiffVision model OCROCR

Troubleshooting Ingestion Failures

7. Model Management

Recommended Models

CivicRecords AI is optimized for the Gemma 4 family of open-weight models from Google. These models provide strong instruction-following, good factual accuracy, and are released under permissive licenses suitable for government use.

The installer picker presents all four supported Gemma 4 tags. The default is gemma4:e4b. Only e2b and e4b are supportable at the 32 GB baseline target profile; 26b and 31b require stronger hardware and are gated behind an explicit confirmation in the installer.

ModelClassParametersDiskRAM (advisory)Supportable at baseline
gemma4:e2bEdge2.3B effective (5.1B w/ embeddings)7.2 GB~16 GBYes
gemma4:e4b (default)Edge4.5B effective (8B w/ embeddings)9.6 GB~20 GBYes
gemma4:26bWorkstation MoE25.2B total / 3.8B active18 GB48+ GB recommendedNo (32 GB baseline)
gemma4:31bWorkstation dense30.7B20 GB64+ GB; GPU recommendedNo (32 GB baseline)

Pulling Models via Ollama

# Pull the embedding model (required)
docker compose exec ollama ollama pull nomic-embed-text

# Pull the default chat/vision model
docker compose exec ollama ollama pull gemma4:e4b
# OR a smaller edge model for tighter installs:
docker compose exec ollama ollama pull gemma4:e2b
# OR workstation models (require 48+ GB RAM for 26b, 64+ GB RAM and GPU for 31b):
docker compose exec ollama ollama pull gemma4:26b
docker compose exec ollama ollama pull gemma4:31b

# If using host-native Ollama (Windows GPU):
ollama pull nomic-embed-text
ollama pull gemma4:e4b

# Verify installed models
docker compose exec ollama ollama list

Model Registry

The model registry tracks compliance metadata for every model used in the system. This supports audit and governance requirements by recording which model version produced each analysis.

# View registered models
curl http://localhost:8000/admin/models/registry \
  -H "Authorization: Bearer <admin-jwt-token>"

# Register a new model with compliance metadata
curl -X POST http://localhost:8000/admin/models/registry \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "gemma4:e4b",
    "version": "latest",
    "license": "Apache 2.0",
    "provider": "Google",
    "purpose": "Records query analysis and exemption reasoning"
  }'

Switching Models

To change the active model, update the CHAT_MODEL and/or VISION_MODEL variables in .env and restart the API service:

# Edit .env to set the new model (any Ollama tag; picker defaults are four Gemma 4 tags)
# CHAT_MODEL=gemma4:e2b
# VISION_MODEL=gemma4:e2b

# Restart API to pick up the change
docker compose restart api

Embedding Model

The embedding model (nomic-embed-text by default) converts document chunks into vector representations for semantic search. Changing the embedding model requires re-ingesting all documents to rebuild the vector index.

Warning
Changing EMBEDDING_MODEL invalidates all existing document embeddings. You must re-ingest every data source after changing this value. This can be a time-consuming operation for large document corpora.

Model Resource Requirements

ModelDisk SpaceRAM (inference)GPU VRAMAvg. Query Time (CPU)
nomic-embed-text~275 MB~1 GB~512 MB<1 sec / chunk
gemma4:e2b7.2 GB~16 GB~8 GB8–15 sec
gemma4:e4b (default)9.6 GB~20 GB~10 GB10–20 sec
gemma4:26b18 GB48+ GB recommended~20 GB20–40 sec
gemma4:31b20 GB64+ GB recommended~24 GB30–60 sec

8. Exemption Rules Administration

How the Rules Engine Works

CivicRecords AI uses a rules-primary, LLM-secondary architecture for exemption detection. When a document is scanned for potential exemptions (e.g., personal privacy, law enforcement confidentiality, trade secrets):

  1. Deterministic rules are evaluated first. These are pattern-based rules tied to specific state statutes. Each rule defines keywords, patterns, and the statutory citation it implements.
  2. LLM analysis runs second, using the language model to identify exemptions that pattern matching may miss. The LLM provides reasoning for each flagged exemption.
  3. Human review is always required. The system flags potential exemptions but never automatically redacts or denies records. A human reviewer must accept, modify, or dismiss each flag.
Tip
The rules-primary approach ensures that well-known exemption patterns are caught consistently regardless of LLM behavior. The LLM layer adds coverage for nuanced cases that rules alone would miss.

50-State Coverage

The system ships with approximately 180 exemption rules covering all 50 U.S. states plus federal FOIA categories. Rules are seeded from the backend/scripts/seed_rules.py script and stored in the exemption_rules database table.

Viewing Rules by State

# List all exemption rules (filterable by state)
curl "http://localhost:8000/exemptions/rules/?state=CO" \
  -H "Authorization: Bearer <admin-jwt-token>"

Creating Custom Rules

curl -X POST http://localhost:8000/exemptions/rules/ \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "CO",
    "statute": "CRS 24-72-204(3)(a)(IV)",
    "category": "Trade Secrets",
    "description": "Trade secrets, privileged information, confidential commercial data",
    "keywords": ["trade secret", "proprietary", "confidential commercial"],
    "rule_type": "pattern",
    "enabled": true
  }'

Enabling / Disabling Rules

# Disable a specific rule
curl -X PATCH http://localhost:8000/exemptions/rules/{rule_id} \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Running the Seed Script

To populate or reset the exemption rules database with the built-in rule set:

docker compose exec api python -m scripts.seed_rules
Warning
Running the seed script on an existing installation will recreate default rules. Custom rules you have created will not be affected, but modified default rules may be overwritten. Back up your rules before re-seeding.

9. Compliance & Audit

Hash-Chained Audit Log Architecture

Every significant action in the system (login, document access, search queries, exemption decisions, request status changes) generates an immutable audit log entry. Each entry includes a SHA-256 hash of its contents chained to the previous entry's hash, creating a tamper-evident log structure similar to a blockchain.

If any entry is modified or deleted, the chain verification will fail at that point, providing cryptographic proof of tampering. This design satisfies records retention and legal discovery requirements.

Audit Log Retention

The default retention period is 1,095 days (3 years), configurable via the AUDIT_RETENTION_DAYS environment variable. Entries older than this threshold are eligible for archival.

Security
Consult your municipality's records retention schedule before reducing the default retention period. Many jurisdictions require minimum retention periods of 3–7 years for government records.

Verifying Audit Chain Integrity

# Verify the audit log hash chain is intact
curl http://localhost:8000/audit/verify \
  -H "Authorization: Bearer <admin-jwt-token>"
{"valid": true, "entries_checked": 14523, "chain_intact": true}

Exporting Audit Logs

# Export audit logs (supports CSV and JSON formats)
curl "http://localhost:8000/audit/export?format=csv&start_date=2026-01-01&end_date=2026-03-31" \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -o audit-q1-2026.csv

# JSON export
curl "http://localhost:8000/audit/export?format=json" \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -o audit-full.json

Compliance Template Documents

The system includes five compliance template documents stored at backend/compliance_templates/:

TemplateFilePurpose
AI Governance Policyai-governance-policy.mdOrganizational AI use policy framework
AI Use Disclosureai-use-disclosure.mdPublic notice that AI is used in records processing
CAIA Impact Assessmentcaia-impact-assessment.mdColorado AI Act risk classification assessment
Data Residency Attestationdata-residency-attestation.mdCertification that all data remains on-premises
Response Letter Disclosureresponse-letter-disclosure.mdAI disclosure language for inclusion in response letters

Rendering Templates with City Profile

Templates contain placeholder variables (city name, contact information, etc.) that are populated from the city profile:

# Set up the city profile first
curl -X POST http://localhost:8000/city-profile \
  -H "Authorization: Bearer <admin-jwt-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "city_name": "City of Boulder",
    "state": "CO",
    "contact_email": "records@bouldercolorado.gov"
  }'

# Render a template with city profile data
curl http://localhost:8000/exemptions/templates/{template_id}/render \
  -H "Authorization: Bearer <admin-jwt-token>"

Data Sovereignty Verification

Run the sovereignty verification script to confirm that no data leaves the local system:

# Linux / macOS
bash scripts/verify-sovereignty.sh

# Windows
.\scripts\verify-sovereignty.ps1

This script inspects Docker network configuration, DNS resolution settings, and container firewall rules to verify that no container has outbound internet access.

CAIA Classification

Under the Colorado Artificial Intelligence Act (SB 24-205), CivicRecords AI is classified as not high-risk. The system does not autonomously make or substantially contribute to consequential decisions. All exemption flags and document classifications require human review and approval before any action is taken. The system is an assistive tool, not a decision-maker. The full impact assessment is documented in backend/compliance_templates/caia-impact-assessment.md.

10. Security

Network Configuration

By default, only the API (port 8000) and frontend (port 8080) services expose ports to the host. All other services (PostgreSQL, Redis, Ollama) communicate over Docker's internal network and are not accessible from outside the Docker Compose stack.

Security
The system is designed for deployment on a private, trusted network behind a firewall. Do not expose ports 8000 or 8080 directly to the public internet. Use a reverse proxy (nginx, Caddy, or HAProxy) with TLS termination for any external access.

HTTPS / TLS Setup

CivicRecords AI does not include a built-in TLS certificate. To enable HTTPS, place a reverse proxy in front of the application:

# Example: nginx reverse proxy configuration
server {
    listen 443 ssl;
    server_name records.city.gov;

    ssl_certificate     /etc/ssl/certs/records.city.gov.crt;
    ssl_certificate_key /etc/ssl/private/records.city.gov.key;

    location /api/ {
        proxy_pass http://127.0.0.1:8000/;
    }
    location / {
        proxy_pass http://127.0.0.1:8080/;
    }
}

JWT Secret Management

Login Rate Limiting

The login endpoint (POST /auth/jwt/login) is rate-limited to 5 requests per minute per IP address. Exceeding this limit returns HTTP 429 (Too Many Requests). The rate limiter uses in-memory tracking with a maximum of 10,000 tracked IPs and automatic expiry of stale entries.

No Default Passwords

The installer generates a random JWT secret and forces the administrator to set the initial admin password before the first start. The application will not start with placeholder credentials.

No Telemetry or Outbound Connections

CivicRecords AI makes zero outbound network connections during normal operation. There is no telemetry, no analytics, no update checking, and no "phone home" behavior. All AI inference happens locally via Ollama. The system can operate on a completely air-gapped network after initial Docker image pulls.

Role-Based Access Control

All API endpoints enforce role-based access control (RBAC) using the require_role() dependency. See the User & Department Administration section for the full role permissions matrix. Key restrictions:

Service Account API Key Hashing

Service account API keys are hashed with bcrypt before storage. The plaintext key is returned exactly once at creation time and is never stored or logged. Authentication of service account requests validates the provided key against the stored hash.

11. Backup & Recovery

What to Back Up

ComponentLocationContentsPriority
PostgreSQL databasepgdata Docker volumeAll application data: users, documents, audit logs, search indices, rulesCritical
Environment files.env, .env.hardwareConfiguration, JWT secret, database credentialsCritical
Uploaded documentsuploads Docker volumeOriginal files uploaded through the admin panelHigh
Ollama modelsollamadata Docker volumeDownloaded model weights (~8–17 GB)Low (re-downloadable)

PostgreSQL Backup

# Create a full database dump
docker compose exec -T postgres pg_dump -U civicrecords civicrecords > backup-$(date +%Y%m%d).sql

# Compressed backup (recommended for large databases)
docker compose exec -T postgres pg_dump -U civicrecords -Fc civicrecords > backup-$(date +%Y%m%d).dump

# Back up environment files
cp .env .env.backup-$(date +%Y%m%d)

# Back up uploaded documents volume
docker run --rm -v civicrecords-ai_uploads:/data -v $(pwd)/backups:/backup alpine \
  tar czf /backup/uploads-$(date +%Y%m%d).tar.gz -C /data .

Restore Procedure

# 1. Stop the application
docker compose down

# 2. Start only the database
docker compose up -d postgres

# 3. Wait for database readiness
docker compose exec -T postgres pg_isready -U civicrecords

# 4. Drop and recreate the database
docker compose exec -T postgres dropdb -U civicrecords civicrecords
docker compose exec -T postgres createdb -U civicrecords civicrecords

# 5a. Restore from SQL dump
docker compose exec -T postgres psql -U civicrecords civicrecords < backup-20260401.sql

# 5b. OR restore from compressed dump
docker compose exec -T postgres pg_restore -U civicrecords -d civicrecords < backup-20260401.dump

# 6. Start all services
docker compose up -d

# 7. Verify health
curl http://localhost:8000/health

Backup Schedule Recommendations

Backup TypeFrequencyRetention
Full database dumpDaily30 days
Environment file backupOn every changeIndefinite
Uploaded documentsWeekly90 days
Off-site copyWeekly1 year
Tip
Automate backups with a cron job. Store backups on a separate physical disk or network share. Test restore procedures quarterly.

12. Monitoring & Troubleshooting

Health Endpoint

curl http://localhost:8000/health
{"status":"ok","version":"1.4.1"}

The GET /health endpoint returns HTTP 200 with the application version when the API is operational. Use this for load balancer health checks and monitoring systems.

Docker Service Health Checks

# View health status of all services
docker compose ps

# View health check logs for a specific service
docker inspect --format='{{json .State.Health}}' civicrecords-api-1

Common Issues

SymptomLikely CauseResolution
API returns 500 on startup Database migration failure Check logs: docker compose logs api. Run migrations manually: docker compose exec api alembic upgrade head
Search returns no results Embedding model not pulled Pull the model: docker compose exec ollama ollama pull nomic-embed-text
Ingestion stuck at "processing" Worker service crashed or not running Check and restart: docker compose restart worker
Login returns 429 Rate limit exceeded (5/min per IP) Wait 60 seconds. If persistent, check for brute-force attempts in audit logs.
Ollama timeout errors Insufficient RAM for model Switch to a smaller model or add RAM. Check: docker stats
"JWT_SECRET is set to an insecure default" .env not configured Generate a secret: openssl rand -hex 32 and set in .env
Frontend shows blank page API not reachable from browser Verify CORS_ORIGINS in .env includes the frontend URL
PostgreSQL won't start Corrupted data volume or port conflict Check logs: docker compose logs postgres. Verify port 5432 is not in use.

Log Locations

# API server logs
docker compose logs api --tail 100 -f

# Worker (ingestion) logs
docker compose logs worker --tail 100 -f

# Database logs
docker compose logs postgres --tail 50

# Ollama inference logs
docker compose logs ollama --tail 50

# All services combined
docker compose logs --tail 200 -f

Checking Ollama Model Status

docker compose exec ollama ollama list
NAME                  ID           SIZE     MODIFIED
nomic-embed-text      274b5616...  274 MB   3 days ago
gemma4:e4b            c6eb396d...  9.6 GB   3 days ago

Database Connectivity Checks

# Test database connectivity from the API container
docker compose exec api python -c "
from app.database import engine
import asyncio
async def check():
    async with engine.connect() as conn:
        result = await conn.execute(text('SELECT 1'))
        print('Database OK:', result.scalar())
from sqlalchemy import text
asyncio.run(check())
"

# Direct PostgreSQL check
docker compose exec postgres pg_isready -U civicrecords
/var/run/postgresql:5432 - accepting connections

13. Upgrading

Upgrade Procedure

  1. Back up the database before any upgrade:
    docker compose exec -T postgres pg_dump -U civicrecords -Fc civicrecords > pre-upgrade-backup.dump
  2. Pull the latest code:
    git pull origin main
  3. Rebuild Docker images:
    docker compose build
  4. Run database migrations:
    docker compose run --rm api alembic upgrade head
  5. Restart all services:
    docker compose up -d
  6. Verify health:
    curl http://localhost:8000/health
Tip
The API service automatically runs Alembic migrations on startup. However, running migrations explicitly as a separate step (step 4) allows you to catch migration errors before bringing the full stack online.

Database Migrations

CivicRecords AI uses Alembic for database schema migrations. Migration files are stored in backend/alembic/versions/. Key commands:

# Apply all pending migrations
docker compose exec api alembic upgrade head

# Check current migration version
docker compose exec api alembic current

# View migration history
docker compose exec api alembic history

Version Compatibility

Always upgrade sequentially through minor versions. Skipping major versions may result in migration conflicts. Check the CHANGELOG.md for breaking changes between versions.

Rollback Procedure

  1. Stop all services:
    docker compose down
  2. Restore the database from the pre-upgrade backup:
    docker compose up -d postgres
    docker compose exec -T postgres dropdb -U civicrecords civicrecords
    docker compose exec -T postgres createdb -U civicrecords civicrecords
    docker compose exec -T postgres pg_restore -U civicrecords -d civicrecords < pre-upgrade-backup.dump
  3. Check out the previous version:
    git checkout v1.0.0  # Replace with the target version tag
  4. Rebuild and restart:
    docker compose build
    docker compose up -d
Warning
Rolling back Alembic migrations with alembic downgrade is possible but risky in production. Prefer a full database restore from backup for rollback scenarios.

14. Appendices

Appendix A: Complete API Endpoint Reference

MethodPathAuthDescription
Health
GET/healthNoneApplication health check and version
Authentication
POST/auth/jwt/loginNoneAuthenticate and receive JWT token
POST/auth/jwt/logoutJWTInvalidate current session
Users
GET/users/meJWTGet current authenticated user profile
PATCH/users/meJWTUpdate current user profile
Administration
GET/admin/statusAdminSystem status overview
GET/admin/usersAdminList all users
POST/admin/usersAdminCreate a new user account
GET/admin/modelsAdminList available Ollama models
GET/admin/models/registryAdminList registered models with compliance metadata
POST/admin/models/registryAdminRegister a model with compliance metadata
PATCH/admin/models/registry/{model_id}AdminUpdate model registry entry
DELETE/admin/models/registry/{model_id}AdminRemove model from registry
Audit
GET/audit/logsAdminQuery audit log entries with filters
GET/audit/verifyAdminVerify audit log hash chain integrity
GET/audit/exportAdminExport audit logs (CSV/JSON)
Data Sources
POST/datasources/AdminCreate a new data source
GET/datasources/JWTList data sources
PATCH/datasources/{source_id}AdminUpdate data source configuration
POST/datasources/{source_id}/ingestAdminTrigger ingestion for a data source
POST/datasources/uploadJWTUpload a file for ingestion
GET/datasources/statsJWTData source ingestion statistics
Documents
GET/documents/JWTList ingested documents
GET/documents/{document_id}JWTGet document details and metadata
GET/documents/{document_id}/chunksJWTGet document chunks with embeddings info
Search
POST/search/queryJWTExecute a semantic + keyword search query
GET/search/sessionsJWTList search sessions
GET/search/sessions/{session_id}JWTGet search session details and results
GET/search/filtersJWTGet available search filter options
Records Requests
POST/requests/JWTCreate a new records request
GET/requests/JWTList records requests
GET/requests/statsJWTRequest processing statistics
GET/requests/{request_id}JWTGet request details
PATCH/requests/{request_id}JWTUpdate request details
POST/requests/{request_id}/documentsJWTAttach documents to a request
GET/requests/{request_id}/documentsJWTList documents attached to a request
DELETE/requests/{request_id}/documents/{doc_id}JWTRemove document from a request
POST/requests/{request_id}/submit-reviewJWTSubmit request for review
POST/requests/{request_id}/ready-for-releaseReviewerMark request ready for release
POST/requests/{request_id}/approveReviewerApprove and finalize request
POST/requests/{request_id}/rejectReviewerReject request with reason
GET/requests/{request_id}/timelineJWTGet request timeline / activity log
POST/requests/{request_id}/timelineJWTAdd timeline entry
GET/requests/{request_id}/messagesJWTGet request messages / correspondence
POST/requests/{request_id}/messagesJWTSend a message on a request
GET/requests/{request_id}/feesJWTGet fee schedule for a request
POST/requests/{request_id}/feesJWTAdd fee line items to a request
POST/requests/{request_id}/response-letterJWTGenerate response letter
GET/requests/{request_id}/response-letterJWTGet response letter
PATCH/requests/{request_id}/response-letter/{letter_id}JWTUpdate response letter
Exemptions
POST/exemptions/rules/AdminCreate an exemption rule
GET/exemptions/rules/JWTList exemption rules (filterable by state)
PATCH/exemptions/rules/{rule_id}AdminUpdate or disable an exemption rule
POST/exemptions/scan/{request_id}JWTRun exemption scan on request documents
GET/exemptions/flags/{request_id}JWTGet exemption flags for a request
PATCH/exemptions/flags/{flag_id}ReviewerAccept, modify, or dismiss an exemption flag
GET/exemptions/dashboardJWTExemption analytics dashboard data
GET/exemptions/dashboard/accuracyJWTExemption detection accuracy metrics
GET/exemptions/dashboard/exportAdminExport exemption dashboard data
GET/exemptions/templates/JWTList disclosure templates
POST/exemptions/templates/AdminCreate a disclosure template
GET/exemptions/templates/{template_id}/renderJWTRender template with city profile data
PUT/exemptions/templates/{template_id}AdminUpdate a disclosure template
Departments
POST/departments/AdminCreate a department
GET/departments/JWTList departments
GET/departments/{dept_id}JWTGet department details
PATCH/departments/{dept_id}AdminUpdate department details or membership
DELETE/departments/{dept_id}AdminDelete a department
Service Accounts
POST/service-accounts/AdminCreate a service account (returns API key once)
GET/service-accounts/AdminList service accounts
PATCH/service-accounts/{account_id}AdminUpdate or deactivate a service account
Analytics
GET/analytics/operationalJWTOperational analytics and dashboard metrics
City Profile
GET/city-profileJWTGet the city profile
POST/city-profileAdminCreate the city profile
PATCH/city-profileAdminUpdate the city profile
Systems Catalog
GET/catalog/domainsJWTList catalog domains (functional areas)
GET/catalog/systemsJWTList cataloged municipal systems
POST/catalog/loadAdminLoad/reload systems catalog from seed data
Notifications
GET/notifications/templatesJWTList notification templates
GET/notifications/logJWTView notification delivery log

Appendix B: Database Schema Overview

The system uses 29 PostgreSQL tables. Key tables and their relationships:

TablePurposeKey Relationships
usersUser accounts and authenticationHas many requests, belongs to department
departmentsOrganizational departmentsHas many users, has many data sources
data_sourcesRegistered document data sourcesHas many documents, belongs to department
documentsIngested document recordsBelongs to data source, has many chunks
document_chunksText chunks with vector embeddingsBelongs to document (pgvector column)
model_registryAI model compliance trackingReferenced by audit entries
search_sessionsSearch session containersHas many queries, belongs to user
search_queriesIndividual search queriesBelongs to session, has many results
search_resultsSearch result rankingsBelongs to query, references document chunks
records_requestsOpen records request lifecycleHas many documents, flags, timeline entries
request_documentsDocuments attached to requestsJoin table: requests ↔ documents
document_cacheCached document analysis resultsBelongs to request
exemption_rulesStatutory exemption rule definitionsReferenced by exemption flags
exemption_flagsFlagged exemptions on request documentsBelongs to request, references rule
disclosure_templatesResponse letter disclosure templatesReferenced by response letters
request_timelinesRequest activity timeline entriesBelongs to request
request_messagesCorrespondence on requestsBelongs to request, belongs to user
response_lettersGenerated response lettersBelongs to request
fee_schedulesFee schedule definitionsHas many line items
fee_line_itemsIndividual fee entriesBelongs to fee schedule
notification_templatesEmail/notification templatesReferenced by notification log
notification_logNotification delivery recordsReferences template and user
prompt_templatesLLM prompt templatesUsed by search and exemption modules
city_profilesMunicipality configuration dataReferenced by template rendering
system_catalogMunicipal IT systems catalogHas many connector templates
connector_templatesIntegration connector definitionsBelongs to catalog system
service_accountsMachine-to-machine API accountsHashed API key, audit-logged
audit_logTamper-evident hash-chained audit logReferences user, hash chain to previous entry
alembic_versionDatabase migration version trackingManaged by Alembic

Appendix C: Docker Compose Environment Variables

The following environment variables are set within docker-compose.yml for infrastructure services:

ServiceVariableValue
postgresPOSTGRES_USERcivicrecords
postgresPOSTGRES_PASSWORDcivicrecords
postgresPOSTGRES_DBcivicrecords
Security
The default PostgreSQL credentials (civicrecords/civicrecords) are acceptable because the database is only accessible within the Docker internal network. However, for defense-in-depth, consider changing them in production by overriding the postgres service environment and updating DATABASE_URL accordingly.

Application services (api, worker, beat) load their configuration from the .env file via the env_file directive in docker-compose.yml.

Appendix D: Supported Platforms Matrix

PlatformArchitectureDockerGPU SupportInstallerStatus
Ubuntu 22.04 / 24.04x86_64Docker EngineROCm (AMD), CUDA (NVIDIA)install.shFully supported
Debian 12x86_64Docker EngineROCm (AMD), CUDA (NVIDIA)install.shFully supported
RHEL 9 / Rocky 9x86_64Docker EngineROCm (AMD), CUDA (NVIDIA)install.shCommunity tested
Windows 10/11 Prox86_64Docker DesktopDirectML (native Ollama)install.ps1Fully supported
macOS 13+ (Intel)x86_64Docker DesktopNone (CPU only)install.shSupported
macOS 13+ (Apple Silicon)ARM64Docker DesktopNone (CPU only)install.shSupported