Skip to content
OpenJevX wiki
Every statement on this page is cited and was checked against jev-cloud origin/main (10a744c), openjevx v0.5.9 (ee2a1f4) on 2026-10-04.

Troubleshooting

Each entry gives the symptom, the cause and the fix. Server entries come first, then model storage, then fine-tuning.

Serving

Section titled “Serving”

The server refuses the model: sha256 does not match

Section titled “The server refuses the model: sha256 does not match”
  • Symptom: the error sha256 … does not match config.json. 1
  • Cause: the server refuses a graph whose sha256 does not match config.json. 2
  • Fix: build the folder again from the ONNX file and its eval report with make_model_folder.py. 3

No model found at startup

Section titled “No model found at startup”
  • Symptom: the error no model found: set "model" in openjevx.json, or put a model folder at …. 4
  • Cause: with "model" unset, the server looks next to itself for model/, then models/openjevx/, then openjevx.w8.onnx, and stops with this error when none of them exists. 5 6 7 8 The binary holds no model. 9
  • Fix: set "model" in openjevx.json to a model folder, or unpack the model archive into the same directory as the binary, where the server finds model/. 4 10 11

The configured model path does not exist

Section titled “The configured model path does not exist”
  • Symptom: the error not found (openjevx.json "model" must be a model folder or a .onnx file). 12
  • Cause: os.Stat fails on the configured "model" path. 13
  • Fix: the server loads a folder from "model" in openjevx.json, -model, or OPENJEVX_MODEL; "model" must be a model folder or a .onnx file. 14 12

device gpu will not start

Section titled “device gpu will not start”
  • Symptom: the error device gpu was set and no GPU provider ran the model. 15
  • Cause: device is gpu, and gpu does not fall back. 16 gpu refuses to start unless one of the GPU providers (CUDA, CoreML on macOS, DirectML on Windows) loads. 17 18 19
  • Other settings: "device": "auto" logs no GPU provider ran the model, using CPU and serves on CPU. 20 21 "device": "cpu" returns the CPU session directly. 22

The Docker container exits at once

Section titled “The Docker container exits at once”
  • Symptom: openjevx: set OPENJEVX_PASSWORD or mount your own openjevx.json; refusing to start without a dashboard password. 23 Or openjevx: set OPENJEVX_API_KEY (16+ characters, e.g. openssl rand -hex 24); refusing to start with an open decision API. 24
  • Cause: the image has no default credentials: it refuses to start without OPENJEVX_PASSWORD and OPENJEVX_API_KEY, or your own openjevx.json mounted at /data/openjevx.json. 25
  • Fix: pass both, and make a key with openssl rand -hex 24. 26 27
Terminal window
OPENJEVX_PASSWORD=<12+ characters> OPENJEVX_API_KEY=<16+ characters> docker compose up -d --build

With the prebuilt image, pass both as -e OPENJEVX_PASSWORD=<12+ characters> -e OPENJEVX_API_KEY=<16+ characters>. 28

Terminal window
docker run -d -p 127.0.0.1:21118:21118 \
  -e OPENJEVX_PASSWORD=<12+ characters> -e OPENJEVX_API_KEY=<16+ characters> \
  ghcr.io/deemwar-products/openjevx:v0.5.9

The container rejects the password or the key

Section titled “The container rejects the password or the key”
  • Symptom: the error openjevx: OPENJEVX_PASSWORD must be at least 12 characters. 29 Or openjevx: OPENJEVX_API_KEY must be at least 16 characters. 30
  • Cause: when no openjevx.json is mounted, the entrypoint checks the password’s length (-ge 12) and, unless OPENJEVX_ALLOW_NO_API_KEY=1, the key’s length (-ge 16). 31
  • Fix: use OPENJEVX_PASSWORD=<12+ characters> and OPENJEVX_API_KEY=<16+ characters>. 26

401 missing or wrong API key from /v1/systemone

Section titled “401 missing or wrong API key from /v1/systemone”
  • Symptom: 401 with the body {"error":"missing or wrong API key"}. 32
  • Cause: the server listens beyond loopback, so the API key is on. 33
  • Fix: send Authorization: Bearer <key>; a generated key is in openjevx.api-key beside openjevx.json. 34 For jevx, add --header 'Authorization: Bearer $OPENJEVX_API_KEY' to the profile. 35

The server will not start: API key too short or not creatable

Section titled “The server will not start: API key too short or not creatable”
  • Symptom: api key from … is … characters; use at least 16. 36
  • Symptom: listening on … needs an API key: cannot create openjevx.api-key in any of: …; set OPENJEVX_DATA to a writable folder. 37 38
  • Cause: a needed credential that is unset is generated beside openjevx.json (or the executable when there is none), else in $OPENJEVX_DATA, else the working folder, whichever is writable. 39
  • Fix: set OPENJEVX_DATA to a writable folder. 38 Or set OPENJEVX_API_KEY (at least 16 characters) and OPENJEVX_PASSWORD. 40 37 41

The log does not show the generated key or password

Section titled “The log does not show the generated key or password”
  • Symptom: the log says new … generated into … (sha256 ...…); not printed: stderr is not a terminal. 42
  • Cause: a start that creates a credential shows the value once, and only on a terminal; when stderr is a log (Docker, systemd, ECS, CloudWatch), it logs the file and a fingerprint, never the secret. 43
  • Fix: read it from the file; the log names the file. 43

I never set a dashboard password, but the dashboard asks for one

Section titled “I never set a dashboard password, but the dashboard asks for one”
  • Cause: there is no default password; unset, the server generates one on its first start, shows it once on a terminal, and keeps it in openjevx.password. 44
  • Fix: read openjevx.password (the npx install keeps it in ~/.local/share/openjevx/), or set "password" in openjevx.json or OPENJEVX_PASSWORD. 44 Any user name works. 45

adminadmin no longer works

Section titled “adminadmin no longer works”
  • Symptom: the log says password: … has the old published default "adminadmin"; ignoring it. 46 47
  • Cause: adminadmin shipped in openjevx.json up to v0.5.6; it is public, so it counts as no password. 48
  • Fix: use the generated password in openjevx.password, or set your own. 49 44

Crash in CreateOrtEnv in a distroless image

Section titled “Crash in CreateOrtEnv in a distroless image”
  • Symptom: SIGSEGV in CreateOrtEnv in distroless images. 50
  • Cause: ORT 1.29’s Linux build has telemetry on by default and runs popen("echo `blkid; hostname`") when /etc/machine-id is missing, which crashes without /bin/sh. 50
  • Fix: server 0.5.3 sets ORT_DISABLE_TELEMETRY=1 before creating the environment and calls DisableTelemetry after. 51

Slow on AWS Fargate

Section titled “Slow on AWS Fargate”
  • Symptom: a 1 vCPU Fargate task running 2 threads was 4x slower. 52
  • Cause: Fargate limits CPU with shares that neither the quota nor Go can see, so a 1 vCPU task reports 2 CPUs. 52
  • Fix: jev-ecs and openjevx-server-ecs set OPENJEVX_THREADS from the task’s CPU; at 1 vCPU the log then shows intra-op 1 (from env). 53 OPENJEVX_THREADS wins over the config. 54 Check the startup line threads: intra-op 1 (from ecs) or (from env). 55

Measured on Fargate with model 0.5.2, p50 of run for a 51-token request: 1 vCPU at intra-op 2 (from GOMAXPROCS) 1510 ms; 1 vCPU with OPENJEVX_THREADS=1 362 ms. 56

Slow on some x86 hosts

Section titled “Slow on some x86 hosts”
  • Symptom: on Linux, the second startup line names the CPU and its SIMD flags, e.g. has avx2; lacks avx512f avx512_vnni …. 57
  • Cause: on x86, AVX2-only hosts are about 2x slower than AVX-512 VNNI ones, and Fargate hands out both. 58

Measured with model 0.5.2 on ONNX Runtime 1.29.0 CPU, short (40 tokens) / long (512 tokens). 59 AMD EPYC 7763, AVX2, 1 thread: 531 / 6953 ms. Intel 8573C, AVX-512 VNNI, 2 threads: 173 / 2104 ms. 60

Old answers after a model upgrade

Section titled “Old answers after a model upgrade”
  • Symptom: after upgrading from an older model, jevx still serves cached answers. 61
  • Cause: jevx caches answers by model name. 61
  • Fix: run jevx cache clear, or give the profile a versioned model name such as --model openjevx-v0.5.2. 61

Intel Mac

Section titled “Intel Mac”
  • Symptom: task setup fails on an Intel Mac. 62
  • Cause: ORT 1.29 has no Intel-Mac build. 62
  • Symptom (npx): the installer stops with OpenJevX: no server build for darwin/…: ONNX Runtime 1.29 has no Intel macOS build, so OpenJevX runs on Apple Silicon only (or use Docker). 63

The npx install says there is no server build

Section titled “The npx install says there is no server build”
  • Symptom: OpenJevX: no server build for <os>/<cpu>: there is no OpenJevX release for it (use Docker, or build from source). 63
  • Cause: releases exist only for macOS on Apple Silicon, Linux amd64 and arm64, and Windows amd64. 64
  • Fix: build from source: task run fetches ONNX Runtime and the model folder, builds, and starts the server. 65

Model from S3

Section titled “Model from S3”

Missing objects and s3:ListBucket

Section titled “Missing objects and s3:ListBucket”
  • Why it matters: with s3:ListBucket on the prefix, a missing object is a 404. 66 67
  • Fix: grant s3:GetObject on the objects and s3:ListBucket on the prefix. 68

MinIO, Ceph or R2 does not resolve the bucket

Section titled “MinIO, Ceph or R2 does not resolve the bucket”
  • Symptom: the model is on an S3-compatible store (MinIO, Ceph, Cloudflare R2) rather than AWS S3. 69
  • Cause: the store has no wildcard bucket DNS, so it needs path-style: bucket in the path, not the host name. 70 69
  • Fix: set AWS_ENDPOINT_URL_S3 and turn on path-style with AWS_S3_USE_PATH_STYLE=true or "model_s3_path_style": true; for R2 also set AWS_REGION=auto. 71

A pinned server never reloads

Section titled “A pinned server never reloads”
  • Symptom: a server pinned with model_sha256 rejects every new upload to the bucket. 72
  • Cause: the pin (model_sha256) and reload are at odds: a pinned server rejects every new upload. 72
  • Fix: pin for a fixed version, reload for a moving current/. 72

A config-only promote downloads the whole graph

Section titled “A config-only promote downloads the whole graph”
  • Symptom: changing only config.json triggers a full download. 73
  • Cause: a promote that changes only config.json still downloads the graph again (about 600 MB). 74
  • Fix: expect it: reusing unchanged files by ETag is a later step if promotes get frequent. 73

The server will not start with the store down

Section titled “The server will not start with the store down”
  • Symptom: with no valid cache, startup fails when the store is unreachable, denies access or holds a bad upload. 75
  • Cause: with no valid cache the server refuses to start. 75
  • Fix: with a valid cache in model_cache, the server starts from it with a WARNING when the store is down. 76 75

Fine-tuning

Section titled “Fine-tuning”

The importer rejects two rows as conflicting

Section titled “The importer rejects two rows as conflicting”
  • Symptom: the importer error same state and question as line … but a different answer. 77
  • Cause: two rows with the same state and question have different answers: a fact that decides the answer is missing from the state. 78
  • Fix: never give the same state and question two different answers: a fact is missing. 79

The training box silently trains on CPU

Section titled “The training box silently trains on CPU”
  • Symptom: the session provider is not CUDAExecutionProvider. 80
  • Cause: version 1.30 needs CUDA 13 and silently falls back to CPU on this image. 80
  • Fix: install onnxruntime-gpu==1.22.0 with --force-reinstall --no-deps, and abort unless the session provider is CUDAExecutionProvider. 80

The GPU box is stuck loading

Section titled “The GPU box is stuck loading”
  • Symptom: the GPU box is stuck loading. 81
  • Cause: some hosts are slow to start. 81
  • Fix: a box not running after 15 minutes is destroyed and the next machine is tried, up to 3; change the wait with VAST_START_MINUTES. 82

The box cannot clone your fork

Section titled “The box cannot clone your fork”
  • Symptom: you work from a fork, and the box clones your checkout’s origin remote. 83
  • Cause: the box clones origin over HTTPS and with no credentials. 84
  • Fix: origin must be a public repo you can push to: check it with git remote get-url origin. 85

The gate fails

Section titled “The gate fails”
  • Symptom: ft.py gate exits 1. 86
  • Cause and fix: look at which file failed; for your file, add near-miss rows and check the state holds every fact; for the basics, lower --repeat so your data does not crowd them out. 87

SageMaker job: too few rows

Section titled “SageMaker job: too few rows”
  • Symptom: the job fails with too few rows to train: the … shard has N train rows … refuses fewer than 100 trainable items. 88 89
  • Cause: each CSV row is one item, and the smoke shard holds your train rows ×2 and the full shard ×3. 90
  • Fix: give at least 34 train rows for a full run, or 50 for --smoke. 91 OpenJevX’s own examples/decisions.csv has had 53 train rows since 2026-10-03; the 39-row version before it (24 train rows) failed both. 92

SageMaker full run: no held-out rows

Section titled “SageMaker full run: no held-out rows”
  • Symptom: the job fails with no held-out rows (split test/gate): openjevx's accuracy gate (…) would fail at accuracy 0 after training. 93
  • Cause: with no held-out rows (split test or gate), the 0.55 gate would fail at accuracy 0 after the whole run. 94
  • Fix: leave split empty (10% go to gate) or mark some rows test or gate. 95

SageMaker job killed with no model

Section titled “SageMaker job killed with no model”
  • Symptom: the job ends at the time limit and produces no model. 96
  • Cause: hitting MaxRuntimeInSeconds kills the job with no model. 96
  • Fix: set MAX_HOURS below MaxRuntimeInSeconds, so train_job.py stops on its own and still calibrates, evaluates and exports. 97

More on each path: Fine-tune. Terms: Glossary.

Sources

Section titled “Sources”

Footnotes

Section titled “Footnotes”

  1. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/model.go L133 ↩

  2. openjevx @ v0.5.9 (ee2a1f4) · README.md L81–83 ↩

  3. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L247–252 ↩

  4. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/model.go L70 ↩ ↩2

  5. openjevx @ v0.5.9 (ee2a1f4) · README.md L81 ↩

  6. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/model.go L64–70 ↩

  7. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/source.go L90–93 ↩

  8. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L122–124 ↩

  9. openjevx @ v0.5.9 (ee2a1f4) · README.md L65 ↩

  10. openjevx @ v0.5.9 (ee2a1f4) · README.md L79–81 ↩

  11. openjevx @ v0.5.9 (ee2a1f4) · scripts/package.sh L7 ↩

  12. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/model.go L60 ↩ ↩2

  13. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/model.go L58–61 ↩

  14. openjevx @ v0.5.9 (ee2a1f4) · docs/RELEASE_PROCESS.md L67 ↩

  15. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L383 ↩

  16. openjevx @ v0.5.9 (ee2a1f4) · README.md L24–25 ↩

  17. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L574–576 ↩

  18. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L372 ↩

  19. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L381–383 ↩

  20. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L381–386 ↩

  21. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L574–575 ↩

  22. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/main.go L365–367 ↩

  23. openjevx @ v0.5.9 (ee2a1f4) · deploy/docker-entrypoint.sh L7–9 ↩

  24. openjevx @ v0.5.9 (ee2a1f4) · deploy/docker-entrypoint.sh L11–13 ↩

  25. openjevx @ v0.5.9 (ee2a1f4) · README.md L169–170 ↩

  26. openjevx @ v0.5.9 (ee2a1f4) · README.md L166 ↩ ↩2

  27. openjevx @ v0.5.9 (ee2a1f4) · README.md L170–171 ↩

  28. openjevx @ v0.5.9 (ee2a1f4) · README.md L156–160 ↩

  29. openjevx @ v0.5.9 (ee2a1f4) · deploy/docker-entrypoint.sh L10 ↩

  30. openjevx @ v0.5.9 (ee2a1f4) · deploy/docker-entrypoint.sh L14 ↩

  31. openjevx @ v0.5.9 (ee2a1f4) · deploy/docker-entrypoint.sh L7–14 ↩

  32. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L182–186 ↩

  33. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L23 ↩

  34. openjevx @ v0.5.9 (ee2a1f4) · README.md L31 ↩

  35. openjevx @ v0.5.9 (ee2a1f4) · README.md L185 ↩

  36. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L76–78 ↩

  37. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L84–87 ↩ ↩2

  38. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L172 ↩ ↩2

  39. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L25–28 ↩

  40. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L33 ↩

  41. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L103–106 ↩

  42. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L207–208 ↩

  43. openjevx @ v0.5.9 (ee2a1f4) · README.md L38–40 ↩ ↩2

  44. openjevx @ v0.5.9 (ee2a1f4) · README.md L202–204 ↩ ↩2 ↩3

  45. openjevx @ v0.5.9 (ee2a1f4) · README.md L204 ↩

  46. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L35 ↩

  47. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L94–97 ↩

  48. openjevx @ v0.5.9 (ee2a1f4) · cmd/openjevx/auth.go L34–35 ↩

  49. openjevx @ v0.5.9 (ee2a1f4) · README.md L42–43 ↩

  50. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0011-inference-runtime.md L32–34 ↩ ↩2

  51. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0011-inference-runtime.md L34–35 ↩

  52. openjevx @ v0.5.9 (ee2a1f4) · README.md L52–53 ↩ ↩2

  53. jev-cloud @ origin/main (10a744c) · docs/fargate-retest-2026-10-03.md L23–25 ↩

  54. openjevx @ v0.5.9 (ee2a1f4) · README.md L49 ↩

  55. openjevx @ v0.5.9 (ee2a1f4) · README.md L54–55 ↩

  56. jev-cloud @ origin/main (10a744c) · docs/fargate-retest-2026-10-03.md L14–20 ↩

  57. openjevx @ v0.5.9 (ee2a1f4) · README.md L55–57 ↩

  58. openjevx @ v0.5.9 (ee2a1f4) · README.md L56–57 ↩

  59. openjevx @ v0.5.9 (ee2a1f4) · llmresults/14-x86-cpu-latency.md L3–5 ↩

  60. openjevx @ v0.5.9 (ee2a1f4) · llmresults/14-x86-cpu-latency.md L26–31 ↩

  61. openjevx @ v0.5.9 (ee2a1f4) · README.md L188 ↩ ↩2 ↩3

  62. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0011-inference-runtime.md L30 ↩ ↩2

  63. openjevx @ v0.5.9 (ee2a1f4) · bin/platform.js L8–11 ↩ ↩2

  64. openjevx @ v0.5.9 (ee2a1f4) · bin/platform.js L1–7 ↩

  65. openjevx @ v0.5.9 (ee2a1f4) · README.md L220 ↩

  66. openjevx @ v0.5.9 (ee2a1f4) · README.md L92 ↩

  67. jev-cloud @ origin/main (10a744c) · docs/model-s3-contract.md L39 ↩

  68. openjevx @ v0.5.9 (ee2a1f4) · README.md L91–92 ↩

  69. openjevx @ v0.5.9 (ee2a1f4) · README.md L103–105 ↩ ↩2

  70. openjevx @ v0.5.9 (ee2a1f4) · README.md L101 ↩

  71. openjevx @ v0.5.9 (ee2a1f4) · README.md L103–106 ↩

  72. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0012-model-from-object-store.md L39–40 ↩ ↩2 ↩3

  73. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0012-model-from-object-store.md L37–38 ↩ ↩2

  74. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0012-model-from-object-store.md L37 ↩

  75. openjevx @ v0.5.9 (ee2a1f4) · README.md L109–110 ↩ ↩2 ↩3

  76. openjevx @ v0.5.9 (ee2a1f4) · README.md L98 ↩

  77. openjevx @ v0.5.9 (ee2a1f4) · finetuning/dataprep/import_csv.py L170–171 ↩

  78. jev-cloud @ origin/main (10a744c) · client/README.md L90–91 ↩

  79. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L123 ↩

  80. openjevx @ v0.5.9 (ee2a1f4) · docs/adr/0002-training-run.md L18 ↩ ↩2 ↩3

  81. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L306 ↩ ↩2

  82. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L306–308 ↩

  83. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L189–190 ↩

  84. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L189–191 ↩

  85. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L190–194 ↩

  86. openjevx @ v0.5.9 (ee2a1f4) · finetuning/ft.py L14–15 ↩

  87. openjevx @ v0.5.9 (ee2a1f4) · docs/book/train-your-own-jev.md L313–314 ↩

  88. jev-cloud @ origin/main (10a744c) · train/sagemaker/sm_train.py L52 ↩

  89. jev-cloud @ origin/main (10a744c) · train/sagemaker/sm_train.py L249–256 ↩

  90. jev-cloud @ origin/main (10a744c) · train/sagemaker/README.md L64–66 ↩

  91. jev-cloud @ origin/main (10a744c) · client/README.md L94–95 ↩

  92. jev-cloud @ origin/main (10a744c) · train/sagemaker/README.md L66–68 ↩

  93. jev-cloud @ origin/main (10a744c) · train/sagemaker/sm_train.py L257–260 ↩

  94. jev-cloud @ origin/main (10a744c) · train/sagemaker/README.md L69 ↩

  95. jev-cloud @ origin/main (10a744c) · train/sagemaker/sm_train.py L259–260 ↩

  96. jev-cloud @ origin/main (10a744c) · train/sagemaker/README.md L55–56 ↩ ↩2

  97. jev-cloud @ origin/main (10a744c) · train/sagemaker/README.md L54–55 ↩