Upgrade to 5090 in pegasus (production server) causes preprocessor failures due to old versions of pytorch

[jeffbl]

Production server has been down since we upgraded it to a 5090 from a 4090 gpu.

TLDR: sm_120 required for 5090, but pytorch version on the below preprocessors/services support max sm_90

• object-detection (if we deprecate YOLO, this would be resolved, pending testing of qwen Select and validate the production vision-language model #1167)
• multilang-support
• espnet-tts
• semantic-segmentation: error only comes up when making a request (see below), whereas others come up on imageup
• PROBABLY espnet-tts-fr : I don't see error in logs, but maybe it isn't hitting warmup or something. if it is working, then that would point to a fix for espnet-tts!

OPTIONS:

• put 4090 back in to unicorn (if we can't solve relatively quickly), since it supported earlier pytorch. Note this may require driver rollback as well. Not sure. To get 5090 working, we had to update pegasus OS to 24.04 and the nvidia drivers to a much more current version.
• fix these preprocessors/services.
• deprecate these preprocessors/services in favor of newer tools.

The last option is obviously the best since we're moving in that direction anyway, but having production down means anyone downloading the IMAGE extension and trying it is just getting errors back.

Looking for this problem in pegasus logs after ./bin/imageup

jeffbl@pegasus /v/d/image [SIGPIPE|SIGINT]> ./bin/imagelogs | grep sm_120
multilang-support-1            | 2026-07-06T14:03:49.165789243Z NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
object-detection-1                        | 2026-07-06T14:03:49.688917810Z NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
espnet-tts-1                              | 2026-07-06T14:19:42.217363324Z NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.

semantic-segmentation logs when making request to pegasus:

semantic-segmentation-1                   | 2026-07-06T14:23:11.059494651Z   warnings.warn(
semantic-segmentation-1                   | 2026-07-06T14:23:11.463412398Z load checkpoint from local path: /app/upernet_beit-base_8x2_640x640_160k_ade20k-eead221d.pth
semantic-segmentation-1                   | 2026-07-06T14:23:11.971749695Z /opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:145: UserWarning: 
semantic-segmentation-1                   | 2026-07-06T14:23:11.971771727Z NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
semantic-segmentation-1                   | 2026-07-06T14:23:11.971774141Z The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 compute_37.
semantic-segmentation-1                   | 2026-07-06T14:23:11.971776215Z If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
semantic-segmentation-1                   | 2026-07-06T14:23:11.971778169Z 
semantic-segmentation-1                   | 2026-07-06T14:23:11.971780113Z   warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
semantic-segmentation-1                   | 2026-07-06T14:23:40.854954991Z [2026-07-06 14:23:40 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
orchestrator-1                            | 2026-07-06T14:23:41.857241377Z req=a7da6dcd-c7ad-4928-ac6d-2b1779165906 Error occurred while fetching from preprocessor "semantic-segmentation"
orchestrator-1                            | 2026-07-06T14:23:41.857272957Z req=a7da6dcd-c7ad-4928-ac6d-2b1779165906 timestamp=2026-07-06T14:23:41.857Z label=Preprocessor "semantic-segmentation" execution_time_ms=31405.87ms cpu_time_ms=59.65ms normalized_cpu_usage_percent=0.01%
orchestrator-1                            | 2026-07-06T14:23:41.857386151Z req=a7da6dcd-c7ad-4928-ac6d-2b1779165906 FetchError: request to http://semantic-segmentation:5000/preprocessor failed, reason: read ECONNRESET
orchestrator-1                            | 2026-07-06T14:23:41.857730192Z req=a7da6dcd-c7ad-4928-ac6d-2b1779165906 TotalRequestExecutionTime execution_time_ms=32012.69ms
image-pegasus-cim-mcgill-ca-1             | 2026-07-06T14:23:41.857991697Z 174.91.220.117 - - [06/Jul/2026:14:23:41 +0000] "POST /render HTTP/1.1" 500 101 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36"
semantic-segmentation-1                   | 2026-07-06T14:23:42.082208826Z [2026-07-06 14:23:42 +0000] [1] [ERROR] Worker (pid:8) was sent SIGKILL! Perhaps out of memory?
semantic-segmentation-1                   | 2026-07-06T14:23:42.083686692Z [2026-07-06 14:23:42 +0000] [197] [INFO] Booting worker with pid: 197
semantic-segmentation-1                   | 2026-07-06T14:23:43.097145846Z INFO:root:Environment Pegasus: PII logging is disabled.
semantic-segmentation-1                   | 2026-07-06T14:23:43.097162527Z DEBUG:root:Suppressed verbose logging for OpenAI client libraries

Example multilang-support logs from pegasus:

multilang-support-1                       | 2026-07-06T14:03:49.086767009Z 26-07-06 14:03 UTC [INFO]: Translator(en, fr) instantiated!                                                                                                                        
multilang-support-1                       | 2026-07-06T14:03:49.165772291Z /opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py:287: UserWarning:                                                                                                   
multilang-support-1                       | 2026-07-06T14:03:49.165789243Z NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.                                                                       
multilang-support-1                       | 2026-07-06T14:03:49.165791287Z The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_37 sm_90.                                                                            
multilang-support-1                       | 2026-07-06T14:03:49.165792940Z If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/                                         
multilang-support-1                       | 2026-07-06T14:03:49.165794583Z                                                                                                                                                                                    
multilang-support-1                       | 2026-07-06T14:03:49.165796006Z   warnings.warn(                                                                                                                                                                   
espnet-tts-1                              | 2026-07-06T14:03:49.266483830Z 2026-07-06 14:03:49,266 Fallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_ty
pe = 'latest'.                                                                                                                                                                                                                                                
espnet-tts-1                              | 2026-07-06T14:03:49.266498107Z 2026-07-06 14:03:49,266 Fallback to conformer_self_attn_layer_type = 'legacy_rel_selfattn' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_l
ayer_type = 'latest'.                                                                                                                                                                                                                                         
multilang-support-1                       | 2026-07-06T14:03:49.335366898Z 26-07-06 14:03 UTC [INFO]: Translator(en, fr) running on NVIDIA GeForce RTX 5090                                                                                                   
multilang-support-1                       | 2026-07-06T14:03:49.335384902Z 26-07-06 14:03 UTC [INFO]: Translation service is instantiated and ready!                                                                                                          
multilang-support-1                       | 2026-07-06T14:03:49.335387907Z 26-07-06 14:03 UTC [DEBUG]: (1) Tokenizing input.                                                                                                                                  
multilang-support-1                       | 2026-07-06T14:03:49.335695097Z 26-07-06 14:03 UTC [DEBUG]: (2) Generating tensor.                                                                                                                                 
multilang-support-1                       | 2026-07-06T14:03:49.336234854Z [transformers] The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.                               
multilang-support-1                       | 2026-07-06T14:03:49.344989341Z [2026-07-06 14:03:49 +0000] [7] [ERROR] Exception in worker process                                                                                                                
multilang-support-1                       | 2026-07-06T14:03:49.344996825Z Traceback (most recent call last):                                                                                                                                                 
multilang-support-1                       | 2026-07-06T14:03:49.344998478Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/arbiter.py", line 713, in spawn_worker
multilang-support-1                       | 2026-07-06T14:03:49.344999981Z     worker.init_process()                           
multilang-support-1                       | 2026-07-06T14:03:49.345001163Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/workers/base.py", line 136, in init_process
multilang-support-1                       | 2026-07-06T14:03:49.345002446Z     self.load_wsgi()                                
multilang-support-1                       | 2026-07-06T14:03:49.345003538Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/workers/base.py", line 148, in load_wsgi
multilang-support-1                       | 2026-07-06T14:03:49.345004710Z     self.wsgi = self.app.wsgi()                     
multilang-support-1                       | 2026-07-06T14:03:49.345005852Z                 ^^^^^^^^^^^^^^^                     
multilang-support-1                       | 2026-07-06T14:03:49.345006984Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/base.py", line 66, in wsgi
multilang-support-1                       | 2026-07-06T14:03:49.345015149Z     self.callable = self.load()                     
multilang-support-1                       | 2026-07-06T14:03:49.345016342Z                     ^^^^^^^^^^^                     
multilang-support-1                       | 2026-07-06T14:03:49.345017434Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 57, in load
multilang-support-1                       | 2026-07-06T14:03:49.345018616Z     return self.load_wsgiapp()                      
multilang-support-1                       | 2026-07-06T14:03:49.345019698Z            ^^^^^^^^^^^^^^^^^^^                      
multilang-support-1                       | 2026-07-06T14:03:49.345020810Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
multilang-support-1                       | 2026-07-06T14:03:49.345022022Z     return util.import_app(self.app_uri)
multilang-support-1                       | 2026-07-06T14:03:49.345023155Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345024277Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/util.py", line 411, in import_app
multilang-support-1                       | 2026-07-06T14:03:49.345025459Z     mod = importlib.import_module(module)
multilang-support-1                       | 2026-07-06T14:03:49.345026581Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345027873Z   File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
multilang-support-1                       | 2026-07-06T14:03:49.345029046Z     return _bootstrap._gcd_import(name[level:], package, level)
multilang-support-1                       | 2026-07-06T14:03:49.345030178Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345031310Z   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
multilang-support-1                       | 2026-07-06T14:03:49.345032773Z   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
multilang-support-1                       | 2026-07-06T14:03:49.345034075Z   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
multilang-support-1                       | 2026-07-06T14:03:49.345035638Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
multilang-support-1                       | 2026-07-06T14:03:49.345036800Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
multilang-support-1                       | 2026-07-06T14:03:49.345038003Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
multilang-support-1                       | 2026-07-06T14:03:49.345039185Z   File "/app/src/translate.py", line 4, in <module>
multilang-support-1                       | 2026-07-06T14:03:49.345040397Z     from .utils import LOGGER, Translator, SUPPORTED_LANGS
multilang-support-1                       | 2026-07-06T14:03:49.345041519Z   File "/app/src/utils.py", line 182, in <module>
multilang-support-1                       | 2026-07-06T14:03:49.345042711Z     .translate([ready_message])[0].pop())
multilang-support-1                       | 2026-07-06T14:03:49.345044024Z      ^^^^^^^^^^^^^^^^^^^^^^^^^^                     
multilang-support-1                       | 2026-07-06T14:03:49.345045196Z   File "/app/src/utils.py", line 137, in translate
multilang-support-1                       | 2026-07-06T14:03:49.345046358Z     output_tensor = self.generate_output_tensor(input_tensor)
multilang-support-1                       | 2026-07-06T14:03:49.345047500Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345048633Z   File "/app/src/utils.py", line 98, in generate_output_tensor
multilang-support-1                       | 2026-07-06T14:03:49.345049785Z     return self.MODEL.generate(input_ids,
multilang-support-1                       | 2026-07-06T14:03:49.345051087Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345054193Z   File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
multilang-support-1                       | 2026-07-06T14:03:49.345055445Z     return func(*args, **kwargs)                    
multilang-support-1                       | 2026-07-06T14:03:49.345056578Z            ^^^^^^^^^^^^^^^^^^^^^                    
multilang-support-1                       | 2026-07-06T14:03:49.345057690Z   File "/opt/conda/lib/python3.11/site-packages/transformers/generation/utils.py", line 2442, in generate
multilang-support-1                       | 2026-07-06T14:03:49.345058912Z     self._prepare_special_tokens(generation_config, kwargs_has_attention_mask, device=device)
multilang-support-1                       | 2026-07-06T14:03:49.345060104Z   File "/opt/conda/lib/python3.11/site-packages/transformers/generation/utils.py", line 2017, in _prepare_special_tokens
multilang-support-1                       | 2026-07-06T14:03:49.345061327Z     if eos_token_tensor is not None and torch.isin(eos_token_tensor, pad_token_tensor).any():
multilang-support-1                       | 2026-07-06T14:03:49.345062489Z                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-06T14:03:49.345063631Z RuntimeError: CUDA error: no kernel image is available for execution on the device
multilang-support-1                       | 2026-07-06T14:03:49.345064783Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
multilang-support-1                       | 2026-07-06T14:03:49.345065965Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1
multilang-support-1                       | 2026-07-06T14:03:49.345067077Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
multilang-support-1                       | 2026-07-06T14:03:49.345068220Z                                                     
multilang-support-1                       | 2026-07-06T14:03:49.345069322Z CUDA error: no kernel image is available for execution on the device
multilang-support-1                       | 2026-07-06T14:03:49.345070464Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
multilang-support-1                       | 2026-07-06T14:03:49.345071636Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1
multilang-support-1                       | 2026-07-06T14:03:49.345072778Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
multilang-support-1                       | 2026-07-06T14:03:49.345073920Z                                                     
multilang-support-1                       | 2026-07-06T14:03:49.345082216Z [2026-07-06 14:03:49 +0000] [7] [INFO] Worker exiting (pid: 7)

[jeffbl]

did a quick attempt to put the problematic services on CPU rather than GPU. Pegasus is responding to requests again, with notes/caveats:

1. multilang service is still failing, so i don't think French will work.
2. semantic segmentation is removed from production profile, to prevent it trying and failing.
3. I'm pointing it at gemma4:31b since I didn't want to roll back models, and yolo object detection would take up too much GPU memory without reducing gemma's context window.
4. It is in a hacked docker-compose.yml in my homedir, since an override apparently can't knock out parts of the YAML.

Performance is still good, thanks to pegasus being very fast, and the 5090 GPU upgrade. There may be boundary cases that still don't work, but I figure this state is better than completely down and just giving an error when the production extension makes requests.

Currently implemented in a custom docker-compose.yml that can be set up on pegasus with commands below. When resolving this issue, will need to migrate the differences to the main docker-compose.yml, but how much transfers depends on how everything is eventually resolved:

cd /var/docker/image
ln -s ~/docker-compose.yml.20260714cpu ./docker-compose.yml
./bin/imageup

multilang-support log errors:

multilang-support-1                       | 2026-07-14T15:24:14.821900226Z 26-07-14 15:24 UTC [INFO]: Loading model from local path: /app/models/opus-mt-en-fr                                                 
espnet-tts-1                              | 2026-07-14T15:24:15.242152533Z 2026-07-14 15:24:15,242 Fallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' due to the compatibility. If you want to use the
 new one, please use conformer_pos_enc_layer_type = 'latest'.                                                                                                                                                  
espnet-tts-1                              | 2026-07-14T15:24:15.242175517Z 2026-07-14 15:24:15,242 Fallback to conformer_self_attn_layer_type = 'legacy_rel_selfattn' due to the compatibility. If you want to 
use the new one, please use conformer_pos_enc_layer_type = 'latest'.                                                                                                                                           
Loading weights: 100%|██████████| 254/254 [00:00<00:00, 11078.27it/s]2318Z                                                                                                                                     
multilang-support-1                       | 2026-07-14T15:24:15.865672304Z 26-07-14 15:24 UTC [INFO]: Translator(en, fr) instantiated!                                                                         
multilang-support-1                       | 2026-07-14T15:24:15.866667400Z 26-07-14 15:24 UTC [ERROR]: 'Translator' object has no attribute 'DEVICE_NAME'                                                      
multilang-support-1                       | 2026-07-14T15:24:15.866704871Z 26-07-14 15:24 UTC [INFO]: Failed to instantiate Translator(en, fr)!                                                                
multilang-support-1                       | 2026-07-14T15:24:15.866716663Z 26-07-14 15:24 UTC [DEBUG]: Expected model path: /app/models/opus-mt-en-fr                                                          
multilang-support-1                       | 2026-07-14T15:24:15.870263580Z 26-07-14 15:24 UTC [INFO]: Translation service is instantiated and ready!                                                           
multilang-support-1                       | 2026-07-14T15:24:15.870743675Z [2026-07-14 15:24:15 +0000] [7] [ERROR] Exception in worker process                                                                 
multilang-support-1                       | 2026-07-14T15:24:15.870749156Z Traceback (most recent call last):                                                                                                  
multilang-support-1                       | 2026-07-14T15:24:15.870750578Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/arbiter.py", line 713, in spawn_worker                                     
multilang-support-1                       | 2026-07-14T15:24:15.870752051Z     worker.init_process()                                                                                                           
multilang-support-1                       | 2026-07-14T15:24:15.870753223Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/workers/base.py", line 136, in init_process                                
multilang-support-1                       | 2026-07-14T15:24:15.870754506Z     self.load_wsgi()                                                                                                                
multilang-support-1                       | 2026-07-14T15:24:15.870755798Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/workers/base.py", line 148, in load_wsgi                                   
multilang-support-1                       | 2026-07-14T15:24:15.870757050Z     self.wsgi = self.app.wsgi()                                                                                                     
multilang-support-1                       | 2026-07-14T15:24:15.870758193Z                 ^^^^^^^^^^^^^^^                                                                                                     
multilang-support-1                       | 2026-07-14T15:24:15.870759305Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/base.py", line 66, in wsgi                                             
multilang-support-1                       | 2026-07-14T15:24:15.870760767Z     self.callable = self.load()                                                                                                     
multilang-support-1                       | 2026-07-14T15:24:15.870761880Z                     ^^^^^^^^^^^                                                                                                     
multilang-support-1                       | 2026-07-14T15:24:15.870763012Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 57, in load
multilang-support-1                       | 2026-07-14T15:24:15.870764214Z     return self.load_wsgiapp()                                                                                                      
multilang-support-1                       | 2026-07-14T15:24:15.870765306Z            ^^^^^^^^^^^^^^^^^^^                                                                                                      
multilang-support-1                       | 2026-07-14T15:24:15.870766408Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 47, in load_wsgiapp
multilang-support-1                       | 2026-07-14T15:24:15.870767610Z     return util.import_app(self.app_uri)
multilang-support-1                       | 2026-07-14T15:24:15.870768732Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-14T15:24:15.870769915Z   File "/opt/conda/lib/python3.11/site-packages/gunicorn/util.py", line 411, in import_app
multilang-support-1                       | 2026-07-14T15:24:15.870777699Z     mod = importlib.import_module(module)
multilang-support-1                       | 2026-07-14T15:24:15.870778952Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-14T15:24:15.870780094Z   File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
multilang-support-1                       | 2026-07-14T15:24:15.870781286Z     return _bootstrap._gcd_import(name[level:], package, level)
multilang-support-1                       | 2026-07-14T15:24:15.870782919Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multilang-support-1                       | 2026-07-14T15:24:15.870784081Z   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
multilang-support-1                       | 2026-07-14T15:24:15.870785434Z   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
multilang-support-1                       | 2026-07-14T15:24:15.870786726Z   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
multilang-support-1                       | 2026-07-14T15:24:15.870787969Z   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
multilang-support-1                       | 2026-07-14T15:24:15.870789131Z   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
multilang-support-1                       | 2026-07-14T15:24:15.870790303Z   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
multilang-support-1                       | 2026-07-14T15:24:15.870791465Z   File "/app/src/translate.py", line 4, in <module>
multilang-support-1                       | 2026-07-14T15:24:15.870792748Z     from .utils import LOGGER, Translator, SUPPORTED_LANGS
multilang-support-1                       | 2026-07-14T15:24:15.870794531Z   File "/app/src/utils.py", line 182, in <module>
multilang-support-1                       | 2026-07-14T15:24:15.870825019Z     .translate([ready_message])[0].pop())
multilang-support-1                       | 2026-07-14T15:24:15.870830168Z      ^^^^^^^^^                                                                                                                      
multilang-support-1                       | 2026-07-14T15:24:15.870834787Z AttributeError: 'NoneType' object has no attribute 'translate'
multilang-support-1                       | 2026-07-14T15:24:15.870847932Z 'NoneType' object has no attribute 'translate'
multilang-support-1                       | 2026-07-14T15:24:15.870861407Z [2026-07-14 15:24:15 +0000] [7] [INFO] Worker exiting (pid: 7)
multilang-support-1                       | 2026-07-14T15:24:16.519287525Z [2026-07-14 15:24:16 +0000] [1] [ERROR] Worker (pid:7) exited with code 3.
multilang-support-1                       | 2026-07-14T15:24:16.519697277Z [2026-07-14 15:24:16 +0000] [1] [ERROR] Shutting down: Master
multilang-support-1                       | 2026-07-14T15:24:16.519713969Z [2026-07-14 15:24:16 +0000] [1] [ERROR] Reason: Worker failed to boot.
multilang-support-1 exited with code 3                                                                 

[Miliya-Ai]

Current understanding of how language information might propagate through the pipeline

Browser Extension
        │
        │  Language specified by user?
        │  (Request parameter / Accept-Language header?)
        ▼
POST /render
        │
        ▼
Request Model
────────────────────────────────────────────
Knows current language?     ❓ Need to verify
Needs modification?         ✓ Probably
Already supports it?        ❓ Unknown
────────────────────────────────────────────
        │
        ▼
Pipeline Context
────────────────────────────────────────────
Knows current language?     Should store it
Needs modification?         ✓ Probably
Already supports it?        ❓ Unknown
────────────────────────────────────────────
        │
        ├──────────────┐
        │              │
        ▼              ▼

OCR                    Caption Model
──────────────────     ──────────────────
Needs language? ✓      Needs language? ✓
Current state?  ?      Current state?  ?
Can choose locale? ?   Can generate directly
                       in target language?

        │
        └──────────────┐
                       ▼

LLM Prompt Builder
────────────────────────────────────────────
Needs language?         ✓ Yes
Current prompt?         English only?
Should prompt specify:
"Respond in <language>" ?
────────────────────────────────────────────
                       │
                       ▼

Translation Layer?
────────────────────────────────────────────
Does it exist?          ❓
Needed at all?          ❓
Or should models generate
the target language directly?
────────────────────────────────────────────
                       │
                       ▼

Response
────────────────────────────────────────────
Contains language field?     ❓
Returns localized content?   ?
Needs schema update?         ✓ Maybe
────────────────────────────────────────────

[jeffbl]

The existing multilang service is much more straightforward, I think, in that (from my understanding) it just takes a string in one language and translates it to the target. See details here:
https://github.com/Shared-Reality-Lab/IMAGE-server/tree/main/services/multilang-support

From a brief search, it looks like the approach we were using is still valid, so just fixing this service might be the fastest route to success, if we can get it running on CPU-only. (Which we can develop&test on unicorn). Apart from paid APIs like deepl, and azure, which has a huge free quota if we went that router, and numerous MCP endpoints, a copilot chat search also suggested https://docs.libretranslate.com/ as an alternative.