Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
80e0549
feat: initial queue implementation
danielboloc Jun 9, 2026
8bae61d
docs: update version
danielboloc Jun 11, 2026
200ff1f
feat: create queues from scratch
danielboloc Jun 15, 2026
0bdaf58
refactor: update ux
danielboloc Jun 15, 2026
5381769
refactor: update queue
danielboloc Jun 15, 2026
5de372a
refactor: add message for azure
danielboloc Jun 15, 2026
6d5535a
refactor: add default option
danielboloc Jun 15, 2026
742b969
review: initial ai review
danielboloc Jun 16, 2026
587f8f6
pytest: fix
danielboloc Jun 16, 2026
69f7689
docs: update changelog
danielboloc Jun 16, 2026
ba62050
Apply suggestions from code review
danielboloc Jun 16, 2026
7c5e3e0
Apply suggestions from code review
danielboloc Jun 16, 2026
812a3f7
Apply suggestions from code review
danielboloc Jun 16, 2026
8955d16
Apply suggestions from code review
danielboloc Jun 16, 2026
8e8ecb7
Apply suggestions from code review
danielboloc Jun 16, 2026
b8e6670
Potential fix for pull request finding
danielboloc Jun 16, 2026
bc94381
Apply suggestions from code review
danielboloc Jun 16, 2026
08c49ec
Add test for NoJobQueuesAvailableException in fetch_job_queue_id
Copilot Jun 16, 2026
824eb48
Potential fix for pull request finding
danielboloc Jun 16, 2026
345d4e8
docs: update README with queue generation
danielboloc Jun 19, 2026
c7aee29
refactor: remove unused method and its tests
danielboloc Jun 19, 2026
6a74d60
style: move import at the top
danielboloc Jun 19, 2026
668a62d
refactor: remove --add-compute-env
danielboloc Jun 19, 2026
1a8bc77
Merge branch 'main' into create-queue
danielboloc Jun 23, 2026
03d1cbb
feat: add all instances
danielboloc Jun 23, 2026
549905f
revert: disable --from-scratch until api is open
danielboloc Jun 26, 2026
2f629ef
test: disable --from-scratch tests
danielboloc Jun 26, 2026
4b92200
refactor: remove --from-scratch helpers
danielboloc Jun 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
## lifebit-ai/cloudos-cli: changelog

## v2.94.0 (2026-06-23)

### Feat:

- Adds `cloudos queue create` with preset templates (e.g. `standard-stable`, `standard-gpu`)
- Adds `--set-default` to mark a new queue as the workspace default

## v2.93.1 (2026-06-12)

### Fix:
Expand Down
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Python package for interacting with Lifebit Platform
- [Create Projects](#create-projects)
- [Queue](#queue)
- [List Queues](#list-queues)
- [Create Queue](#create-queue)
- [Workflow](#workflow)
- [List All Available Workflows](#list-all-available-workflows)
- [Import a Nextflow Workflow](#import-a-nextflow-workflow)
Expand Down Expand Up @@ -441,6 +442,35 @@ cloudos queue list --profile my_profile --output-format csv

> NOTE: The queue name that is visible in Lifebit Platform and must be used with the `--job-queue` parameter is the one in the `label` field.

#### Create Queue

You can create a new AWS batch job queue in your Lifebit Platform workspace using the `queue create` command. By default the queue is built from a preset template, selected with the `--preset` option. Available presets are:

- **standard-stable** (default): On-demand stable instances
- **standard-cost-saving**: Spot instances (up to ~80% cheaper, at risk of premature termination)
- **read-write-optimised**: On-demand instances with increased disk I/O performance
- **standard-gpu**: On-demand standard instances plus GPU instances

To create a queue from a preset:

```bash
cloudos queue create --profile my_profile --label "my-new-queue" --description "Queue for RNA-seq jobs" --preset standard-stable
```

Before creating the queue, a summary is shown and confirmation is requested. To skip the confirmation prompt, add the `-y`/`--yes` flag. You can also set the new queue as the workspace default with `--set-default`.

The expected output is something similar to:

```console
Executing queue create...
Queue "my-new-queue" created successfully.
Queue ID : 64f1a23b8e4c9d001234abcd
View at : https://cloudos.lifebit.ai/app/job-queues/64f1a23b8e4c9d001234abcd
```

> [!NOTE]
> **Azure Platform**: Batch job queues are an AWS-only feature and are not available in Azure or HPC workspaces.

**Job queues for platform workflows**

Platform workflows (those provided by Lifebit Platform in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, Lifebit Platform will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
Expand Down
2 changes: 1 addition & 1 deletion cloudos_cli/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '2.93.1'
__version__ = '2.94.0'
2 changes: 1 addition & 1 deletion cloudos_cli/queue/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
from .queue import Queue


__all__ = ['queue']
__all__ = ['Queue']
197 changes: 196 additions & 1 deletion cloudos_cli/queue/cli.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,48 @@
"""CLI commands for Lifebit Platform job queue management."""

import sys
import rich_click as click
import json
from cloudos_cli.queue.queue import Queue
from rich.console import Console
from cloudos_cli.queue.queue import (
Queue,
QUEUE_PRESETS,
MAX_WORKSPACE_COMPUTE_ENVS,
WORKSPACE_CE_LIMIT_REACHED_MESSAGE,
)
from cloudos_cli.utils.resources import ssl_selector
from cloudos_cli.configure.configure import with_profile_config, CLOUDOS_URL
from cloudos_cli.utils.cli_helpers import pass_debug_to_subcommands
from cloudos_cli.utils.details import create_queue_list_table


# Workflow executors supported for job queues.
_EXECUTORS = ["nextflow", "cromwell"]


def _check_workspace_ce_limit(console, j_queue, queues=None):
"""Exit with a warning if the workspace has reached its CE limit.

Parameters
----------
console : rich.console.Console
The console used for rich output.
j_queue : Queue
The queue client used to count compute environments.
queues : list or None, optional
A pre-fetched list of job queue dicts. If ``None``, the queues are
fetched.
"""
count = j_queue.count_workspace_compute_environments(queues=queues)
if count >= MAX_WORKSPACE_COMPUTE_ENVS:
console.print(
f"[yellow]Warning:[/yellow] {WORKSPACE_CE_LIMIT_REACHED_MESSAGE}"
)
sys.exit(0)




# Create the queue group
@click.group(cls=pass_debug_to_subcommands())
def queue():
Expand Down Expand Up @@ -50,6 +84,10 @@ def queue():
@click.option('--exclude-system-queues',
help='Exclude system job queues from the list.',
is_flag=True)
@click.option('--execution-platform',
help='Name of the execution platform implemented in your Lifebit Platform. Default=aws.',
type=click.Choice(['aws', 'azure', 'hpc']),
default='aws')
@click.option('--disable-ssl-verification',
help=('Disable SSL certificate verification. Please, remember that this option is ' +
'not generally recommended for security reasons.'),
Expand All @@ -67,13 +105,24 @@ def list_queues(ctx,
output_format,
all_fields,
exclude_system_queues,
execution_platform,
disable_ssl_verification,
ssl_cert,
profile):
"""Collect and display all available job queues from a Lifebit Platform workspace."""
# apikey, cloudos_url, and workspace_id are now automatically resolved by the decorator

verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)

# Batch job queues are an AWS-only feature; they are not available in
# Azure or HPC workspaces.
if execution_platform in ('azure', 'hpc'):
Console().print(
'[yellow]Warning:[/yellow] Batch job queues are not available in '
f'{execution_platform.upper()} workspaces.'
)
sys.exit(0)
Comment thread
danielboloc marked this conversation as resolved.

print('Executing list...')
j_queue = Queue(cloudos_url, apikey, None, workspace_id, verify=verify_ssl)
my_queues = j_queue.get_job_queues(exclude_system_queues=exclude_system_queues)
Expand All @@ -93,3 +142,149 @@ def list_queues(ctx,
o.write(json.dumps(my_queues))
print(f'\tJob queue list collected with a total of {len(my_queues)} queues.')
print(f'\tJob queue list saved to {outfile}')


@queue.command('create')
@click.option('-k',
'--apikey',
help='Your Lifebit Platform API key',
required=True)
@click.option('-c',
'--cloudos-url',
help=(f'The Lifebit Platform url you are trying to access to. Default={CLOUDOS_URL}.'),
default=CLOUDOS_URL,
required=True)
@click.option('--workspace-id',
help='The specific Lifebit Platform workspace id.',
required=True)
@click.option('--label',
help='Name (label) for the new job queue.',
required=False,
default=None)
@click.option('--description',
help='Short description of the new job queue.',
default='',
required=False)
@click.option('--preset',
help=(
'Preset template to use. Choices: '
+ ', '.join(QUEUE_PRESETS.keys())
+ '. Default=standard-stable.'
),
type=click.Choice(list(QUEUE_PRESETS.keys()), case_sensitive=False),
default='standard-stable',
show_default=True,
required=False)
@click.option('--executor',
help='Workflow executor for the queue. Default=nextflow.',
type=click.Choice(_EXECUTORS, case_sensitive=False),
default='nextflow',
show_default=True,
required=False)
@click.option('-y',
'--yes',
'skip_confirmation',
help='Skip the confirmation prompt and proceed immediately.',
is_flag=True)
@click.option('--set-default',
help='Set the new job queue as the workspace default. Default=False.',
is_flag=True)
@click.option('--execution-platform',
help='Name of the execution platform implemented in your Lifebit Platform. Default=aws.',
type=click.Choice(['aws', 'azure', 'hpc']),
default='aws')
@click.option('--disable-ssl-verification',
help=('Disable SSL certificate verification. Please, remember that this option is '
'not generally recommended for security reasons.'),
is_flag=True)
@click.option('--ssl-cert',
help='Path to your SSL certificate file.')
@click.option('--profile', help='Profile to use from the config file', default=None)
@click.pass_context
@with_profile_config(required_params=['apikey', 'workspace_id'])
def create_queue(ctx,
apikey,
cloudos_url,
workspace_id,
label,
description,
preset,
executor,
skip_confirmation,
set_default,
execution_platform,
disable_ssl_verification,
ssl_cert,
profile):
"""Create a new job queue in a Lifebit Platform workspace.

A preset template is used to create the queue.
"""

verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)

# Batch job queues are an AWS-only feature; they are not available in
# Azure or HPC workspaces.
if execution_platform in ('azure', 'hpc'):
Console().print(
'[yellow]Warning:[/yellow] Batch job queues are not available in '
f'{execution_platform.upper()} workspaces.'
)
sys.exit(0)
Comment thread
danielboloc marked this conversation as resolved.

# --description is required when creating a queue.
if not description:
raise click.UsageError('Missing option --description.')

if label is None:
raise click.UsageError('Missing option --label.')

# Resolve the preset to show the user what will be created
preset_info = QUEUE_PRESETS[preset]
ce_name = preset_info['computeEnvironmentName']
cr = preset_info['computeResources']
resource_type = cr.get('type', 'EC2')
max_vcpus_preset = cr.get('maxvCpus', 'N/A')
instance_count = len(cr.get('instanceTypes', []))
template_name = preset_info['templateName']

j_queue = Queue(cloudos_url, apikey, None, workspace_id, verify=verify_ssl)

# Creating a queue creates a compute environment; enforce the workspace limit.
_check_workspace_ce_limit(Console(), j_queue)

if not skip_confirmation:
click.echo('\nYou are about to create the following job queue:')
click.echo(f' Label : {label}')
click.echo(f' Description : {description or "(none)"}')
click.echo(f' Preset : {template_name}')
click.echo(f' Compute env name : {ce_name}')
click.echo(f' Resource type : {resource_type}')
click.echo(f' Max vCPUs : {max_vcpus_preset}')
click.echo(f' Instance types : {instance_count} types')
click.echo(f' Executor : {executor}')
click.echo(f' Set as default : {set_default}')
click.echo(f' Workspace : {workspace_id}')
click.echo('')
if not click.confirm('Proceed with queue creation?'):
click.echo('Aborted.')
sys.exit(0)

console = Console()
print('Executing queue create...')

try:
queue_id = j_queue.create_job_queue(
label=label,
description=description,
preset_name=preset,
executor=executor,
is_default=set_default,
)
console.print(f'\t[green]Queue "{label}" created successfully.[/green]')
print(f'\tQueue ID : {queue_id}')
print(f'\tView at : {cloudos_url}/app/job-queues/{queue_id}')
except Exception as e:
console.print(f'\t[red]Error creating queue:[/red] {str(e)}')
sys.exit(1)

Loading
Loading