Skip to content

Add multi-storage support with managed upload workflow #49

Description

@mgajek-cern

Summary

Extend k8s-tutorial to demonstrate Rucio data management across multiple storage technologies (S3, dCache) using the managed upload pattern.

Motivation

Current tutorial only demonstrates XRootD storage. Production Rucio deployments typically integrate with diverse storage backends (object stores, grid storage, distributed filesystems).

Proposed Changes

  • Add S3-like storage, e.g. Localstack (2 instances) - cloud-native object storage
  • Add dCache storage (2 instances) - grid storage with WebDAV
  • Update workflow script demonstrating managed upload pattern
  • Update deployment scripts to handle all storage types preserving previous behavior and opt-in feature flags for multi-storage demonstration

Workflow demonstrated:

  1. Upload files to source RSEs (XRD1, S3_1, DCACHE_1)
  2. Create datasets and containers
  3. Create replication rules to destination RSEs
  4. FTS3 orchestrates third-party copy transfers
  5. Monitor transfer completion

Testing

./scripts/deploy-rucio.sh --enable-all-storage

# Auto-detects and configures enabled storages
./scripts/use-rucio.sh

Future Work

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

discussionTopics to talk through before actingenhancementNew feature or request

Type

No type

Fields

Size

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions