diff --git a/docs/integrations/backup/acronis-cyber-protect.mdx b/docs/integrations/backup/acronis-cyber-protect.mdx new file mode 100644 index 0000000..94db506 --- /dev/null +++ b/docs/integrations/backup/acronis-cyber-protect.mdx @@ -0,0 +1,38 @@ +--- +title: Acronis Cyber Protect +description: Configure a Fil One storage location for Acronis backup and restore. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Acronis Cyber Protect + +Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 (AuthV4) access key and secret scoped to the bucket's region | + +## Configure + +In the Acronis administration console, add an S3-compatible storage location. Enter the regional endpoint, the matching access key, and AuthV4 signing from [Connection settings](/integrations/connection-settings). Select an existing Fil One bucket when prompted, then assign the location to a protection plan. + +## Examples + +1. Protect the fixture directory with the plan, run it, and confirm the activity completes. +2. Restore to a new location and compare size and SHA-256 for every restored file. +3. Confirm that retention and cleanup operations affect only the test backup set. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection failure | Recheck endpoint, regional key, AuthV4, and TLS validation. | +| Missing bucket | Create it first and ensure the key is scoped to it. | +| Immutability option | This depends on Object Lock semantics. Review [S3 Compatibility](/reference/s3-compatibility) and validate it with the exact Acronis release before enabling it. | diff --git a/docs/integrations/backup/comet-backup.mdx b/docs/integrations/backup/comet-backup.mdx new file mode 100644 index 0000000..b05dbf6 --- /dev/null +++ b/docs/integrations/backup/comet-backup.mdx @@ -0,0 +1,39 @@ +--- +title: Comet Backup +description: Configure and validate a Comet S3-compatible Storage Vault with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Comet Backup + +Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (some Comet fields expect the hostname without `https://`; follow the field label) | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +Create either a Comet Storage Template or a user's Custom Storage Vault, select the S3-compatible type, and enter the regional hostname, region, existing bucket, and matching credentials. Enable encrypted transmission. Use the same values described in [Connection settings](/integrations/connection-settings). + +## Examples + +1. Run Comet's connection test, protect the fixture directory, and complete an on-demand backup. +2. Restore to a separate directory, run Comet's vault integrity check, and inspect the job history for partial failures. +3. Remove only the test snapshot and prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection test fails | Confirm whether the Comet field expects a hostname or a full URL. | +| Access errors | Check that the key can read, write, list, and delete within the selected bucket. | +| Object Lock vault type | Use the standard S3-compatible storage type unless the Object Lock workflow has been validated with your Comet release; see [S3 Compatibility](/reference/s3-compatibility). | +| Slow or failing large backups | Test concurrency and recovery time from the same network where the Comet server runs. | diff --git a/docs/integrations/backup/msp360.mdx b/docs/integrations/backup/msp360.mdx new file mode 100644 index 0000000..cfcbff3 --- /dev/null +++ b/docs/integrations/backup/msp360.mdx @@ -0,0 +1,38 @@ +--- +title: MSP360 +description: Add Fil One as S3-compatible storage for MSP360 backup plans. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# MSP360 + +MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 (signature version 4) access key and secret scoped to the bucket's region | + +## Configure + +Add an S3-compatible storage account. Put the regional URL in the HTTPS endpoint field, choose signature version 4, keep certificate validation enabled, and select an existing bucket. Enable native multipart upload after validating it with your MSP360 release. The desktop agent exposes the same core settings. + +## Examples + +1. Create a test plan for the fixture directory and run it. +2. Confirm the remote objects are listed, restore to a new directory, and compare size and SHA-256. +3. Delete only the isolated test backup set. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection or signature errors | Use the S3-compatible provider rather than an AWS preset, confirm path-style addressing, and keep HTTPS enabled. | +| Bucket discovery fails | Check the bucket's region and the key's scope. | +| Backup succeeds but restore fails | Preserve the job logs and rerun the round trip after correcting the settings. | diff --git a/docs/integrations/backup/nakivo.mdx b/docs/integrations/backup/nakivo.mdx new file mode 100644 index 0000000..b29dfbe --- /dev/null +++ b/docs/integrations/backup/nakivo.mdx @@ -0,0 +1,37 @@ +--- +title: NAKIVO Backup and Replication +description: Configure Fil One as an S3-compatible NAKIVO repository. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# NAKIVO Backup and Replication + +NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +Add an S3-compatible repository. Enter the regional endpoint, region, matching credentials, and an existing bucket. Keep HTTPS certificate validation enabled. Ensure the Transporter that performs data movement can reach the endpoint. + +## Examples + +1. Back up a disposable workload with known file hashes, then perform file-level recovery to a different location and compare size and SHA-256. +2. Run NAKIVO's repository verification if available. +3. Exercise retention against only the test recovery point and confirm unrelated prefixes remain untouched. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection errors | Endpoint, key region, and path-style settings must match. | +| Retention or immutability options | Validate against the named NAKIVO release and current [S3 Compatibility](/reference/s3-compatibility) behavior. | diff --git a/docs/integrations/connection-settings.mdx b/docs/integrations/connection-settings.mdx new file mode 100644 index 0000000..3f68241 --- /dev/null +++ b/docs/integrations/connection-settings.mdx @@ -0,0 +1,20 @@ +--- +title: Integration connection settings +description: Shared endpoint, region, authentication, and addressing settings for Fil One integrations. +--- + +# Integration connection settings + +Use the values shown in your bucket's **Access endpoints** panel. Keep endpoint values in configuration rather than embedding them in scripts or application code. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (also shown in the bucket's **Access endpoints** panel) | +| Region | The bucket's region: `eu-west-1` or `us-east-1` | +| Authentication | AWS Signature Version 4 (SigV4) | +| Addressing | Path style required | +| Access key and secret | A key created for the same region, scoped to the target bucket | + +Create the bucket before configuring an integration. Endpoint, region, key, and bucket must belong to the same region. Keep TLS certificate validation enabled. Never place a secret key in documentation, screenshots, source control, or command-line history. + +Third-party tools may attempt optional S3 operations after a transfer. Review [S3 Compatibility](/reference/s3-compatibility) for current regional behavior and [Encryption](/security/encryption) for security settings. Feature availability differs between the S3 API and the app.fil.one dashboard; see S3 Compatibility for API behavior. diff --git a/docs/integrations/data-ai/apache-iceberg.mdx b/docs/integrations/data-ai/apache-iceberg.mdx new file mode 100644 index 0000000..1f5cace --- /dev/null +++ b/docs/integrations/data-ai/apache-iceberg.mdx @@ -0,0 +1,38 @@ +--- +title: Apache Iceberg +description: Evaluate Apache Iceberg object storage workflows with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Apache Iceberg + +Apache Iceberg reads and writes table data in Fil One through `S3FileIO`, with `s3.endpoint` set to the regional endpoint, `s3.path-style-access=true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`client.region`) | +| Addressing | Path style (`s3.path-style-access=true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the engine's secret mechanism | + +## Configure + +Use Iceberg's `S3FileIO` when your engine supports it. Configure the warehouse on an existing bucket, the regional endpoint and region, path-style access, and credentials through the engine's secret mechanism. Catalog configuration is separate: REST, JDBC, Hive, Glue-compatible, and filesystem catalogs have different correctness and concurrency properties. + +## Examples + +1. Create a disposable table, append and update data, and read it from a fresh session. +2. Validate partition pruning and inspect metadata and manifest objects under the dedicated warehouse prefix. +3. Run concurrent commits and confirm one writer does not silently overwrite another. +4. Preserve the metadata tree, recover the table from its catalog, and compare row counts and content hashes. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Some components reach an AWS hostname | An engine may use more than one S3 client; apply endpoint and path-style settings to every execution component. | +| Reads work but commits fail | Check the catalog configuration and the engine and catalog versions. | diff --git a/docs/integrations/data-ai/databricks.mdx b/docs/integrations/data-ai/databricks.mdx new file mode 100644 index 0000000..54f2eb1 --- /dev/null +++ b/docs/integrations/data-ai/databricks.mdx @@ -0,0 +1,37 @@ +--- +title: Databricks +description: Evaluate Databricks access to data stored in Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Databricks + +Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region, stored in a Databricks secret scope | + +## Configure + +Use a cluster or SQL environment that permits a custom S3-compatible endpoint. Store credentials in a Databricks secret scope and inject them into the supported filesystem or connector configuration. Set the regional endpoint, region, and path-style access from [Connection settings](/integrations/connection-settings). Keep secret values out of notebooks and cluster policies. + +## Examples + +1. Read a disposable Parquet dataset, write results to a dedicated prefix, terminate the compute, and read the output from a fresh cluster. +2. Compare row counts, schema, and deterministic aggregates. +3. For managed table formats, validate their transaction and concurrency behavior separately. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Serverless or governed mode blocks custom endpoints | Use a compute mode that permits them, or treat the integration as unavailable for that workspace. | +| Driver works but jobs fail | Confirm that executors as well as the driver receive endpoint and credential settings. | diff --git a/docs/integrations/data-ai/delta-lake.mdx b/docs/integrations/data-ai/delta-lake.mdx new file mode 100644 index 0000000..c8bb482 --- /dev/null +++ b/docs/integrations/data-ai/delta-lake.mdx @@ -0,0 +1,37 @@ +--- +title: Delta Lake +description: Evaluate Delta Lake transaction-log workflows on Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Delta Lake + +Delta Lake on Spark reads and writes Fil One through the `s3a` connector, with `fs.s3a.endpoint` set to the regional endpoint, `fs.s3a.path.style.access=true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`fs.s3a.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`fs.s3a.endpoint.region`) | +| Addressing | Path style (`fs.s3a.path.style.access=true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the platform's secret mechanism | + +## Configure + +Configure Spark's S3 connector with the existing bucket, regional endpoint and region, path-style access, and credentials supplied through the platform's secret mechanism. Keep a dedicated table prefix. Ensure the driver and every executor receive the same settings. + +## Examples + +1. Create a disposable Delta table, append and update rows, produce a checkpoint, and read the table from a fresh session. +2. Run concurrent writers and verify conflicts do not cause silent data loss. +3. Restore a prior table version after confirming the required log and data objects exist, then compare row counts and deterministic hashes. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Parquet reads work but Delta commits fail | Connector retry, rename, conditional-write, and consistency assumptions vary by release; check the Delta, Spark, and connector version set. | +| Executors reach an AWS hostname | Apply the endpoint and path-style settings to executors as well as the driver. | diff --git a/docs/integrations/data-ai/duckdb.mdx b/docs/integrations/data-ai/duckdb.mdx new file mode 100644 index 0000000..da6496d --- /dev/null +++ b/docs/integrations/data-ai/duckdb.mdx @@ -0,0 +1,51 @@ +--- +title: DuckDB +description: Read and write Fil One objects with DuckDB httpfs. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# DuckDB + +DuckDB reads and writes Fil One through the `httpfs` extension with an S3 secret that sets `ENDPOINT` to the regional hostname, `URL_STYLE 'path'`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `eu-west-1.s3.fil.one` or `us-east-1.s3.fil.one`, matching the bucket's region (`ENDPOINT`, hostname without `https://`, with `USE_SSL true`) | +| Region | `eu-west-1` or `us-east-1` (`REGION`) | +| Addressing | Path style (`URL_STYLE 'path'`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`KEY_ID` and `SECRET`) | + +## Configure + +Install and load DuckDB's `httpfs` extension. Create an S3 secret with the regional key, region, endpoint hostname, `USE_SSL true`, and `URL_STYLE 'path'`. Use a secret provider appropriate to your environment and keep credentials out of SQL files. + +```sql +INSTALL httpfs; +LOAD httpfs; +CREATE SECRET filone ( + TYPE S3, + KEY_ID 'ACCESS_KEY', + SECRET 'SECRET_KEY', + REGION 'eu-west-1', + ENDPOINT 'eu-west-1.s3.fil.one', + USE_SSL true, + URL_STYLE 'path' +); +``` + +## Examples + +1. Read a disposable Parquet file from the bucket and compare row count and deterministic aggregates. +2. Use `COPY` to write a result under a unique prefix, start a fresh DuckDB process, and read the output back. Compare schema, row count, byte size, and content-level checks before removing the prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Requests target a bucket subdomain | Set `URL_STYLE 'path'`. | +| Endpoint rejected | The `ENDPOINT` field expects a hostname rather than a full URL. | +| Large datasets | Test range reads and multipart writes separately. | diff --git a/docs/integrations/data-ai/hugging-face-datasets.mdx b/docs/integrations/data-ai/hugging-face-datasets.mdx new file mode 100644 index 0000000..4c46189 --- /dev/null +++ b/docs/integrations/data-ai/hugging-face-datasets.mdx @@ -0,0 +1,37 @@ +--- +title: Hugging Face Datasets +description: Evaluate Hugging Face Datasets access to files stored in Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Hugging Face Datasets + +Hugging Face `datasets` reads files in Fil One through `fsspec` and `s3fs`, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`client_kwargs={"endpoint_url": ...}`) | +| Region | `eu-west-1` or `us-east-1` (`client_kwargs={"region_name": ...}`) | +| Addressing | Path style (`config_kwargs={"s3": {"addressing_style": "path"}}`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`key` and `secret`) | + +## Configure + +Pass the regional endpoint, region, matching credentials, and path-style behavior through the storage options supported by the installed `s3fs` release. Load credentials from a secret store or environment and keep them out of notebooks. Use explicit object paths under an existing bucket. Pin and record the complete Python package set, because endpoint and addressing options can change across `datasets`, `fsspec`, and `s3fs`. + +## Examples + +1. Load a disposable dataset normally and in streaming mode. Compare feature schema, row count, and deterministic sample hashes. +2. Clear the local cache and load it again to confirm the remote path is sufficient. +3. If preprocessing writes cache or transformed data remotely, isolate that output under a unique prefix and validate its cleanup. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Load works once, then fails | A local cache can mask remote errors; clear the cache and repeat. | +| Requests target a bucket subdomain | Set the path-style addressing option for the installed `s3fs` release. | diff --git a/docs/integrations/data-ai/ray-data.mdx b/docs/integrations/data-ai/ray-data.mdx new file mode 100644 index 0000000..b2f8e8d --- /dev/null +++ b/docs/integrations/data-ai/ray-data.mdx @@ -0,0 +1,37 @@ +--- +title: Ray Data +description: Evaluate distributed Ray Data reads and writes with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Ray Data + +Ray Data reads and writes Fil One through a PyArrow `S3FileSystem` with `endpoint_override` set to the regional endpoint, `region` set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`endpoint_override`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (PyArrow uses path style with `endpoint_override` unless `force_virtual_addressing` is set) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, distributed through the cluster's secret mechanism | + +## Configure + +Create the S3 filesystem with the regional endpoint, region, path-style access, and matching credentials. Distribute credentials through the cluster's secret mechanism and confirm every worker receives the endpoint configuration. Use a dedicated input and output prefix. + +## Examples + +1. Read a disposable multi-file Parquet dataset across several workers, apply a deterministic transformation, and write the result to a unique prefix. +2. Start a fresh Ray job, read the output, and compare schema, row count, partition count where relevant, and deterministic aggregates. +3. Exercise worker loss and retry under a bounded test before relying on recovery. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Driver succeeds, workers fail | Inspect worker logs for bucket-subdomain requests, credential errors, and retry loops. | +| Results differ between runs | Validate the exact Ray and PyArrow versions at production concurrency. | diff --git a/docs/integrations/data-movement/aws-cli-sdk.mdx b/docs/integrations/data-movement/aws-cli-sdk.mdx new file mode 100644 index 0000000..0819a9d --- /dev/null +++ b/docs/integrations/data-movement/aws-cli-sdk.mdx @@ -0,0 +1,36 @@ +--- +title: AWS CLI and SDKs +description: Use existing Fil One AWS CLI and SDK guides for data movement workflows. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# AWS CLI and SDKs + +The AWS CLI and SDKs are Fil One's reference clients; see the [SDK guides](/sdk/overview) for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (`forcePathStyle: true` in JavaScript, `UsePathStyle: true` in Go, `addressing_style: path` in boto3 and the CLI) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +Follow the guide for your client: [AWS CLI](/sdk/cli), [Python with boto3](/sdk/python), [JavaScript with AWS SDK v3](/sdk/javascript), or [Go with aws-sdk-go-v2](/sdk/go). Apply the [connection settings](/integrations/connection-settings) to each client. + +## Examples + +Include a file above the client's multipart threshold so automatic multipart upload is exercised. For supported operations and regional differences, see [S3 Compatibility](/reference/s3-compatibility); for object and part sizes, see [Limits & Quotas](/limits). + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Requests go to a bucket subdomain | Set the client's path-style option. | +| Signature mismatch or `403` | Endpoint, region, and key must belong to the same region and the key must be scoped to the bucket. | +| Optional operation returns `403` or `NotImplemented` | Compare against [S3 Compatibility](/reference/s3-compatibility). | diff --git a/docs/integrations/data-movement/cyberduck-mountain-duck.mdx b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx new file mode 100644 index 0000000..14621ad --- /dev/null +++ b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx @@ -0,0 +1,42 @@ +--- +title: Cyberduck and Mountain Duck +description: Connect Cyberduck or Mountain Duck to a Fil One bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Cyberduck and Mountain Duck + +Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (enter the hostname without a bucket name or path) | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (Cyberduck's path-style S3 connection profile) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +1. Install Cyberduck's path-style S3 connection profile from the profiles list. +2. Open **New Bookmark**, choose that profile, and enter the hostname from the bucket's regional endpoint. +3. Enter the access key and secret for the same region. +4. Keep HTTPS and certificate validation enabled. Save the bookmark. + +Mountain Duck uses the same bookmark settings, then mounts the bucket through Finder or File Explorer. + +## Examples + +1. Rename and delete a test object and confirm the remote listing reflects each operation. +2. Test keys containing spaces before a larger migration. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Client requests a bucket subdomain | Use the path-style S3 connection profile rather than the default Amazon S3 profile. | +| `403` | Key, endpoint, and region must belong to the same region and the key must be scoped to the bucket. | +| Unexpected filename encoding | Test keys with spaces and review regional behavior in [S3 Compatibility](/reference/s3-compatibility). | diff --git a/docs/integrations/data-movement/rclone.mdx b/docs/integrations/data-movement/rclone.mdx new file mode 100644 index 0000000..e2ef0ef --- /dev/null +++ b/docs/integrations/data-movement/rclone.mdx @@ -0,0 +1,57 @@ +--- +title: rclone +description: Configure rclone for upload, restore, and integrity checks with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# rclone + +rclone connects to Fil One as an S3 remote with `provider = Other`, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (`force_path_style = true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | +| Provider | `Other` | + +## Configure + +Run `rclone config`, create a new remote, choose **Amazon S3 Compliant Storage Providers**, and select **Other** as the provider. Enter the bucket's region and endpoint when prompted, enable path-style access, and enter credentials at rclone's prompts. The resulting section in `rclone.conf` looks like this: + +```ini +[filone] +type = s3 +provider = Other +access_key_id = ACCESS_KEY +secret_access_key = SECRET_KEY +region = eu-west-1 +endpoint = https://eu-west-1.s3.fil.one +force_path_style = true +``` + +Test discovery without changing remote data: + +```bash +rclone lsd filone: +``` + +## Examples + +1. Copy with `rclone copy ./fixture filone:BUCKET/example/` and list with `rclone lsjson filone:BUCKET/example/`. +2. Restore with `rclone copy filone:BUCKET/example/ ./restored`. +3. Run `rclone check --download ./restored filone:BUCKET/example/`. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Signature mismatch | Endpoint, region, and key must refer to the same region; confirm `force_path_style = true`. | +| Bucket not found | Confirm the bucket already exists and the key is scoped to it. | +| TLS error | Keep certificate validation enabled; inspect local proxy or trust-store configuration. | +| Cleanup leaves data | List the exact prefix and remove only that prefix. | diff --git a/docs/integrations/devops/harbor.mdx b/docs/integrations/devops/harbor.mdx new file mode 100644 index 0000000..bc46879 --- /dev/null +++ b/docs/integrations/devops/harbor.mdx @@ -0,0 +1,39 @@ +--- +title: Harbor +description: Configure Harbor registry storage on a Fil One S3-compatible bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Harbor + +Harbor stores image layers in Fil One through the registry's `s3` storage driver, with `regionendpoint` set to the regional endpoint, `forcepathstyle: true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`regionendpoint`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`forcepathstyle: true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`v4auth: true`) | +| TLS | `secure: true` | + +## Configure + +Set Harbor's registry storage driver to `s3`. Provide the existing bucket, `regionendpoint`, `region`, `secure: true`, and `forcepathstyle: true`. Store credentials in the deployment's secret mechanism. Use a dedicated bucket or prefix so registry garbage collection cannot affect unrelated objects. Back up Harbor's database separately; the registry bucket holds only image layers. + +## Examples + +1. Push a disposable image with several layers, remove the local image, and pull it again. Compare the image digest and start a container. +2. Inspect Harbor and registry logs. +3. Delete the test artifact, run garbage collection in dry-run mode first, then confirm only unreferenced test layers are removed. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Redirects or bucket-subdomain requests | Set `forcepathstyle: true`. | +| `403` | Key region does not match the endpoint, or the key is not scoped to the bucket. | +| Multipart or garbage-collection failures | Rerun the validation with the exact Harbor release before production rollout. | diff --git a/docs/integrations/devops/terraform.mdx b/docs/integrations/devops/terraform.mdx new file mode 100644 index 0000000..b8ab064 --- /dev/null +++ b/docs/integrations/devops/terraform.mdx @@ -0,0 +1,50 @@ +--- +title: Terraform S3 backend +description: Evaluate a Fil One bucket for Terraform remote state. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Terraform S3 backend + +Terraform's `s3` backend stores state in Fil One with `endpoints { s3 = ... }` set to the regional endpoint and `use_path_style = true` (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`endpoints { s3 = ... }`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`use_path_style = true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied outside the configuration | + +## Configure + +Use the `s3` backend with an existing bucket, a dedicated state key, the regional endpoint, matching region, and path-style addressing. Keep credentials outside the Terraform configuration. Evaluate the backend's non-AWS options `skip_credentials_validation`, `skip_requesting_account_id`, `skip_metadata_api_check`, and `skip_s3_checksum` for your Terraform release, and enable only the ones that release needs. + +```hcl +terraform { + backend "s3" { + bucket = "BUCKET_NAME" + key = "terraform/ENVIRONMENT.tfstate" + region = "REGION" + endpoints = { s3 = "ENDPOINT_URL" } + use_path_style = true + } +} +``` + +## Examples + +1. Initialize against a disposable configuration, apply it, and confirm a subsequent `terraform plan` is empty. +2. Copy state to an offline recovery location, remove the local working directory, reinitialize, and confirm state can be read. +3. Simulate two concurrent writers before relying on locking. Test any S3 lock-file option with the exact Terraform version and region before enabling it. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Initialization contacts an AWS hostname | Check the custom endpoint and path-style options supported by your Terraform release. | +| Account or credential lookups fail | Evaluate the `skip_*` options listed above. | +| Backend migration | Preserve an offline state backup before migrating. | diff --git a/docs/integrations/devops/velero.mdx b/docs/integrations/devops/velero.mdx new file mode 100644 index 0000000..83d85c5 --- /dev/null +++ b/docs/integrations/devops/velero.mdx @@ -0,0 +1,38 @@ +--- +title: Velero +description: Configure Velero object storage for Kubernetes backup and restore with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Velero + +Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets `s3Url` to the regional endpoint, `s3ForcePathStyle: "true"`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3Url`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`s3ForcePathStyle: "true"`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through a Kubernetes Secret | + +## Configure + +Install Velero with its AWS object-store plugin. Configure a backup storage location with the existing bucket, `region`, `s3Url`, and `s3ForcePathStyle: "true"`. Supply the key through a Kubernetes Secret rather than a manifest committed to source control. Volume-snapshot support is separate from object storage and requires its own provider. + +## Examples + +1. Create a disposable namespace containing representative objects and persistent data. +2. Run a backup, inspect `velero backup describe --details` and the logs, delete the namespace, then restore it. +3. Verify resource counts, application data, and file hashes. Confirm that expiration or cleanup affects only the test backup. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Storage location unavailable | Endpoint, region, secret, bucket scope, or `s3ForcePathStyle` is wrong. | +| Version mismatch | Check Velero, plugin, and Kubernetes compatibility as a version set. | +| Metadata restores but volumes do not | Volume recovery needs a snapshot or file-system backup provider; test it separately. | diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx new file mode 100644 index 0000000..2bff1a3 --- /dev/null +++ b/docs/integrations/index.mdx @@ -0,0 +1,32 @@ +--- +title: Integrations +description: Configure third-party tools with Fil One object storage. +--- + +# Integrations + +These guides show how to point common transfer, backup, NAS, DevOps, analytics, and AI tools at Fil One. Start with [Connection settings](/integrations/connection-settings), then open the guide for your tool. For API behavior, see the [S3 Compatibility reference](/reference/s3-compatibility). + +| Category | Tool | +| --- | --- | +| Data movement | [rclone](/integrations/data-movement/rclone) | +| Data movement | [Cyberduck and Mountain Duck](/integrations/data-movement/cyberduck-mountain-duck) | +| Data movement | [AWS CLI and SDKs](/integrations/data-movement/aws-cli-sdk) | +| Backup and disaster recovery | [Acronis Cyber Protect](/integrations/backup/acronis-cyber-protect) | +| Backup and disaster recovery | [Comet Backup](/integrations/backup/comet-backup) | +| Backup and disaster recovery | [MSP360](/integrations/backup/msp360) | +| Backup and disaster recovery | [NAKIVO Backup and Replication](/integrations/backup/nakivo) | +| NAS and file services | [Synology Cloud Sync](/integrations/nas/synology-cloud-sync) | +| NAS and file services | [QNAP HBS](/integrations/nas/qnap-hbs) | +| NAS and file services | [TrueNAS Cloud Sync](/integrations/nas/truenas-cloud-sync) | +| Kubernetes and DevOps | [Velero](/integrations/devops/velero) | +| Kubernetes and DevOps | [Harbor](/integrations/devops/harbor) | +| Kubernetes and DevOps | [Terraform S3 backend](/integrations/devops/terraform) | +| SIEM and logging | [Elasticsearch searchable snapshots](/integrations/logging/elasticsearch-snapshots) | +| SIEM and logging | [Splunk SmartStore](/integrations/logging/splunk-smartstore) | +| Data and AI | [Apache Iceberg](/integrations/data-ai/apache-iceberg) | +| Data and AI | [Databricks](/integrations/data-ai/databricks) | +| Data and AI | [Delta Lake](/integrations/data-ai/delta-lake) | +| Data and AI | [DuckDB](/integrations/data-ai/duckdb) | +| Data and AI | [Hugging Face Datasets](/integrations/data-ai/hugging-face-datasets) | +| Data and AI | [Ray Data](/integrations/data-ai/ray-data) | diff --git a/docs/integrations/logging/elasticsearch-snapshots.mdx b/docs/integrations/logging/elasticsearch-snapshots.mdx new file mode 100644 index 0000000..7ff036e --- /dev/null +++ b/docs/integrations/logging/elasticsearch-snapshots.mdx @@ -0,0 +1,39 @@ +--- +title: Elasticsearch searchable snapshots +description: Configure Elasticsearch S3 snapshots with Fil One for recovery testing. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Elasticsearch searchable snapshots + +Elasticsearch stores snapshots in Fil One through an `s3` repository whose named client sets `endpoint`, `region`, and `path_style_access: true`, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes `repository-s3`; on 7.x install the plugin on every node. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3.client.CLIENT.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`s3.client.CLIENT.region`) | +| Addressing | Path style (`s3.client.CLIENT.path_style_access: true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, stored in the keystore as `s3.client.CLIENT.access_key` and `s3.client.CLIENT.secret_key` | + +## Configure + +Store the access and secret keys in the Elasticsearch keystore under a named S3 client. Configure that client's endpoint, region, and path-style access, then register an `s3` snapshot repository using an existing bucket and a dedicated `base_path`. Run the repository verification API from every master and data node before taking a snapshot. Searchable snapshots require an Enterprise subscription; ordinary snapshot storage does not. + +## Examples + +1. Create a disposable index with known documents, take a snapshot, delete the index, and restore it under a new name. +2. Compare document counts and selected content hashes. +3. If searchable snapshots are in scope, mount the snapshot and execute representative queries. +4. Test cleanup only against the dedicated repository path. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Repository verification fails on some nodes | Confirm every node can reach the endpoint and has the same client settings and keystore entries. | +| Endpoint syntax errors | Review the endpoint format for your Elasticsearch release. | +| Verification passes but restore fails | Repository verification checks connectivity only; rerun the snapshot and restore steps. | diff --git a/docs/integrations/logging/splunk-smartstore.mdx b/docs/integrations/logging/splunk-smartstore.mdx new file mode 100644 index 0000000..045b63a --- /dev/null +++ b/docs/integrations/logging/splunk-smartstore.mdx @@ -0,0 +1,37 @@ +--- +title: Splunk SmartStore +description: Evaluate Fil One as remote storage for Splunk SmartStore. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Splunk SmartStore + +Splunk SmartStore uses Fil One as a `remote` volume in `indexes.conf`, with `remote.s3.endpoint` set to the regional endpoint, `remote.s3.auth_region` set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`remote.s3.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`remote.s3.auth_region`) | +| Addressing | Path style (`remote.s3.url_version = v1`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`remote.s3.signature_version = v4`) | + +## Configure + +Define a `remote` volume in `indexes.conf` with the existing bucket, regional endpoint, auth region, path-style URL mode, and versioning behavior appropriate to the bucket. Reference the volume from each intended index with `remotePath`. Store credentials through the deployment's protected configuration workflow and apply the settings consistently across an indexer cluster. SmartStore licensing and topology requirements apply. + +## Examples + +1. Index disposable events, roll data to remote storage, and confirm searches can fetch evicted buckets. +2. Restart or replace a test indexer and verify remote recovery. Compare event counts and representative hashes or fields. +3. Exercise cache eviction under a bounded test load before choosing concurrency and cache-size settings. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Remote errors | Use `splunk btool` to inspect effective configuration; check endpoint, auth region, URL style, clock skew, and credential scope. | +| Uploads succeed but searches fail after eviction | Review indexer and cache-manager logs and rerun the search-after-eviction step. | diff --git a/docs/integrations/nas/qnap-hbs.mdx b/docs/integrations/nas/qnap-hbs.mdx new file mode 100644 index 0000000..473ed2c --- /dev/null +++ b/docs/integrations/nas/qnap-hbs.mdx @@ -0,0 +1,38 @@ +--- +title: QNAP HBS +description: Configure QNAP Hybrid Backup Sync with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# QNAP HBS + +QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +In HBS 3, create a storage space of the S3-compatible type. Enter the regional endpoint, region, SigV4 credentials, and existing bucket. Enable SSL and certificate validation. Create a backup or sync job targeting a dedicated remote prefix. + +## Examples + +1. Run the backup job for the fixture, delete the local fixture, and restore it to a new folder; compare size and SHA-256. +2. Review the HBS job log for skipped files. +3. Test retention or sync deletion only inside the validation prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Timestamp or signature errors | Enable NTP on the NAS; choose SigV4 and the correct region. | +| Connection fails | Confirm the service address includes HTTPS and does not construct a bucket subdomain. | +| Large-file failures | Capture a repeatable failure before reducing concurrency or chunk size. | diff --git a/docs/integrations/nas/synology-cloud-sync.mdx b/docs/integrations/nas/synology-cloud-sync.mdx new file mode 100644 index 0000000..73d2388 --- /dev/null +++ b/docs/integrations/nas/synology-cloud-sync.mdx @@ -0,0 +1,38 @@ +--- +title: Synology Cloud Sync +description: Configure Synology Cloud Sync with a Fil One bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Synology Cloud Sync + +Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +## Configure + +Install Cloud Sync, add a connection, and choose S3 storage with a custom server URL. Enter the regional endpoint, SigV4 credentials, and existing bucket. Choose a local path, remote path, and one of bidirectional, upload-only, or download-only synchronization. Keep TLS validation enabled. + +## Examples + +1. Sync the fixture with an upload-only task to a dedicated remote prefix and wait for the task to report up to date. +2. Remove the local fixture and create a download-only recovery task to a separate directory; compare size and SHA-256. +3. Test rename and delete semantics before choosing bidirectional sync, because that mode can propagate deletions. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Empty bucket dropdown | Endpoint, region, credentials, or bucket scope do not match. | +| Signature failures | Correct NAS time with NTP. | +| Large files stall | Test a smaller chunk size within the multipart guidance in [Limits & Quotas](/limits). | diff --git a/docs/integrations/nas/truenas-cloud-sync.mdx b/docs/integrations/nas/truenas-cloud-sync.mdx new file mode 100644 index 0000000..9db7b91 --- /dev/null +++ b/docs/integrations/nas/truenas-cloud-sync.mdx @@ -0,0 +1,38 @@ +--- +title: TrueNAS Cloud Sync +description: Configure a TrueNAS Cloud Sync push, pull, or sync task with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# TrueNAS Cloud Sync + +TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region (leave signature version 2 disabled) | + +## Configure + +In TrueNAS SCALE 22 and later, create the credential under Backup Credentials; in TrueNAS CORE, use the System cloud credentials page. Choose Amazon S3 with a custom endpoint, enter the regional key and region, keep signature version 2 disabled, and verify the credential. Create a Cloud Sync task using an existing bucket and a dedicated folder. + +Choose the COPY transfer mode for the first test. SYNC can delete destination files that are absent at the source, and MOVE deletes source files after transfer. + +## Examples + +1. Run a PUSH of the fixture, then a PULL into a separate dataset; compare size and SHA-256. +2. Inspect the task logs and confirm cleanup stays within the test folder. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Credential verification fails | Recheck endpoint, region, key scope, SigV4, and system time. | +| Task succeeds with missing files | Compare the source and restored manifests before increasing concurrency. | diff --git a/docs/integrations/validate-an-integration.mdx b/docs/integrations/validate-an-integration.mdx new file mode 100644 index 0000000..b169a35 --- /dev/null +++ b/docs/integrations/validate-an-integration.mdx @@ -0,0 +1,16 @@ +--- +title: Check a setup +description: Optional steps to copy a small fixture, restore it, and compare checksums. +--- + +# Check a setup + +If you want to confirm a tool can write to a bucket and read the same bytes back, use disposable data and a prefix you can delete. + +1. Create a local folder with a small text file, a nested file, and a file large enough to trigger multipart upload in the tool (above the tool's part or chunk size; see [Limits & Quotas](/limits)). +2. Copy it to a unique prefix in the bucket, for example `check-YYYYMMDD-HHMM/`. +3. List the prefix and confirm every file is present. +4. Remove the local copy. +5. Restore the prefix to a new local directory. +6. Compare size and SHA-256 for every restored file against the values recorded before upload. +7. Delete only that prefix, then list it again to confirm it is empty. diff --git a/package.json b/package.json index ebfeca8..775443f 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,8 @@ "swizzle": "docusaurus swizzle", "deploy": "docusaurus deploy", "clear": "docusaurus clear", - "serve": "docusaurus serve" + "serve": "docusaurus serve", + "check:integrations": "node scripts/check-integrations.mjs" }, "dependencies": { "@docusaurus/core": "^3.9.2", diff --git a/scripts/check-integrations.mjs b/scripts/check-integrations.mjs new file mode 100644 index 0000000..a81135d --- /dev/null +++ b/scripts/check-integrations.mjs @@ -0,0 +1,70 @@ +import fs from "node:fs"; +import path from "node:path"; + +const root = path.resolve("docs/integrations"); +const guides = []; + +function walk(directory) { + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { + const absolute = path.join(directory, entry.name); + if (entry.isDirectory()) walk(absolute); + else if (entry.name.endsWith(".mdx") && path.dirname(absolute) !== root) guides.push(absolute); + } +} + +walk(root); +guides.sort(); +const failures = []; +const allowedStatuses = new Set(["Verified", "Compatible", "Preview", "Not currently tested", "Not currently supported"]); +const sidebar = fs.readFileSync("sidebars.js", "utf8"); +const mirrors = Object.fromEntries(["static/llms.txt", "static/llms-full.txt"].map((m) => [m, fs.readFileSync(m, "utf8")])); + +for (const file of guides) { + const source = fs.readFileSync(file, "utf8"); + const route = path.relative("docs", file).replace(/\.mdx$/, ""); + const match = source.match(/^---\n([\s\S]*?)\n---/); + if (!match) { + failures.push(`${file}: missing frontmatter`); + continue; + } + const value = (key) => match[1].match(new RegExp(`^${key}:\\s*(.*)$`, "m"))?.[1]?.trim(); + const status = value("integration_status"); + const body = source.slice(match[0].length); + if (!status) failures.push(`${file}: missing integration_status`); + if (status && !allowedStatuses.has(status)) failures.push(`${file}: invalid integration_status`); + const blocked = status === "Not currently supported"; + if (blocked && value("draft") !== "true") failures.push(`${file}: blocked guide must be draft: true`); + if (status === "Verified") { + for (const key of ["tested_versions", "verified_regions", "verification_date", "evidence"]) { + const field = value(key); + if (!field || field === "[]" || field === "null") failures.push(`${file}: Verified requires ${key}`); + } + if (!/^\[[^\]]*\S[^\]]*\]$/.test(value("tested_versions") ?? "")) failures.push(`${file}: Verified versions must be a non-empty inline list`); + if (!/^\[(eu-west-1|us-east-1)(,\s*(eu-west-1|us-east-1))*\]$/.test(value("verified_regions") ?? "")) failures.push(`${file}: Verified regions must use public region IDs`); + if (!/^\d{4}-\d{2}-\d{2}$/.test(value("verification_date") ?? "")) failures.push(`${file}: Verified requires an ISO verification date`); + if (!/^https:\/\//.test(value("evidence") ?? "")) failures.push(`${file}: Verified evidence must be an HTTPS link`); + } + if (status !== "Verified" && /(^|\W)Verified(\W|$)/.test(body)) { + failures.push(`${file}: body uses Verified without Verified metadata`); + } + const statusLines = body.split("\n").filter((line) => line.startsWith("Validation status: ")); + if (status === "Verified" || status === "Not currently supported") { + if (statusLines.length !== 1 || !statusLines[0].startsWith(`Validation status: ${status}.`)) { + failures.push(`${file}: body must contain exactly one line starting with "Validation status: ${status}."`); + } + } else if (statusLines.length) { + failures.push(`${file}: untested guides must not include a Validation status line`); + } + if (blocked && sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: blocked guide must not be listed: ${route}`); + if (!blocked && !sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: missing ${route}`); + for (const [mirror, text] of Object.entries(mirrors)) { + if (blocked && text.includes(`/${route}`)) failures.push(`${mirror}: blocked guide must not be mirrored: /${route}`); + if (!blocked && !text.includes(`/${route}`)) failures.push(`${mirror}: missing /${route}`); + } +} + +if (failures.length) { + console.error(failures.join("\n")); + process.exit(1); +} +console.log(`Checked ${guides.length} integration guides.`); diff --git a/sidebars.js b/sidebars.js index 9f232b3..22740c5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -77,6 +77,72 @@ const sidebars = { "sdk/cli", ], }, + { + type: "category", + label: "Integrations", + items: [ + "integrations/index", + "integrations/connection-settings", + "integrations/validate-an-integration", + { + type: "category", + label: "Data movement", + items: [ + "integrations/data-movement/rclone", + "integrations/data-movement/cyberduck-mountain-duck", + "integrations/data-movement/aws-cli-sdk", + ], + }, + { + type: "category", + label: "Backup and disaster recovery", + items: [ + "integrations/backup/acronis-cyber-protect", + "integrations/backup/comet-backup", + "integrations/backup/msp360", + "integrations/backup/nakivo", + ], + }, + { + type: "category", + label: "NAS and file services", + items: [ + "integrations/nas/synology-cloud-sync", + "integrations/nas/qnap-hbs", + "integrations/nas/truenas-cloud-sync", + ], + }, + { + type: "category", + label: "Kubernetes and DevOps", + items: [ + "integrations/devops/velero", + "integrations/devops/harbor", + "integrations/devops/terraform", + ], + }, + { + type: "category", + label: "SIEM and logging", + items: [ + "integrations/logging/elasticsearch-snapshots", + "integrations/logging/splunk-smartstore", + ], + }, + { + type: "category", + label: "Data and AI", + items: [ + "integrations/data-ai/apache-iceberg", + "integrations/data-ai/databricks", + "integrations/data-ai/delta-lake", + "integrations/data-ai/duckdb", + "integrations/data-ai/hugging-face-datasets", + "integrations/data-ai/ray-data", + ], + }, + ], + }, "faq", "limits", ], diff --git a/static/llms-full.txt b/static/llms-full.txt index d6edb6e..97d10ea 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2913,3 +2913,146 @@ Embeddings and extracted chunk text are held in storage Fil One manages, separat from the bucket. Objects themselves are not moved or modified. Extracted chunk text is retained in the index so answers can quote it — enable Bucket Intelligence deliberately per bucket rather than across everything. +# Integrations + +Source: https://docs.fil.one/integrations + +Each integration guide gives the connection settings, setup steps, and troubleshooting for one tool. For API behavior, see the S3 Compatibility reference. + +# Integration connection settings + +Source: https://docs.fil.one/integrations/connection-settings + +Endpoint: https://eu-west-1.s3.fil.one or https://us-east-1.s3.fil.one, matching the bucket's region (also shown in the bucket's Access endpoints panel). Region: eu-west-1 or us-east-1. Authentication: SigV4 with a key created for the same region and scoped to the bucket. Addressing: path style required. Endpoint, region, key, and bucket must belong to the same region. Keep TLS certificate validation enabled and secret keys out of docs, screenshots, source control, and command-line history. Feature availability differs between the S3 API and the app.fil.one dashboard; the S3 Compatibility reference covers API behavior. + +# Check a setup + +Source: https://docs.fil.one/integrations/validate-an-integration + +Optional steps to copy a small fixture (including a file large enough for multipart upload), restore it, and compare size and SHA-256. Use a unique prefix you can delete. + +# rclone + +Source: https://docs.fil.one/integrations/data-movement/rclone + +rclone connects to Fil One as an S3 remote with provider = Other, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Cyberduck and Mountain Duck + +Source: https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck + +Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# AWS CLI and SDKs + +Source: https://docs.fil.one/integrations/data-movement/aws-cli-sdk + +The AWS CLI and SDKs are Fil One's reference clients; see the SDK guides for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Acronis Cyber Protect + +Source: https://docs.fil.one/integrations/backup/acronis-cyber-protect + +Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Comet Backup + +Source: https://docs.fil.one/integrations/backup/comet-backup + +Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# MSP360 + +Source: https://docs.fil.one/integrations/backup/msp360 + +MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# NAKIVO Backup and Replication + +Source: https://docs.fil.one/integrations/backup/nakivo + +NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Synology Cloud Sync + +Source: https://docs.fil.one/integrations/nas/synology-cloud-sync + +Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# QNAP HBS + +Source: https://docs.fil.one/integrations/nas/qnap-hbs + +QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# TrueNAS Cloud Sync + +Source: https://docs.fil.one/integrations/nas/truenas-cloud-sync + +TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Velero + +Source: https://docs.fil.one/integrations/devops/velero + +Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets s3Url to the regional endpoint, s3ForcePathStyle: "true", and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Harbor + +Source: https://docs.fil.one/integrations/devops/harbor + +Harbor stores image layers in Fil One through the registry's s3 storage driver, with regionendpoint set to the regional endpoint, forcepathstyle: true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Terraform S3 backend + +Source: https://docs.fil.one/integrations/devops/terraform + +Terraform's s3 backend stores state in Fil One with endpoints { s3 = ... } set to the regional endpoint and use_path_style = true (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Elasticsearch searchable snapshots + +Source: https://docs.fil.one/integrations/logging/elasticsearch-snapshots + +Elasticsearch stores snapshots in Fil One through an s3 repository whose named client sets endpoint, region, and path_style_access: true, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes repository-s3; on 7.x install the plugin on every node. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Splunk SmartStore + +Source: https://docs.fil.one/integrations/logging/splunk-smartstore + +Splunk SmartStore uses Fil One as a remote volume in indexes.conf, with remote.s3.endpoint set to the regional endpoint, remote.s3.auth_region set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Apache Iceberg + +Source: https://docs.fil.one/integrations/data-ai/apache-iceberg + +Apache Iceberg reads and writes table data in Fil One through S3FileIO, with s3.endpoint set to the regional endpoint, s3.path-style-access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Databricks + +Source: https://docs.fil.one/integrations/data-ai/databricks + +Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Delta Lake + +Source: https://docs.fil.one/integrations/data-ai/delta-lake + +Delta Lake on Spark reads and writes Fil One through the s3a connector, with fs.s3a.endpoint set to the regional endpoint, fs.s3a.path.style.access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# DuckDB + +Source: https://docs.fil.one/integrations/data-ai/duckdb + +DuckDB reads and writes Fil One through the httpfs extension with an S3 secret that sets ENDPOINT to the regional hostname, URL_STYLE 'path', and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Hugging Face Datasets + +Source: https://docs.fil.one/integrations/data-ai/hugging-face-datasets + +Hugging Face datasets reads files in Fil One through fsspec and s3fs, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Ray Data + +Source: https://docs.fil.one/integrations/data-ai/ray-data + +Ray Data reads and writes Fil One through a PyArrow S3FileSystem with endpoint_override set to the regional endpoint, region set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. Configuration, tool-specific validation steps, and troubleshooting are on the page. diff --git a/static/llms.txt b/static/llms.txt index e768105..f241b48 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -174,6 +174,33 @@ Critical integration notes for developers: download, paginated listing, presigned URLs, and error handling; requires `UsePathStyle: true` +## Integrations + +- [Integrations](https://docs.fil.one/integrations): Setup guides for common transfer, backup, NAS, DevOps, analytics, and AI tools +- [Integration connection settings](https://docs.fil.one/integrations/connection-settings): Endpoints (https://eu-west-1.s3.fil.one, https://us-east-1.s3.fil.one), region, SigV4, path-style, and credential settings +- [Check a setup](https://docs.fil.one/integrations/validate-an-integration): Optional steps to copy a small fixture, restore it, and compare checksums +- [rclone](https://docs.fil.one/integrations/data-movement/rclone): provider Other, force_path_style, config example, rclone check validation +- [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): bookmark, mount, restore, and troubleshooting guidance +- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): the reference clients, with setup in the SDK guides +- [Acronis Cyber Protect](https://docs.fil.one/integrations/backup/acronis-cyber-protect): S3-compatible storage, backup, restore, and integrity validation +- [Comet Backup](https://docs.fil.one/integrations/backup/comet-backup): Storage Vault setup and restore validation +- [MSP360](https://docs.fil.one/integrations/backup/msp360): S3-compatible storage account and recovery workflow +- [NAKIVO Backup and Replication](https://docs.fil.one/integrations/backup/nakivo): repository, recovery, and verification guidance +- [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): custom S3 task and download recovery +- [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): HBS storage space, backup, and restore +- [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): credential, push, pull, and integrity workflow +- [Velero](https://docs.fil.one/integrations/devops/velero): Kubernetes backup, restore, and volume-scope validation +- [Harbor](https://docs.fil.one/integrations/devops/harbor): registry push, pull, digest, and garbage-collection guidance +- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): endpoints and use_path_style backend settings, state migration, recovery, and concurrency validation +- [Elasticsearch searchable snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): s3 repository client settings, repository verification, restore, and searchable-snapshot validation +- [Splunk SmartStore](https://docs.fil.one/integrations/logging/splunk-smartstore): remote storage, eviction, search, and recovery validation +- [Apache Iceberg](https://docs.fil.one/integrations/data-ai/apache-iceberg): S3FileIO, catalog, commits, and recovery validation +- [Databricks](https://docs.fil.one/integrations/data-ai/databricks): custom-endpoint compute, read, write, and fresh-cluster recovery +- [Delta Lake](https://docs.fil.one/integrations/data-ai/delta-lake): transaction log, checkpoints, concurrency, and recovery +- [DuckDB](https://docs.fil.one/integrations/data-ai/duckdb): httpfs read, write, and cache-independent validation +- [Hugging Face Datasets](https://docs.fil.one/integrations/data-ai/hugging-face-datasets): load, streaming, cache clearing, and package-version guidance +- [Ray Data](https://docs.fil.one/integrations/data-ai/ray-data): distributed reads, writes, worker configuration, and retry validation + ## Optional The browser console at app.fil.one, one page per console screen. No code