Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 48 additions & 62 deletions src/content/docs/snowflake/capabilities/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,30 @@ label:

LocalStack exposes various configuration options to control its behaviour.

These options can be passed to LocalStack as environment variables like so:
With `lstk`, these options can be passed as `LOCALSTACK_`-prefixed environment variables when starting the container:

```bash
DEBUG=1 localstack start --stack snowflake
LOCALSTACK_DEBUG=1 lstk start
```

Alternatively, set them as named environment profiles in your config file and reference them from the container block:

```toml
# .lstk/config.toml
[[containers]]
type = "snowflake"
env = ["debug"]

[env.debug]
DEBUG = "1"
```

```bash
lstk start
```

See [Passing environment variables to the container](/aws/developer-tools/running-localstack/lstk/configuration/#passing-environment-variables-to-the-container) for details.

## Core

Options that affect the core Snowflake emulator functionality.
Expand All @@ -35,11 +53,20 @@ Options that affect the core Snowflake emulator functionality.

By default, the Snowflake emulator accepts requests for hostnames such as `snowflake.localhost.localstack.cloud` and other `*.snowflake.*` hostnames.
If you expose the emulator through a custom DNS name, for example in Kubernetes or behind an ingress, set `SF_HOSTNAMES` to the exact hostnames clients use to reach the emulator.
When you use the `localstack` CLI, add the `LOCALSTACK_` prefix so the CLI passes the variable to the container:
When you use `lstk`, set this as a named environment profile in your config file:

```toml
# .lstk/config.toml
[[containers]]
type = "snowflake"
env = ["custom"]

[env.custom]
SF_HOSTNAMES = "snowflake.internal.example.com,snowflake.internal,snowflake.localhost.localstack.cloud"
```

```bash
Comment thread
quetzalliwrites marked this conversation as resolved.
LOCALSTACK_SF_HOSTNAMES=snowflake.internal.example.com,snowflake.internal,snowflake.localhost.localstack.cloud \
localstack start --stack snowflake
lstk start
```

The first hostname in `SF_HOSTNAMES` is used as the primary hostname for local connection defaults and generated URLs.
Expand Down Expand Up @@ -71,81 +98,40 @@ If you previously used `SF_HOSTNAME_REGEX`, migrate to `SF_HOSTNAMES` and list e

If your custom hostname also needs a matching TLS certificate, use LocalStack's standard certificate configuration options:

```toml
# .lstk/config.toml
[[containers]]
type = "snowflake"
env = ["custom"]

[env.custom]
SF_HOSTNAMES = "snowflake.internal.example.com"
CUSTOM_SSL_CERT_PATH = "/var/lib/localstack/custom/cert.pem"
SKIP_SSL_CERT_DOWNLOAD = "1"
```

```bash
LOCALSTACK_SF_HOSTNAMES=snowflake.internal.example.com \
CUSTOM_SSL_CERT_PATH=/var/lib/localstack/custom/cert.pem \
SKIP_SSL_CERT_DOWNLOAD=1 \
localstack start --stack snowflake
lstk start
Comment thread
quetzalliwrites marked this conversation as resolved.
```

The file referenced by `CUSTOM_SSL_CERT_PATH` must contain a certificate and private key that match the hostname used by your Snowflake clients.
For more general guidance on adding trusted certificates to LocalStack, see [Custom TLS certificates](/aws/developer-tools/security-testing/custom-tls-certificates/).

## CLI

These options are applicable when using the CLI to start LocalStack.

| Variable | Example Values | Description |
| - | - | - |
| `LOCALSTACK_VOLUME_DIR` | `~/.cache/localstack/volume` (on Linux) | The location on the host of the LocalStack volume directory mount. |
| `CONFIG_PROFILE` | | The configuration profile to load. See [Profiles](#profiles) |
| `CONFIG_DIR` | `~/.localstack` | The path where LocalStack can find configuration profiles and other CLI-specific configuration |
`lstk` is configured through its config file rather than through environment variables.
See [Configuration](/aws/developer-tools/running-localstack/lstk/configuration/) on the `lstk` page for the config file search order, the field reference, and how to define named environment profiles.

## Docker

Options to configure how LocalStack interacts with Docker.

| Variable | Example Values | Description |
| - | - | - |
| `LOCALSTACK_VOLUME_DIR` | `~/.cache/localstack/volume` (on Linux) | The location on the host of the LocalStack volume directory mount. |
| `DOCKER_FLAGS` | | Allows to pass custom flags (e.g., volume mounts) to "docker run" when running LocalStack in Docker. |
| `DOCKER_SOCK` | `/var/run/docker.sock` | Path to local Docker UNIX domain socket |
| `DOCKER_BRIDGE_IP` | `172.17.0.1` | IP of the Docker bridge used to enable access between containers |
| `LEGACY_DOCKER_CLIENT` | `0`\|`1` | Whether LocalStack should use the command-line Docker client and subprocess execution to run Docker commands, rather than the Docker SDK. |
| `DOCKER_CMD` | `docker` (default), `sudo docker`| Shell command used to run Docker containers (only used in combination with `LEGACY_DOCKER_CLIENT`) |
| `FORCE_NONINTERACTIVE` | | When running with Docker, disables the `--interactive` and `--tty` flags. Useful when running headless. |

## Profiles

LocalStack supports configuration profiles which are stored in the `~/.localstack` config directory.
A configuration profile is a set of environment variables stored in a `*.env` file in the LocalStack config directory.

Here is an example of what configuration profiles might look like:

```bash
tree ~/.localstack
/home/username/.localstack
├── default.env
├── dev.env
└── pro.env
```

Here is an example of what a specific environment profile looks like

```bash
cat ~/.localstack/pro-debug.env
LOCALSTACK_AUTH_TOKEN=XXXXX
SF_LOG=trace
SF_S3_ENDPOINT=s3.localhost.localstack.cloud:4566
```

You can load a profile by either setting the environment variable `CONFIG_PROFILE=<profile>` or the `--profile=<profile>` CLI flag when using the CLI.
Let's take an example to load the `dev.env` profile file if it exists:

```bash
localstack --profile=dev start --stack snowflake
```

If no profile is specified, the `default.env` profile will be loaded.
If explicitly specified, any environment variables will overwrite the configurations defined in the profile.

To display the config environment variables, you can use the following command:

```bash
localstack --profile=dev config show
```

:::note
The `CONFIG_PROFILE` is a CLI feature and cannot be used directly with a docker-compose setup.
You can look at [alternative means of setting environment variables](https://docs.docker.com/compose/environment-variables/set-environment-variables/) for your Docker Compose setups.
For Docker setups, we recommend passing the environment variables directly to the `docker run` command.
:::
22 changes: 16 additions & 6 deletions src/content/docs/snowflake/capabilities/init-hooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ CREATE DATABASE test123;
SHOW DATABASES;
```

Mount the script into `/etc/localstack/init/ready.d/` using Docker Compose or the `localstack` CLI:
Mount the script into `/etc/localstack/init/ready.d/` using Docker Compose or `lstk`:

<Tabs>
<TabItem label="docker-compose.yml">
Expand All @@ -51,12 +51,22 @@ services:
- "/var/run/docker.sock:/var/run/docker.sock"
```
</TabItem>
<TabItem label="CLI">
<TabItem label="lstk">
Declare the bind mount and the `DEBUG` profile in your config file, then start LocalStack:

```toml
# .lstk/config.toml
[[containers]]
type = "snowflake"
env = ["debug"]
volumes = ["/path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql"]

[env.debug]
Comment thread
quetzalliwrites marked this conversation as resolved.
DEBUG = "1" # Optionally enable DEBUG, not required for init hooks but helps diagnose init hook issues
```

```bash
# DOCKER_FLAGS are additional parameters to the `docker run` command of localstack start
DOCKER_FLAGS='-v /path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql' \
DEBUG=1 \
localstack start --stack snowflake
lstk start
```
</TabItem>
</Tabs>
Expand Down
26 changes: 18 additions & 8 deletions src/content/docs/snowflake/capabilities/state-management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,24 @@ State Management is an essential feature that supports various use-cases, such a

LocalStack’s Persistence mechanism enables the saving and restoration of the entire LocalStack state. It functions as a **pause and resume** feature, allowing you to take a snapshot of your LocalStack instance and save this data to disk. This mechanism ensures a quick and efficient way to preserve and continue your work with Snowflake resources locally.

To start snapshot-based persistence, launch LocalStack with the configuration option `PERSISTENCE=1`. This setting instructs LocalStack to save all local Snowflake resources and their respective application states into the LocalStack Volume Directory. Upon restarting LocalStack, you'll be able to resume your activities exactly where you left off.
To start snapshot-based persistence, launch LocalStack with the `--persist` command-line option, or the configuration option `PERSISTENCE=1`. This setting instructs LocalStack to save all local Snowflake resources and their respective application states into the LocalStack Volume Directory. Upon restarting LocalStack, you'll be able to resume your activities exactly where you left off.

<Tabs>
<TabItem label="LocalStack CLI">
<TabItem label="lstk">
```bash
export LOCALSTACK_AUTH_TOKEN=<your_auth_token>
PERSISTENCE=1 \
localstack start --stack snowflake
lstk start --persist
```
Comment thread
quetzalliwrites marked this conversation as resolved.

To resume a persisted session after the emulator has stopped, or to restart a session while keeping persistence on, pass `--persist` again:

```bash
lstk stop
lstk start --persist
```

```bash
lstk restart --persist
```
</TabItem>
<TabItem label="Docker Compose">
Expand Down Expand Up @@ -69,17 +79,17 @@ The Export/Import State feature enables you to export the state of your LocalSta
To export the state, you can run the following command:

```bash
localstack state export '<file-name>'
lstk snapshot save '<path-to-snapshot-file>'
```

You can use the `<file-name>` argument to specify a file path to export the state to. If you do not specify a file path, the state will be exported to the current working directory into a file named `ls-state-export`.
You can use the `<path-to-snapshot-file>` argument to specify a file path to export the state to. If you do not specify a file path, the state will be exported to the current working directory into an auto-named snapshot file. See the [lstk snapshot documentation](/aws/developer-tools/running-localstack/lstk/snapshots/) for the full list of supported save destinations, including local files and remote Cloud Pods.

### Import the State

To import the state, you can run the following command:

```bash
localstack state import '<file-name>'
lstk snapshot load '<path-to-snapshot-file>'
```

The `<file-name>` argument is required and specifies the file path to import the state from. The file should be generated from a previous export.
The `<path-to-snapshot-file>` argument is required and specifies the file path to import the state from. The file should be generated from a previous export.
4 changes: 2 additions & 2 deletions src/content/docs/snowflake/features/dynamic-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,10 @@ A Dynamic Iceberg Table consists of three key concepts:

### Create an S3 bucket

Create a local S3 bucket using the `mb` command with the `awslocal` CLI:
Create a local S3 bucket using the `mb` command with `lstk aws`:

```bash
awslocal s3 mb s3://test-bucket
lstk aws s3 mb s3://test-bucket
```

### Create an external volume
Expand Down
10 changes: 5 additions & 5 deletions src/content/docs/snowflake/features/glue-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The Snowflake emulator can connect to this Glue Iceberg REST endpoint through a

## Getting started

This guide walks through creating an Iceberg table in S3 Tables through the Glue Iceberg REST endpoint, registering that table with the Snowflake emulator through a Glue catalog integration, and querying it with SQL. It assumes basic knowledge of the AWS CLI, our [`awslocal`](/aws/connecting/infrastructure-as-code/deprecated-wrapper-scripts/#awslocal) wrapper, and Snowflake.
This guide walks through creating an Iceberg table in S3 Tables through the Glue Iceberg REST endpoint, registering that table with the Snowflake emulator through a Glue catalog integration, and querying it with SQL. It assumes basic knowledge of the AWS CLI, our [`lstk aws`](/aws/developer-tools/running-localstack/lstk/cloud-and-iac-commands/#aws) command, and Snowflake.

In this guide, you will:

Expand All @@ -29,7 +29,7 @@ Start your Snowflake emulator and connect to it using an SQL client in order to
The Glue Iceberg REST endpoint serves tables stored in S3 Tables. Create a table bucket:

```bash
awslocal s3tables create-table-bucket --name my-table-bucket
lstk aws s3tables create-table-bucket --name my-table-bucket
```

```bash title="Output"
Expand All @@ -41,7 +41,7 @@ awslocal s3tables create-table-bucket --name my-table-bucket
Now create a namespace to hold the Iceberg table:

```bash
awslocal s3tables create-namespace \
lstk aws s3tables create-namespace \
--table-bucket-arn arn:aws:s3tables:us-east-1:000000000000:bucket/my-table-bucket \
--namespace my_namespace
```
Expand All @@ -60,7 +60,7 @@ awslocal s3tables create-namespace \
Glue exposes S3 Tables buckets through a federated catalog. Register a catalog named `s3tablescatalog` that federates to all S3 Tables buckets in the account:

```bash showLineNumbers
awslocal glue create-catalog \
lstk aws glue create-catalog \
--name s3tablescatalog \
--catalog-input '{
"FederatedCatalog": {
Expand All @@ -75,7 +75,7 @@ awslocal glue create-catalog \
Confirm the catalog was registered:

```bash
awslocal glue get-catalogs
lstk aws glue get-catalogs
```

The response includes a `CatalogList` entry with `Name: s3tablescatalog` and a `FederatedCatalog` block pointing at S3 Tables. Snowflake will reference this catalog through its `WAREHOUSE` identifier in the form `<account-id>:s3tablescatalog/<table-bucket-name>`.
Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/snowflake/features/iceberg-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ In this guide, you will create an external volume, and an Iceberg table to store

### Create an S3 bucket

You can create a local S3 bucket using the `mb` command with the `awslocal` CLI.
You can create a local S3 bucket using the `mb` command with `lstk aws`.

```bash
awslocal s3 mb s3://test-bucket
lstk aws s3 mb s3://test-bucket
```

### Create an external volume
Expand Down Expand Up @@ -81,5 +81,5 @@ The output should be:
You can also list the content of the S3 bucket:

```bash
awslocal s3 ls --recursive s3://test-bucket/
lstk aws s3 ls --recursive s3://test-bucket/
```
6 changes: 3 additions & 3 deletions src/content/docs/snowflake/features/polaris-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,10 @@ curl -s -X PUT http://localhost:8181/api/management/v1/catalogs/polaris/catalog-

### Create a bucket

Create a bucket using the `awslocal` command:
Create a bucket using the `lstk aws` command:

```bash
awslocal s3 mb s3://$BUCKET_NAME
lstk aws s3 mb s3://$BUCKET_NAME
```

### Create an external volume
Expand Down Expand Up @@ -181,7 +181,7 @@ The output should be:
All data will be persisted under:

```bash
awslocal s3 ls s3://$BUCKET_NAME/test/test_namespace/
lstk aws s3 ls s3://$BUCKET_NAME/test/test_namespace/
```

You will see:
Expand Down
8 changes: 4 additions & 4 deletions src/content/docs/snowflake/features/snowpipe.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ In this guide, you will create a stage, and a pipe to load data from a local S3

### Create an S3 bucket

You can create a local S3 bucket using the `mb` command with the `awslocal` CLI.
You can create a local S3 bucket using the `mb` command with `lstk aws`.

```bash
awslocal s3 mb s3://test-bucket
lstk aws s3 mb s3://test-bucket
```

### Create a stage
Expand Down Expand Up @@ -70,7 +70,7 @@ Retrieve the `notification_channel` value from the output of the `DESC PIPE` que
You can use the [`PutBucketNotificationConfiguration`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketNotificationConfiguration.html) API to create a bucket notification configuration that sends notifications to Snowflake when new files are uploaded to the S3 bucket.

```bash showLineNumbers
awslocal s3api put-bucket-notification-configuration \
lstk aws s3api put-bucket-notification-configuration \
--bucket test-bucket \
--notification-configuration file://notification.json
```
Expand Down Expand Up @@ -104,7 +104,7 @@ Copy a JSON file to the S3 bucket to trigger the pipe to load the data into the
Upload the file to the S3 bucket:

```bash
awslocal s3 cp test.json s3://test-bucket/
lstk aws s3 cp test.json s3://test-bucket/
```

### Check the data
Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/snowflake/features/stages.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,11 @@ The expected output is:

## Loading files from S3

You can also load data from an S3 bucket using the `CREATE STAGE` command. Create a new S3 bucket named `testbucket` and upload the [employees CSV files](/artifacts/getting-started.zip) to the bucket. You can use LocalStack's `awslocal` CLI to create the S3 bucket and upload the files.
You can also load data from an S3 bucket using the `CREATE STAGE` command. Create a new S3 bucket named `testbucket` and upload the [employees CSV files](/artifacts/getting-started.zip) to the bucket. You can use LocalStack's `lstk aws` command to create the S3 bucket and upload the files.

```bash
awslocal s3 mb s3://testbucket
awslocal s3 cp employees0*.csv s3://testbucket
lstk aws s3 mb s3://testbucket
lstk aws s3 cp employees0*.csv s3://testbucket
```

In this example, you can create a stage called `my_s3_stage` to load data from an S3 bucket:
Expand Down
Loading