Conversation
🛡️ Jit Security Scan Results
✅ No security findings were detected in this PR
Security scan by Jit
|
dwdougherty
left a comment
dwdougherty
left a comment
There was a problem hiding this comment.
One minor nitpick; otherwise, language LGTM.
|
|
||
| RDI uses the [RIOTX](https://redis.github.io/riotx/) collector to stream data from Snowflake to Redis. | ||
| During the [snapshot]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) phase, RDI reads the current state of the database using the JDBC driver. In the | ||
| [streaming]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) |
There was a problem hiding this comment.
The sets of bullets that this link points to doesn't have "streaming" as a keyword. Maybe add that streaming is CDC or something.
|
|
||
| During the [snapshot]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) phase, RDI reads the current state of the database using the JDBC driver. In the | ||
| [Change data capture (CDC)]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) | ||
| phase, RDI uses [Snowflake Streams](https://docs.snowflake.com/en/user-guide/streams) to |
There was a problem hiding this comment.
Both the initial snapshot and CDC both use a Snowflake STREAM. When initial snapshot is desired the SHOW_INITIAL_ROWS = TRUE option is set on the Snowflake STREAM.
|
|
||
| ## 1. Set up Snowflake permissions | ||
|
|
||
| The RDI user requires the following permissions to connect and capture data from Snowflake: |
There was a problem hiding this comment.
These are likely not enough for this to work - see the "minimally required permission" section here: https://redis.github.io/riotx/databases/snowflake.html - I dont think we should link to that page but use it as a reference and copy what is required here
|
|
||
| ## 2. Configure authentication | ||
|
|
||
| RDI supports two authentication methods for Snowflake. You must configure one of these methods. |
There was a problem hiding this comment.
Many production snowflake accounts require MFA for username and passwords. If you wish to create the RDI account using username and password you must designate it as a service account so it does not require MFA. See https://docs.snowflake.com/en/user-guide/security-mfa-rollout for more information
|
|
||
| ### Performance tuning | ||
|
|
||
| **High Snowflake API usage** |
There was a problem hiding this comment.
Maybe a different way to word this besides High Snowflake API usage, maybe High Snowflake Warehouse usage
It is also worth noting that on each poll interval the RDI Snowflake connector uses a system function STREAM_HAS_DATA (https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data) to check if there is new data on the stream. This system function does not start a warehouse so will not incur warehouse compute time. If this function returns true, indicating there is new data to import then the next queries issued from the connector to read the data from the stream automatically spin up a warehouse as needed.
|
Added two follow-up commits to keep this docs PR aligned with the current Snowflake source implementation direction in
|
❌ Jit Scanner failed - Our team is investigatingJit Scanner failed - Our team has been notified and is working to resolve the issue. Please contact support if you have any questions. 💡 Need to bypass this check? Comment |
3 participants
Based on https://github.com/RedisLabs/redis-data-integration/blob/main/docs/snowflake-connector.md
The AI tool has expanded on the SQL in the original document, so please check that this is OK. Any other corrections or suggestions are, of course, most welcome :-)
Note
Low Risk
Adds a new documentation page only; no runtime code changes, so the main risk is inaccurate setup instructions for Snowflake permissions/auth.
Overview
Adds a new
Prepare Snowflake for RDIguide describing how to use Snowflake as an RDI source (private preview) when deployed on Kubernetes/Helm.The page documents required Snowflake grants (including change tracking prerequisites), supported auth methods (password or key-pair), Kubernetes secret setup, an example
config.yamlfor the Snowflake source, and troubleshooting/performance tips.Reviewed by Cursor Bugbot for commit f7eb9f4. Bugbot is set up for automated code reviews on this repo. Configure here.