setup-openflow-privatelink

Overview

Connect OpenFlow on SPCS to private data sources (RDS, on-prem DBs reachable via Direct Connect/VPN) through AWS PrivateLink. The flow:

OpenFlow (SPCS) → EAI → Network Rule (PRIVATE_HOST_PORT)
                  → Outbound PrivateLink Endpoint
                  → AWS VPC Endpoint Service → Internal NLB
                  → Target Group(s) → RDS / on-prem DB (via TCP proxy if needed)

Each NLB listener uses a unique port (any port > 1024) so multiple instances sharing the same backend port (e.g., two MySQL on 3306) can be disambiguated.

Prerequisites

• AWS permissions: NLB, target groups, endpoint services, security groups
• Snowflake ACCOUNTADMIN
• OpenFlow already deployed on SPCS
• One or more data sources in a private VPC (or on-prem with VPC connectivity)

Phase 1 — AWS infrastructure

1. Target groups (IP type) — one per database, TCP on the DB port, registered with the private IP of each instance (or a TCP proxy EC2 IP if the source is on-prem with non-RFC1918 addresses).
2. Internal NLB — TCP listeners on unique custom ports (e.g., 3301, 3302), each forwarding to its target group. Enable cross-zone load balancing. Set dns_record.client_routing_policy=any_availability_zone.
3. VPC Endpoint Service — fronts the NLB. Keep manual acceptance enabled (default).

⚠️ STOPPING POINT: Confirm describe-target-health shows healthy and the NLB state is active before continuing. Record nlb_dns_name, endpoint_service_name, and listener ports.

On-prem targets outside RFC1918/RFC6598 ranges require a single TCP proxy EC2 (NGINX/HAProxy/Envoy in TCP mode) registered into the target groups.

Phase 2 — Snowflake side

1. Get Snowflake's account principal:

   SELECT key, value FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$GET_PRIVATELINK_CONFIG())));
   

   Add the privatelink-account-principal ARN to the endpoint service's allowed principals via aws ec2 modify-vpc-endpoint-service-permissions.

2. Provision the endpoint:

   USE ROLE ACCOUNTADMIN;
   SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT('<endpoint_service_name>', '<nlb_dns_name>');
   

3. Accept the pending connection in AWS (aws ec2 accept-vpc-endpoint-connections).

⚠️ STOPPING POINT: Run SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO(); and wait until status is available (30–40s) before creating the network rule.

1. Network rule + EAI:

   CREATE OR REPLACE NETWORK RULE <rule_name>
     MODE = EGRESS
     TYPE = PRIVATE_HOST_PORT
     VALUE_LIST = ('<nlb_dns_name>:<port1>', '<nlb_dns_name>:<port2>');
   
   CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION <eai_name>
     ALLOWED_NETWORK_RULES = (<rule_name>)
     ENABLED = TRUE;
   

2. In OpenFlow UI: Runtime console → ... → Associate External Access Integration → pick <eai_name>.

3. Validate by pointing a connector at <nlb_dns_name>:<listener_port>.

Common Mistakes

• Reusing a backend port as the NLB listener port when multiple instances share it — listeners must be unique.
• Uppercase letters in VALUE_LIST — the NLB FQDN must be lowercase, otherwise Invalid VALUE_LIST.
• Forgetting to accept the AWS endpoint connection — the endpoint stays in pendingAcceptance forever.
• Skipping the account-principal allowlist — SYSTEM$PROVISION_PRIVATELINK_ENDPOINT will fail.
• Security group on RDS not allowing the NLB's SG on the DB port — target health flips unhealthy.
• Deleting the NLB before deprovisioning the Snowflake endpoint — leaves a dangling endpoint.
• On-prem IPs outside RFC1918/RFC6598 registered directly — NLB rejects; use a TCP proxy EC2 instead.
• Using PRIVATE_HOST_PORT against the DB hostname — use the NLB DNS name, not the database's.

Troubleshooting quick reference

Symptom	Check
Endpoint stuck pendingAcceptance	Accept it in AWS VPC > Endpoint Services
Targets unhealthy	RDS/proxy SG inbound from NLB SG on DB port
Invalid VALUE_LIST	NLB DNS must be lowercase
Endpoint failed	SYSTEM$DEPROVISION_PRIVATELINK_ENDPOINT then re-provision; SYSTEM$RESTORE_PRIVATELINK_ENDPOINT recovers within 7 days
Traffic drops despite healthy targets	VPC NACLs: inbound listener ports, outbound 1024–65535

Cleanup

⚠️ STOPPING POINT: Confirm with the user before running any DROP / delete-* commands below.

Order matters — Snowflake first, then AWS:

1. OpenFlow UI: disassociate EAI from runtime.
2. DROP EXTERNAL ACCESS INTEGRATION <eai_name>; DROP NETWORK RULE <rule_name>;
3. SELECT SYSTEM$DEPROVISION_PRIVATELINK_ENDPOINT('<endpoint_service_name>');
4. aws ec2 delete-vpc-endpoint-service-configurations, then delete NLB, then target groups.

Stopping Points

• Phase 1 — wait for healthy targets and active NLB before creating the endpoint service
• Phase 2 step 3 — wait until SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO() returns available before creating the network rule
• Cleanup — confirm with the user before running any destructive DROP / delete-* commands

Cost note

NLB (hourly + LCU), VPC Endpoint Service (hourly), PrivateLink endpoint (hourly + per-GB), and cross-zone data transfer all incur AWS charges. Check current AWS pricing.