Connecting To Data Source

Connect to Data Source

Register an external data source with AWS Glue so downstream skills (ingesting-into-data-lake) can move data from it. A Glue connection stores the network config, driver, and credential reference for one source. Create once per source, reuse across jobs.

Philosophy

A connection is a named pipe, not a pipeline. This skill produces a tested, reusable Glue connection. It does not move data.

Common Tasks

You MUST execute commands using AWS MCP server tools when connected -- they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.

Workflow

1. Verify Dependencies and Context

• You MUST check whether AWS MCP tools or AWS CLI are available and inform the user if missing
• You MUST confirm target AWS region and verify credentials with aws sts get-caller-identity

2. Classify the Source

Ask the user which source type they want to connect to, or infer from hints:

If the user names DynamoDB or a local file, stop and tell them: DynamoDB is read directly by Glue without a connection, and local files belong in the ingesting-into-data-lake skill's local-upload workflow.

3. Gather Connection Hints from the User

You MUST ask for hints the user can provide -- do not guess.

For all sources:

• Desired connection name (lowercase, hyphens: oracle-prod-sales, snowflake-analytics)
• Existing Secrets Manager secret, or create one
• Is source reachable from a Glue VPC (same, peered, VPN, Direct Connect)

JDBC: hostname/endpoint, port, database, whether RDS/Aurora/self-managed, IAM DB auth enabled (Aurora/RDS MySQL/Postgres), SSL required.

Snowflake: account identifier, warehouse, role, default database, auth (password, key-pair, OAuth).

BigQuery: GCP project ID, location, whether service account JSON is provisioned.

4. Discover Existing Connections and Candidate Sources

Check what exists before creating.

Existing Glue connections:

aws glue get-connections --filter ConnectionType=<TYPE> --region <REGION>

If a suitable one exists, confirm and skip to Step 7.

Candidate sources in account (JDBC/Redshift only):

• RDS: aws rds describe-db-instances
• Aurora: aws rds describe-db-clusters
• Redshift: aws redshift describe-clusters

Present candidates to user; let them pick. See discovery.md.

5. Register Credentials

You MUST encourage AWS Secrets Manager over plaintext passwords. You SHOULD prefer IAM database authentication where supported (Aurora/RDS MySQL and PostgreSQL, Redshift). See credential-security.md.

• You MUST confirm with user before creating a new Secrets Manager secret
• You MUST NOT write plaintext credentials into chat or logs
• For IAM DB auth, no secret is needed

6. Create the Glue Connection

Follow the source-specific reference for connection properties:

aws glue create-connection --connection-input '<JSON>' --region <REGION>

Private sources require PhysicalConnectionRequirements (SubnetId, SecurityGroupIdList, AvailabilityZone). See network-setup.md.

7. Test the Connection

You MUST test before handing off. Testing is two-phase: a quick API check, then an engine-level verification.

Phase A: Glue TestConnection (network and credential sanity check)

aws glue test-connection --connection-name <NAME> --region <REGION>

This validates that Glue can reach the source and authenticate. It does NOT prove the connection works end-to-end with the query engine the user plans to use.

Phase B: Engine-level verification

After TestConnection passes, verify the connection works with the user's intended engine by running a minimal query through it:

• Glue ETL (default): Run a smoke-test Glue job that reads one row via the connection. See troubleshooting.md.
• Athena: If the user plans to query via Athena with a federated connector, run a SELECT 1 through the Athena connection to confirm the Lambda-based connector can reach the source.
• Glue Crawler: If the user plans to crawl the source, run a test crawl on a single table.

Phase B catches issues that TestConnection misses: driver compatibility at job runtime, catalog configuration, Spark-level serialization, and engine-specific auth flows (e.g., Snowflake SNOWFLAKE type works in ETL but not via JDBC crawlers).

On success in both phases, tell user the connection name is ready for ingesting-into-data-lake. On failure in either phase, Step 8.

8. Troubleshoot (only if test failed)

Diagnose in order: network, credentials, driver. See troubleshooting.md.

Constraints:

• You MUST check VPC routing, security groups, and S3 VPC endpoint before blaming credentials
• You MUST verify Glue role can read the Secrets Manager secret
• You MUST NOT rotate credentials without user confirmation

Argument Routing

• No args: Walk through Steps 1-7 interactively
• Source type keyword (e.g., snowflake, oracle): Skip to Step 2 with the type prefilled
• Existing connection name: Skip to Step 7 (test) then Step 8 if failing
• Hostname or RDS endpoint: Skip to Step 4 with the candidate prefilled

Gotchas

• Glue's SNOWFLAKE connection type is distinct from JDBC configured for Snowflake. You MUST use SNOWFLAKE for Spark ETL jobs; do not use JDBC.
• Connection names are immutable. Choose carefully.
• PhysicalConnectionRequirements.AvailabilityZone MUST match the subnet's AZ or the connection fails at job runtime, not creation time.
• IAM database authentication tokens expire in 15 minutes. The Glue job generates a fresh token on each connection; do not cache.
• An S3 VPC gateway endpoint MUST exist in the VPC used by private-source connections. Without it, Glue jobs cannot read their scripts or write results to S3.

Troubleshooting

References

• jdbc-setup.md -- Oracle, SQL Server, PostgreSQL, MySQL, RDS, Redshift
• snowflake-setup.md -- Glue SNOWFLAKE type, auth modes
• bigquery-setup.md -- Glue BIGQUERY type, GCP service accounts
• discovery.md -- Finding existing connections and candidate sources
• credential-security.md -- Secrets Manager and IAM DB auth
• network-setup.md -- VPC, subnets, security groups, endpoints
• troubleshooting.md -- Connection errors and diagnostic flow

BigQuery Connection Setup

AWS Glue native BigQuery connection (type BIGQUERY). Authentication is via a GCP service account; credentials flow through AWS Secrets Manager.

Contents

• Prerequisites
• Service Account Setup
• Secrets Manager Storage
• Connection JSON Template
• Further Reading

Prerequisites

• GCP project with BigQuery enabled
• Service account in that project with BigQuery access (typically roles/bigquery.dataViewer plus roles/bigquery.jobUser for running jobs)
• Service account JSON key file from GCP
• AWS Secrets Manager secret in the same region as the Glue job

Service Account Setup

Service account and key generation happen in GCP, not AWS. For current steps see GCP service account docs and BigQuery access control.

Minimum GCP IAM roles for read-only ingestion:

• roles/bigquery.dataViewer on the target dataset
• roles/bigquery.jobUser on the project (to run queries)

For cross-project reads, grant both roles in each source project.

Secrets Manager Storage

Base64-encode the service account JSON and store in Secrets Manager. The Glue BigQuery connection expects the secret value to be the base64 string directly, not a JSON wrapper.

base64 -i <service-account>.json | tr -d '\n' > sa.b64
aws secretsmanager create-secret \
  --name glue/bigquery/<project-id>/credentials \
  --secret-string file://sa.b64 \
  --region <region>
rm sa.b64

Rotate by creating a new key in GCP and updating the secret value. Glue picks up the new value on next job run.

Connection JSON Template

{
  "Name": "bigquery-<project-id>",
  "ConnectionType": "BIGQUERY",
  "ConnectionProperties": {
    "SECRET_ID": "glue/bigquery/<project-id>/credentials"
  }
}

Glue's BigQuery connection talks to Google APIs over the internet. No PhysicalConnectionRequirements needed unless the Glue job itself must run in a specific VPC for other reasons (e.g., also reading from a private RDS). In that case, ensure the subnet has NAT gateway egress so Glue can reach bigquery.googleapis.com.

Further Reading

• AWS Glue: Creating a BigQuery connection
• AWS Glue: Creating a BigQuery source node
• GCP service account keys

Credential Security

Order of preference for authenticating Glue connections to data sources:

1. IAM database authentication (where supported) 2. AWS Secrets Manager (SECRET_ID) 3. Plaintext USERNAME/PASSWORD in connection properties (not recommended)

Contents

• IAM Database Authentication
• AWS Secrets Manager
• Plaintext Credentials
• Rotation

IAM Database Authentication

Supported sources:

• Aurora MySQL, Aurora PostgreSQL
• RDS MySQL, RDS PostgreSQL
• Amazon Redshift (via GetClusterCredentials / GetCredentials)

Benefits:

• No long-lived database passwords
• No secret to rotate
• Database access controlled by IAM policies
• Audit trail via CloudTrail

RDS / Aurora Setup

1. Enable IAM DB auth on the cluster or instance:

   aws rds modify-db-instance \
     --db-instance-identifier <ID> \
     --enable-iam-database-authentication \
     --apply-immediately

2. Create a DB user that authenticates via IAM (MySQL):

   CREATE USER 'etl_user'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
   GRANT SELECT ON app_db.* TO 'etl_user'@'%';

PostgreSQL:

   CREATE USER etl_user;
   GRANT rds_iam TO etl_user;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO etl_user;

3. Grant the Glue job role the rds-db:connect action:

   {
     "Effect": "Allow",
     "Action": "rds-db:connect",
     "Resource": "arn:aws:rds-db:<region>:<account>:dbuser:<resource-id>/etl_user"
   }

4. In the Glue connection, omit SECRET_ID, USERNAME, and PASSWORD. Glue generates an auth token on each connection.

Redshift Setup

Grant the Glue role redshift:GetClusterCredentials (provisioned) or redshift-serverless:GetCredentials (serverless), scoped to the cluster/workgroup and DB user.

Configure the connection with the Redshift endpoint and a DB user. No password.

AWS Secrets Manager

When IAM DB auth is not available (Oracle, SQL Server, Snowflake, BigQuery, self-managed), use Secrets Manager.

Create Secret

JDBC sources:

aws secretsmanager create-secret \
  --name glue/<connection-name>/credentials \
  --secret-string '{"username":"etl_user","password":"<password>"}' \
  --region <region>

Snowflake (key names are Glue-specific):

aws secretsmanager create-secret \
  --name glue/snowflake-analytics/credentials \
  --secret-string '{"snowflakeUser":"ETL_USER","snowflakePassword":"<password>"}' \
  --region <region>

BigQuery (base64 of service account JSON, stored as the secret string directly):

base64 -i <sa>.json | tr -d '\n' | \
aws secretsmanager create-secret \
  --name glue/bigquery/<project-id>/credentials \
  --secret-string file:///dev/stdin \
  --region <region>

Grant Glue Role Access

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:<region>:<account>:secret:glue/<connection-name>/credentials-*"
}

The -* suffix matches the random 6-character suffix Secrets Manager appends.

Reference in Connection

"ConnectionProperties": {
  "JDBC_CONNECTION_URL": "...",
  "SECRET_ID": "glue/<connection-name>/credentials"
}

Omit USERNAME and PASSWORD. Glue reads them from the secret at job runtime.

Plaintext Credentials

Not recommended. Use only for:

• Disposable developer sandboxes
• Sources where Secrets Manager integration is not supported by the Glue connector

If you must, use USERNAME and PASSWORD in ConnectionProperties. The password is encrypted at rest in the Data Catalog but visible in get-connection responses to any principal with glue:GetConnection.

Rotation

Secrets Manager rotation:

• Enable automatic rotation on the secret (7, 30, 60, or 90 days)
• Rotation Lambda updates the password in the source database and writes the new value to the secret
• Glue picks up the new value on the next job run; no connection update needed
• For Aurora/RDS, use the AWS-provided rotation template

IAM DB auth: no rotation -- tokens are minted per-connection and expire in 15 minutes.

Service account keys (BigQuery) / key-pairs (Snowflake): rotate by generating a new key at the source, updating the Secrets Manager value, and letting the old key expire or be deleted in the source.

Discovering Connections and Candidate Sources

Before creating a new Glue connection, check what exists and what the user has available in their account. Users often forget about previously registered connections or don't realize they already have running databases that can be registered.

Contents

• Existing Glue Connections
• RDS and Aurora Candidates
• Redshift Candidates
• Presenting Candidates

Existing Glue Connections

List all connections, optionally filtered by type:

# All connections
aws glue get-connections --region <REGION> --query 'ConnectionList[].{Name:Name,Type:ConnectionType,LastUpdated:LastUpdatedTimestamp}'

# Filter by type
aws glue get-connections --filter ConnectionType=JDBC --region <REGION>
aws glue get-connections --filter ConnectionType=SNOWFLAKE --region <REGION>
aws glue get-connections --filter ConnectionType=BIGQUERY --region <REGION>

Inspect a specific connection's properties (credentials are redacted in the response):

aws glue get-connection --name <NAME> --region <REGION>

If a connection matching the user's intent already exists, confirm with the user and skip creation. Re-test it (Step 7 of the skill) before handing off.

RDS and Aurora Candidates

RDS instances:

aws rds describe-db-instances \
  --query 'DBInstances[].{Id:DBInstanceIdentifier,Endpoint:Endpoint.Address,Port:Endpoint.Port,Engine:Engine,DBName:DBName,VpcId:DBSubnetGroup.VpcId,Status:DBInstanceStatus,IAMAuth:IAMDatabaseAuthenticationEnabled}' \
  --region <REGION>

Aurora clusters:

aws rds describe-db-clusters \
  --query 'DBClusters[].{Id:DBClusterIdentifier,Endpoint:Endpoint,ReaderEndpoint:ReaderEndpoint,Port:Port,Engine:Engine,DatabaseName:DatabaseName,IAMAuth:IAMDatabaseAuthenticationEnabled}' \
  --region <REGION>

Prefer the Aurora reader endpoint for ETL reads to avoid impacting the writer. The reader endpoint is load-balanced across reader instances.

Note IAMDatabaseAuthenticationEnabled: true -- if set, recommend IAM DB auth over password per credential-security.md.

Redshift Candidates

Provisioned clusters:

aws redshift describe-clusters \
  --query 'Clusters[].{Id:ClusterIdentifier,Endpoint:Endpoint.Address,Port:Endpoint.Port,DBName:DBName,VpcId:VpcId,IAMRoles:IamRoles[*].IamRoleArn,Status:ClusterStatus}' \
  --region <REGION>

Serverless workgroups:

aws redshift-serverless list-workgroups \
  --query 'workgroups[].{Name:workgroupName,Endpoint:endpoint.address,Port:endpoint.port,Status:status}' \
  --region <REGION>

Presenting Candidates

When you find candidates, present them as a numbered list and let the user pick. Example:

I found these databases in your account. Which would you like to register?

1. RDS PostgreSQL: analytics-prod (analytics-prod.abc123.us-east-1.rds.amazonaws.com:5432, DB: analytics, IAM auth: enabled)
2. Aurora MySQL cluster: orders-writer (orders.cluster-abc123.us-east-1.rds.amazonaws.com, reader: orders.cluster-ro-abc123..., DB: orders)
3. Redshift: warehouse-prod (warehouse-prod.abc123.us-east-1.redshift.amazonaws.com:5439, DB: analytics)
4. None of these -- I want to register a source outside my account.

Never auto-select. The user may have multiple candidates or want to register a source that isn't visible to these discovery APIs (on-premises, peered account, Snowflake, BigQuery).

Snowflake and BigQuery sources are not discoverable via AWS APIs -- always ask the user for account/project details directly.

JDBC Connection Setup

AWS Glue JDBC connections for Oracle, SQL Server, PostgreSQL, MySQL, MariaDB, Amazon RDS, Amazon Aurora, and Amazon Redshift.

Contents

• URL Formats and Drivers
• Built-in Drivers
• Custom Driver Upload
• Connection JSON Template
• Redshift
• RDS and Aurora Considerations

URL Formats and Drivers

For Oracle, prefer the service name form (@//host:port/service). SID form (@host:port:SID) works but is deprecated in Oracle 12c+.

Built-in Drivers

Glue includes drivers for Oracle, SQL Server, PostgreSQL, MySQL, and Redshift. No JDBC_DRIVER_JAR_URI needed.

Custom Driver Upload

For driver versions not built into Glue, upload the JAR to S3 and reference:

aws s3 cp ojdbc8-21.jar s3://<scripts-bucket>/jdbc-drivers/

Add to connection properties:

"JDBC_DRIVER_JAR_URI": "s3://<scripts-bucket>/jdbc-drivers/ojdbc8-21.jar",
"JDBC_DRIVER_CLASS_NAME": "oracle.jdbc.OracleDriver"

Connection JSON Template

{
  "Name": "<connection-name>",
  "ConnectionType": "JDBC",
  "ConnectionProperties": {
    "JDBC_CONNECTION_URL": "<url>",
    "SECRET_ID": "<secrets-manager-arn-or-name>",
    "JDBC_ENFORCE_SSL": "true"
  },
  "PhysicalConnectionRequirements": {
    "SubnetId": "subnet-xxxxx",
    "SecurityGroupIdList": ["sg-xxxxx"],
    "AvailabilityZone": "<region>-<az>"
  }
}

The secret should contain username and password keys. Omit USERNAME/PASSWORD from properties when using SECRET_ID.

Redshift

Redshift accepts both JDBC password auth and IAM-based GetClusterCredentials.

Password-based: use the JDBC template above.

IAM-based (preferred for human/role users): search AWS docs for "Redshift GetClusterCredentials Glue". The Glue role needs redshift:GetClusterCredentials on the cluster; no Secrets Manager secret.

For Redshift Serverless, use the workgroup endpoint and redshift-serverless:GetCredentials.

RDS and Aurora Considerations

• RDS endpoint format: <instance-id>.<hash>.<region>.rds.amazonaws.com
• Aurora cluster endpoint (writer): <cluster-id>.cluster-<hash>.<region>.rds.amazonaws.com
• Aurora reader endpoint (read-only, load balanced): <cluster-id>.cluster-ro-<hash>.<region>.rds.amazonaws.com -- prefer for ETL reads
• Aurora custom endpoints: target a subset of instances, useful for dedicated ETL reader pools

IAM database authentication (Aurora MySQL, Aurora PostgreSQL, RDS MySQL, RDS PostgreSQL):

• Enable on the DB cluster/instance: --enable-iam-database-authentication
• Create a DB user CREATE USER etl_user IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'
• No Secrets Manager secret needed; the Glue role calls rds-db:connect at runtime to get a 15-minute token
• See credential-security.md for the full IAM policy

Prefer IAM auth over password auth where supported.

Network Setup

VPC, subnet, and security group configuration for Glue connections to private data sources. Skip this reference if the source is reachable over the public internet (Snowflake default, BigQuery, public RDS).

Contents

• When Networking Is Required
• VPC and Subnet
• Security Group Rules
• S3 VPC Endpoint
• NAT Gateway
• Cross-VPC and On-Prem

When Networking Is Required

Required:

• RDS/Aurora in private subnets
• Redshift in private subnets
• Self-managed databases in a VPC
• Snowflake with PrivateLink
• BigQuery if the Glue job also needs private AWS resources (then the Glue subnet needs NAT egress for Google APIs)

Not required:

• Public Snowflake endpoints
• Public BigQuery (default)
• Public RDS instances (not recommended for production)

VPC and Subnet

The Glue connection's SubnetId determines where Glue provisions ENIs at job runtime. Constraints:

• MUST be in the same VPC as the source (or a peered/VPN-connected VPC)
• SHOULD be a private subnet with NAT gateway egress (Glue needs internet access to pull dependencies and write to CloudWatch)
• MUST have route to source's VPC
• AvailabilityZone in PhysicalConnectionRequirements MUST match the subnet's AZ

Match AZ to source for lower latency:

aws rds describe-db-instances --db-instance-identifier <ID> \
  --query 'DBInstances[0].AvailabilityZone'

Security Group Rules

Two security groups are involved: Glue's and the source's.

Glue security group (outbound):

• Allow TCP to source port (1521 Oracle, 1433 SQL Server, 5432 Postgres, 3306 MySQL, 5439 Redshift)
• Destination: source's security group ID
• Self-referencing rule on all ports: Glue ENIs must talk to each other during a job. Required even for single-worker jobs.

Source security group (inbound):

• Allow TCP on source port from Glue's security group ID (not CIDR -- ENIs change)

Verify:

aws ec2 describe-security-groups --group-ids <glue-sg> \
  --query 'SecurityGroups[0].IpPermissionsEgress'
aws ec2 describe-security-groups --group-ids <source-sg> \
  --query 'SecurityGroups[0].IpPermissions'

S3 VPC Endpoint

Glue jobs read their scripts from S3 and write results to S3. The Glue subnet MUST have either a NAT gateway or an S3 VPC gateway endpoint; endpoint is preferred (no NAT costs, stays on AWS backbone).

Check:

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=<VPC_ID> Name=service-name,Values=com.amazonaws.<region>.s3

Create if missing:

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --service-name com.amazonaws.<region>.s3 \
  --route-table-ids <RTB_ID>

Without this, Glue jobs fail at startup with UnableToFindVpcEndpoint.

NAT Gateway

Required if:

• Glue needs to reach the internet (BigQuery, public Snowflake, external APIs)
• The subnet has no S3 VPC endpoint

Not required if:

• Source is in the same VPC AND S3 VPC endpoint exists AND no other internet access needed

NAT gateway costs per-hour plus per-GB processed. For pure private-VPC ETL with S3 endpoint, omit it.

Cross-VPC and On-Prem

Peered VPCs: Glue subnet's route table MUST have a route to the source VPC's CIDR via the peering connection. Both VPCs must be in the same region.

Transit Gateway: Route tables in both VPCs attached to the TGW MUST have routes to each other's CIDR.

On-premises via VPN/Direct Connect: Route table for Glue subnet MUST have a route to on-prem CIDR via virtual private gateway (VPN) or transit gateway (DX). Source firewall must allow inbound from Glue's ENI IPs (which change per-job -- use subnet CIDR).

Test reachability from an EC2 instance in the same subnet before creating the Glue connection:

# From EC2 in Glue's intended subnet
telnet <source-host> <source-port>

If EC2 can't reach the source, neither will Glue. Fix routing first.

Snowflake Connection Setup

AWS Glue native Snowflake connection (type SNOWFLAKE, not JDBC). Required for Glue for Spark ETL jobs reading from or writing to Snowflake.

Contents

• Connection Type
• Authentication Modes
• Connection JSON Template
• PrivateLink
• Further Reading

Connection Type

Use ConnectionType: SNOWFLAKE. Do NOT use a JDBC connection configured with the Snowflake JDBC URL -- that path is for Glue crawlers only and cannot be used by Glue for Spark ETL jobs. The two credential types are stored separately in the Data Catalog.

Authentication Modes

OAuth 2.0 for Glue Snowflake connections was released April 2026. For current Snowflake OAuth setup steps, cite Snowflake's OAuth docs rather than repeating them.

Connection JSON Template

Password-based:

{
  "Name": "snowflake-analytics",
  "ConnectionType": "SNOWFLAKE",
  "ConnectionProperties": {
    "HOST": "<account>.<region>.snowflakecomputing.com",
    "WAREHOUSE": "<warehouse-name>",
    "ROLE": "<role-name>",
    "DATABASE": "<default-database>",
    "SECRET_ID": "<secrets-manager-arn>"
  }
}

The secret must contain snowflakeUser and snowflakePassword keys per Glue's Snowflake connection convention.

Account identifier formats vary -- see Snowflake account identifier docs for the correct form for your region/cloud.

Private sources add PhysicalConnectionRequirements as in jdbc-setup.md.

PrivateLink

Snowflake accounts configured for AWS PrivateLink have a different hostname pattern. Glue jobs use the privatelink hostname directly. Configure the Glue connection's security group to allow outbound to the privatelink endpoint. See Snowflake PrivateLink docs.

Further Reading

• AWS Glue: Creating a Snowflake connection
• AWS Glue: Snowflake connections (programming)

Connection Troubleshooting

Diagnose Glue connection failures. Run checks in order: network → credentials → driver → SSL. Most failures are network.

Contents

• Test Decision Tree
• Network
• Credentials
• Driver
• SSL
• Smoke-Test Glue Job Template

Test Decision Tree

1. Run aws glue test-connection --connection-name <NAME>. If it fails, read the error message. 2. If error mentions timeout, unreachable, UnableToFindVpcEndpoint, or ENI -- go to Network. 3. If error mentions authentication, Access denied, invalid username/password, ORA-01017, 28000 -- go to Credentials. 4. If error mentions No suitable driver, ClassNotFoundException -- go to Driver. 5. If error mentions SSL handshake, certificate, TLS -- go to SSL. 6. If TestConnection passes but the engine-level smoke test fails, the issue is engine-specific (driver version, catalog config, Spark serialization). Run the smoke-test Glue job for a more informative error. See Smoke-Test Glue Job Template.

Network

Most connection failures are network. Check in order:

1. Subnet and routing

aws glue get-connection --name <NAME> \
  --query 'Connection.PhysicalConnectionRequirements'

Note the SubnetId. Check its route table:

aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=<SUBNET_ID>

Verify: route to source's VPC CIDR exists.

2. Security groups

Verify Glue SG allows outbound to source port AND has self-referencing rule:

aws ec2 describe-security-groups --group-ids <GLUE_SG>

Verify source SG allows inbound from Glue SG:

aws ec2 describe-security-groups --group-ids <SOURCE_SG>

3. S3 VPC endpoint

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=<VPC_ID> Name=service-name,Values=com.amazonaws.<region>.s3

If missing and subnet has no NAT gateway, create the endpoint. See network-setup.md.

4. Test from EC2 in the same subnet

Launch or use an existing EC2 in the Glue subnet with the Glue SG attached:

telnet <source-host> <source-port>
nc -zv <source-host> <source-port>

If EC2 can't reach the source, fix routing/SG/NACL before blaming Glue.

5. Database firewall

Source-side ACLs beyond AWS SGs:

• Oracle: listener.ora restricts connecting hosts
• SQL Server: Windows Firewall on the host
• PostgreSQL: pg_hba.conf
• MySQL: user host restrictions (SELECT user, host FROM mysql.user)
• Self-managed in a VPC: NACLs on the subnet

Credentials

Run through this checklist:

1. Secrets Manager access

# Impersonate the Glue role and fetch the secret
aws sts assume-role --role-arn <GLUE_ROLE_ARN> --role-session-name test \
  | jq -r '.Credentials'
# then with those creds:
aws secretsmanager get-secret-value --secret-id <SECRET_ID>

If AccessDenied: Glue role lacks secretsmanager:GetSecretValue on the secret ARN. See credential-security.md.

2. Secret contents match expected keys

• JDBC: username, password
• Snowflake: snowflakeUser, snowflakePassword
• BigQuery: bare base64 string (no JSON keys)

3. IAM DB auth (if enabled)

Verify the Glue role has rds-db:connect on arn:aws:rds-db:<region>:<account>:dbuser:<resource-id>/<db-user>.

Verify the DB user exists with IDENTIFIED WITH AWSAuthenticationPlugin (MySQL) or GRANT rds_iam TO <user> (PostgreSQL).

4. Direct credential test

From EC2 in the Glue subnet:

# Oracle
sqlplus <user>/<password>@//host:1521/service
# PostgreSQL
PGPASSWORD=<password> psql -h host -U user -d db -c "SELECT 1"
# MySQL
mysql -h host -u user -p<password> -e "SELECT 1"

5. Password edge cases

• Special characters (@, #, %, :) in the password can break JDBC URL parsing. Store in Secrets Manager (avoids URL encoding entirely).
• Expired password: Oracle SELECT account_status FROM dba_users; MySQL / Postgres check user's password expiry.
• Locked account: Oracle ALTER USER <user> ACCOUNT UNLOCK.

Driver

For built-in drivers (Oracle, SQL Server, PostgreSQL, MySQL, Redshift), no action needed.

For custom drivers:

1. JAR accessible

Verify the Glue role can read the JAR:

aws s3 head-object --bucket <SCRIPTS_BUCKET> --key jdbc-drivers/<driver>.jar

2. Driver class name matches

3. Driver version compatibility

Driver major version must match or exceed the database major version. Downgrading works for minor versions, not major.

SSL

1. Enforcement mismatch

Source requires SSL but connection doesn't enable it:

"JDBC_ENFORCE_SSL": "true"

2. Self-signed certificates

Source uses a cert not in the default Java truststore:

• Import the cert into a custom truststore
• Upload truststore to S3
• Add to Glue job args: --extra-jars s3://... and JVM args pointing at the truststore

For AWS RDS and Aurora, the default truststore includes the RDS CA bundle.

3. TLS version

Older databases may require TLS 1.0/1.1; Glue 5.1 or higher defaults to 1.2+. Update database or use connection property to downgrade (not recommended).

Smoke-Test Glue Job Template

When test-connection passes but the engine-level verification fails (or when test-connection fails with an unhelpful message), a minimal Glue job produces a clearer error.

Save to s3://<scripts>/test-connection.py:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'connection_name', 'source_type'])
sc = SparkContext()
glueContext = GlueContext(sc)

test_queries = {
    'oracle': '(SELECT 1 FROM DUAL) AS t',
    'sqlserver': '(SELECT 1) AS t',
    'postgresql': '(SELECT 1) AS t',
    'mysql': '(SELECT 1) AS t',
    'redshift': '(SELECT 1) AS t',
}

source_type = args['source_type']
if source_type not in test_queries:
    raise ValueError(
        f"Unsupported source_type '{source_type}'. "
        "This JDBC smoke test supports: oracle, sqlserver, postgresql, mysql, redshift. "
        "For Snowflake/BigQuery, use their native connection_type."
    )

try:
    df = glueContext.create_dynamic_frame.from_options(
        connection_type='jdbc',
        connection_options={
            'useConnectionProperties': 'true',
            'connectionName': args['connection_name'],
            'dbtable': test_queries[args['source_type']]
        }
    ).toDF()
    print(f"SUCCESS: {df.collect()}")
except Exception as e:
    print(f"FAIL: {type(e).__name__}: {e}")
    raise

Create and run the job:

aws glue create-job \
  --name test-connection-smoke \
  --role <GLUE_ROLE_ARN> \
  --command Name=glueetl,ScriptLocation=s3://<scripts>/test-connection.py,PythonVersion=3 \
  --connections Connections=<CONNECTION_NAME> \
  --glue-version 5.1 \
  --number-of-workers 2 \
  --worker-type G.1X

aws glue start-job-run \
  --job-name test-connection-smoke \
  --arguments '{"--connection_name":"<CONNECTION_NAME>","--source_type":"<TYPE>"}'

Read CloudWatch logs for the specific JDBC error. Most common errors are more descriptive in logs than in get-connection-test output.

Delete the test job after use.