Skip to main content
HashnodeThe Cybergavin StackThe Cybergavin Stack

Command Palette

Search for a command to run...

Bypassing the Bottleneck: Architecting an AWS Express Lane for Heavy On-Premises Workloads

How we used PAN-OS Policy-Based Forwarding, selective SNAT, and an isolated Transit Gateway routing domain to bypass centralized firewall inspection for trusted high-volume database transfers—without weakening our security posture.

Updated
•10 min read•View as Markdown
Bypassing the Bottleneck: Architecting an AWS Express Lane for Heavy On-Premises Workloads
G
Gavin Satur
If you are migrating or replicating large databases to AWS, you may eventually run into an unexpected bottleneck: not the VPN itself, but everything sitting in the traffic path. For months, our Database Administrators (DBAs) battled timeouts and failures while using Oracle RMAN DUPLICATE to seed an approximately 1 TB database from our on-premises datacenter into AWS over Site-to-Site VPN. One obvious answer was AWS Direct Connect. But in our environment, our estimated monthly connectivity cost for two Site-to-Site VPN connections was less than half that of comparable redundant Direct Connect connectivity. These are environment-specific estimates rather than generally applicable AWS prices, but they gave us a good reason to first determine whether we could solve the problem using VPN. It turned out that the VPN wasn't the primary bottleneck.
Our default hybrid-cloud security architecture routes traffic through centralized Palo Alto Networks firewalls running in an AWS Security VPC. That's an appropriate default security posture, but forcing a known, tightly scoped, sustained database-transfer flow through a stateful inspection tier imposed a significant reliability and performance penalty. For communication between explicitly approved database systems, we concluded that the incremental security benefit of centralized inspection did not justify that reliability and performance impact. Instead, we could constrain the exception through IPsec encryption, tightly scoped source and destination addresses, PAN-OS policy, AWS routing, VPC security controls, and database-level controls.
What we needed was therefore an explicitly controlled exception to an otherwise secure-by-default network. AWS's introduction of Large Bandwidth Site-to-Site VPN tunnels, supporting up to 5 Gbps per tunnel, made that much more practical. Standard AWS Site-to-Site VPN tunnels support up to 1.25 Gbps per tunnel; when using Transit Gateway and ECMP, multiple tunnels can provide higher aggregate capacity. Large Bandwidth Tunnels raise the individual tunnel capacity to as much as 5 Gbps and are specifically positioned by AWS for bandwidth-intensive hybrid applications and large data migrations.

Here's how we built our AWS "express lane" using Palo Alto Networks PAN-OS firewalls, AWS Transit Gateway, an isolated TGW routing domain, and a dedicated Large Bandwidth Site-to-Site VPN.

The Baseline: Inspection by Default

Our AWS network follows a hub-and-spoke model with centralized firewall inspection. Two Transit Gateway route tables provide the core of the inspection architecture:

TGW-RT-TO-FW - Pre-Inspection: Standard spoke VPC and hybrid-connectivity attachments enter this routing domain and are directed toward the Security VPC.

TGW-RT-FROM-FW - Post-Inspection: The Security VPC attachment uses this table to reach the appropriate destination after inspection.

AWS Transit Gateway associates each attachment with one TGW route table, while an attachment can propagate routes into multiple route tables. That distinction is important: the associated table determines the routing policy applied to traffic entering the Transit Gateway from that attachment.

Our standard Site-to-Site VPN, VPN-A, remains the normal hybrid connectivity path. It uses standard-bandwidth AWS VPN tunnels — up to 1.25 Gbps per tunnel, with higher aggregate capacity possible through ECMP, while our Security VPC remains the default inspection point.

Fig. 1: Standard on-premises-to-AWS traffic traverses the Security VPC for inspection.

The Exception: A Dedicated High-Bandwidth VPN

For high-volume trusted database transfers, we introduced a second Site-to-Site VPN, VPN-B, using AWS Large Bandwidth Tunnels with up to 5 Gbps of capacity per tunnel. The 5 Gbps VPN alone wasn't the solution. The more important design decision was selectively steering only approved traffic onto it while keeping centralized inspection as the default for everything else.

Fig. 2: Approved high-volume database traffic enters through VPN-B and bypasses the AWS Security VPC

In our implementation, both the VPN-B attachment and Database VPC attachment are associated with TGW-RT-BYPASS-FW. This is an isolated TGW routing domain: the Database VPC is not generally exempt from inspection simply because it participates in a bypass path.

The governing principle is:

Bypass must be explicit. Inspection remains the default.

The Solution: PBF, SNAT and Isolated TGW Routing Domain

1. Steering the Traffic On-Premises

The first routing decision happens before the packets leave our datacenter. Our Palo Alto border firewalls use Policy-Based Forwarding (PBF) to identify approved on-premises-to-AWS database traffic and direct it through VPN-B rather than the standard path. PAN-OS PBF can match attributes including source and destination addresses and then override the normal routing decision. Separately, our NAT policy source-NATs selected on-premises database traffic into a dedicated bypass prefix, for example: 192.168.250.0/24 This gives bypassed traffic a unique routing identity. Instead of AWS having to distinguish whether the original on-premises networks should return through VPN-A or VPN-B, it can simply know: 192.168.250.0/24 → VPN-B SNAT isn't an inherent requirement of AWS Site-to-Site VPN. In our architecture it acts as a route-disambiguation mechanism, allowing us to introduce the bypass without changing the routing behavior of the original shared on-premises CIDRs. PBF determines the path; NAT determines the translated identity. They are separate PAN-OS policy mechanisms.

2. Creating an AWS Bypass Routing Domain

We created a dedicated Transit Gateway route table: TGW-RT-BYPASS-FW and associated both the VPN-B attachment and the Database VPC attachment with it. The table is intentionally sparse. Conceptually:

Approved on-prem DB prefixes   → VPN-B
On-prem bypass SNAT prefix     → VPN-B
Database VPC CIDR              → Database VPC attachment
0.0.0.0/0                      → Security VPC
  • The first routes implement explicit exceptions.
  • The final route is the critical guardrail: 0.0.0.0/0 → Security VPC
  • Anything that does not match an approved bypass route goes back through our normal PAN-OS inspection architecture.
  • That means a database EC2 instance attempting to communicate with an ordinary VPC, an unapproved on-premises network, or an internet destination continues through the centralized firewall path.

3. Keep the Bypass Route Table Sparse

There is an important operational detail here. The security property above works only if TGW-RT-BYPASS-FW does not acquire more-specific routes that unintentionally bypass the Security VPC. Transit Gateway propagation can automatically install attachment routes into selected route tables. Because more-specific routes beat the 0.0.0.0/0 fail-safe, blindly propagating every spoke VPC into the bypass table could silently turn: Database VPC → Application VPC into direct, uninspected communication. AWS supports independently controlling route-table association and propagation, so we deliberately control propagation into the bypass routing domain.

The rule is simple:

Only destinations explicitly approved for bypass should have more-specific routes in the bypass table. Everything else falls through to the Security VPC.

That guardrail is arguably more important than the VPN itself.

Bidirectional Traffic Is Not the Same as Bidirectional Session Initiation

Our first use case was straightforward: an on-premises Oracle system initiated a high-volume RMAN operation toward AWS. PBF selected VPN-B, SNAT assigned the dedicated bypass identity, and AWS returned traffic for that SNAT prefix through VPN-B. Conceptually:

---------------------
ON-PREMISES INITIATED
---------------------

On-prem Oracle
      |
      | PBF → VPN-B
      | SNAT → bypass prefix
      v
    VPN-B
      |
      v
AWS Oracle

---------
RETURN
---------

AWS Oracle
      |
      | destination = bypass SNAT prefix
      v
    VPN-B
      |
      v
On-prem Oracle
That is symmetric and deterministic. Later, the requirement evolved. The DBAs needed the Oracle systems to establish sessions in either direction, and Oracle Enterprise Manager also needed to communicate with the database systems. That exposed an important distinction. A VPN is inherently bidirectional. There is nothing about AWS Site-to-Site VPN that prevents AWS from initiating a TCP session toward on-premises. The challenge is routing the response back through the same VPN.

Consider an AWS-initiated Oracle connection:

AWS Oracle:ephemeral-port
        |
        | SYN
        v
      VPN-B
        |
        v
On-prem Oracle:1521

The response looks like:

On-prem Oracle:1521
        |
        | SYN/ACK
        v
AWS Oracle:ephemeral-port

If the on-premises PBF policy only identifies traffic by destination service — for example TCP/1521 — the SYN/ACK no longer matches that rule because its destination is an ephemeral client port. The firewall can therefore select the normal VPN-A route for the response:

AWS ---- SYN ----> VPN-B ----> On-prem

AWS <--- SYN/ACK -- VPN-A <---- On-prem

That's asymmetric routing, exactly what stateful firewalls don't want. PAN-OS documents that PBF can be applied to the initial SYN or the first SYN/ACK of a session, which is particularly relevant to this scenario.

Make PBF Endpoint-Based, Not Oracle-Port-Based

Traffic between these explicitly approved systems belongs to the high-bandwidth hybrid path.

PAN-OS supports using addresses as PBF match criteria, and Palo Alto explicitly documents that setting Service to Any causes all otherwise matching traffic to use the PBF rule.

So the cleaner PBF is conceptually:

Source:
    approved on-prem DB/OEM endpoints

Destination:
    approved AWS DB endpoints

Service:
    Any

Forwarding:
    VPN-B
Now the policy doesn't care whether Oracle, RMAN, OEM, SSH for troubleshooting, or a TCP return packet happens to use 1521, 3872, or an ephemeral port. It cares about which systems are communicating. This also means an AWS-initiated SYN/ACK returning from an approved on-premises endpoint toward an approved AWS endpoint remains eligible for VPN-B rather than falling back to VPN-A simply because its destination port is ephemeral.

Service=Any in PBF does not mean Allow Any. PAN-OS Security Policy can still independently allow only the Oracle, RMAN, OEM, administrative, or other services actually required.

In other words:

PBF       = path
NAT       = identity
Security  = permission

This separation makes the design both simpler and easier to govern.

Completing Bidirectional Initiation

Endpoint-based PBF solves only the on-premises path-selection half of the problem. AWS must also know that connections initiated in AWS toward the real on-premises database addresses should enter VPN-B. So TGW-RT-BYPASS-FW also needs approved real on-premises endpoint routes pointing to VPN-B, in addition to the existing SNAT return-prefix route:
Approved real on-prem DB CIDRs → VPN-B
Bypass SNAT prefix             → VPN-B
0.0.0.0/0                      → Security VPC

This produces two valid traffic models.

For on-premises-initiated sessions:

On-premises
  → PBF
  → SNAT
  → VPN-B
  → AWS

AWS
  → bypass SNAT prefix
  → VPN-B
  → On-premises

For AWS-initiated sessions:

AWS
  → real on-prem DB prefix
  → VPN-B
  → On-premises

On-premises
  → endpoint-based PBF
  → VPN-B
  → AWS

Both sides can initiate, and both directions remain on VPN-B.

The Result

By combining PAN-OS Policy-Based Forwarding, selective SNAT, a dedicated Large Bandwidth Site-to-Site VPN and an intentionally isolated Transit Gateway routing domain, we decoupled a high-volume database workload from our standard centralized inspection path without deploying Direct Connect solely to solve this problem. The DBAs can perform RMAN database seeding over a path with up to 5 Gbps of tunnel capacity, without forcing those sustained transfers through the AWS PAN-OS inspection tier. At the same time, we haven't made the Database VPC generally uninspected. The default route still sends non-bypass traffic to the Security VPC, and only explicitly approved prefixes escape that path. Our default inspection posture therefore remains intact; we've created a controlled exception rather than removing the control. And as the requirement evolved from one-way database seeding to Oracle, RMAN and OEM sessions that could be initiated from either AWS or on-premises, another lesson became clear:

Don't encode a workload-routing policy as a growing list of application ports when the real policy is about trusted endpoints.

Routing should determine the path. NAT should solve address-identity problems. Security Policy should determine what's allowed. When it comes to hybrid cloud networking, you don't always have to choose between strict security, high performance and cost control. Sometimes you just need a better toll lane.
#aws#networking#vpn#security#hybrid-cloud

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

T

The Cybergavin Stack

2 posts

Architecture, automation, and lessons from building real systems.