Inter-container traffic always NAT'd post-packer change?

[james-masson]

Pre-packer ( eg. box 237 ) the NAT rules between containers was something like...

Chain warden-postrouting (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       10.244.0.0/19       !10.244.0.0/19       

This meant all inter-container traffic was direct routed.

Post-packer ( eg. head or box 293 ), the NAT rules seem to be SNATing all inter-container traffic.

Chain w--postrouting (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 SNAT       all  --  *      *       10.244.0.0/16        0.0.0.0/0            to:10.0.2.15

This means that all traffic comes with a source of the wrong IP.

root@ad1f9e9f-7384-470a-af05-0eb5949fb407:~# tcpdump -nni w5movi9acqir-1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on w5movi9acqir-1, link-type EN10MB (Ethernet), capture size 65535 bytes
10:32:15.631798 IP 10.0.2.15 > 10.244.10.6: ICMP echo request, id 149, seq 18, length 64
10:32:15.631938 IP 10.244.10.6 > 10.0.2.15: ICMP echo reply, id 149, seq 18, length 64
10:32:16.631908 IP 10.0.2.15 > 10.244.10.6: ICMP echo request, id 149, seq 19, length 64
10:32:16.632160 IP 10.244.10.6 > 10.0.2.15: ICMP echo reply, id 149, seq 19, length 64
10:32:17.633047 IP 10.0.2.15 > 10.244.10.6: ICMP echo request, id 149, seq 20, length 64
10:32:17.633268 IP 10.244.10.6 > 10.0.2.15: ICMP echo reply, id 149, seq 20, length 64
10:32:18.632100 IP 10.0.2.15 > 10.244.10.6: ICMP echo request, id 149, seq 21, length 64
10:32:18.632331 IP 10.244.10.6 > 10.0.2.15: ICMP echo reply, id 149, seq 21, length 64
10:32:19.633623 IP 10.0.2.15 > 10.244.10.6: ICMP echo request, id 149, seq 22, length 64
10:32:19.633883 IP 10.244.10.6 > 10.0.2.15: ICMP echo reply, id 149, seq 22, length 64

This may work with traffic that doesn't imply meaning from source/destination IP, but will confuse applications that rely on this.

Is this SNAT deliberate? I wouldn't normally expect this behaviour on real networks.

thanks

James M

[sykesm]

FWIW, the recipes that used to build the boxes would apply replace-SNAT-with-MASQ.patch while the new builds seem to use warden-linux without changes.

I've actually wondered about the use of SNAT vs MASQ in warden/garden in the past; might be interesting to see if there are opinions there about changes to enable this use case without breaking the required encapsulation of containers on the DEAs.

[james-masson]

I understand why you'd want this on DEAs - but BOSH-Lite is supposed to be a general purpose tool, used for developing more than CF core.

For example, using a Cassandra bosh-release through this SNAT is likely to result in Weird Stuff Happening.

[cppforlife]

We have recently changed how bosh lite boxes are built and this feature was skipped. None of our tests caught this so we decided to postpone it until someone asks for it.

It seems that a proper place for this would be as a configuration for garden-linux. Recently we have experimented with splitting off garden into its own bosh release and London team taking over maintaining it. Bosh-lite did not switch over to using that release yet pending some changes.

I will talk with that team and see how hard would this be to support it.

I'll update this issue with more info shortly. If you need a temporary solution you can always apply patch that was mentioned above via your vagrantfile.

-dmitriy

On Sep 25, 2014, at 1:16 AM, James Masson [email protected] wrote:

I understand why you'd want this on DEAs - but BOSH-Lite is supposed to be a general purpose tool, used for developing more than CF core.

For example, using a Cassandra bosh-release through this SNAT is likely to result in Weird Stuff Happening.

—
Reply to this email directly or view it on GitHub.

[james-masson]

Thanks Dmitriy.

Yes, I have a workaround for my use case - just concerned that other people will run in to mysterious problems related to this change and not know the cause - seems this happened to CF London with cassandra thrift.

[aramprice]

+1 for addressing this, somehow.

cassandra (thrift ?) doesn't like the NAT between bosh-lite VMS, changing. making the following changes (thanks @james-masson) on our bosh-lite (vagrant) VM fixed the problem:

# remove post-routing NAT rule
iptables  -t nat -F w--postrouting
# skip NAT for internal network
iptables -t nat -A w--postrouting --source 10.244.0.0/16 ! --destination 10.244.0.0/16 --jump MASQUERADE

[cppforlife]

Tracker story in CF Garden: https://www.pivotaltracker.com/story/show/79557632

[james-masson]

In the tracker story @aramprice mentioned that his problems went away on current versions of bosh-lite.

Mine haven't - had to apply the fix mentioned by @aramprice above on box v370

[bosh-ci-push-pull]

@james-masson no changes have been made to garden that bosh-lite uses so patch is still required to make this work.

-dmitriy

On Oct 14, 2014, at 6:20 AM, James Masson [email protected] wrote:

In the tracker story @aramprice mentioned that his problems went away on current versions of bosh-lite.

Mine haven't - had to apply the fix mentioned by @aramprice above on box v370

—
Reply to this email directly or view it on GitHub.

[jpalermo]

@cppforlife We're still having this problem, but now it is inconsistent across infrastructures. Everything works fine on virtualbox but when using bosh lite on aws we are still seeing the above problem.

[jpalermo]

We were wrong about it being inconsistent across infrastructures. Created a PR to address the issue:
#364