Plz help me with my atc, it successfully runned but actually not working

[JaewookYou]

ATC has a few requirements, this section is used to make sure the requirements are met. Until you can check all the boxes, ATC will not work on your system.

• [o ] atcd runs on Linux
• [ o] atcd has 2 physical interfaces
• [o ] 1 that connects to the internet (WAN)
• [o ] 1 that connects to the local network (LAN)
• [o ] the traffic from the devices that are being shaped is routed through atcd
• [x ] atcd sees the real IP of the devices (e.g there is no NAT on the LAN segment)

Steps to reproduce the issue:
I succesffuly run ATC, but actually it doesn't work. I gave packet loss, delay, modify bandwidth, but device's network works well.

1. Here's my network environment.

                             ATC Ubuntu
   

ap1 ---------------------eth0 | eth1 ---------------------------- ap2 ------------ adsl-----outside
(mobile use this) atcd (connect lan)

2. This is my ifconfig set

eht0 -- lan
eth1 -- wan

$ ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:100.100.100.1 Bcast:100.100.100.255 Mask:255.255.255.0
inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:284744 errors:0 dropped:0 overruns:0 frame:0
TX packets:341465 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:215253744 (215.2 MB) TX bytes:365172508 (365.1 MB)

eth1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.xxx.xxx.250 Bcast:172.xxx.xxx.255 Mask:255.255.255.0
inet6 addr: xxxx::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:361268 errors:0 dropped:0 overruns:0 frame:0
TX packets:299660 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:383028468 (383.0 MB) TX bytes:215711661 (215.7 MB)

eth2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:21756 errors:0 dropped:0 overruns:0 frame:0
TX packets:21756 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:3055869 (3.0 MB) TX bytes:3055869 (3.0 MB)

3. This is my command when run ATC

(venv)a@a-Z68A-D3H-B3:~/dev/atc/atcui$ sudo atcd --thrift-host 127.0.0.1 --atcd-lan eth0 --atcd-wan eth1 --atcd-dont-drop-packets
INFO:AtcdVService.AtcdNBServerTask:AtcdNBServerTask Server Started on 127.0.0.1:9090
INFO:AtcdVService.AtcdLinuxShaper:Calling initialize_shaping_system
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -F FORWARD
INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth0
INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth0
INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth1
INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth1
INFO:AtcdVService.AtcdLinuxShaper:Restoring shaped connection from DB
INFO:AtcdVService.AtcdLinuxShaper:Request startShaping TrafficControl(device=TrafficControlledDevice(controllingIP='100.100.100.1', controlledIP='100.100.100.1'), timeout=85805.38567900658, settings=TrafficControlSetting(down=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0)), up=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0))))
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth1
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth1, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth1, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth1, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -d 100.100.100.1 -i eth1 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth0
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth0, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth0, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth0, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -s 100.100.100.1 -i eth0 -j MARK --set-mark 2
DEBUG:AtcdVService:All tasks started
DEBUG:AtcdVService:VService Active. Awaiting graceful shutdown.
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)

4. This is my iptables set

$sudo iptables -nvL

Chain INPUT (policy ACCEPT 39203 packets, 22M bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 280K packets, 344M bytes)
pkts bytes target prot opt in out source destination
216K 206M ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 38460 packets, 4955K bytes)
pkts bytes target prot opt in out source destination

• I've done this command set on iptables

$sudo iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE

5. This is my ~/dev/atc/atcui/atcui/settings.py and urls.py set

• settings.py

ALLOWED_HOSTS = ['*']

Application definition

INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'atc_api',
'bootstrap_themes',
'django_static_jquery',
'atc_demo_ui',
'atc_profile_storage',
]

• urls.py

from django.conf.urls import url, include
from django.contrib import admin
from django.views.generic.base import RedirectView

urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^api/v1/', include('atc_api.urls')),
url(r'^atc_demo_ui/', include('atc_demo_ui.urls')),
url(r'^api/v1/profiles/', include('atc_profile_storage.urls')),
url(r'^$', RedirectView.as_view(url='/atc_demo_ui/', permanent=False)),
]

6. Ubuntu's info

######### uname -a #########

Linux x-xxxx-xxx-xx 4.4.0-47-generic #68~14.04.1-Ubuntu SMP Wed Oct 26 19:42:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

######### cat /etc/os-release #########

NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

######### python -V #########

Python 2.7.6

######### pip freeze #########

Django==1.10.3
PAM==0.4.2
Pillow==2.3.0
Twisted-Core==13.2.0
Twisted-Web==13.2.0
adium-theme-ubuntu==0.3.4
apt-xapian-index==0.45
argparse==1.2.1
atc-thrift==0.1.6
atcd==0.1.6
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
daemonize==2.4.7
debtagshw==0.1
defer==1.0.6
dirspec==13.10
django-atc-api==0.1.6
django-atc-demo-ui==0.1.6
django-atc-profile-storage==0.1.6
django-bootstrap-themes==3.3.6
django-static-jquery==1.11.1
djangorestframework==3.5.3
duplicity==0.6.23
futures==3.0.5
html5lib==0.999
httplib2==0.8
lockfile==0.8
lxml==3.3.3
oauthlib==0.6.1
oneconf==0.3.7.14.04.1
pexpect==3.1
piston-mini-client==0.7.5
pyOpenSSL==0.13
pycrypto==2.6.1
pycups==1.9.66
pygobject==3.12.0
pyotp==1.4.1
pyroute2==0.3.3
pyserial==2.6
pysmbc==1.0.14.1
python-apt==0.9.3.5ubuntu2
python-debian==0.1.21-nmu2ubuntu2
pyxdg==0.25
reportlab==3.0
requests==2.2.1
sessioninstaller==0.0.0
six==1.5.2
software-center-aptd-plugins==0.0.0
sparts==0.7.1
system-service==0.1.6
thrift==0.9.3
unity-lens-photos==1.0
urllib3==1.7.1
virtualenv==1.11.4
wheel==0.24.0
wsgiref==0.1.2
xdiagnose==3.6.3build2
zope.interface==4.0.5

######### ip r #########

default via 172.xxx.xxx.1 dev eth1 proto static
100.100.100.0/24 dev eth0 proto kernel scope link src 100.100.100.1 metric 1
172.xxx.xxx.0/24 dev eth1 proto kernel scope link src 172.xxx.xxx.250 metric 1

######### iptables-save #########

######### ip r #########

default via 172.xxx.xxx.1 dev eth1 proto static
100.100.100.0/24 dev eth0 proto kernel scope link src 100.100.100.1 metric 1
172.xxx.xxx.0/24 dev eth1 proto kernel scope link src 172.xxx.xxx.250 metric 1

######### ip a #########

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
inet 100.100.100.1/24 brd 100.100.100.255 scope global eth0
valid_lft forever preferred_lft forever
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
inet 172.xxx.xxx.250/24 brd 172.xxx.xxx.255 scope global eth1
valid_lft forever preferred_lft forever
valid_lft forever preferred_lft forever
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000

######### tc qdisc show dev eth1 #########

qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 109455 direct_qlen 1000
qdisc netem 8009: parent 1:3 limit 1000 delay 1.0s loss 100%

######### tc class show dev eth1 #########

class htb 1:3 root leaf 8009: prio 0 rate 1000Kbit ceil 1000Kbit burst 1600b cburst 1600b

######### tc filter show dev eth1 #########

filter parent 1: protocol ip pref 1 fw
filter parent 1: protocol ip pref 1 fw handle 0x3 classid 1:3

Always thanks for providing useful and helpful programs, and kind answer for questions.
Thank you so much, I'll wait your kind answer.

[smorad]

Are you sure [o ] atcd sees the real IP of the devices (e.g there is no NAT on the LAN segment)? Can you dump the IP addresses of your shaped devices as seen from your linux machine?

[JaewookYou]

@smorad Thanks for applying

ap1 ---------------------eth0 | eth1 ---------------------------- ap2 ------------ adsl-----outside

in this environment,

mobile device

• 123.123.123.2~

ap1

• internal IP : 123.123.123.1
  net mask : 255.255.255.0
• external IP : 100.100.100.10
  net mask : 255.255.255.0
  gateway : 100.100.100.1

eth0

• IP : 100.100.100.1
  net mask : 255.255.255.0
  gateway : 0.0.0.0

eth1

• IP : 172.4.29.250
  net mask : 255.255.255.0
  gateway : 172.4.29.1

ap2

• internal IP : 172.4.29.1
  net mask : 255.255.255.0
  DNS : 8.8.8.8
• external IP : 172.16.31.3
  subnet mask : 255.255.255.0
  gateway : 172.16.31.1
  DNS : 8.8.8.8

[chantra]

@JaewookYou

Can you tell us how you trigger the shaping?

TrafficControl(device=TrafficControlledDevice(controllingIP='100.100.100.1', controlledIP='100.100.100.1'), timeout=85805.38567900658, 

this show that the shaping is triggered from the atc box, not from a mobile client on the network.

[JaewookYou]

@chantra Ok, then how can I change that the shaping is triggered from the atc box?

[chantra]

Ok, then how can I change that the shaping is triggered from the atc box?

@JaewookYou what do you mean? Do you want to remove it? or update it? If removing, there is an option to start from clean DB, --help should help you finding it out. I dont remember on top of my head. If you leave it around, it will eventually clear and in any cases, it should not cause any issue.

Did you manage to shape the traffic from a remote device? is it working for you? if now, can you send the logs.

[JaewookYou]

@chantra Sorry, I misunderstood. I just run atc server by

sudo atcd --thrift-host 127.0.0.1 --atcd-lan eth0 --atcd-wan eth1

and run UI page by

sudo python manage.py runserver 100.100.100.1:8000

I turned on ATC by UI Page's 'Turn On' button.

[JaewookYou]

@chantra
This is when I run ATC server / UI Page server and press 'Turn On' button on UI Page.

(venv)a@a-Z68A-D3H-B3:~/dev/atc/atcui$ sudo atcd --thrift-host 127.0.0.1 --atcd-lan eth0 --atcd-wan eth1
[sudo] password for a:
INFO:AtcdVService.AtcdNBServerTask:AtcdNBServerTask Server Started on 127.0.0.1:9090
INFO:AtcdVService.AtcdLinuxShaper:Calling initialize_shaping_system
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -F FORWARD
INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth0
INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth0
INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth1
INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth1
INFO:AtcdVService.AtcdLinuxShaper:Restoring shaped connection from DB
INFO:AtcdVService.AtcdLinuxShaper:Request startShaping TrafficControl(device=TrafficControlledDevice(controllingIP='100.100.100.1', controlledIP='100.100.100.1'), timeout=81521.30441403389, settings=TrafficControlSetting(down=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0)), up=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0))))
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth1
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth1, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth1, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth1, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -d 100.100.100.1 -i eth1 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth0
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth0, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth0, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth0, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -s 100.100.100.1 -i eth0 -j MARK --set-mark 2
DEBUG:AtcdVService:All tasks started
DEBUG:AtcdVService:VService Active. Awaiting graceful shutdown.
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)
INFO:AtcdVService.AtcdLinuxShaper:Request stopShaping for ip 100.100.100.1
INFO:AtcdVService.AtcdLinuxShaper:Unshaping ip 100.100.100.1 on interface eth1
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -D FORWARD -d 100.100.100.1 -i eth1 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:deleting filter on IFID eth1, handle 2
INFO:AtcdVService.AtcdLinuxShaper:deleting class on IFID eth1, classid 1:2
INFO:AtcdVService.AtcdLinuxShaper:Unshaping ip 100.100.100.1 on interface eth0
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -D FORWARD -s 100.100.100.1 -i eth0 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:deleting filter on IFID eth0, handle 2
INFO:AtcdVService.AtcdLinuxShaper:deleting class on IFID eth0, classid 1:2
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)
INFO:AtcdVService.AtcdLinuxShaper:Request startShaping TrafficControl(device=TrafficControlledDevice(controllingIP='100.100.100.1', controlledIP='100.100.100.1'), timeout=86400, settings=TrafficControlSetting(down=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0)), up=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0))))
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth1
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth1, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth1, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth1, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -d 100.100.100.1 -i eth1 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 100.100.100.1 on interface eth0
INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth0, classid 1:2,parent 1:0, rate 4194303kbits
INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth0, parent 1:2, loss 0.0%, delay 0
INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth0, classid 1:2, handle 2, rate: Nonekbits
INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -s 100.100.100.1 -i eth0 -j MARK --set-mark 2
INFO:AtcdVService.AtcdLinuxShaper:Request requestToken(100.100.100.1, 259200)

[chantra]

@JaewookYou
sorry for the late reply:

This is when I run ATC server / UI Page server and press 'Turn On' button on UI Page.

Are you doing this from the phone? Or from the ATC host?

Theses logs:

TrafficControl(device=TrafficControlledDevice(controllingIP='100.100.100.1', controlledIP='100.100.100.1'), timeout=81521.30441403389, settings=TrafficControlSetting(down=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0)), up=Shaping(loss=Loss(percentage=0.0, correlation=0.0), delay=Delay(delay=0, jitter=0, correlation=0.0), rate=None, iptables_options=[], corruption=Corruption(percentage=0.0, correlation=0.0), reorder=Reorder(percentage=0.0, correlation=0.0, gap=0))))

shows that the controllingIP (ATC box) is the same as the controlledIP (suppose to be the device you shape).

So there is 2 possibilities here.

1. As @smorad suggested, there is nat involved and the traffic coming to ATC appear as if it came from ATC, this will prevent ATC from being able to shape the traffic. As mentioned in the requirements: atcd sees the real IP of the devices (e.g there is no NAT on the LAN segment), you need to get rid of this.
2. You are connect to 100.100.100.1 directly from the machine itself, this causes ATC to try to shape itself, which will actually not happen and also no shape the device you intent to shape.