local services accessable with/without ncat in public-bridge #53
Comments
|
I think this issue is more of a Layer8 problem. Therefore I am closing it. Still I would be interested, how I can oversee portmappings or redirects edit: works. |
|
@einsiedlerkrebs Sorry for not being responsive, busy days around here. You are correct that the tooling for anchors in pfctl is a bit annoying. What you can do is: which will list port redirects of known running pots to you (making use of pfctl). If you want to see pot's anchors (e.g., for redirects) without using pot itself, you can do (case sensitive). And then poke into each of them, e.g.: |
|
Hi @grembo, No problem. I am seeing my issue here more as a support case, not a bug any more. Still I wanted to keep it in order to document a little. and BTW: When |
einsiedlerkrebs
changed the title
We run a cluster without ncat by using custom networking scripts - this is configured in pot.conf by setting POT_EXPORT_PORTS_PF_RULES_HOOK, see https://github.com/bsdpot/pot/blob/master/etc/pot/pot.default.conf#L53-L63 This allows us to have local consul on the host talk to services as well as services talk to each other in a well controlled way. The script I use is not published yet, as it's quite custom for what we do (it consists of creating a reflect jail on the host, which runs pf itself). If you're interested I can share more details (it's a bit on the complex side though).
|
|
@einsiedlerkrebs tagging you again to make sure you won't miss my response above. |
|
Hi Grembo.
Yes. That seems friendly!
I have seen this hook in the code and considered indeed to use it, but later thought I might come up with a dedicated On a glance on
Of course I would like to archive a least proprietary solution (In a frame of using the hole cloud system on one host ;) ). Maybe the POT_EXPORT_PORTS_PF_RULES_HOOK and FreeBSD new capabilities of filter on |
|
Another Idea that came up and could solve the issue I have is running consul in a jail with vnet. That should allow the firewall to apply the redirects as well. |
|
The Idea above seem to lead into the same issue (of course ?) because the consul server can not be reached from localhost. |
|
@grembo since I didn't had any success yet, would you mind, to share the script you mentioned? |
|
@einsiedlerkrebs Ok, so here we go (might also be interesting to @pizzamig): This setup scales pretty well, which means:
Example is from one compute node in the cluster. Configuration used: pot.conf: make-pf-rules.sh: #!/bin/sh
if [ -e /usr/local/etc/pot/pfrules.conf ]; then
# shellcheck disable=SC1091
. /usr/local/etc/pot/pfrules.conf
fi
if [ -z "$APP_IP" ]; then
1>&2 echo "Error: Please configure APP_IP"
exit 1
fi
: "${APP_BITMASK:="24"}"
if [ $# -ne 8 ]; then
1>&2 echo "Usage: $0 POT_EXTIF _bridge _pot_net _pot_gateway _proto_port _host_port _ip _pot_port"
1>&2 echo "Example: $0 cluster bridge1 10.192.0.0/10 10.192.0.1 tcp 32732 10.192.0.10 80"
exit 1
fi
set -e
# shellcheck disable=SC3040
set -o pipefail
SCRIPT=$(readlink -f "$0")
SCRIPTDIR=$(dirname "$SCRIPT")
if ! jls -j pot-reflect >/dev/null 2>&1; then
"$SCRIPTDIR/make-reflect-jail.sh" >/dev/null
fi
POT_EXTIF=$1; shift
_bridge=$1; shift
POT_NETWORK=$1; shift
POT_GATEWAY=$1; shift
_proto_port=$1; shift
_host_port=$1; shift
_ip=$1; shift
_pot_port=$1; shift
# custom redirect rules / can contain various ports depending on required payload
echo "rdr on $_bridge proto {tcp, udp} from $_ip to $POT_GATEWAY port 53 -> 127.0.0.1"
echo "rdr on $_bridge proto udp from $_ip to $POT_GATEWAY port 514 -> 127.0.0.1"
echo "rdr pass log on pot-reflect-a proto $_proto_port from 10.9.9.2 to $APP_IP port $_host_port -> $_ip port $_pot_port"
echo "rdr pass log on $POT_EXTIF inet proto tcp from $APP_IP/$APP_BITMASK to $APP_IP port $_host_port -> $_ip port $_pot_port"
echo "rdr log on $_bridge proto tcp from $POT_NETWORK to $APP_IP port $_host_port tag potnat -> $_ip port $_pot_port"
echo "nat log on $_bridge proto tcp from $POT_NETWORK to $_ip port $_pot_port tagged potnat -> $APP_IP"Content of make-reflect-jail.sh: #!/bin/sh
if [ -e /usr/local/etc/pot/pfrules.conf ]; then
# shellcheck disable=SC1091
. /usr/local/etc/pot/pfrules.conf
fi
for var in APP_BITMASK APP_IP CLUSTER_IF COMPUTE_IF NOMAD_IF UNTRUSTED_IF; do
if [ -z "$(eval echo "\${$var}")" ]; then
log "Error: Please configure $var"
exit 1
fi
done
set -e
# shellcheck disable=SC3040
set -o pipefail
SCRIPT=$(readlink -f "$0")
SCRIPTDIR=$(dirname "$SCRIPT")
fibs=$(sysctl -n net.fibs)
if [ "$fibs" -lt 2 ]; then
sysctl net.fibs=2 >/dev/null
fi
JAILDIR="/root/reflectjail"
# cleanup existing jail if necessary
if jls -j pot-reflect >/dev/null 2>&1; then
"$SCRIPTDIR"/destroy-reflect-jail.sh "$APP_IP/$APP_BITMASK" "$NOMAD_IF"
fi
# create jail and config
mkdir -p "$JAILDIR"
tar -C / -cf - bin etc/pf.os lib libexec sbin | tar -xf - -C "$JAILDIR/."
chflags -R noschg "$JAILDIR"
mkdir -p "$JAILDIR/dev"
cat <<EOF >"$JAILDIR/etc/pf.conf"
nat log on pot-reflect-b to $APP_IP -> pot-reflect-b:0
pass
EOF
# start jail
jail -c vnet name=pot-reflect host.hostname=pot-reflect \
path="$JAILDIR" mount.devfs persist
# build vnet
epair_a=$(ifconfig epair create)
epair_b=$(echo "$epair_a" | sed "s/a$/b/")
ifconfig "$epair_a" inet 10.9.9.1/30 name pot-reflect-a >/dev/null
ifconfig "$epair_b" name pot-reflect-b >/dev/null
ifconfig pot-reflect-b vnet pot-reflect
jexec pot-reflect ifconfig pot-reflect-b 10.9.9.2/30
jexec pot-reflect sysctl net.inet.ip.forwarding=1 >/dev/null
jexec pot-reflect sysctl net.inet.ip.redirect=0 >/dev/null
jexec pot-reflect route add default 10.9.9.1 >/dev/null
# tune devfs and start pf
devfs -m "$JAILDIR/dev" rule apply path pf unhide
jexec pot-reflect pfctl -f /etc/pf.conf -eq
# route nomad range
route add "$APP_IP" 10.9.9.2 >/dev/null
# create pseudo interface for nomad to pickup
ifconfig lo create \
inet "$APP_IP/$APP_BITMASK" \
name "$NOMAD_IF" fib 1 >/dev/null
# add NAT for outbound cluster (no "sort before uniq" on purpose)
pfctl -qa reflect -F all
echo "
nat on $COMPUTE_IF to $APP_IP/$APP_BITMASK -> $APP_IP
nat on $CLUSTER_IF from 10.192/10 to ! $APP_IP/$APP_BITMASK -> $CLUSTER_IF:0
nat on $UNTRUSTED_IF from 10.192/10 to ! $APP_IP/$APP_BITMASK \
-> $UNTRUSTED_IF:0
pass out quick log to $APP_IP rtable 0
pass from 10.192/10 to 10.192/10
pass from 10.9.9.2/32 to 10.192/10
pass from $APP_IP/32 to 10.192/10
" | uniq | pfctl -qa reflect -f -Example pfrules.conf: NAT/RDR rules in hosts /etc/pf.conf: (no additional anchors, the ones created by pot by default don't apply) Args to nomad: Additional interfaces involved in our setup: Now, the way this works is:
Now, these are a lot bits and pieces, which might cause many more questions - I hope the general concept is clear. If it is, we could see how we can streamline this into an example that works out of the box. Besides all the syntactic complication, the underlying concept is actually quite simple. Traffic flow consul: Traffic flow internal: Traffic flow external: It's been a while since I wrote it and it has been working stable in production for about two years now. Main advantages are:
All of it is part of a bigger setup, which also features inbound and outbound load balancers and inter-service authentication using rotating client certificates - all of it is left out here for simplicity. If you could share your basic setup, we could think about how to turn it into something like this without all the "big environment" elements, which make our setup potentially too complex for your needs. |
|
@pizzamig Maybe we could enable the discussions feature on the pot project, discussing this in an issue feels wrong. |
|
@einsiedlerkrebs So yeah, if you could share the essence of your setup, I could see if our approach could be scaled down to meet your needs and then maybe even prepackaged, so it's useful to many. |
|
Cool. I will write down something tomorrow.
…On 30 April 2024 17:03:26 CEST, grembo ***@***.***> wrote:
@einsiedlerkrebs So yeah, if you could share the essence of your setup, I could see if our approach could be scaled down to meet your needs and then maybe even prepackaged, so it's useful to many.
|
|
Hey @grembo, thanks for the insights to your setup. It looks very interesting, sophisticated and comfortable. Happy to have tickle it out finally. Maybe it should get a more prominent location than this issue here.
The setup is very basic and straight forward.
My setup is, that a service is reachable under the task name. The task name is set as an alias in the nomad-pot task configurations. The alias configuration is red by potnet and can as an option been included into the When Nomad/Pot starts or stops a services, it checks for all pots, if they are set to have dynamic hosts enabled and adds/removes the When a pot-service came up, the No redirects, no extra (ncat) processes. This approach is definitely something else, as what you described about your setup, but for a small single host setup it still brings a lot of pots/nomad/consuls comfort and to me it also looks as a clean and transparent way. I would of course be happy, if you would see my approach useful enough (at least for me and others), to get it merged into the necessary components. |
|
@einsiedlerkrebs It would be good if you could be more specific about your setup (like, give one or two example job configurations, how they interact etc.), as I would like to replicate your setup locally and see what we can do to make it work smoothly. The pure /etc/hosts workaround is nice, but IMHO in its current form it is too limited, as it will break a couple of nomad's features. I think your setup is a good real-life sized example we could derive from though, so maybe taking all the input from this thread + a good example setup, we could forge a solution that is more general purpose (or at least write some good howtos). Just to confirm: |
|
Okay, @grembo . I will try; Nomad is running on the host consulI am running consul manually: set sysver=13.2.2 build=117 template=consul_template ipaddr=10.192.0.254 name=consul
pot prepare -U file:///opt/pot-images/ -p $template -t ${sysver}.${build} -c "/usr/local/bin/cook-${name}" -N public-bridge -a ${build} -i ${ipaddr} -n ${name} -v
pot mount-in -r -d /opt/pot-mounts/${name} -m /config -p ${name}_${build}
pot mount-in -d /opt/pot-mounts/${name}data/ -m /data -p ${name}_${build}
pot set-attribute -A dynamic-etc-hosts -V YES -p ${name}_${build}
pot set-attribute -A no-etc-hosts -V NO -p ${name}_${build}
pot export-ports -p ${name}_${build} -e tcp:8500:8500 -e udp:8600:8600
pot info -vp ${name}_${build}
pot run ${name}_${build}
# ...(the cook script runs consul) some service (http)This runs nginx job "dev" {
datacenters = ["dc1"]
type = "service"
reschedule {
# this disables rescheduling
attempts = 0
unlimited = true
}
group "dev" {
count = 1
restart {
attempts = 3
delay = "15s"
interval = "1m"
mode = "delay"
}
network {
# https://developer.hashicorp.com/consul/docs/install/ports
port "sensor_proxy" {
static = 80
}
}
task "sensorProxy" {
driver = "pot"
service {
address = "${NOMAD_TASK_NAME}.local"
tags = ["sensors", "transport"]
name = "sensorProxy"
port = "sensor_proxy"
check {
type = "http"
name = "http for sensors"
port = "sensor_proxy"
path = "/"
interval = "30s"
timeout = "1s"
}
}
config {
image = "file:///opt/pot-images/"
pot = "access"
tag = "13.2.2.117"
command = "/usr/local/bin/cook-sensors"
network_mode = "public-bridge"
port_map = {"sensor_proxy" = "80"}
mount_read_only = [ "/opt/pot-mounts/sensors:/config" ]
attributes = ["localhost-tunnel:NO", "no-etc-hosts:NO", "dynamic-etc-hosts:YES"]
aliases = ["${NOMAD_TASK_NAME}.local"]
}
}
}
}Next are application containers. That are depending on the http service and can reach it by name (alias). This can already be tested within the consul pot. pot run consul_117
# (inside the consul)
ping sensorProxy.localconsul can check against a name as well: |
|
@einsiedlerkrebs Thank you for sharing. Questions:
For comparison, this is what our setup looks like:
The minimum setup you're looking for is:
|
|
Hi Grembo, thanks for your reply.
Yes.
Iterface stove is where nomad binds to
Yes, exactly. Didn't change the name.
Yes.
This is for agnostic reasons. In this case it helps, that pot controls its /etc/hosts file. It is not controlled by nomad because of Nomads dependency on Consul.
Yes.
The main issue I am struggling with is, that it should be reachable from another lets say application pot next to the nginx pot. Or a database Pot next to it should be reachable from within an application pot, as the application pot should be reachable from the Nginx pot. Consul -> DB <- App <- Nginx <- WWW. The Nginx should be reachable from the outside, but I don't mind to maintain this firewall rule.
Yes. |
|
Hi @grembo , after the time passed, I am curious if there are any news. Wish you a pleasant day and thanks. einsiedlerkrebs |
|
HI, just as an update, I spend a few hours last week to prepare a small example, but it's not self-contained/reproducible yet. |
|
Hi. I am happy to hear! Do you have questions or can I help out with something? |
I wonder if I have the correct understanding at all;
for
public-bridgemode, I expect the created nomad-pot to be reachable via the<POT.PUPLIC.BRIDGE.IP>:<PORT>. Looking at the code this would be realized via a deamonized ncat, which should be copied to the pot's filesystem during start.To me it looks that does not work. I can not see the daemonized process nor can I find the binary in the jails root. (the required setting
localhost-tunnelis correctly set to yes.Do I assume correctly, that the services should be reachable via the public bridge? And if not, how can for instance consul check on them, when the IP's a varying
thanks for help!
pot 0.15.6
nomad-pot-driver 0.9.1
freebsd 13.2-RELEASE-p6
The text was updated successfully, but these errors were encountered: