anyone deployed on AWS Lightsail

S

s2svoip

Member
Dec 9, 2019
259
8
18
44
Been playing around with Django and cant get it to work on lightsail which is my cloud host of choice, got it working on digital ocean so its something to do with the 'cloud build' aws use. i went and got an output of everything installed on a base deployment in both platforms but its hard to compare.

The issue I see on AWS is after enabling nftables you get locked out, I tested uninstalling iptables first but no change - wondering if anyone else has tried this
 
Adrian Fretwell

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,414
376
83
I'll just give this one a nudge. Any AWS experts out there?

I have been involved in this for @s2svoip, I have logged in to the machine and it is still not clear to me exactly what is going on.

This debian12 deployment used by AWS appears to have iptables installed despite nft now being the debian default firewall. If we unistall iptables, and enable nft the system works fine, the white lists for ssh access are all fine. We can log out, close the connection and then log back in again, with no issues.

The problem is, once you reboot the machine, all access is lost, so my question is: What is AWS doing when we reboot?
Maybe there is a script running form somewhere, that is reverting the firewall configuration, if there is, I have not found it.

I don't use AWS, so I'm not the best person to try and help with this - someone must have an idea what is going on...
 
  • Like
Reactions: s2svoip
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
With LightSail you're trading control for convenience, try an EC2 instance and you won't get any surprises or 'help' from an automated admin.

Actually, maybe just look in the AWS console instead of on the host itself: Use a launch script to configure your Lightsail instance when it starts up - Amazon Lightsail

Spinning up a lightsail instance to see for myself.
1. Create Instance
Linux/Unix > Operating System (OS) only > Debian 12.5.
2. Follow instructions to wget, edit, and run install script from djangopbx-install.sh repository


Will try installing from the instructions and report back here.
 
Last edited:
S

s2svoip

Member
Dec 9, 2019
259
8
18
44
ardyhash said:
With LightSail you're trading control for convenience, try an EC2 instance and you won't get any surprises or 'help' from an automated admin.

Actually, maybe just look in the AWS console instead of on the host itself: Use a launch script to configure your Lightsail instance when it starts up - Amazon Lightsail

Spinning up a lightsail instance to see for myself.
1. Create Instance
Linux/Unix > Operating System (OS) only > Debian 12.5.
2. Follow instructions to wget, edit, and run install script from djangopbx-install.sh repository


Will try installing from the instructions and report back here.
Click to expand...
I just tried with EC2 using the AMI - debian-12-amd64-20231013-1532 - and the exact same thing happened, after enabling nftables and reboot, I get totally locked out (triple check my IP is in the white list which it is) so what ever it's its on both lightsale and ec2
 
Last edited:
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
Thanks for the input about EC2 and sorry I was wrong about lightsail messing with stuff. The problem is that AWS doesn't assign instances public IP addresses, and the djangoPBX install script uses the IP address it detects for configuring nginx. This IP is not routable from outside your VPC.

@Adrian Fretwell I don't know if there's any concern about using 0.0.0.0 for nginx to listen on, but if there isn't making the change to use 0.0.0.0 instead of the detected IP in the installer ought to fix this issue for AWS users.

@s2svoip you can simply
sudo vi /etc/nginx/sites-enabled/djangopbx

and change all occurances of 172.x.x.x to 0.0.0.0 then
sudo systemctl restart nginx

and that ought to do it.
 
S

s2svoip

Member
Dec 9, 2019
259
8
18
44
ardyhash said:
Thanks for the input about EC2 and sorry I was wrong about lightsail messing with stuff. The problem is that AWS doesn't assign instances public IP addresses, and the djangoPBX install script uses the IP address it detects for configuring nginx. This IP is not routable from outside your VPC.

@Adrian Fretwell I don't know if there's any concern about using 0.0.0.0 for nginx to listen on, but if there isn't making the change to use 0.0.0.0 instead of the detected IP in the installer ought to fix this issue for AWS users.

@s2svoip you can simply
sudo vi /etc/nginx/sites-enabled/djangopbx

and change all occurances of 172.x.x.x to 0.0.0.0 then
sudo systemctl restart nginx

and that ought to do it.
Click to expand...
gave this a try and no dice, same outcome

it should be noted its not just connectivity with the box via http/https that is lost, you cant ssh into it either, you totally loose access - so it might be more than just nginx - more likley a nftables issue ? I also uninstalled iptables piror to reboot. the box does respond to ping after a reboot.
 
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
s2svoip said:
gave this a try and no dice, same outcome

it should be noted its not just connectivity with the box via http/https that is lost, you cant ssh into it either, you totally loose access - so it might be more than just nginx - more likley a nftables issue ? I also uninstalled iptables piror to reboot. the box does respond to ping after a reboot.
Click to expand...

I'm not able to reproduce following these steps:

1. Install fresh debian 12.5 os-only lightsail instance
2. give it a static ip and configure dns to point to it (not sure about lightsail but with EC2 unless you explicitly assign a static IP every time you reboot you could be assigned a different IP address)
3. download, edit, and run the install script
4. update nginx to listen on 0.0.0.0 instead of the 172.x.x.x ip address
5. reboot
6. open a browser to the admin interface

no matter how many reboots i haven't lost access via ssh or browser.
 
S

s2svoip

Member
Dec 9, 2019
259
8
18
44
ardyhash said:
I'm not able to reproduce following these steps:

1. Install fresh debian 12.5 os-only lightsail instance
2. give it a static ip and configure dns to point to it (not sure about lightsail but with EC2 unless you explicitly assign a static IP every time you reboot you could be assigned a different IP address)
3. download, edit, and run the install script
4. update nginx to listen on 0.0.0.0 instead of the 172.x.x.x ip address
5. reboot
6. open a browser to the admin interface

no matter how many reboots i haven't lost access via ssh or browser.
Click to expand...
did you systemctl enable nftables ?
 
S

s2svoip

Member
Dec 9, 2019
259
8
18
44
ardyhash said:
enabling nftables effectively killed the instance, I'll start over and stop there
Click to expand...
yeah thats what I have been finding, I have a snapshot of the instance right up until I enable nftables, so ive been reverting to that - but have not figured out whats causing the issue because your either locked out, or as you say it kills it so cant get any visibility even via browser ssh
 
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
edit: I spoke too soon, didn't work after reboot

Try during the installation process (or whenever I guess) commenting out the 172.16.0.0/12 subnet from the block whose definition is:
Drop IPv4s not in internet (RFC 1918, RFC 5735 etc.)

Your line numbers may vary, in my /etc/nftables.conf its line 279:

Code: 
admin@ip-172-26-12-131:~$ cat -n /etc/nftables.conf | grep '172\.' -B12 -A21
   267          chain ingress {
   268                  type filter hook ingress device ens5 priority -500; policy accept;
   269                  # Drop all fragments.
   270                  ip frag-off & 0x1fff != 0 counter drop
   271
   272                  # Drop IPv4s not in internet (RFC 1918, RFC 5735 etc.)
   273                  ip saddr {
   274                  0.0.0.0/8,
   275                  10.0.0.0/8,
   276                  100.64.0.0/10,
   277                  127.0.0.0/8,
   278                  169.254.0.0/16,
   279                  #172.16.0.0/12,
   280                  192.0.0.0/24,
   281                  192.0.2.0/24,
   282                  198.18.0.0/15,
   283                  198.51.100.0/24,
   284                  203.0.113.0/24,
   285                  224.0.0.0/3
   286                  } counter drop
   287
   288                  # Drop bad addresses.
   289                  ip saddr @ipv4_block_list counter name ipv4-block-list-dropped drop
   290                  ip6 saddr @ipv6_block_list counter name ipv6-block-list-dropped drop
   291
   292                  # Drop XMAS packets.
   293                  tcp flags & (fin|syn|rst|psh|ack|urg) == fin|syn|rst|psh|ack|urg counter drop
   294
   295                  # Drop NULL packets.
   296                  tcp flags & (fin|syn|rst|psh|ack|urg) == 0x0 counter drop
   297
   298                  # Drop uncommon MSS values.
   299                  tcp flags syn tcp option maxseg size 1-535 counter drop
   300          }
 
Last edited:
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
I keep speaking too soon, the below config lets me ssh back in but the webui is still blocked, but at least it's been narrowed down to specific configuration, not nftables in general. Oh yeah, to validate that just rename /etc/nftables.conf to something else enable and reboot.

It can certainly use an audit/review and cleanup as I butchered the original without really undoing the attempts that didn't work... but the following nftables.conf works even after several reboots with the service being enabled (also did a sudo apt-get purge iptables way before but i'm fairly certain it was just adding my ip to the whitelist (or that in combination with another edit) that did the trick, shame on me for not following instructions!):

Code: 
#!/usr/sbin/nft -f

flush ruleset


define ipv4_sip_gateways = { 192.168.13.5, 192.168.88.0/24 }
define ipv6_sip_gateways = { fc00:6b2:0211:fb0f::/64 }
define ipv4_sip_customers = {}
define ipv6_sip_customers = {}
define ipv4_white_list = { MYPUBLICIPADDRESS }
define ipv6_white_list = { fc00:04c8:0214:fc81::/64 }
define sip_service_ports = { 5060, 5061 }
define private_tcp_services = { 22 }
define public_tcp_services = { 80, 443 }
define private_udp_services = {}
define public_udp_services = {}


table inet filter {
        counter sip-ipv4-inbound-gateway {
                comment "IPv4 SIP received from Gateway"
        }

        counter sip-ipv4-udp-authorised {
                comment "IPv4 UDP SIP packets authorised"
        }

        counter sip-ipv4-tcp-authorised {
                comment "IPv4 TCP SIP packets authorised"
        }

        counter sip-ipv4-udp-unauthorised {
                comment "IPv4 UDP SIP packets unauthorised"
        }

        counter sip-ipv4-tcp-unauthorised {
                comment "IPv4 TCP SIP packets unauthorised"
        }

        counter sip-ipv4-udp-dropped {
                comment "IPv4 UDP SIP packets dropped"
        }

        counter sip-ipv4-tcp-dropped {
                comment "IPv4 TCP SIP packets dropped"
        }

        counter sip-ipv6-inbound-gateway {
                comment "IPv6 SIP received from Gateway"
        }

        counter sip-ipv6-udp-authorised {
                comment "IPv6 UDP SIP packets authorised"
        }

        counter sip-ipv6-tcp-authorised {
                comment "IPv6 TCP SIP packets authorised"
        }

        counter sip-ipv6-udp-unauthorised {
                comment "IPv6 UDP SIP packets unauthorised"
        }

        counter sip-ipv6-tcp-unauthorised {
                comment "IPv6 TCP SIP packets unauthorised"
        }

        counter sip-ipv6-udp-dropped {
                comment "IPv6 UDP SIP packets dropped"
        }

        counter sip-ipv6-tcp-dropped {
                comment "IPv6 TCP SIP packets dropped"
        }

        counter web-ipv4-tcp-dropped {
                comment "IPv4 TCP Web packets dropped"
        }

        counter web-ipv6-tcp-dropped {
                comment "IPv6 TCP Web packets dropped"
        }

        set ipv4_white_list {
                type ipv4_addr
                flags interval
                comment "Accept packets from these IPv4 hosts"
                elements = $ipv4_white_list
        }

        set ipv6_white_list {
                type ipv6_addr
                flags interval
                comment "Accept packets from these IPv6 hosts"
                elements = $ipv6_white_list
        }

        set ipv4_sip_gateway_list {
                type ipv4_addr
                flags interval
                comment "IPv4 SIP Gateways"
                elements = $ipv4_sip_gateways
        }

        set ipv6_sip_gateway_list {
                type ipv6_addr
                flags interval
                comment "IPv6 SIP Gateways"
                elements = $ipv6_sip_gateways
        }

        set ipv4_sip_customer_list {
                type ipv4_addr
                comment "SIP Customer IPv4 Addresses"
                elements = $ipv4_sip_customers
        }

        set ipv6_sip_customer_list {
                type ipv6_addr
                comment "SIP Customer IPv6 Addresses"
                elements = $ipv6_sip_customers
        }

        set ipv4_web_block_list {
                type ipv4_addr
                comment "IPs banned from web service"
        }

        set ipv6_web_block_list {
                type ipv6_addr
                comment "IPs banned from web service"
        }

        set ipv4_sip_meter {
                type ipv4_addr
                size 65535
                flags timeout, dynamic
        }

        set ipv6_sip_meter {
                type ipv6_addr
                size 65535
                flags timeout, dynamic
        }

        set sip_service_ports {
                type inet_service
                comment "Destination ports of SIP services we offer"
                elements = $sip_service_ports
        }

        set private_tcp_services {
                type inet_service
                comment "Destination ports of whitelist TCP only services we offer"
                elements = $private_tcp_services
        }

        set public_tcp_services {
                type inet_service
                comment "Destination ports of public TCP services we offer"
                elements = $public_tcp_services
        }

        set private_udp_services {
                type inet_service
                comment "Destination ports of whitelist only UDP services we offer"
                elements = $private_udp_services
        }

        set public_udp_services {
                type inet_service
                comment "Destination ports of public UDP services we offer"
                elements = $public_udp_services
        }

        chain inbound_ipv4 {
                udp dport 16384-32768 accept
                icmp type echo-request limit rate 5/second accept
                icmp type { destination-unreachable, echo-reply, source-quench, time-exceeded } accept

                ip saddr @ipv4_sip_gateway_list counter name sip-ipv4-inbound-gateway accept

                ip saddr @ipv4_sip_customer_list udp dport @sip_service_ports counter name sip-ipv4-udp-authorised accept
                ip saddr @ipv4_sip_customer_list tcp dport @sip_service_ports counter name sip-ipv4-tcp-authorised accept
                udp dport @sip_service_ports update @ipv4_sip_meter { ip saddr timeout 2h limit rate 20/hour } counter name sip-ipv4-udp-unauthorised accept
                udp dport @sip_service_ports counter name sip-ipv4-udp-dropped drop
                tcp dport @sip_service_ports update @ipv4_sip_meter { ip saddr timeout 2h limit rate 20/hour } counter name sip-ipv4-tcp-unauthorised accept
                tcp dport @sip_service_ports counter name sip-ipv4-tcp-dropped drop

                tcp dport { 80, 443 } counter name web-ipv4-tcp-dropped drop
                # Allow traffic from established and related packets
                ct state { established, related } accept
                tcp dport @public_tcp_services accept
                udp dport @public_udp_services accept
                ip saddr @ipv4_white_list tcp dport @private_tcp_services accept
                ip saddr @ipv4_white_list udp dport @private_udp_services accept
        }

        chain inbound_ipv6 {
                udp dport 16384-32768 accept
                # accept neighbour discovery otherwise connectivity breaks
                icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
                icmpv6 type { destination-unreachable, echo-reply, packet-too-big, parameter-problem, time-exceeded } accept
                icmpv6 type echo-request limit rate 5/second accept

                ip6 saddr @ipv6_sip_gateway_list counter name sip-ipv6-inbound-gateway accept

                ip6 saddr @ipv6_sip_customer_list udp dport @sip_service_ports counter name sip-ipv6-udp-authorised accept
                ip6 saddr @ipv6_sip_customer_list tcp dport @sip_service_ports counter name sip-ipv6-tcp-authorised accept
                udp dport @sip_service_ports update @ipv6_sip_meter { ip6 saddr timeout 2h limit rate 20/hour } counter name sip-ipv6-udp-unauthorised accept
                udp dport @sip_service_ports counter name sip-ipv6-udp-dropped drop
                tcp dport @sip_service_ports update @ipv6_sip_meter { ip6 saddr timeout 2h limit rate 20/hour } counter name sip-ipv6-tcp-unauthorised accept
                tcp dport @sip_service_ports counter name sip-ipv6-tcp-dropped drop

                ip6 saddr @ipv6_web_block_list tcp dport { 80, 443 } counter name web-ipv6-tcp-dropped

}
        chain inbound {
                type filter hook input priority filter; policy drop;

                # Allow all from loopback
                iifname "lo" accept
                # Jump to chain according to layer 3 protocol using a verdict map
                meta protocol vmap { ip : jump inbound_ipv4, ip6 : jump inbound_ipv6 }
        }

        chain outbound {
                type filter hook output priority filter; policy accept;

                # Set differential services bits for RTP Priority
                udp sport 16384-32768 ip dscp set 0x2e

                # Set differential services bits for SIP Priority
                udp sport @sip_service_ports ip dscp set 0x1a
                tcp sport @sip_service_ports ip dscp set 0x1a
        }
}


table netdev filter {
        counter ipv4-block-list-dropped {
                comment "IPv4 Block list dropped"
        }

        counter ipv6-block-list-dropped {
                comment "IPv6 Block list dropped"
        }

        set ipv4_block_list {
                type ipv4_addr
                flags interval
                comment "Drop all packets from these hosts"
        }

        set ipv6_block_list {
                type ipv6_addr
                flags interval
                comment "Drop all packets from these hosts"
        }

        chain ingress {
                type filter hook ingress device ens5 priority -500; policy accept;
                # Drop all fragments.
                ip frag-off & 0x1fff != 0 counter drop

                # Drop IPv4s not in internet (RFC 1918, RFC 5735 etc.)
                #ip saddr {
                #0.0.0.0/8,
                #10.0.0.0/8,
                #100.64.0.0/10,
                #127.0.0.0/8,
                #169.254.0.0/16,
                #172.16.0.0/12,
                #192.0.0.0/24,
                #192.0.2.0/24,
                #198.18.0.0/15,
                #198.51.100.0/24,
                #203.0.113.0/24,
                #224.0.0.0/3
                #} counter drop

                # Drop bad addresses.
                ip saddr @ipv4_block_list counter name ipv4-block-list-dropped drop
                ip6 saddr @ipv6_block_list counter name ipv6-block-list-dropped drop

                # Drop XMAS packets.
                tcp flags & (fin|syn|rst|psh|ack|urg) == fin|syn|rst|psh|ack|urg counter drop

                # Drop NULL packets.
                tcp flags & (fin|syn|rst|psh|ack|urg) == 0x0 counter drop

                # Drop uncommon MSS values.
                tcp flags syn tcp option maxseg size 1-535 counter drop
        }
}


table inet mangle {
        chain prerouting {
                type filter hook prerouting priority -150;

                # Drop invalid
                ct state invalid counter drop

                # TCP SYN (Drop new without SYN)
                tcp flags & (fin|syn|rst|ack) != syn ct state new counter drop
        }
}
 
Last edited:
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
Thanks Adrian, haven't made it that far yet but good to know in case it comes up.

Restarted from scratch and took notes along the way, was able to reboot and get back in, here's my notes from last night/this morning:

# on a fresh install of debian as root:
mkdir -p /usr/src/djangopbx-install
cd /usr/src/djangopbx-install
wget https://raw.githubusercontent.com/djangopbx/djangopbx-install.sh/master/install.sh
chmod +x install.sh
# edit install.sh then run it
# update /etc/nginx/sites-available/djangopbx to replace every occurance of 172.x.x.x with 0.0.0.0 and restart nginx
# allow traffic on 443 through the aws console
# edit /etc/nftables.conf to add own public IP to whitelist and comment out the block that restricts non-internet addresses (this is kind of dangerous, residential IP addresses are dynamic so one may get locked out if their IP changes)
# enable nftables and reboot

I actually haven't tested just adding my IP to the whitelist without commenting out the nftables.conf block blocking non-internet addresses, but I imagine yall have beat that horse to death already!
 
S

s2svoip

Member
Dec 9, 2019
259
8
18
44
ardyhash said:
Thanks Adrian, haven't made it that far yet but good to know in case it comes up.

Restarted from scratch and took notes along the way, was able to reboot and get back in, here's my notes from last night/this morning:

# on a fresh install of debian as root:
mkdir -p /usr/src/djangopbx-install
cd /usr/src/djangopbx-install
wget https://raw.githubusercontent.com/djangopbx/djangopbx-install.sh/master/install.sh
chmod +x install.sh
# edit install.sh then run it
# update /etc/nginx/sites-available/djangopbx to replace every occurance of 172.x.x.x with 0.0.0.0 and restart nginx
# allow traffic on 443 through the aws console
# edit /etc/nftables.conf to add own public IP to whitelist and comment out the block that restricts non-internet addresses (this is kind of dangerous, residential IP addresses are dynamic so one may get locked out if their IP changes)
# enable nftables and reboot

I actually haven't tested just adding my IP to the whitelist without commenting out the nftables.conf block blocking non-internet addresses, but I imagine yall have beat that horse to death already!
Click to expand...
thanks for your testing, so i followed this and indeed was able to retain access after a reboot, one issue I am seeing is Server Error (500) when trying to access the firewall block list or whitelist - when I saw this before it was due to nftables not being enabled, but it is this time..

So these changes basically open up the server for local addresses only, all external internet traffic is still filtered and blocked ?
 
Last edited:
Adrian Fretwell

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,414
376
83
Do those lists exist now?
What output do you see if you issues the following commands as root?

Code: 
/usr/sbin/nft list set netdev filter ipv4_block_list
/usr/sbin/nft list set inet filter ipv4_white_list
 
A

ardyhash

Member
Jan 7, 2021
80
9
8
44
s2svoip said:
thanks for your testing, so i followed this and indeed was able to retain access after a reboot, one issue I am seeing is Server Error (500) when trying to access the firewall block list or whitelist - when I saw this before it was due to nftables not being enabled, but it is this time..

So these changes basically open up the server for local addresses only, all external internet traffic is still filtered and blocked ?
Click to expand...
I had the same issue when I used a bare IP address in the whitelist, changing it to CIDR format resolved that (add a /32 to the end of the ip address)


The nftables.conf is pretty restrictive (the internet is a bad neighborhood), you'd need to modify it to be more permissive according to your needs.
 
You must log in or register to reply here.
Top Bottom