New alert: PVAC_a041_virtual_clamper_triggered

[Nexarian]

Saw this in my monitoring of my system, not sure what it means or where it comes from though.

[jasonacox]

Interesting. It looks like we have had PVAC_a041_excess_PV_clamp_triggered reported before:

https://github.com/jasonacox/pypowerwall?tab=readme-ov-file#pvac---photovoltaic-ac---solar-inverter

But not this one, so I will add it. Can you describe yours system? Also, do you see any odd power metrics happening at the time of the alert?

[Nexarian]

System:

• 2x 7.6 kW Tesla Standalone Inverters (No Powerwalls)
• 20.5 kW (50x 410W Rec Alpha panels)
• 8 strings of various sizes, the smallest being 3 panels (the minimum)

As to odd metrics: Yes. The 3-panel string is dodgy right now. Sometimes the MCI works properly, sometimes it shuts down the string entirely for no apparent reason, and sometimes it does what I call "grey but producing" meaning in Tesla One the string is greyed out but power is still being reported on it. When that happens I get:

alerts = [
    "PVS_a018_MciStringB",
    "PVS_a027_Mci2PvVoltage",
    "PVS_a043_InactiveUnsafePvStrings"
]

Presumably (not a Tesla engineer and not privy to extra info): Power is still being shoved through the string even though the MCI should be open, but because it's not, it means control of the MCI on string B has been lost; ergo, "unsafe."

I have noticed if I go into Tesla One and "disable" and "enable" the inverter again, sometimes the strings will come back in a better state, it is during this cycling that I saw the alert in the issue title. So I suspect it's a startup alert related to initialization of the system, something that doesn't last long. HOWEVER, it might not be normal, as I also saw PVAC_a014_PVS_disabled_relay at the same time.

[Nexarian]

One more note: When I set the timestamp to Tesla (they usually require that you send your installer a timestamp and the error codes your seeing, and it has to be a very sunny day at close to solar noon for them to even look at your case), they said MCIs getting into this state may be related to a "surge" of some sort. They didn't elaborate.

The techs are scheduled to come out sometime within the next month or two. They are slow :/

[jasonacox]

Do you have the Dashboard graphing your alerts? I noticed that for my system, the "PVS_a027_Mci*PvVoltage" alerts happen at sundown right before "PVS_a059_MciOpen". The "PVAC_a014_PVS_disabled_relay " alert happens at sunrise. Here is my system and alerts over time:

[Nexarian]

@jasonacox See discussion here: https://gist.github.com/jasonacox/91479957d0605248d7eadb919585616c#gistcomment-5364463

Without an intelligent way to query BOTH inverters, I'll only ever at best see half the picture, I'm blocked.

I've gone to war trying to figure out the proper settings to make this query mechanism possible, but alas, no dice.

[jasonacox]

Ah, thanks for the reminder. Understood... If it were me, I believe I would set up the Dashboard for at least the problematic string inverter. I have a few RPIs around so I would probably set up a dashboard for each inverter (I suppose two VMs would also work). With some customization, the two pypowerwall containers could be set up to write to the same InfluxDB, or use Grafana to pull from two sources to provide a combined view. Ideally there is some networking wizardry that would allow it to work on the same host, but it is beyond my skill or ability to test.

[Nexarian]

@jasonacox You're clearly a manager :). I will eventually give up/in on that and follow that approach, but for now I want my cake and eat it, too!

[jasonacox]

Ha! That usually requires two cakes. 😜

[Nexarian]

I solved it; though, it only works on Linux; the scripts are below. Regardless, this issue was about logging the new error code, and I still don't have the proxy working to be able to do that just yet. We can follow up on the rest here: #119

You can resolve this issue.

Bash Version

#!/bin/bash

set -ex


# Function to check if a command succeeded
check_command() {
    if [ $? -ne 0 ]; then
        echo "Error: $1"
        exit 1
    fi
}

# Function to clean up existing rules and configurations
cleanup() {
    echo "Cleaning up existing rules and configurations..."
    
    # Remove virtual IP addresses
    ip addr del 192.168.92.100/24 dev eth0 2>/dev/null || true
    ip addr del 192.168.92.101/24 dev eth0 2>/dev/null || true

    # Flush NAT table
    iptables -t nat -F
    check_command "Failed to flush NAT table"

    # Flush mangle table
    iptables -t mangle -F
    check_command "Failed to flush mangle table"

    # Remove all rules in filter table
    iptables -F
    check_command "Failed to flush filter table"

    # Remove non-default chains
    iptables -X
    check_command "Failed to delete non-default chains"

    # Remove all ip rules (except default)
    ip rule show | grep -v "from all lookup" | cut -d: -f1 | xargs -r -n1 ip rule del prio
    check_command "Failed to remove ip rules"

    # Remove routing tables
    ip route flush table 100 || true
    ip route flush table 101 || true
    check_command "Failed to flush routing tables"

    echo "Cleanup completed successfully."
}

# Main setup function
setup() {
    echo "Setting up routing and NAT rules..."

    # Enable IP forwarding
    echo 1 > /proc/sys/net/ipv4/ip_forward
    check_command "Failed to enable IP forwarding"

    # Add virtual IP addresses
    ip addr add 192.168.92.100/24 dev eth0
    check_command "Failed to add virtual IP 192.168.92.100"
    ip addr add 192.168.92.101/24 dev eth0
    check_command "Failed to add virtual IP 192.168.92.101"

    # Set up routing tables
    ip route add 192.168.91.1/32 via 192.168.1.67 dev eth0 onlink table 100
    check_command "Failed to add route to table 100"
    ip route add 192.168.91.1/32 via 192.168.1.250 dev eth0 onlink table 101
    check_command "Failed to add route to table 101"

    # Set up NAT rules
    iptables -t nat -A OUTPUT -d 192.168.92.100 -j DNAT --to-destination 192.168.91.1
    check_command "Failed to add DNAT rule for 192.168.92.100"
    iptables -t nat -A OUTPUT -d 192.168.92.101 -j DNAT --to-destination 192.168.91.1
    check_command "Failed to add DNAT rule for 192.168.92.101"

    iptables -t nat -A PREROUTING -d 192.168.92.100 -j DNAT --to-destination 192.168.91.1
    check_command "Failed to add DNAT rule for 192.168.92.100"
    iptables -t nat -A PREROUTING -d 192.168.92.101 -j DNAT --to-destination 192.168.91.1
    check_command "Failed to add DNAT rule for 192.168.92.101"

    # Set up packet marking
    iptables -t mangle -A OUTPUT -d 192.168.92.100 -j MARK --set-mark 1
    check_command "Failed to add mark for 192.168.92.100"
    iptables -t mangle -A OUTPUT -d 192.168.92.101 -j MARK --set-mark 2
    check_command "Failed to add mark for 192.168.92.101"

    # Set up ip rules
    ip rule add fwmark 1 table 100
    check_command "Failed to add ip rule for mark 1"
    ip rule add fwmark 2 table 101
    check_command "Failed to add ip rule for mark 2"

    # Set up return traffic NAT
    iptables -t nat -A POSTROUTING -s 192.168.92.100 -d 192.168.91.1 -j SNAT --to-source 192.168.1.225
    check_command "Failed to add return SNAT rule for 192.168.92.100"
    iptables -t nat -A POSTROUTING -s 192.168.92.101 -d 192.168.91.1 -j SNAT --to-source 192.168.1.225
    check_command "Failed to add return SNAT rule for 192.168.92.101"

    echo "Setup completed successfully."
}

# Main execution
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root" 
   exit 1
fi

# Cleanup first
cleanup

# Then setup
setup

echo "All operations completed successfully."

Python Version

#!/usr/bin/env python3

import os
import socket
import subprocess
import sys
from typing import Any, Dict, Final, List, Tuple

import iptc
import psutil
from pypowerwall import scan
from pyroute2 import IPRoute
from pyroute2.netlink.exceptions import NetlinkError

# Define constant for destination IP
DESTINATION_IP: Final[str] = "192.168.91.1"

# Generate inverter configurations dynamically
BASE_TABLE_ID: Final[int] = 101
BASE_MARK: Final[int] = 1

def check_root() -> None:
    if os.geteuid() != 0:
        print("This script must be run as root")
        sys.exit(1)

def get_network_interface() -> Tuple[str, int, Any]:
    """Find the default network interface."""
    ATTRIBUTES: Final[str] = 'attrs'
    with IPRoute() as ip_route:
        return next(
            (
                (ip_route.link('get', index=attr[1])[0][ATTRIBUTES][0][1], attr[1])
                for route in ip_route.get_routes(family=socket.AF_INET)  # IPv4 routes
                    if route.get('dst_len', None) == 0  # Default route
                        for attr in route[ATTRIBUTES]
                            if attr[0] == 'RTA_OIF'  # Outgoing interface
            ),
            None
        )

def get_local_ip(interface: str) -> str:
    """
    Get the LAN IP address of a specific network interface.

    Args:
        interface (str): The name of the network interface (e.g., 'eth0').

    Returns:
        str: The IP address of the specified interface, or an error message if not found.
    """
    try:
        addrs = psutil.net_if_addrs().get(interface)
        if not addrs:
            return f"Interface '{interface}' not found."
        
        for addr in addrs:
            if addr.family == socket.AF_INET:
                return addr.address
        
        return f"No IPv4 address found for interface '{interface}'."
    except Exception as e:
        return f"Error retrieving IP for interface '{interface}': {e}"

def cleanup(inverters: List[Dict[str, str | int]]) -> None:
    print("Cleaning up existing rules and configurations...")
    interface = get_network_interface()
    with IPRoute() as ip_route:
        try:
            # Remove virtual IPs
            for inverter in inverters:
                try:
                    ip_route.addr('delete', index=interface[1], address=inverter['ip'], prefixlen=24)
                except NetlinkError:
                    pass

            # Flush iptables rules
            for table_name in ["nat", "mangle", "filter"]:
                table = iptc.Table(table_name)
                for chain in table.chains:
                    chain.flush()
                    for rule in chain.rules:
                        chain.delete_rule(rule)

            # Flush routes for each inverter
            for inverter in inverters:
                ip_route.flush_routes(table=inverter['table_id'])

            # Remove ip rules
            rules: List[Dict] = ip_route.get_rules()
            for rule in rules:
                for inverter in inverters:
                    if rule['table'] != inverter['table_id']:
                        continue
                    ip_route.rule('delete', **rule)

            print("Cleanup completed successfully.")

        except Exception as e:
            print(f"Error during cleanup: {e}")
            
def enable_ip_forwarding():
    try:
        subprocess.run(["sudo", "sysctl", "-w", "net.ipv4.ip_forward=1"], check=True)
        print("IP forwarding enabled successfully.")
    except subprocess.CalledProcessError:
        print("Error enabling IP forwarding. Do you have sudo privileges?")

def setup(inverters: List[Dict[str, str | int]]) -> None:
    print("Setting up routing, NAT, and marking rules...")
    interface = get_network_interface()
    local_ip = get_local_ip(interface=interface[0])
    print(f"Local IP is: {local_ip}")
    with IPRoute() as ip_route:
        try:
            # Enable IP forwarding
            enable_ip_forwarding()

            interface_index: Final[int] = interface[1]
            print(f"Interface index is: {interface_index}\n")

            # Add virtual IPs, routing tables, and marking rules
            for inverter in inverters:
                try:
                    ip = inverter['ip']
                    mark = inverter['mark']
                    table_id = inverter['table_id']

                    print(f"Adding address: {ip}")
                    ip_route.addr('add', index=interface_index, address=ip, prefixlen=24)
                    print("Adding route")
                    ip_route.route('add', dst=f'{DESTINATION_IP}/32', gateway=inverter['gateway'], table=table_id, flags="onlink", oif=interface[1])
                    print("Adding rule mark")
                    ip_route.rule('add', table=table_id, fwmark=mark)

                    print("Mangle table")
                    # Mangle table rules
                    mangle_table = iptc.Table(iptc.Table.MANGLE)
                    output_chain = iptc.Chain(mangle_table, "OUTPUT")
                    rule = iptc.Rule()
                    rule.dst = ip
                    target = iptc.Target(rule, "MARK")
                    target.set_mark = str(mark)
                    rule.target = target
                    output_chain.append_rule(rule)

                    # NAT rules
                    nat_table = iptc.Table(iptc.Table.NAT)
                    output_chain = iptc.Chain(nat_table, "OUTPUT")
                    prerouting_chain = iptc.Chain(nat_table, "PREROUTING")
                    postrouting_chain = iptc.Chain(nat_table, "POSTROUTING")

                    # DNAT rules
                    for chain in [output_chain, prerouting_chain]:
                        rule = iptc.Rule()
                        rule.dst = ip
                        target = iptc.Target(rule, "DNAT")
                        target.to_destination = DESTINATION_IP
                        rule.target = target
                        chain.insert_rule(rule)

                    # SNAT rule
                    rule = iptc.Rule()
                    rule.src = ip
                    target = iptc.Target(rule, "SNAT")
                    target.to_source = local_ip
                    rule.target = target
                    postrouting_chain.insert_rule(rule)
                    
                    print(f"Address {ip} setup successfully.\n")
                except (NetlinkError, iptc.IPTCError) as e:
                    print(f"Error setting up {inverter['ip']}: {e}")

            print("Setup completed successfully.")

        except Exception as e:
            print(f"Error during setup: {e}")

if __name__ == "__main__":
    check_root()
    
    devices = scan.scan(max_threads=256)

    inverters: List[Dict[str, str | int]] = [
        {
            "ip": f"192.168.92.{100 + i}",
            "table_id": BASE_TABLE_ID + i,
            "gateway": ip,
            "mark": BASE_MARK + i
        }
        for i, ip in enumerate([i['ip'] for i in devices])
    ]

    cleanup(inverters)
    setup(inverters)