I'm not saying it's rocket science. Many people do NOT have the problems I had.
Probably I had something VERY wrong in my network setup (maybe even more than one thing).
The biggest clue was that disabling the OPNSense Firewall for a quick testing (pfctl -d) made iperf3 work.
You should only do this in case you have an upstream ISP Router that has a firewall enabled ! Otherwise you would expose yourself to the Internet (Dangerous) !!!
Schematics of the issue:
ESP32 Devices (172.22.x.y) <---> Rock 5B (172.22.1.1) WiFi AP Hostapd + DNSmasq <---> Rock 5B (192.168.4.x) <---> Switch <---> Main Router/Gateway (192.168.1.1) <--> Ubuntu Client (192.168.x.y)
Schematics of the solution with Static Routes:
ESP32 Devices (172.22.x.y) <---> Rock 5B (172.22.1.1) WiFi AP Hostapd + DNSmasq <---> Rock 5B (192.168.4.x) <---> Ubuntu Client (192.168.x.y) with Static Route of 192.168.4.x for the 172.22.0.0/16 Network
The most likely issue is that the OPNSense Router was getting traffic IN from one Network Interface and then having to send it OUT to the SAME Network Interface. Even with some settings (e.g. Firewall -> Settings -> Advanced -> Static route filtering: Bypass firewall rules for traffic on the same interface [CHECKED]) it just wouldn't work properly. The strange part is that it "kinda" worked with ping, traceroute and even more surprisingly tcptraceroute (since this uses TCP packets ! , whereas ping ICMP is usually allowed no matter what). The main hint was running iperf3 between the AP and the Ubuntu Client (Rock 5B/Raspberry Pi AP running server iperf3 -s while Ubuntu Client running client iperf3 -c 192.168.4.x -P 1).
iperf3 would just show MISERABLE performance below 0.1 Mbit/s. As soon as you disable the firewall on OPNSense (just for testing, anyway I already have firewall enabled on the ISP Router upstream !) with pfctl -d then it would start working. Enable with pfctl -e then the iperf3 performance would go again off a cliff.
I solved with RFC3442 and DHCP Option 121 to automatically push these static routes to DHCP clients: see
https://github.com/luckylinux/rfc3442-dhcp-option-121 for a script that can build the DHCP Option 121 in case you are interested (I based it on an existing Github gist in Python + some fixes based on some Web Based Javascript Calculators I found online).
But otherwise you can just define a Static Route manually for your Linux Hosts:
https://www.cyberciti.biz/tips/configuring-static-routes-in-debian-or-red-hat-linux-systems.html.
Proper solution will be to have VLAN and VLAN routing, but I am currently facing many issues on that front, so it's not a day (or week) fix. Getting Mikrotik Switches work with Openwrt Switches work with Zyxel Switches working with OPNSense in a VM on Proxmox VE (with Mellanox ConnectX-2 or Mellanox ConnectX-3 card and the Linux Driver VLAN limitation of 128 VLANs) is more complicated than you might think (but will need to be done at some point) ....