Using the Fibocom FM350-GL 5G module on Banana Pi BPI-R4

BPI-R4 is a high-performance router development kit from Banana Pi based on the MediaTek MT7988 (Filogic 880). It has unrivaled high performance IO scalability. It provides a wide range of high-speed interfaces from SFP+ to PCI-E(m.2).

The Fibocom FM350-GL is a 5G NR (SA/NSA) high speed (4670 Mbps DL / 1250 Mbps UL) 5G module that can be purchased at a relatively low price in the market today (2024). It is based on the Mediatek T700 chipset. And a full reference manual as well as an AT command manual can be found on the web.

In the first half of 2024, the Linux driver for the FM350-GL was merged into the mainline. Just last week, the driver for the BPI-R4-specific BPI-R4-NIC-BE14 Wi-Fi 7 module was also merged into OpenWrt’s latest Snapshot, which constitutes a necessary prerequisite for us to implement this project.

Installation of the module is very simple & straightforward, just insert it into the PCI-E2 slot labeled SIM1 and tighten the screws, remembering to install the proper antenna. Tip: Since the FM350 generates a lot of heat during operation, it is recommended to install a larger heatsink to ensure that no speed degradation occurs during operation due to overheating of the module.

Hardware readiness. Since neither the BSP-based firmware nor the OpenWrt Snapshot pre-compiled firmware will work properly with our current configuration, the next step is to compile the firmware.

First we install a compilation environment according to the official OpenWrt documentation (https://openwrt.org/docs/guide-developer/toolchain/install-buildsystem). I chose Ubuntu 24.04.3 Server.

Then pull the required source code according to this documentation (https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem). And select the kernel and luci modules we need when make menuconfig.

I checked the following:

mt7986-wo-firmware
mt7988-2p5g-phy-firmware
kmod-mtk-t7xx
kmod-phy-aquantia
kmod-sfp
kmod-wwan
kmod-usb-net-rndis
kmod-usb-serial
kmod-usb-serial-option
kmod-usb3
kmod-mt7915e
kmod-mt7986-firmware
kmod-mt7996-233-firmware
kmod-mt7996e
luci-proto-mbim
luci-proto-modemmanager
luci-proto-ncm
comgt
pciutils
usbutils

Then start the compilation. Then you can start the compilation in . /openwrt/bin/targets/mediatek/filogic/openwrt-mediatek-filogic-bananapi_bpi-r4-squashfs-sysupgrade.itb to find the compiled image.

Flash it in.

The first thing you can see is that the 3 Wi-Fi are working fine.

Since the FM350 is now working in PCI-E mode, create a new ModemManager interface inside the Interface tab. Then you can directly access the Internet.

But it doesn’t last long, you’ll soon run into the problem shown in the image and it will soon lead to Kernel Panic and then an automatic reboot.

For this issue, I tried patch backporting the fix for the latest 6.11 kernel to Openwrt’s current 6.6 kernel. But the result is very little, just can alleviate, but not completely solve the problem of Tx Queue full.

https://github.com/torvalds/linux/commit/d785ed945de6955361aafc2d540d9bb7c6a69a65.patch
https://github.com/torvalds/linux/commit/7d5a7dd5a35876f0ecc286f3602a88887a788217.patch

For this, I first chose a workaround (for now) to get the FM350 to work in USB3 mode by modifying the dtsi to get around the PCI-E driver bug, which is why I had to go the extra mile to select various USB as well as NCM kmods before.

Now let’s go back to our compilation system. Execute the following command.

make -j$(nproc) defconfig download 

Then modify the dtsi, path following

./build_dir/target-aarch64_cortex-a53_musl/linux-mediatek_filogic/linux-6.6.52/arch/arm64/boot/dts/mediatek/mt7988a-bananapi-bpi-r4.dts
/* M.2 key-B SIM1 */
&pcie2 {
	pinctrl-names = "default";
	pinctrl-0 = <&pcie2_pins>;
	status = "okay";
};

Modify line 290 to “disabled”.

/* M.2 key-B SIM1 */
&pcie2 {
	pinctrl-names = "default";
	pinctrl-0 = <&pcie2_pins>;
	status = "disabled";
};

Note that at this point in the compilation do not execute make clean or download, otherwise it will cause our changes to be overwritten.

make -j$(nproc) world 

Then flash the newly compiled firmware obtained into the BPI-R4, and then power off and reboot. This is to completely reboot the chipset inside the FM350. Don’t ask me why, just do it, this is very important!

After poking around for a while, I’ve discovered that the best practice (for now) for using the FM350 in USB3 mode is to use a package called atc. Installation is very simple with the following command

wget https://github.com/mrhaav/openwrt/raw/master/atc/fib-fm350_gl/atc-fib-fm350_gl_2024-08-04-0.2_all.ipk
opkg install atc-fib-fm350_gl_2024-08-04-0.2_all.ipk
wget https://github.com/mrhaav/openwrt/raw/master/atc/luci-proto-atc_20230813-0.2_all.ipk
opkg install luci-proto-atc_20230813-0.2_all.ipk

Similarly, create a new ATC proto port in the interface tab. And don’t forget to fill the APN.

In my case the modem device is /dev/ttyUSB4, if it doesn’t work you can use minicom to send an AT command to confirm which ttyUSB is the real AT interface, it could be any of 0-7.

So far, you get a Wi-Fi7 router with 5G high-speed WWAN.

Thanks to my friend Ananas🍍 who provided me with various ideas.

继续阅读Using the Fibocom FM350-GL 5G module on Banana Pi BPI-R4

Build a customized iPXE firmware and deploy a chain loading environment

As the number of devices in my network has increased, so has the workload of performing maintenance on these systems. From time to time I need to use some rescue systems, such as WinPE and Linux LiveCD. Mounting boot images for them often requires the use of a corresponding management tool or out-of-band management, such as IPMI or BMC manager. The physical machine even needs to burn a USB boot disk.

Preboot eXecution Environment, the abbreviation is PXE, provides a mechanism for booting a computer using the NIC. This mechanism allows the computer to boot without relying on a local data storage device (such as a hard disk) or a locally installed operating system.

So I noticed that there is an open-source PXE firmware called iPXE. Building a customized iPXE firmware is simple according to the wiki page of iPXE.

sudo apt update
sudo apt install git gcc binutils make perl liblzma-dev mtools
git clone git://git.ipxe.org/ipxe.git
cd ipxe/src
make

I need the chain loading function, so I need to add some parameters during the building process like this.

make bin/nm-undionly-20211002-1a.kpxe EMBED=nmboot.ipxe
make bin-x86_64-efi/nm-ipxe-20211002-1a.efi EMBED=nmboot.ipxe

Now I get both legacy BIOS and UEFI PXE firmware embedded my own chain loading script nmboot.ipxe. The script is quiet simple, just chaining the entry point from my HTTP server.

#!ipxe
   
dhcp
chain http://172.16.0.11/pxe/ipxe/boot.php

The advantage of this is that it minimizes the size of iPXE firmware for the NIC PXE to be downloaded from the TFTP Server, i.e., the Next Server that has been given by DHCP server. Because of the slow loading speed over TFTP.

My home lab use pfSense as my gateway and DHCP server, the DHCP server config is quite simple.

The next step is just to prepare the file on the http server. Refer here.

It’s done!

阅读更多:Build a customized iPXE firmware and deploy a chain loading environment

Reference:

https://ipxe.org/

https://gist.github.com/robinsmidsrod/2234639

https://fogproject.org/

Provision a CAP-only SKU Aruba wireless AP to VC

I recently had a problem with my newly purchased Aruba AP-305 not being able to be added to the VC. The solution was discovered through research.

Firstly, use the following command to show the provisioning log:

show log provision

If you got something like “ADP info: CAP-only sku. Will set it as standalone mode“. This is due to the lack of CCODE (i.e. Country-Code)  in this AP. The solution is also simple.

  1. Connect the serial cable to the console port of Aruba AP
  2. Reset the AP
  3. Hit any key to break into apboot >
proginv system ccode CCODE-[CC]-[SHA1]

(replace [SHA1] with the SHA-1 hash of “[CC]-[SN]” ([CC] is your AP’s Restricted Regulatory Domain, like RW, US, JP and IL), and [SN] is the serial number)

US – Restricted Regulatory Domain – US
JP – Restricted Regulatory Domain – Japan
IL – Restricted Regulatory Domain – Israel
RW or UNRST – Rest of the World (Unrestricted)

For example CCODE-RW-de6fdb363ff04c13ee261ec04fbb01bdd482d1cd

5.

invent -w

6.

dhcp

7.

setenv serverip [TFTP_Server_IP]

8.

upgrade os 0 [ArubaInstant_Firmware]

9.

upgrade os 1 [ArubaInstant_Firmware]

10.

factory_reset

11.

saveenv

12.

reset

Now it’s done.

Solution for unable resolving an A record with an intranet address when using pfSense

If you are having trouble resolving a domain name with an A record as an intranet address when using pfSense, add the following field to the Custom options of your DNS Resolver settings.

server:
private-domain: "example.com"
继续阅读Solution for unable resolving an A record with an intranet address when using pfSense

Routing local traffic to a remote outbound with pfSense Firewall and OpenVPN

The purpose of this article is to realize the local machine X can communicate over local router A running pfSense through the WAN gateway of the remote router B also with pfSense.


The lab environment

Local machine X:
[IP_ADDR]=192.168.0.254 [Mask]=255.255.255.0 [GW]=192.168.0.1

Local router A:
[IP_ADDR]=192.168.0.1 [Mask]=255.255.255.0 [GW]=Router_A_WAN_ADDR

Remote router B:
[IP_ADDR]=192.168.11.1 [Mask]=255.255.255.0 [GW]=Router_B_WAN_ADDR

OpenVPN client on router A:
[Mode]=TUN [Interface]=OVPN_A  [IPv4 Tunnel Network]=192.168.30.49/30

OpenVPN server on router B:
[Mode]=TUN [Interface]=OVPN_B [IPv4 Tunnel Network]=192.168.30.50/30

1. Establish an OpenVPN TUN tunnel between router A and B anyway. This is not the focus of this article.

2. Set a allow all rule for OVPN_B on router B

[Area]=OVPN_B
[Action]=Pass
[Interface]=OVPN_B
[Protocol]=Any
[Src]=any
[Dst]=any

3. Set an Outbound NAT on the firewall of router B

[Interface]=WAN
[Protocol]=Any
[Src]=192.168.0.0/24 (or any area you want)
[Dst]=any
[Translation addr]=Interface Address

4. Set a rule for routing traffic to OVPN_A on the LAN firewall table of router A

# Route all traffic of local machine X
[Area]=LAN
[Action]=Pass
[Interface]=LAN
[Protocol]=Any
[Src]=192.168.0.254
[Dst]=any
[Gateway]=OVPN_A
# Route specific destnation IP traffic
[Area]=LAN
[Action]=Pass
[Interface]=LAN
[Protocol]=Any
[Src]=any
[Dst]=192.168.11.1 (example)
[Gateway]=OVPN_A

Result: (Tested on local machine X)

> tracert 192.168.11.1

Tracing route to 192.168.11.1 over a maximum of 30 hops

1 20 ms 20 ms 20 ms 192.168.30.49
2 30 ms 30 ms 30 ms 192.168.11.1

Trace complete.

Use Doxygen to generate documents and diagrams or graphs for your source code

1. apt install required paackages

$ sudo apt update
$ sudo apt install doxygen dia graphviz

2. generate the Doxyfile

$ cd [SOME_PATH]
$ doxygen -g

3. edit the Doxyfile

OUTPUT_DIRECTORY = [YOUR_OUTPUT_DIRECTORY]
INPUT = [YOUR_SOURCE_CODE]
RECURSIVE = YES
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_STATIC = YES
CLASS_DIAGRAMS = YES
DIA_PATH = /usr/bin/dia
HAVE_DOT = YES
CLASS_GRAPH = YES
COLLABORATION_GRAPH = YES

4. run doxygen

$ doxygen [YOUR_Doxyfile]

5. result examlple