AT&T U-verse → [AT&T Fiber] Questions regarding BGW210 and removing it replaced w PFSense

So, after weeks of reading and rereading, I have succeeded in extracting two certs and the mfg.dat from my BGW210 gateway by using a USB TTL. I am still having an issue with EAP authentication failures in WPA supplicant, however.

I am thinking I missed something simple that can be corrected by sharing the steps I took:

First, my setup:

ATT 1 gigabit fiber
Arris BGW210-700 (ATT issued)
My firewall appliance is the 6 port Protectli 6 port device
TPLink 24port fully managed switch (T2600G-28TS)

Initial:
Come out of LAN port 1 on the gateway going into the WAN port of a Protectli appliance (interface em0) and out of the appliance on LAN port (interface em1) to the managed switch.

Post Extraction:
Connect the ONT cable directly to the WAN port (em0) of the Protectli appliance.

Process:
(skipping extraction steps as I believe they are not relevant)

After extraction and decoding mfg.dat I am left with 3 files:

attroot2031.cer - CA Cert in DER binary format
attsubca2021.cer - Another cert in .der format
EAP-TLS_8021x[REMOVED CHARS].tar

The tar file contains 4 files of note: Another CA cert (Arris), Client cert, PrivateKey cert, and a wpa_supplicant.conf.

As per instructions, I extract the certs from the tar archive and move them to the Protectli appliance running PFSense. The wpa_supplicant file gets stored in /etc on the PFSense box.

Next, I adjust the mac address of the WAN interface in PFSense (em0) to be the same as the mac address referenced in the wpa_supplicant.conf file which mirrors the mac of the AT&T gateway... Lastly, before launching wpa_supplicant command line I edit wpa_supplicant.conf adjusting the referenced interface to em0 since the ONT connection from the wall is now plugged directly into this port on PFSense. The final edit to wpa_supplicant.conf is to update the references to the 3 certs mentioned earlier (CA cert (Arris), Client cert, PrivateKey cert) and I specify the full path when editing the certs.

My wpa_supplicant.conf:

ca_cert="/root/CA_xxxxxxxxxxxxxxxxxxxxx.pem"
client_cert="/root/Client_xxxxxxxxxxxxxxxxxxx.pem"
eap=TLS
eapol_flags=0
identity="XX:XX:X:XX:XX:XX" (this mac is the gateway mac from ATT device, entered as cloned mac in PFSense)
key_mgmt=IEEE8021X
phase1="allow_canned_success=1"
private_key="/root/PrivateKey_xxxxxxxxxxxxxxxxxxxxxxx.pem"

As you can see my certs are stored in the base home dir of the PFSense account.

WPA_supplicant command line:

/usr/sbin/wpa_supplicant -Dwired -iem0 -c/etc/wpa_supplicant.conf

At this point, I switch back to PFSense GUI and initiate a lease renew while looking keeping an eye on the PFSense shell which is starting to give diagnostic info. After about 20-30 seconds I see 'EAP authentication failure' printed to stderr.

What you see here is literally everything I've done. I haven't tagged any interfaces with VLAN ids (remember reading this somewhere) or taken any other steps.

Any help would be forever appreciated.

I don’t run pfsense but I’ve seen this on my ER-4 initially, but then it tries again and succeeds in authenticating. Could you maybe get it make another authentication attempt after the initial failure?

Not familiar with pfsence, but a few questions:
Is wpa_supplicant running as root? If not, does it have enough rights to connect to em0 and access /root/ ?
What is the output of wpa_supplicant when you run it? Does it complete authentication? If not, what is the error message?

Also, what is em0? Is it a physical NIC/passthrough device, or is it an virtual emulated NIC? Virtual NIC may not work with 802.1X BPDUs.

Yesem0: Associated with xx:xx:xx:xx:xx:xx (Mac address of the physical port em0)
em0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
em0: CTRL-EVENT-EAP-FAILURE EAP authentication failedem0 - em5 is the device nodes PFSense assigned to the physical NICs. I did just notice something suspect:

The output:

em0: Associated with xx:xx:xx:xx:xx:xx (Mac address of the physical port em0)

This shows the mac address of the physical NIC port and not the mac address that I am trying to clone to the port (the gateway mac) though I don't know if this is relevant.Will try.

Is there a better way to get wpa_supplicant to show more detailed data?

Yes, try -dd in command line.
It is clearly failing to authenticate.

I put this in my config file and then call up wpa_cli at boot:
ctrl_interface=/var/run/wpa_supplicant

I believe the specifics vary by platform so not sure what pfsense would use.

wpa_supplicant v2.8
Successfully initialized wpa_supplicant
Initializing interface 'em0' conf '/etc/wpa_supplicant.conf' driver 'wired' ctrl_interface 'N/A' bridge 'N/A'
Configuration file '/etc/wpa_supplicant.conf' -> '/etc/wpa_supplicant.conf'
Reading configuration file '/etc/wpa_supplicant.conf'
eapol_version=1
ap_scan=0
fast_reauth=1
Line: 9 - start of a new network block
ca_cert - hexdump_ascii(len=34):
2f 72 6f 6f 74 2f 43 41 5f 30 30 31 45 34 36 2d /root/CA_001E46-
(lines trimmed)
client_cert - hexdump_ascii(len=38):
2f 72 6f 6f 74 2f 43 6c 69 65 6e 74 5f 30 30 31 /root/Client_001
(lines trimmed)
eap methods - hexdump(len=16): 00 00 00 00 0d 00 00 00 00 00 00 00 00 00 00 00
eapol_flags=0 (0x0)
identity - hexdump_ascii(len=17):
xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx:xx:xx:xx:xx:xx (ATT gateway mac)
1
key_mgmt: 0x8
phase1 - hexdump_ascii(len=22):
61 6c 6c 6f 77 5f 63 61 6e 6e 65 64 5f 73 75 63 allow_canned_suc
63 65 73 73 3d 31 cess=1
private_key - hexdump_ascii(len=48):
2f 72 6f 6f 74 2f 50 72 69 76 61 74 65 4b 65 79 /root/PrivateKey
(lines trimmed)
Priority group 0
id=0 ssid=''
driver_wired_init_common: Added multicast membership with SIOCADDMULTI
driver_wired_init_common: waiting for link to become active
Add interface em0 to a new radio N/A
em0: Failed to attach pkt_type filter
em0: Own MAC address: xx:xx:xx:xx:xx:xx (mac address of em0 on Protectli, the wan/ont port)
em0: RSN: flushing PMKID list in the driver
em0: Setting scan request: 0.100000 sec
em0: WPS: UUID based on MAC address: 3fff5ce4-7f27-5314-8fbf-8e0069ae34cd
ENGINE: Loading dynamic engine
ENGINE: Loading dynamic engine
EAPOL: SUPP_PAE entering state DISCONNECTED
EAPOL: Supplicant port status: Unauthorized
EAPOL: KEY_RX entering state NO_KEY_RECEIVE
EAPOL: SUPP_BE entering state INITIALIZE
EAP: EAP entering state DISABLED
em0: Added interface em0
em0: State: DISCONNECTED -> DISCONNECTED
EAPOL: External notification - EAP success=0
EAPOL: External notification - EAP fail=0
EAPOL: External notification - portControl=Auto
em0: Already associated with a configured network - generating associated event
em0: Event ASSOC (0) received
em0: Association info event
FT: Stored MDIE and FTIE from (Re)Association Response - hexdump(len=0):
em0: State: DISCONNECTED -> ASSOCIATED
em0: Associated to a new BSS: BSSID=01:80:c2:00:00:03 0
EAPOL: SUPP_PAE entering state CONNECTING
EAPOL: txStart
TX EAPOL: dst=01:80:c2:00:00:03
TX EAPOL - hexdump(len=4): 01 01 00 00
EAPOL: startWhen --> 0
EAPOL: SUPP_PAE entering state CONNECTING
EAPOL: txStart
TX EAPOL: dst=01:80:c2:00:00:03
TX EAPOL - hexdump(len=4): 01 01 00 00
EAPOL: idleWhile --> 0
EAP: EAP entering state FAILURE
em0: CTRL-EVENT-EAP-FAILURE EAP authentication failed
EAPOL: SUPP_PAE entering state AUTHENTICATING
EAPOL: SUPP_BE entering state FAIL
EAPOL: SUPP_PAE entering state HELD
EAPOL: Supplicant port status: Unauthorized
EAPOL: SUPP_BE entering state IDLE
EAPOL authentication completed - result=FAILURE
EAPOL: startWhen --> 0
EAPOL: heldWhile --> 0
EAPOL: disable timer tick
EAPOL: SUPP_PAE entering state CONNECTING
EAPOL: enable timer tick

Referring to the OS boot, correct?

You enter wpa_cli into your command prompt and it gives you the current status of wpa_supplicant if you have it specified correctly in your config file. It should tell you it’s entering interactive mode or something like that after you run wpa_cli

wpa_supplicant is sending EAPol-Start abd not getting response (usually, server cert). Check your connectivity and Ethernet interface configuration. 802.1x BPDUs are not being passed through.

Also try getting tcpdump capture to see if there is any traffic coming your way after EAPol-Start

Look at pfatt on github, FreeBSD doesn’t support vlan 0 and have to use Netgraph.

He could not even authenticate, that happens well before VLAN0 issues.

wpa_supllicant output:

EAPOL: External notification - portEnabled=0
EAPOL: External notification - portValid=0
EAPOL: External notification - portEnabled=1
EAPOL: SUPP_PAE entering state CONNECTING
EAPOL: SUPP_BE entering state IDLE
EAP: EAP entering state INITIALIZE
EAP: EAP entering state IDLE
em0: Cancelling scan request
em0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
EAPOL: startWhen --> 0
EAPOL: SUPP_PAE entering state CONNECTING
EAPOL: txStart
TX EAPOL: dst=01:80:c2:00:00:03
TX EAPOL - hexdump(len=4): 01 01 00 00

tcpdump output: (with -v switch)

01:00:22.636002 EAP packet (0) v1, len 15, Request (1), id 1, len 15
Type Identity (1), Identity: User Name:

Is the ident/username supposed to be empty?

If with the ONT connection plugged into em0 (and em0's mac cloned to match the gateway mac) I'm not seeing any answer, it souds like ATT is refusing to talk to me. As soon as I plug the ONT back to the modem I am online in 10 seconds or so. My TTL is still attached and I can see all of the modem startup scripts in /etc/init.d. Do you think that could be useful?

The refusal to talk could be a cert and/or mac and I have no way to isolate this to my knowledge...

I am familiar with the pfatt project but ultimate I want the ATT gateway gone just on principle.

Is the traffic from eappol standard tcp? tcpdump would show this show any answer back regardless of proto, correct?

I don't have an issue using opnsense, a Linux distro, or whatever I need to do to achieve the goal of removing the ATT gw. If I end up needing to use pfatt I will it just isn't the idea I started with.

EAPol is BPDU traffic, not TCP, nor even IP, but tcpdump should be able to capture the packets. ONT should respond to EAPol start. The fact that it is not responding points to some kind of network configuration issue. Is your BSD running on bare metal, or as a VM?

Thanks for clarifying this I was s[ecfically wanting to know the answer to that.It's a physical device: Intel i5, 6 NICs etc etc

It really seems like the ONT is just refusing to chat with me even though I am cloning macs. Wish I had tcpdmp for the modem...

I am tempted to try another firewall software just to see if the result differs (to eliminate pfsense from the equation) but I like PFSense a ton.

^^Try using a dumb switch in between the ONT and the pfsense box.

Try capturing traffic with tcpdump. Open in Wireshark and post screenshot

Just to be clear, Wireshark will directly import text output from tcpdump?Can do.

I recall (vaguely) someone using netgraph to allow authentication directly between bare metal pfsense and the ONT due to pfsense not passing vlan 0 traffic (or something to that effect). I can't remember which thread it was... It was fairly recent though, last few weeks.

You can test this theory by using a dumb switch as that effectively filters out the vlan 0 tags.

802.1x authentication is handled without 802.1p or 802.1q encapsulation, so VLANs are irrelevant.

tcpdump can save into .pcap file format, that can be viewed with Wireshark

Sorry so delayed in replying... Life got in the way of plat time.

Have mot tried the 'dumb switch' approach yet, will try that this weekend.

I didn't take a screenshot, but in tcpdump pcap there is no traffic coming back to the ONT interface even with the mac address spoofed... Almost like the certs extracted by the tool (and being used by wpa_supplicant) are incorrect.

Is there any way to verify this? Hell, for that matter I am curious if anyone has successfully gotten direct communication with the ONT working.

My thoughts at this point are:

1) Set up the pfatt method with the ONT wire and modem wire on separate ports of the protectli and then try tcpdump the physical interface that the modem would be connected to and see if I can verify certs through tcpdump pcap in wireshark though I fear this may be beyond the scope of my knowledge how to do

2) downgrade the modem firmware so that I can use the sysreq interrupt to look at the scripts that run as the device is launching with the hope that there is evidence in these that might allow verification of which certs need to be used

Right now I'm at a standstill on how to move forward.

I have an idea that it might be the CA cert (Issued to Arris, not ATT) that wpa_supplicant is using.

Any tips to what path t head down at this point with be great!

If you see no exchange of certs at all, the issue is not with certs, it is with your configuration.
Take a look at wireshark screenshot posted at devicelocksmith.com. Client initiates communication with EAPol start. This message does not contain any identification data, it just initiates communication. ATT must respond to that message. If you are not seeing that, you are not even getting to the certificate exchange part.

Take a look at this image:
»images.app.goo.gl/mN3KNA ··· NQJZCmY7

The Arris certs authenticate fine. That is not the problem. I have had success with many of them. In fact, none have ever failed.

Holy flying pigs... I got it working!

Thanks to this pm:That branch does indeed work! I assume the virtual interface that gets created (ngeth0) (netgraph driven?) somehow addresses the packet issue I was encountering.

Now I just need to start the shell script as the machine is coming up so the ngeth0 interface will be present for the wan configuration.

I cannot thank all of you that gave input (especially dls ) during the process. Wish I could buy you all a drink... or three.

Any idea why the netgraph adapter is needed?

The issue is FreeBSD and the AT&T process...they use a tagged vlan 0 for .1x authentication which isn’t an issue between the ONT and the RG but FreeBSD doesn’t like tagged vlan 0 packets since they don’t conform to the rfc standards. This is where Netgraph comes in and that interface module doesn’t care... so it allows it through. Using the raw interfaces will never work and you can thank FreeBSD for that.

Last post on this...

The final step I wanted to achieve was to have the script (pfatt-supplicant) run on firewall startup using pfsense/opnsense so I put the script in the following location:

/usr/local/etc/rc.syshook.d/early/

This causes the ngether interface to be present before the firewall attempts to configure networks. Works like a dream.

Glad you got this working! I am dying to try this, would love it if you could share more details on extracting the certificates!