When it comes to remote access I would tend to consider two types of remote access, the heavy weight type and the rubbadub type. Let me talk to you about my "King Tubby" styled remote access.
This post is about configuring Proxmox based Ubuntu LXC containers, that will serve as load balanced Guacamole backend hosts, our good old FortiWeb appliance, which as it's name implies would handle web based traffic like no one else, FortiAuthenticator for anything 2FA as well as Single Sign On's and obviously of course FortiOS, federating it all, controlling it all.
The idea is rather simple
- We will craft a Master Guacamole LXC container template within PVE, hosting all we need.
- That same container template will be deployed on each PVE nodes available and ready to host Guacamole clientless remote connections.
- Our FortiWeb appliance will request authentications and MFA from users and grant us Load Balanced access amongst the Guacamole backend LXC Containers.
- FortiAuthenticator will provide strong authentications with 2nd Factor Auths through FortiTokens, interfacing a unique LDAP repository.
- FortiOS will provide the key tools set in order to reach our FortiWeb platform, control authorizations at the policy level and federate our public network spread apparts IPsec connected "locations" and thus, helping us at reaching assets of interests from anywhere within our Security Fabric.
- LDAP based user repository will serve as our unique credentials repository.
This is going to be a long read, hold tight...
1st thing 1st, we will need a PVE based Ubuntu LXC container template. A few simple CLI's from your Proxmox PVE host will get you the goods, and you don't even have to repeat the same things three, four or five times. This WILL work on your 1st attempt provided you have internet connectivity from the given PVE host:
root@pve:/: pveam available --section system
...
system ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
system ubuntu-20.04-standard_20.04-1_amd64.tar.gz
system ubuntu-21.04-standard_21.04-1_amd64.tar.gz
system ubuntu-21.10-standard_21.10-1_amd64.tar.zst
root@pve:/: pveam download local ubuntu-21.10-standard_21.10-1_amd64.tar.zst
downloading http://download.proxmox.com/images/system/ubuntu-21.10-standard_21.10-1_amd64.tar.zst to /var/lib/vz/template/cache/ubuntu-21.10-standard_21.10-1_amd64.tar.zst
....
root@kpve:/: pveam list local
NAME SIZE
local:vztmpl/ubuntu-21.10-standard_21.10-1_amd64.tar.zst 120.06MB
Once this is gathered, simply create a container either from the CLI or from the PVE GUI, here is what I've done:
pct create 601 local:vztmpl/ubuntu-21.10-standard_21.10-1_amd64.tar.zst \
--storage local-lvm --ostype ubuntu \
--arch amd64 --password insecure --unprivileged 1 \
--cores 2 --memory 2048 --swap 1024 \
--hostname gcm-template --searchdomain local.lan \
--nameserver X.X.X.X --net0 name=eth0,bridge=vmbrxx,tag=XXX,ip=dhcp,type=veth \
--start false
In my Fortinet deployments, or say within my networks layouts, I'm always using the Virtual Domains feature available within FortiOS in order to segregate, segment what I refer to firewalling at the function. Hence, anything that concerns VPN/RA is terminated within my VPN VDOM from a FortiOS perspective. Hence here, my LXC containers are obviously mapped within a VLAN that end's up right within the VPN defined VDOM.
Once our container is up, we can start the software setup we need in order to host our Guacamole stack. Here we go, on github here!
# OS generic updates
apt update
apt upgrade
# GCM dependencies and library needed
apt install -y libjpeg62-dev
apt install -y libjpeg-turbo8-dev
apt install -y libcairo2-dev libpng-dev libtool-bin libossp-uuid-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev freerdp2-dev libpango1.0-dev libssh2-1-dev libtelnet-dev libvncserver-dev libwebsockets-dev libpulse-dev libssl-dev libvorbis-dev libwebp-dev
# Tomcat & MariaDB
apt install -y make tomcat9 mariadb-server
# GCM setup
export guacver=1.4.0
echo $guacver
wget --trust-server-names "https://apache.org/dyn/closer.cgi?action=download&filename=guacamole/$guacver/source/guacamole-server-$guacver.tar.gz" -O /usr/src/guacamole-server-$guacver.tar.gz
wget --trust-server-names "https://apache.org/dyn/closer.cgi?action=download&filename=guacamole/$guacver/binary/guacamole-$guacver.war" -O /usr/src/guacamole-$guacver.war
tar xvzf /usr/src/guacamole-server-$guacver.tar.gz -C /usr/src/
cd /usr/src/guacamole-server-$guacver
./configure --with-systemd-dir=/etc/systemd/system
make
make install
ldconfig
mkdir /etc/guacamole
mkdir /etc/guacamole/extensions
mkdir /etc/guacamole/lib
echo GUACAMOLE_HOME=\"/etc/guacamole\" >> /etc/environment
systemctl enable guacd.service
systemctl start guacd.service
systemctl status guacd.service
systemctl restart tomcat9.service
adduser guacd --disabled-password --disabled-login --gecos ""
sed -i -e 24c"\#User=daemon" /etc/systemd/system/guacd.service
sed -i -e 25i"User=guacd" /etc/systemd/system/guacd.service
mkdir /var/lib/guacd
chown -R guacd: /var/lib/guacd
systemctl daemon-reload
systemctl restart guacd
# GCM extensions/auths setup
wget --trust-server-names "https://apache.org/dyn/closer.cgi?action=download&filename=guacamole/$guacver/binary/guacamole-auth-jdbc-$guacver.tar.gz" -O /usr/src/guacamole-auth-jdbc-$guacver.tar.gz
wget "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.21.tar.gz" -O /usr/src/mysql-connector-java-8.0.21.tar.gz
tar xvzf /usr/src/guacamole-auth-jdbc-$guacver.tar.gz -C /usr/src/
tar xvzf /usr/src/mysql-connector-java-8.0.21.tar.gz -C /usr/src/
cp /usr/src/guacamole-auth-jdbc-$guacver/mysql/guacamole-auth-jdbc-mysql-$guacver.jar /etc/guacamole/extensions/
cp /usr/src/mysql-connector-java-8.0.21/mysql-connector-java-8.0.21.jar /etc/guacamole/lib/
# GCM database setup
export dbpw=$(openssl rand -hex 8)
echo $dbpw
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS guacamole DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;"
mysql -u root -p -e "CREATE USER 'guacamole'@'localhost' IDENTIFIED BY '$dbpw';"
mysql -u root -p -e "CREATE DATABASE IF NOT EXISTS guacamole DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;"
mysql -u root -p -e "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON guacamole.* TO 'guacamole'@'localhost' IDENTIFIED BY '$dbpw' WITH GRANT OPTION;"
mysql -u root -p -e "FLUSH PRIVILEGES;"
mysql -uguacamole -p$dbpw guacamole < /usr/src/guacamole-auth-jdbc-$guacver/mysql/schema/001-create-schema.sql
mysql -uguacamole -p$dbpw guacamole < /usr/src/guacamole-auth-jdbc-$guacver/mysql/schema/002-create-admin-user.sql
# GCM properties setup
cat << EOF > /etc/guacamole/guacamole.properties
#
# Hostname and Guacamole server port
#
guacd-hostname: 127.0.0.1
guacd-port: 4822
#
# MySQL properties
#
mysql-hostname: 127.0.0.1
mysql-port: 3306
mysql-database: guacamole
mysql-username: guacamole
mysql-password: $dbpw
EOF
# GCM MariaDB timezone adapt
cp /etc/mysql/mariadb.conf.d/50-server.cnf /etc/mysql/mariadb.conf.d/50-server.cnf.orginal
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql
sed -i '30 i\# Timezone' /etc/mysql/mariadb.conf.d/50-server.cnf
sed -i '31 i\default_time_zone=Europe/Zurich' /etc/mysql/mariadb.conf.d/50-server.cnf
sed -i '32 i\ ' /etc/mysql/mariadb.conf.d/50-server.cnf
systemctl restart mariadb.service
systemctl restart tomcat9.service
# GCM LDAPs auths backend setup
wget --trust-server-names "https://apache.org/dyn/closer.cgi?action=download&filename=guacamole/$guacver/binary/guacamole-auth-ldap-$guacver.tar.gz" -O /usr/src/guacamole-auth-ldap-$guacver.tar.gz
tar xvzf /usr/src/guacamole-auth-ldap-$guacver.tar.gz -C /usr/src/
cp /usr/src/guacamole-auth-ldap-$guacver/guacamole-auth-ldap-$guacver.jar /etc/guacamole/extensions/
BLA="$(printf ' ')"
cat << EOF >> /etc/guacamole/guacamole.properties
#
#LDAP Settings for AD
#
${BLA}
ldap-hostname: ad.ghost.local
${BLA}
ldap-port: 636
${BLA}
ldap-encryption-method: ssl
${BLA}
ldap-search-bind-dn: CN=sGCM,CN=Users,DC=ghost,DC=local
${BLA}
ldap-search-bind-password: XXXXXXXXXXXXXXXXXX
${BLA}
ldap-user-base-dn: DC=ghost,DC=local
${BLA}
ldap-username-attribute: samAccountName
${BLA}
ldap-group-base-dn: DC=ghost,DC=local
${BLA}
ldap-group-name-attribute: cn
${BLA}
ldap-follow-referrals: false
${BLA}
ldap-operation-timeout: 30
${BLA}
ldap-user-search-filter: (&(objectClass=*)(memberOf=CN=gcm_access,CN=Users,DC=ghost,DC=local))
${BLA}
EOF
# GCM root certificate import, trusting AD presented LDAPs cert
keytool -importcert -alias "ldaps" -keystore "/usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts" -storepass "changeit" -file /usr/src/Root_CA.cer -noprompt
systemctl restart tomcat9.service
tail -f -n 50 /var/log/tomcat9/catalina.out
One note onto the Guacamole authentications, Guacamole will try to authenticate locally and remotely, hence it's good practice to create the "guacadmin" user within AD with a different password from the same local default user (change that one as well!). Once logged on with the AD bound "guacadmin" user, you will see all your AD based groups & users if your LDAPs setup is correct.
Obviously, all the user accounts used before (LDAP bind "sGCM" & AD based "guacadmin" (which is a regular user, member of Domain Users & gcm_access groups)) need to be present on your AD.
Now, our Guacamole backend should be up, our Tomcat server, our Guacamole database as well and finally, our LDAPs authentication backend should be addressable by our Guacamole container.
You should now be able to connect to your Guacamole instance:
http://yourip:8080/guacamole/#/
It is now a good time to setup all your wanted connections, VNC, SSH, RDP, you name it. This is well documented and I therefore wont detail this here.
Of course, depending the "standpoint" of your Guacamole hosts (for me behind my VPN VDOM at my Hub location) you'll need all the sets of policies at the FortiOS layer in order for that/these hosts to be able to reach any of the configured remote connections assets of interests.
Once you're happy with your setup connections, generic Guacamole configuration etc. you're ready to convert that container to a PVE Template. You can do that directly from the Proxmox PVE GUI, by converting your container to a Template. You'll find a very good article here.
Once your Template is set, you can deploy/clone it as stand alone LXC containers. I've done so here, once per available PVE Nodes, all bound to a VLAN id within the vVPN VDOM and set with their own DHCP reservations at the FortiOS layer.
Now that all our load balancing Guacamole hosts are up and running, we can move on to the FortiWeb configuration part.
This is typically how I would generally craft my internal traffic and functional design, roughly:
FortiWeb has it's dedicated VDOM, terminating any inbound request from public networks and cloaking/protecting any potentially externally provisioned services. FortiOS would of course police such traffic according to personal selectors, Security Profiles, Virtual Servers, Geo IP's etc, etc...
Traffic wise, we will cross the FGT through a Server Load Balancing vIP (more details in this ghost post here), reach the FWB which will reverse proxy and load balance the Guacamole backend hosts, them seated within the vVPN VDOM. Of course, FortiWeb will also handle all the frontend SSL/TLS connections and present a valid certificate for the requested FQDN.
A quick view from the Guacamole LXC Server Pool at FortiWeb, the Persistence is here set with a cookie which will be added to the initial request and therefore, all subsequent requests with this cookie will hit the same pool member:
On FortiWeb we will address our Guacamole bastions through what is referred as a Site Publishing Policy, coupled with FortiAuthenticator provided RADIUS Multi Factor Authentications.
Above is a view from my current Site Publishing rule bound to my Web Protection Profile. Authentications on the Site Publishing Policy will request a set of known credentials as well as a second authentication factor, namely FortiToken this through a FAC/RADIUS auth cycle. We can also spot an Authentication Delegation setting, delegating the used user's credentials towards the load balanced Guacamole containers after a successful FortiWeb/FortiAuthenticator multifactor authentication cycle.
I've had to edit a few settings on FortiWeb. I.E. the WebSocket Security Settings, which I've set with 8192 for Max Frame Size & Max Message Size. I've also disengaged the csrf-enhancements on the site publishing rule using the "set csrf-enhancement disable" command.
On the Authentication settings seen above, I'm mapping an Authentication Server Pool configuration which target my FortiAuthenticator, using a service dedicated NAS IP in order for me to configure a dedicated RADIUS client policy for that specific FortiWeb RADIUS client/NAS IP at FAC:
The RADIUS client configuration "filtered group" seen above is a local to FAC Users Group, filtering/accepting only the Guacamole Access Group AD members. Membership of that FAC Users Gourp is handled by the following Remote LDAP User Synchronization Rule:
Indeed, the FAC local Guacamole Users Group is populated automatically upon new members being added/removed within AD using the above seen Remote User Sync Rule. Hence a single AD user group membership add/change and the given user will pop in or pop out of our FAC Users Group.
Another interesting view above is the LDAP User Mapping Attributes, each of which might be already filled within AD and therefore gathered at the FAC level. A nice and easy way to gather Mobile phone numbers (2FA dispatched through SMS) and email addresses directly.
In its standing above, 2FA tokens would be dispatched per email (which is insecure) and any users not having an email set won't be synchronized (I'd rather not provide "guacadmin" external access)... You're of course free to choose amongst the given options how you'd want your FortiTokens to be assigned/dispatched etc.
Now, the Guacamole hosts will do their own authentications, based on what FortiWeb provided as a delegated authentication and will therefore authenticate the incoming user against the LDAP directory. Obviously so, to retrieve and present to the user it's administratively allocated Remote Access systems and so on.
What about SAML?
Yes, SAML!!! I've tested the SAML Authentication extension provided here (-sso file). It works absolutely fine in conjunction with FAC as the IdP. My only concern and why I didn't went for SAML yet is corroborated here. Indeed, Guacamole will not initiate a 2nd user lookup to the LDAP directory upon successful SAML authentications. Which means only local to the Guacamole database connections/users setups. Which is a no go for me.
On FortiOS and concerning these Remote Sessions, if you'd want to apply Authentication, Authorization and Accounting, any Remote Sessions fired from any of the available Guacamole containers, will obviously originate from one of these hosts. The rendering and control will be presented to the final users through WebSocket, nevertheless, the original remote connectivity initiations will be comming from these containers stacks.
What this means is that we can use the Guacamole LDAPs authentication in order to bind the currently logged on user on our Guacamole containers, this in order to enable Identity Based Policies at the FortiOS layer such as these:
You would have many possible ways to achieve this and probably the most scalable would imply the FortiAuthenticator DC Agent / Collector Agent installed on your AD infrastructure. We could also use the FAC API's upon a successful user login from our LXC containers (API access enabled on FAC):
curl --noproxy "*" -k -v -u "apiadm:xxxxXXXxxxXXX" --data '{"event":"1","username":"obruno","user_ip":"10.10.10.10"}' -H "Content-Type: application/json" https://fac.local.lan/api/v1/ssoauth/
We might perhaps as well leverage a direct Syslog parsing at the FAC level, parsing the "catalina.out" outputs from our Tomcat Sever logs on our LXC containers, which would provide the logged on user, here is what would be needed on the LXC Containers:
root@cGCM01:/etc: cat rsyslog.conf
...
#################
#### MODULES ####
#################
...
module(load="imfile" PollingInterval="10") # provide support for local files to syslog
...
root@cGCM01:/etc/rsyslog.d: cat tomcat.conf
input(type="imfile"
File="/var/log/tomcat9/catalina.out"
Tag="catalina"
StateFile="/var/spool/catalina"
Severity="info"
Facility="local1")
local1.* @fac.local.lan:514
Displayed below is a syslog entry gathered at FAC. Obviously a Syslog source & its associated Matching rule would be needed for FAC to process such informations.
FAC tcpdump on src:syslog
> execute tcpdump -nnSX src 10.10.10.3 and port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on port1, link-type EN10MB (Ethernet), capture size 262144 bytes
07:41:18.410734 IP 10.10.10.3.50492 > 10.10.10.4.514: SYSLOG local1.info, length: 227
...
0x0080: 5d20 494e 464f 2020 6f2e 612e 672e 722e ].INFO..o.a.g.r.
0x0090: 6175 7468 2e41 7574 6865 6e74 6963 6174 auth.Authenticat
0x00a0: 696f 6e53 6572 7669 6365 202d 2055 7365 ionService.-.Use
0x00b0: 7220 226f 6272 756e 6f6f 2220 7375 6363 r."obrunoo".succ
0x00c0: 6573 7366 756c 6c79 2061 7574 6865 6e74 essfully.authent
0x00d0: 6963 6174 6564 2066 726f icated.from.
...
Although, as I've had this in place already, I went for the Windows Syslog version, gathering and extracting Windows Logons Events out of Windows Syslog logs sent to FortiAuthenticator, as detailed in this ghost post here.
Ultimately, this setup would provide continuous Authorizations and Identity based audit trails across any of your under control FortiOS within your Security Fabric..
And finally, a few screenshots of the results as seen from any client having the possibility to engage communications towards our FortiWeb Guacamole Virtual Server:
Our login screen:
Followed by the 2FA request:
And below, a Guacamole hosted RDP remote session extended over public networks through IPsec, thus reaching a remote DC/remote host within an OT Domain in that example:
I know it's been a long post, involving many different technologies. Although, I hope you've enjoyed it and that it gave you enough envy to test it for yourself.
Another nice thing about such setups is that with a few clicks, if you're AD is well organized, you should be able to provision temporary remote access to any individuals external to your organization. Coupling that with Guacamole available sessions video recordings, you get it all in no time!
On my end, I couldn't do without such services nowadays and I'm always amazed at how easy this is on an end user perspective. A browser, your credentials, your multi-factor token and flying you are.
Cheers and don't bow!!
Obruno
Technical Credits: A huge thank you to Daniel Wydler, from which the Guacamole setup instructions posted here originate along with some mods: https://blog.wydler.eu/2020/10/25/apache-guacamole-remote-desktop-gateway-rdg/
Image Credits: Larry Southberg - https://www.artstation.com/larrysouthberg