https://vince.ca/ Recent content on Vince Hillier Hugo -- gohugo.io en Thu, 14 Sep 2023 09:42:54 -0500 https://vince.ca/posts/kubernetes-fluentd-rsyslog/ Thu, 14 Sep 2023 09:42:54 -0500 https://vince.ca/posts/kubernetes-fluentd-rsyslog/ After a few hours of scouring documentation and reviewing examples, I couldn’t find a complete working example of how to get FluentD running as a Daemonset to forward logs to rsyslog. Below is a complete working example that I cobbled together. Rsyslog config The following additions need to go into /etc/rsyslog.conf or /etc/rsyslog.d/your.conf. Enable the udp listener for rsyslog: # provides UDP syslog reception module(load="imudp") input(type="imudp" port="1514") Create a template for Kubernetes logs $template KubernetesLogs,"/var/log/remote/kubernetes/%msg:R,ERE,1,BLANK:\"namespace_name\"=>\"([a-zA-z0-9_-]+)\",--end:secpath-replace%_%msg:R,ERE,1,BLANK:\"container_name\"=>\"([a-zA-z0-9_-]+)\",--end:secpath-replace%. https://vince.ca/posts/samsung-odyssey-g9-on-qubesos/ Sat, 07 Jan 2023 01:42:54 -0500 https://vince.ca/posts/samsung-odyssey-g9-on-qubesos/ UPDATE: Dump nvidia chipsets and go with a Radeon RX580. All problems disappear and the rest of this post is moot. After months of careful consideration I pulled the trigger on a Samsung Odyssey G9, a personal Christmas present! This screen is very impressive at 49" with an extreme curve. I couldn’t wait to get it set up. QubesOS had quite the difficulty remembering the Display settings for this monitor, I had to reconfigure it every time I booted or unlocked my screen. https://vince.ca/posts/ovh-licensing-windows-inside-proxmox-vm/ Wed, 04 May 2022 04:38:22 -0400 https://vince.ca/posts/ovh-licensing-windows-inside-proxmox-vm/ I can count on one hand with several fingers left how many Windows Server workloads I’ve implemented. I build a lot of infrastructure that supports Windows Server workloads, but rarely involved in the actual configuration of Windows Server itself. Requirements Setup a Windows Server at OVH with SQL Server running in a Proxmox VM. This VM will use a routed network for internet access and nat through the Proxmox server when communicating with the OVH licensing server. https://vince.ca/posts/traefikee-letsencrypt-acme-via-tls/ Sat, 09 Apr 2022 17:16:51 -0400 https://vince.ca/posts/traefikee-letsencrypt-acme-via-tls/ We’re using wildcard certificates with dnsChallenge everywhere, however, we onboarded a service that is delegated to nameservers we don’t control. We had to add another certificateResolver to Traefik using httpChallenge, but actually tlsChallenge. Our first attempt was using httpChallenge with the following static config: certificatesResolvers: le-prod-http: acme: email: "example@example.com" keyType: "RSA4096" httpChallenge: entryPoint: web This resulted in the following log errors: traefikee_controller-0.1.ooby8h4drk1g@ovh03 | time="2022-04-09T21:03:21Z" level=error msg="Unable to obtain ACME certificate for domains \"service. https://vince.ca/posts/traefikee-tcp-router-ipwhitelist-middleware/ Sun, 20 Mar 2022 04:16:50 -0400 https://vince.ca/posts/traefikee-tcp-router-ipwhitelist-middleware/ Service delivery consists of more than just HTTP reverse proxies and SSL termination. This post will explore exposing a TCP service within a cluster as well as limiting access to said service. It’s written specifically for TraefikEE users but easily adapted to CE. We’ll be working with the following components: Proxy stacks Static configuration Dynamic configuration Docker stacks Proxy Stacks If you want your service to access the source IP of incoming connections you’ll need to enable host networking. https://vince.ca/posts/dhcp-option-43-for-aironet-aps/ Mon, 15 Nov 2021 13:46:27 -0500 https://vince.ca/posts/dhcp-option-43-for-aironet-aps/ Just a quick hacky script to generate dhcp option 43 for Cisco LAPs to join Wireless Lan Controllers (WLC). This script outputs a line to add to your dhcpd.conf subnet declaration. It takes a list of IPs as arguments and will convert them into a suitable dhcpd.conf entry. Script invocation example: user@work:~$ ./gen-dhcp-43.py 192.168.10.5 192.168.10.20 Please add the following line to the correct subject declaration. option vendor-encapsulated-options f1:08:c0:a8:0a:05:c0:a8:0a:14; Example subnet declaration subnet 192. https://vince.ca/posts/traefikee-ipwhitelist-behind-cloudflare/ Tue, 09 Nov 2021 17:44:51 -0500 https://vince.ca/posts/traefikee-ipwhitelist-behind-cloudflare/ At Revenni we’re huge fans of Traefik and have used their software for over 3 years. This time last year we decided to deploy TraefikEE for a couple of clients - it has been a fantastic experience and dealing with Traefik on the business end, a pleasure. There are a literal ton of posts just like this one, but not a single one summarized a working solution to use ipwhitelisting[sic] with Cloudflare. https://vince.ca/posts/preventing-mailman-subscription-spam/ Mon, 01 Nov 2021 07:33:41 -0500 https://vince.ca/posts/preventing-mailman-subscription-spam/ Mailman is an opensource software for managing mailing lists. As with everything on the internet, it’s subject to abuse. One of the mailman services we maintain experienced some wild success with 18K new subscriptions in 24 hours… or did it. root@mailman:/var/log/mailman# cat subscribe|wc -l 18099 Let’s see who is responsible for these subscriptions. root@mailman:/var/log/mailman# cat subscribe |awk '{print $NF;}'|sort|uniq -c|sort -nr|head -5 3942 128.199.129.xxx 1209 138.199.50.xxx 514 49.64.213.xxx 266 220.165.247.xxx 261 114. https://vince.ca/posts/tiling-offsets-with-clusterssh/ Thu, 21 Oct 2021 17:41:46 -0400 https://vince.ca/posts/tiling-offsets-with-clusterssh/ Before ansible and a lot of config management tools, we had clusterssh. I’ve been using clusterssh forever and a day at this point. After a long hiatus, I dusted it off and fired it up only to be very disappointed in the tiling across two screens. Some terminals were on the split, while others were overlapping. For the first time in more than a decade I opened up ~/.clusterssh/config and discovered the following options: https://vince.ca/posts/grapheneos-authenticator-plus-to-andotp/ Mon, 23 Aug 2021 00:03:52 -0400 https://vince.ca/posts/grapheneos-authenticator-plus-to-andotp/ A lifetime ago I found myself switching from Android to iPhone. Part of migrating to my shiny new iPhone involved moving 10s of 2fa tokens trapped in Google Authenticator to a new application on the iPhone. Unfortunately, in 2015, GA had no export function and I couldn’t find a way to migrate the tokens. I began what seemed like an insurmountable task of re-enrolling all of my accounts in a new application, Authenticator Plus, which I ensured had an export function in the event I replaced my phone. https://vince.ca/posts/grapheneos-android-to-iphone-and-back-again/ Sun, 22 Aug 2021 00:15:52 -0400 https://vince.ca/posts/grapheneos-android-to-iphone-and-back-again/ More than a decade ago I jumped on the Android wagon with an Acer Liquid before following the Nexus line from One-6P. By 2015 I became really frustrated with Google and their privacy incursions. This lined up perfectly as Apple was being very obtuse with the US Government requests to access an iPhone belonging to a shooter in San Bernardino. I figured any company willing to take that stand was a company I was willing to pour my wallet into. https://vince.ca/posts/reset-windows-password-ovh/ Fri, 05 Feb 2021 07:50:24 -0500 https://vince.ca/posts/reset-windows-password-ovh/ We have a handful of Windows Servers with password expiry policies. This is a short cheatsheet for anyone else encountering ERRCONNECT_PASSWORD_EXPIRED. Error xfreerdp +clipboard /u:admin /p:$(pass servers/windows/revenni) /w:1920 /h:1080 /v:xxx.xxx.xxx.xxx:3389 [07:29:36:722] [11626:11627] [INFO][com.freerdp.client.common.cmdline] - loading channelEx cliprdr [07:29:36:723] [11626:11627] [INFO][com.freerdp.client.x11] - Property 273 does not exist [07:29:37:285] [11626:11627] [ERROR][com.freerdp.core] - freerdp_set_last_error ERRCONNECT_PASSWORD_EXPIRED [0x0002000E] [07:29:37:286] [11626:11627] [ERROR][com.freerdp.core.rdp] - rdp_recv_callback: CONNECTION_STATE_NLA - nla_recv_pdu() fail [07:29:37:286] [11626:11627] [ERROR][com.freerdp.core.transport] - transport_check_fds: transport->ReceiveCallback() - -1 Software Required on debian apt-get install xfreerdp2-x11 openjdk-11-jre icedtea-netx Process Login to OVH’s control panel Navigate Dedicated Servers -> <server> -> IPMI Inside the Remote KVM section, click From a Java applet (KVM). https://vince.ca/posts/benchmarking-wordpress-with-locust/ Fri, 15 Jan 2021 09:45:10 -0500 https://vince.ca/posts/benchmarking-wordpress-with-locust/ Wordpress. Love it or hate it, it runs close to 40% of the internet. So when tasked with increasing it’s performance we need to establish a baseline and then measure our incremental changes. Queue Locust! Install python3-venv and python dev packages apt-get install python3-venv python3-dev Create locust virtual environment and activate it python3 -mvenv ~/.python_envs/locust source ~/.python_envs/locust/bin/activate Install locust pip install locust Create locustfile. https://vince.ca/posts/openstack-lockout-recovery/ Mon, 26 Oct 2020 05:57:49 -0400 https://vince.ca/posts/openstack-lockout-recovery/ Story time The perfect storm culminated in a dire rescue operation. Some ansible managed machines in a hyper secure cloud environment based on OpenStack. No problem, I’m a partner at OVH and manage a lot of infrastructure there for several clients. OpenStack internals and APIs are things I’m intimately familiar with. Fire off a playbook to update the machines and everything looks good. Expected updates marked as changes, no anomalies. https://vince.ca/posts/streaming-pandora-with-pianobar-on-qubesos/ Sun, 17 May 2020 07:54:32 -0400 https://vince.ca/posts/streaming-pandora-with-pianobar-on-qubesos/ The other day Kris Nóva was broadcasting her effort to get pianobar working. Pianobar is a Pandora radio CLI application written in C. Typically my day is filled with some lo-fi hip hop streamed from YouTube. But the constant ads and “Video Paused. Click to continue” prompts are quite annoying. The idea is to have some chill background music not something that requires constant petting throughout the day, when I found out about pianobar I had to get it going. https://vince.ca/posts/file-uploads-with-locustio/ Tue, 07 Apr 2020 09:45:10 -0500 https://vince.ca/posts/file-uploads-with-locustio/ I had to perform some functional load testing of some file upload features. The last time I played with load testing Jmeter was the incumbent, with an entire team dedicated to its usage. My first crack was to harness curl and launch a bunch of loops into the background. This worked well to generate load but offered little in terms of process control and reporting. To change the number of clients I had to rerun my wrappers with new variables. https://vince.ca/posts/intro-to-linux-pass/ Fri, 21 Feb 2020 22:45:10 -0500 https://vince.ca/posts/intro-to-linux-pass/ I’ve been doing some migration and isolation work lately to reduce the attack surface in the event of a supply chain compromise. The original intention of this post was to document the process of switching from a single password store for all services to several password stores each with their own virtual machine, disk resources, as well as pgp keys and passphrases. Half way through that process I realized there should be some context for that, queue this post. https://vince.ca/posts/rss-and-comments/ Sat, 15 Feb 2020 22:24:27 -0500 https://vince.ca/posts/rss-and-comments/ I enjoy posting content about things I find interesting. I also enjoy stepping outside of my echo chamber and hearing your thoughts, feedback, and adventures. In that spirit, it’s now easier to follow content I post with the new RSS feed. Additionally, comments have been enabled via Commento, the least invasive, privacy focused, open source commenting platform I could find. https://vince.ca/posts/eliminating-502-proxy-errors/ Tue, 11 Feb 2020 14:01:15 -0500 https://vince.ca/posts/eliminating-502-proxy-errors/ While working on an infrastructure refresh and consolidation project for one of my clients they had a legacy archive of public data consisting of several hundred gigabytes. There are a couple of approaches to handle this, each with its advantages and disadvantages. Decisions Move the data into the web container Advantages Local data is easy to manage with standard tools Disadvantages Bloated containers Synchronization requirement for each container Cost Move the data into a central nfs server