Metric</th>	raw HTML</th>	GZIP compressed</th>	Brotli compressed</th>	minified HTML</th>	minified + GZIP</th>	minified + Brotli</th></tr></thead>
Size (bytes)</td>	16781</td>	3712</td>	3002</td>	11108</td>	3374</td>	2765</strong></td></tr>
Threads</td>	25</td>	25</td>	25</td>	25</td>	25</td>	25</td></tr>
Connections</td>	75</td>	125</td>	125</td>	125</td>	125</td>	125</td></tr>
Requests per sec</td>	752.82</td>	2963.38</td>	3193.88</td>	1130.91</td>	3070.17</td>	3246.17</strong></td></tr>
Transfer per sec (MB)</td>	12.22</td>	11.27</td>	9.98 MB</td>	12.23</td>	10.69</td>	9.41</td></tr>
P50 (ms)</td>	98.34</td>	40.93</td>	37.61</td>	67.70</td>	38.90</td>	37.39</strong></td></tr>
P90 (ms)</td>	114.28</td>	52.35</td>	50.94</td>	74.82</td>	52.63</td>	50.10</strong></td></tr>
P99 (ms)</td>	151.74</td>	79.53</td>	75.61</td>	79.71</td>	83.91</td>	70.37</strong></td></tr>
Web server CPU 5min load</td>	0.41</td>	0.74</td>	0.87</td>	N/A</td>	N/A</td>	N/A</td></tr> </tbody></table> Takeways§</a> </h2> It should have been pretty obvious, but of course minifying and compressing HTML with Brotli results in the smallest file sizes, and thus the fastest response times.</p> Being able to handle ~3200 users requesting my home page in a single second at $5/month ain't nothin' to sneeze at, that's for sure (and another reason why static site generators are just the best).</p> What is</em> interesting is that the difference between GZIP and Brotli wasn't as significant as I would have expected, but looking at the 5min CPU load and how the "Transfer per sec" dropped</em> as response sizes got smaller hints at why - my server was starting to bottleneck on its single CPU core.</p> To confirm this, I ran another test with a raw text file containing the word "hello", which measured at a total of 6 bytes. </p> The result? `4077.37</code> requests/sec. </p>` `At first glance, this might seem like a big difference when compared against minified Brotli's 3246</code> request/sec. But considering that the minified + Brotli compressed asset is 2765 bytes, or 460x larger</strong> than the 6 byte text file, and both results are within ~800 TPS of each other, it would appear that the CPU on my droplet (which was pegged at 100% pretty much the whole time for both tests) is much more the culprit than the response size or my network speeds.</p>` `This means that I'd expect even better performance out of the minified + Brotli compressed assets with a beefier CPU/multi-core setup even if all other specs remained the same (network, memory, etc).</p>` `Summary`§</a> </h2> In summary, this website really is</em> fast. Because science says so.</p> One thing I've learned after my 9+ years working in software, it's always important to measure and thus validate your assumptions. Before this, my website always seemed</em> fast, but now I know I can easily handle over 3000 visitors in a single second</em>. This means I've got a lot of headroom on this blog before I'd need to worry about upgrading to a bigger droplet.</p> Was this testing methodolgy perfect? No. Will this web server actually be able to handle 3000 visitors in one second? Maybe. There's a lot that can effect that answer, but these results are very much "good enough", at least for now.</p> This website is (slightly) faster 2021-02-07T00:00:00+00:00 As many of you may be aware, the public internet is a bloated, Javascript-riddled mess.</p> If you don't believe me, I'm not the first one</a> to talk about this</a>. And I certainly won't be the last...</p> You'd think with how much internet speeds have increased in the past 20 years, particularly when compared to the dialup internet that many of us may have grown up with, we'd be loading every website with blinding speeds. But that's just not the case, now is it?</p> There's a multitude of reasons for this, including things like advertising, JavaScript, tracking pixels, the list goes on...</p> Today, I'm going to talk about a much smaller piece of this pie - the actual size of the HTML document you're reading, and how it gets delivered to your browser.</p> </span> Background.§</a> </h2> Let's get some quick background.</p> When you point your browser at my website, you kick off a 3 step proccess. I'm hand waving some things here (like TCP connections and TLS handshakes), so just roll with it if you're more technically inclined.</p> Steps:</p> Browser makes a request to the server, while also specifying what types of content encoding it can accept.</li> The server receives the request, analyzes that list of allowed content encodings, and then finds the requested asset (think an HTML web page or CSS stylesheet).</li> Having found the asset, the server makes sure it can "fit" into the desired content encoding specified by the browser, and may take steps to optimize the target asset for a particular content encoding.</li> </ol> Now, when I say "content encoding" here, I'm really meaning "compression".</p> Specifically, many web servers support serving content using gzip</a> compression. This is an old standard, works pretty well for text data, is relatively fast for a web server to do on the fly, and "just works".</p> Now, it's not guaranteed that you receive a gzip-compressed blob of HTMl for every website your visit. It depends on your browser and the web server configuration.</p> So why is this website "faster"?§</a> </h2> Because I made my compression "more better". Yeah, I said it.</p> More better</del> specifically, I optimized 3 key aspects of how my website assets are compressed:</p> Utilized a better gzip compressor known as Zopfli</a>.</li> Added Brotli</a>, an even more efficient compression algorithm than gzip.</li> Compressed all assets before</strong> the web server ever sees them, meaning less work for it to do.</li> </ol> To be clear, I was already using compression via Caddy's encode gzip zstd</code> option, but that requires all compression to be done on the fly (and thus repeated for every duplicate request), doesn't support Brotli, and uses a sub-par gzip compression algorithm (when compared to Zopfli).</p> In summary, I'm increasing existing compression, utilizing more advanced compression, and pre-compressing all of my pertinent text-related assets of this website (things like HTML, CSS, and even SVG images, if I had any...).</p> Great, so how did you do this?§</a> </h2> It all kind of started when I got a little frustrated with my web server, Caddy</a>. </p> Now to be clear, Caddy is an excellent web server. I just wanted to replicate an existing setup I had on my old nginx server where I pre-compressed a number of assets before uploading them.</p> I'd looked at doing this a few times over the past year, but ran into a few roadblocks along the way, including:</p> Older versions of Caddy (v1) made this easy, but this feature was removed in a large rewrite for v2, which is what I use.</li> It was theoretically possible using their new configuration syntax, but no one had fully put the puzzel pieces together yet.</li> </ol> Luckily, if you wait long enough, someone else figures out how to get something done on the internet. Actually, several someones.</p> After doing another web search on handling pre-compressed assets with Caddy, I noticed some forum posts from the past year, and eventually stumbled on this excellent blog post</a>.</p> See the bottom of that blog post for links to the pertinent forum posts.</p> </blockquote> By solving the "serving pre-compressed assets" problem, getting better gzip compression with Zopfli and adding Brotli to the mix was the easy part.</p> Updating Caddy to serve pre-compressed assets.§</a> </h3> Update 2021-05-15: Serving precompressed assets got a LOT easier in Caddy 2.4.0. Read more here</a></p> </blockquote> This is copied pretty much verbatim from the above blog post, but here is the configuration I added in order to get Caddy to correctly serve pre-compressed Brotli and gzip compressed assets whenever possible.</p> example.com { </span> root * /path/to/dir </span> </span> #I specifically removed the following 'encode' option, since I'm doing compression </span> #myself and Caddy was gzip-ing my PNG and JPEG images with it on (which is wasted CPU effort). </span> #encode gzip zstd </span> </span> ### Precompression support </span> @brotli { </span> header Accept-Encoding br </span> file { </span> try_files {path}.br {path}/index.html.br {path}.html.br </span> } </span> } </span> handle @brotli { </span> header { </span> Content-Encoding br </span> Content-Type text/html </span> } </span> rewrite {http.matchers.file.relative} </span> } </span> </span> @gzip { </span> header Accept-Encoding gzip </span> file { </span> try_files {path}.gz {path}/index.html.gz {path}.html.gz </span> } </span> } </span> handle @gzip { </span> header { </span> Content-Encoding gzip </span> Content-Type text/html </span> } </span> rewrite {http.matchers.file.relative} </span> } </span> </span> @html { </span> file </span> path .html / </span> } </span> header @html { </span> Content-Type text/html </span> defer </span> } </span> </span> @css { </span> file </span> path .css </span> } </span> header @css { </span> Content-Type text/css </span> defer </span> } </span> </span> @js { </span> file </span> path .js </span> } </span> header @js { </span> Content-Type text/javascript </span> defer </span> } </span> </span> @svg { </span> file </span> path .svg </span> } </span> header @svg { </span> Content-Type image/svg+xml </span> defer </span> } </span> </span> @xml { </span> file </span> path .xml </span> } </span> header @xml { </span> Content-Type application/xml </span> defer </span> } </span> </span> @json { </span> file </span> path .json </span> } </span> header @json { </span> Content-Type application/json </span> defer </span> } </span>} </span></code></pre> While long, it's actually not a scary config. It basically tells Caddy to serve pre-compressed assets if they exist, while also fixing the Content-Type</code> header so that we correctly identify .html</code> files as text/html</code> instead of compressed binary data. </p> Compressing assets using Zopfli§</a> </h3> Now that the web server can correctly serve my assets, I tested a few iterations of the Zopfli before adding it to my web publishing workflow. I already optimize my website's images using advpng</code> and jpegtran</code>, so I just added to my existing pattern.</p> Also, for those who may be confused - Zopfli is basically a "better gzip compressor". It produces files that any standard gzip utility can decompress, but it does so using more advanced algorithms than the standard gzip</code> utility.</p> If you want to get started, here's some useful commands for testing zopfli.</p> zopfli -c index.html > index.html.gz </span></code></pre> You'll note that I'm using a redirect here (></code>) to produce a new file. By default, zopfli overwrites the existing file, which I want to preserve as a fallback option on my web server in case the user's browser doesn't support compression. So instead, I force zopfli to output to stdout by using the -c</code> flag, and then use shell redirection to create a new file.</p> Do it in parallel§</a> </h4> Here's the fancy fd</code> command that I came up with that will automatically find all of my text assets in my website publishing directory (public/</code>) and compress them using zopfli:</p> fd -e html -e xml -e css -e js -e svg -e txt -e json --exec sh -c 'zopfli -c {} > {}.gz' \; . public/ </span></code></pre> By default, fd</code> will utilize all of your CPU cores to run whatever command you specify in bulk. This allows me to very quickly</em> compress my entire website.</p> Compressing assets with Brotli§</a> </h3> Brotli produces even smaller files than Zopfli, and has been accepted as an IETF standard</a>, meaning that most web browsers support it by default now. </p> In order to create Brotli files for my assets like I did with Zopfli above, I came up with the following:</p> brotli -Z -f index.html </span></code></pre> Unlike zopfli, brotli doesn't overwrite its input file, so the above command will produce an index.html.br</code> file by default. For good measure I added -Z</code> to always use max compression, and -f</code> to force overwriting of any existing files. This is useful because I statically generate this site from a set of Markdown files, and I'd rather not have to delete the existing output directory every time just to keep Brotli happy.</p> Do it in parallel, again§</a> </h4> fd -e html -e xml -e css -e js -e svg -e txt -e json --exec brotli -Z -f {} \; . public/ </span></code></pre> Same style of fd</code> command as above...</p> Results§</a> </h2> For just a quick comparison, I compressed the home page of this blog with Zopfli and Brotli.</p> Here's the number of bytes in four forms: uncompressed, compressed with Brotli, and compressed with Zopfli, and compressed with standard gzip using gzip -9</code> for maximum compression:</p> 17,817 index.html </span>2,607 index.html.br (85.37% improvement over uncompressed) </span>3,308 index.html.gz (81.43% improvement over uncompressed) (zopfli) </span>3,404 index.html.gzip (80.89% improvement over uncompressed) (gzip -9) </span></code></pre> As you can see, compression in general results in significant savings. Brotli unsurprisingly takes the cake, followed by Zopfli and standard gzip as dead last.</p> That said, the improvement of Zopfli over standard gzip is marginal in this case, but when applied to all text assets across the entire website, you can shave a few percentage points off your web traffic.</p> Conclusion§</a> </h2> So yeah, you're likely reading this via a Brotli-compressed HTML page, or maybe even a Zopfli-compressed HTML page.</p> In addition to the total lack of Javascript, trackers, and any other 3rd party requests, you can enjoy a lighting fast "plain text" experience on this website.</p> Cheers!</p> NetworkManager is bloat 2021-01-31T00:00:00+00:00 Or how I came to love dhcpcd</code> and iwd</code> for all of my networking needs...</p> </span>Background§</a> </h2> I have a weird fascination with running as little</em> software as possible on my machines.</p> For whatever reason, minimizing RAM usage scratches some weird itch of mine. And no, it's not infected. At least, I hope not...</p> Anyways, after watching a video on installing Arch Linux</a> for fun (don't judge me), the presenter Ermanno (who's videos I VERY HIGHLY RECOMMEND</strong>) covered a change to the install process that Arch Linux made recently.</p> Or I suppose a more accurate description is a change that Arch Linux made to how to connect to a wireless network when you're installing. Keep in mind that the standard Arch Linux install process doesn't have a GUI or any desktop environment with a graphical installer. There's no comfy icon to click on to configure networking, nothing.</p> Out of curiousity, I started looking into this new replacement sofware - iwd</code>.</p> Once I started falling down that rabbit hole, I discovered just how unnecessary NetworkManager</code> is in my setup.</p> Why this works for me§</a> </h2> Like I said, I run a minimal setup. This means I don't need nor want big fancy GUI's for my tools.</p> To give an example, I run:</p> A tiling window manager (i3, bspwm, sway, river are all on regular rotation...)</li> A CLI or ncurses-UI for connecting to wifi (this was nmtui</code> when I was using NetworkManager</code>).</li> A CLI VPN with no visual feedback or need to graphically configure said VPN.</li> I don't connect to esoteric modems like PPPOE, etc from my laptop or desktop.</li> </ul> With all this in mind, all of the "extra" features (read: bloat) of NetworkManager</code> mean that I'm paying a pretty steep cost in terms of dependencies and memory usage for things I simply don't need.</p> Note that there are some GUI tools for dhcpcd</code> and iwd</code> at this point, so check those out on the Arch Wiki if you're really curious.</p> </blockquote> Arch Linux Wiki entry on iwd</a></li> Arch Linux Wiki entry on dhcpcd</a></li> </ul> How to make the change§</a> </h2> If you want to make the change yourself, it's actually pretty easy. I recommend doing things in this order, because if you can't get iwd/dhcpcd to work to your liking, it's trivial to go back to NetworkManager</code>.</p> Install iwd and dhcpcd (and openresolv)§</a> </h3> We need all three applications. In short, dhcpcd</code> handles the DHCP handshake (and thus giving you an IP address) when a connection is established, like plugging in an ethernet cable or connecting to a wifi network with iwd</code>. As stated iwd</code> handles wireless connections, that's it.</p> Lastly, openresolv</code> (a resolvconf</code> implementation) handles the DNS configuration (mostly the maintenance of your /etc/hosts</code> file). dhcpcd</code> works with openresolv</code> automatically, you can read its documentation</a> for more details.</p> I did mess around with using systemd-resolvd</a> to have one less dependency, but ran into several issues with DNS configuration reliability. Just use openresolv</code> and you'll be much happier.</p> </blockquote> On Arch, I installed all necessary software with pacman like so:</p> # pacman -S iwd dhcpcd openresolv </span></code></pre> Disable NetworkManager and wpa_supplicant§</a> </h3> Now that you've downloaded the necessary software, you've got to shutdown NetworkManager</code> and wpa_supplicant</code> to prevent conflicts between the two sets of software.</p> # systemctl stop NetworkManager wpa_supplicant </span></code></pre> Note: Obviously, you'll lose internet at this point.</strong></p> Start iwd and dhcpcd§</a> </h3> With NetworkManager</code> and its wireless sidekick taken care of, spin up iwd and dhcpcd.</p> # systemctl start iwd dhcpcd </span></code></pre> Wifi setup§</a> </h3> dhcpcd</code> handles all wired connections automatically, so if you're already connected with an ethernet cable you're done.</p> But, if you need to connect to a wireless network (as I'd wager 99.99% of people reading this do...), you'll need to configured iwd</code> to know about your wireless network.</p> To do so you can use the delightful companion for iwd - iwctl</code>.</p> You can run iwctl</code> directly, type help</code>, and then follow the corresponding documentation. Alternatively, you can pass all of your commands directly to iwctl</code>, as I demonstrate at the end of this section.</p> In my case, I simply needed to type the following:</p> $ iwctl </span>[iwd]# device list </span>wlan0 ... </span></code></pre> I simply wanted to ensure that a wireless device was recognized, and it was. This isn't actually required to connect to an access point.</p> [iwd]# station wlan0 scan </span></code></pre> Wait a few seconds as iwd looks up all the networks in your area...</p> [iwd]# station wlan0 connect "MyNetworkName" </span></code></pre> Enter password when prompted and wait ~10 seconds to ensure you don't get an error back. If you do, you likely mistyped your password or your wifi is on the fritz.</p> Note that you can press Tab</code> to autocomplete pretty much all commands, including the wireless name above.</p> </blockquote> [iwd]# quit </span></code></pre> Exit out of the application.</p> Note that instead of using the iwctl</code> REPL, you can just feed the associated commands directly to iwctl</code>, like so:</p> $ iwctl station wlan0 connect "MyNetworkName" </span></code></pre> I do like the REPL, as it holds your hand (if necessary), supports tab completion and just feels polished. Use whatever method you prefer.</p> Ensure you've got an IP address§</a> </h3> You may need to wait a few seconds as iwd</code> and dhcpcd</code> do their corresonding thaaangs, but you should receive an IP address that looks somewhat familiar. You can check this with the ip</code> command.</p> $ ip addr </span>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 </span> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 </span> inet 127.0.0.1/8 scope host lo </span> valid_lft forever preferred_lft forever </span> inet6 ::1/128 scope host </span> valid_lft forever preferred_lft forever </span>3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 </span> link/ether 60:e1:86:87:bc:89 brd ff:ff:ff:ff:ff:ff </span> inet 192.168.1.217/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0 </span> valid_lft 33914sec preferred_lft 28514sec </span> inet6 fd22:f6c3:4e66::a7e/128 scope global dynamic noprefixroute </span> valid_lft 35181sec preferred_lft 35181sec </span> inet6 2600:6c52:7b00:55aa::a7e/128 scope global dynamic noprefixroute </span> valid_lft 35181sec preferred_lft 35181sec </span> inet6 fd22:f6c3:4e66:0:dcb8:7af:18a1:8bd9/64 scope global mngtmpaddr noprefixroute </span> valid_lft forever preferred_lft forever </span> inet6 2600:6c52:7b00:55aa:9af:4ba4:ee9b:d32d/64 scope global dynamic mngtmpaddr noprefixroute </span> valid_lft 573699sec preferred_lft 573699sec </span> inet6 fe80::1f0:1103:da4d:ee9b/64 scope link </span> valid_lft forever preferred_lft forever </span></code></pre> While that might seem like a wall of text, the 192.168.1.217</code> under wlan0</code> is what I'm looking for and tells me that I'm properly connected to my home network.</p> You can confirm network + DNS connectivity with a simple:</p> $ ping austindw.com </span></code></pre> Or use google.com, whatever you want...</p> </blockquote> If everything looks good, you can make the switch permanent:</p> # systemctl enable dhcpcd iwd </span># systemctl disable NetworkManager wpa_supplicant </span></code></pre> Reboot, and you should come up with a perfectly working internet connection.</p> How to revert§</a> </h3> If you run into issues and you want to switch back, simply do:</p> # systemctl disable --now dhcpcd iwd </span># systemctl enable --now NetworkManager wpa_supplicant </span></code></pre> This will immediately stop dhcpcd</code> and iwd</code>, then start NetworkManager</code> and wpa_supplicant</code>. With these commands, you'll even be safe to reboot and still use NetworkManager</code>.</p> Results§</a> </h2> "But Austin, I just replaced two networking programs with two more networking programs, how is this an improvement?"</p> </blockquote> Good question - if you measure the RAM usage before and after the switch, you should notice around a ~20MB reduction with the exact same end result</em>.</p> Additionally, my wireless connections were much</em> more stable with iwd</code> than with NetworkManager</code>. I used to have frequent disconnect/reconnect storms when I placed a decent amount of load on my wifi connection. I never quite tracked down the root cause, but the broader stroke of just replacing NetworkManager</code> completely has yielded a much improved networking experience.</p> So I have less resource usage, and an extremely stable wireless network connection.</p> I'm never going back to NetworkManager</code> if I can help it.</p> Hopefully this has helped convert you. Give it a try, and you just might be sold on the iwd</code> + dhcpcd</code> power couple for life.</p> OpenWRT Router Setup 2020-12-06T00:00:00+00:00 Since it's 2020, I have an abundance of free time on weekends, and a buddy of mine kept (rightfully) bugging me to write up a post on my router setup, I finally got around to doing so.</p> Let's dive in on my home router setup, including firmware, QoS, and the beginnings of ad blocking.</p> </span>Prelaunch§</a> </h2> Before I get started, perhaps I should explain why I run a custom router firmware, and why you may want to as well.</p> Security</strong> The upside to running a firmware like OpenWRT is that I get much more up-to-date code than what ships with the router. Router manufacturers rarely support their router firmware longer than a year or so, so by using something like OpenWRT I recieve regular updates, which is paramount for network security. Also, by flashing my own firmware, I know exactly what's running on the device, instead of potential malware.</li> If you don't believe me, here's just one example of modern routers being sold on Walmart and Amazon that contain backdoors: Cybernews.com</a></li> </ul> </li> Performance</strong> OpenWRT was one of the first operating systems to support bufferbloat mitigation technologies</a> like fq_codel, "cake", etc. It's a long story, but the creators of this cutting edge QoS technology created a custom fork of OpenWRT (CeroWrt</a>) for research purposes. As their research bore fruit, they published their work upstream into the core OpenWRT project. This technology is truly amazing in its ability to reduce or eliminate severe latency on a loaded network connection, and could stand on its own as the sole reason to run OpenWRT.</li> </ul> </li> Ad Blocking</strong> While Pihole is all the rage (I prefer, and have contributed to, Adguard Home</a> instead), you can get pretty dang before when it comes to blocking ads across your entire network with some of the facilities available in OpenWRT. Off the cuff, I know of three different methods of blocking ads network wide with OpenWRT, and we'll cover one today.</li> </ul> </li> Because it's fun!</strong> Let's face it, if you're reading this blog, you're likely a fellow nerd (said with affection). Hacking away on your own device gives a strong sense of accomplishment and a feeling like you really own something.</li> </ul> </li> </ol> Launch§</a> </h2> Okay, first things first - you'll want to install OpenWRT on your device.</p> There's a few methods for doing this, and I'm just going to refer you to the OpenWRT website, which has a Quick Start</a> guide for taking an existing router with stock firmware and installing OpenWRT.</p> The TL;DR version:</p> Make sure your device is supported by OpenWRT</a>.</li> Boot your router, login to the admin page (generally found at http://192.168.0.1</code> or http://192.168.1.1</code>). If you don't know the username/password (if it prompts you), do an internet search for the brand, as they often use standard credentials like admin/admin</code> or root/admin</code> for the username and password combo. You may be able to find it written on your router near the serial number as well.</li> </ul> </li> Find the firmware upgrade button, and select the -factory.bin</code> firmware image you presumably downloaded. I have a few routers (don't judge), so using the TP-Link Archer C7 v2</a> as an example, I'd upload the openwrt-19.07.4-ath79-generic-tplink_archer-c7-v2-squashfs-factory.bin</a> as an example.</li> Note:</strong> One particular quirk I ran into with this router (funny story actually...) is that the company made a few regionalized models, separate from their global models. These are region locked, and require uploading the correct US</a>/EU</a> version when doing the initial flash. </li> </ul> </li> Wait for the router to reboot (this can</em> take a while, so be sure to give it 5-10 minutes before you assume you've failed).</li> Reset your network (unplug/replug your wired cable, assuming you used one).</li> Login to the admin page at 192.168.1.1</code>. Just type that into your browser and go. If your browser tries to get smart with you and perform a web search with that address instead, look for the little prompt that says "did you actually mean to go to 192.168.1.1?". Yes I did, you cheaky bastard of a browser...</em></li> </ul> </li> </ol> There's loads of tutorials, Youtube videos, etc about doing this particular song and dance. I recommend you start searching if you want more information or better tutorials.</p> Configuration§</a> </h2> Okay, now that you've got the guts installed, let's get configuring the cool stuff.</p> Root password§</a> </h3> When you initially login, you'll see a prompt with the username root</code> and an empty password box. Simply click Login</code> the first time, since an administrative password hasn't been set. You'll want to set one, for obvious security reasons.</p> You can change it under System</code> -> Administration</code> -> Router Password</code>.</p> </picture> Wireless§</a> </h3> Not all routers actually have wireless radios, but assuming yours does (and you want to use it), you can get started creating your wireless network by navigating to Network</code> -> Wireless</code>. </p> If you're running a modern router, you likely have two radios (one for 802.11n and one for 802.11ac).</p> Simply click the Edit</code> button for each and fill in details like the (E)SSID (your wireless network name), set the Wireless Security Encryption to WPA2-PSK</code> (strong security), and pick a decent password for the Key</code>.</p> </picture> You may want to experiment with the Channel</code> and Width</code> options over time as well. Generally, the greater the Width</code>, the better your speeds will be, but this depends on the router and what the devices on your network support. Generally you'll have better compatibility with a smaller Width</code> (like 20-40 MHz), but you may be sacrificing some speed. TBH, you probably won't even notice...</p> The Channel</code> option is a little more tricky. When in doubt, leave it at default. Maybe try auto</code> (although I've had some routers just straight fail to spin up the radio on auto</code>...). The best approach is to use an app like Wifi Analyzer</a> (Android) to map out the channels being used by your neighbors. You'll want to aim for a less congested channel.</p> Also, channels can overlap a bit (you'll see that in the graphs generated by Wifi Analyzer), so it's recommended to pick something that's ~3 (or more) channels away from another channel, if you can. In general aim for the largest gap in channels, and hope for the best.</p> This is more of a problem on 2.4Ghz networks, since 5Ghz networks have access to a lot more channels. </p> Again, if you're unsure about any of this, simply stick with the defaults for now. You can always change it later.</p> Firewall§</a> </h3> For a little extra security, you can use this trick to "hide" your router from the public internet. To be fair, it's not really hiding, but simply acts as a deterent to anyone running port scans on the open internet.</p> Run a "ShieldsUP" check against your current router configuration, courtesy of Steve Gibson ShieldsUP</a></li> </ul> </li> Navigate to Network</code> -> Firewall</code> -> General Settings</code></li> Under the Zones</code> section, change the reject</code> value to drop</code> for the Input</code> and Forward</code> columns of the wan => REJECT</code> row. See the screenshot below for reference.</li> Run the "ShieldsUP" scan again, which should show that your "equipment actively rejected our UPnP probes!".</li> Rejoice.</li> </ol> </picture> SQM (aka the bufferbloat killer...)§</a> </h3> For this next part, you'll want to install the SQM package. You can either do so from the Web UI, or if you want to go commando-line (heh), then you can SSH to your router via ssh root@192.168.1.1</code>. The password is whatever you set on the Router Password</code> page beforehand.</p> Command line install of SQM§</a> </h4> Nice and easy:</p> # opkg update </span># opkg install luci-app-sqm </span></code></pre> Note that I'm installing luci-app-sqm</code>, as that brings in the core SQM dependencies AND</em> a useful widget in the Web UI. If you're hardcore and don't want to use the Web UI at all, then just install the sqm-scripts</code> package.</p> But if you're that hardcore, you probably already knew to do that... why are you reading this blog??</p> Web UI install of SQM§</a> </h4> Navigate to System</code> -> Software</code>.</li> Using the Filter</code> field, type luci-app-sqm</code>.</li> It should show up as the top result, so click the Install</code> button and wait.</li> It can take a minute, but if the screen seems to have frozen while waiting for the install, chances are you're fine and the install finished, so simply reload the page.</li> </ol> Setting up SQM§</a> </h4> Note: I have a DOCSIS (cable) modem to connect to my ISP (Charter). Depending on your connection, you may want to change a few of these settings. I recommend reading the linked references.</strong></p> After installing the luci-app-sqm</code> package, and rebooting</strong>, log back in and go to Network</code> -> SQM QoS</code></p> Check the box to enable this SQM instance.</li> From the dropdown, select your WAN</strong> interface (aka, the interface that connects to the public internet, NOT your local LAN). If you open the list, chances are you'll see an interface with (wan,wan6)</code> tacked on the end. This is what you want.</li> Set the download speed/upload speed accordingly. This will require trial and error, I recommend testing using a good speed test tool (make sure you convert the value to kilobits/s, which is what this tool takes).</li> If you're on Arch Linux/Manjaro/etc, I recommend installing ookla-speedtest-bin</code> from the AUR, and running your speedtest like so: speedtest -u kbps</code></li> You'll want to set these values to someting like 90% of the tested rate to start, but then after doing the later tweaks, you can run tests to bring these closer to 95% or even 99% of rated speeds.</li> For my 100 megabit connection from Charter, I use: 111000</code> for download and 11100</code> for upload on my WRT3200ACM, but I have to tune those down to 110000</code> and 10000</code> respectively on my Archer C7, since it's not powerful enough to handle the higher bandwidth settings without spiking the latency by 200ms in the first few packets.</li> You can test your optimal settings by running a ping (ping 1.1.1.1</code>) in one terminal and then a speedtest (speedtest -u kbps</code>) in another.</li> If the ping spikes significantly while running your speedtest (really noticeable when doing uploads), then you'll need to adjust your settings to more conservative values. If you want an idea of what's bad, simply run this test with SQM off - you'll likely see your ping times pike into the 100's of milliseconds (which is insane).</li> Reference - short version: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm</a></li> Reference - long (and very interesting) version: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details</a></li> </ol> </li> Set the Queue discipline to cake</code>.</li> Set the Queue setup script to piece_of_cake</code>.</li> Check the Show and use dangerous configuration</code> checkbox.</li> Set the Explicit congestion control</code> to ECN</code> for ingress</code>.</li> Set the Explicit congestion control</code> to NOECN</code> for egress</code>.</li> Under the Advanced option...for ingress queueing</code> option, set docsis nat ingress</code></li> Under the Advanced option...for egress queueing</code> option, set docsis nat</code> If your up/down bandwidth is greater than or equal to 1x15, you can experiment with adding ack-filter</code> to your egress options.</li> Reference: https://www.reddit.com/r/HomeNetworking/comments/fvhr4w/psa_sqm_cake_nat_and_bufferbloat_tuning/</a></li> You can read more about these advanced options at the following man page: https://man7.org/linux/man-pages/man8/tc-cake.8.html</a></li> </ol> </li> Both of these are assuming you're using a cable (DOCSIS) modem.</li> IF you aren't using the docsis</code> queue option above THEN do the following, otherwise set the link layer adaptation to none</code> (because you're already adapting for it):</li> Under Link Layer Adaptation, set the Link Layer to Ethernet with overhead</code> if using a cable modem (which I am).</li> Much more information can be found here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#making_cake_sing_and_dance_on_a_tight_rope_without_a_safety_net_aka_advanced_features</li> Set the Per Packet Overhead to 18 (or 22, seems to be some discrepancy in recommendations).</li> </ol> </picture> Encrypted DNS (with optional ad blocking)§</a> </h3> I'm a big fan of privacy. I'm also a big fan of ad blocking, which generally improves privacy, security, AND performance. One hell of a trifecta.</p> With that in mind, I wanted to give my brother all of these benefits, without the hastle of dealing with adblock lists, special admin pages on the router, etc.</p> To be fair, I</strong> do this (via an instance of Adguard Home</a>, installed on my router), as I like to have complete control and reap the performance benefits of having an adblocker running directly on my local router. I'll write a proper blog post about that setup another time...</p> But, this just didn't make sense for my brother's use case. So in order to give my brother the benefits of encrypted DNS while also allowing myself to remotely turn on ad blocking if he desires, I set him up with the following:</p> A profile on my NextDNS</a> account. Even if you aren't using their ad blocking functionality, I think that NextDNS is a fanastic service. They provide high speed DNS servers, encrypted traffic support, optional (!!) metrics/logs, and of course - incredibly comprehensive filtering capabilities, with blocking for everything from ads to malware to social media.</li> Plus, their privacy policy</a> is just incredible and I think sets the standard that other companies should be held against.</li> They're primary service is free for 300,000 queries/month. Because I love their service, I go ahead and throw $2 their way a month. Since I pay for their "Pro" plan, if I spill over 300,000 DNS queries a month, then I don't lose ad blocking support for the remainder of the month.</li> </ol> </li> An instance of dnscrypt-proxy</code> (2), running on the router.</li> </ol> This combination allows for strong DNS traffic encryption, blocking via NextDNS, and local device resolution via the onboard dnsmasq</code> instance that comes with OpenWRT.</p> So, without further ado, let's get started.</p> Installing dnscrypt-proxy 2§</a> </h4> Luckily, this is quite easy these days, ever since they added an official package to the OpenWRT repos.</p> The dnscrypt-proxy wiki</a> has detailed instructions.</p> In short:</p> Update your software lists from either the UI (System</code> -> Software</code> -> Update lists...</code>) or an SSH instance (opkg update</code>).</li> Install dnscrypt-proxy2</code>, just like you did with luci-app-sqm</code> above.</li> Go to Network</code> -> DHCP and DNS</code> -> General Settings</code> -> DNS forwardings</code>, and add 127.0.0.53</code>. This tells the running dnsmasq</code> instance to point to the dnscrypt-proxy</code> instance for DNS resolution, instead of using your ISP's servers.</li> </ol> </li> Save & Apply.</li> </ol> Configuring dnscrypt-proxy 2§</a> </h4> Now, before we restart dnsmasq to get the changes to take effect, let's update dnscrypt-proxy</code> to point to our NextDNS instances.</p> You need to SSH into your router to do so. If you know what SSH is, you probably know how to do it (ssh root@192.168.1.1</code>...)</p> Once you've logged in, you'll want to edit the /etc/dnscrypt-proxy2/dnscrypt-proxy.toml</code> file. Chances are your OpenWRT install has either nano</code> or a stripped down verison of vim</code> installed. Pick your favorite editor.</p> In this case, I used vim</code>: </p> vim /etc/dnscrypt-proxy2/dnscrypt-proxy.toml </span></code></pre> Point to NextDNS§</a> </h5> Scroll to the very bottom, you'll see a section for Static entries</code>.</p> If you're using NextDNS, login to your account, select the profile you want to use, and then scroll down in the installation instructions, and you'll see an generated config for DNSCrypt support.</p> Here's an example of my own after I updated it to point to my NextDNS profile:</p> ... </span>######################################## </span># Static entries # </span>######################################## </span> </span>## Optional, local, static list of additional servers </span>## Mostly useful for testing your own servers. </span> </span>[static] </span> </span> [static.'NextDNS-BroRouter'] </span> stamp = 'sdns://AgEAAAAAAAAACImaBunchOgiBberish...' </span></code></pre> Scroll back up to the top of the file, and uncomment and update the server_names</code> entry to point to your new NextDNS record.</p> ... </span>server_names = ['NextDNS-BroRouter'] </span>... </span></code></pre> You could stop right there. I updated a bit more of the config, out of some old lesson's I'd learned over the years.</p> IPv6§</a> </h5> First, if you have an IPv6 connection, you'll want to able the ipv6_servers</code> option. You can tell if you have IPv6 in OpenWRT by viewing your WAN6 interface - if it's up and sending/recieving packets, you have IPv6 configured on the modem and the router.</p> So, find the ipv6_servers</code> option, and set it to true</code> (it's likely false</code> by default).</p> Blacklist§</a> </h5> Lastly, if you're going to be using NextDNS for ad blocking/domain filtering, I recommend disabling the default blacklist file that dnscrypt-proxy</code> comes with.</p> To do that, find the [blacklist]</code> section.</p> Under that, you should see an entry for blacklist_file</code>. Simply comment it out, like so:</p> [blacklist] </span> </span> ## Path to the file of blocking rules (absolute, or relative to the same directory as the... </span>#blacklist_file = 'blacklist.txt' </span></code></pre> This will prevent dnscrypt-proxy</code> from doing any of its own filtering on box. While this might sound advantageous at first, if you're using NextDNS to configure your filters, you don't want an extra set of filtering rules running on your router that might conflict with your downstream settings. Trust me, this is confusing as hell when you run into it, took me a little while to figure out why certain websites were't working that should have been.</p> Caching§</a> </h5> You don't have to do anything! </p> Older versions of dnscrypt-proxy</code> ran with less-than-optimal caching settings, but the author conducted some interesting research</a>, found pretty optimal settings, and set those as the defaults in dnscrypt-proxy.toml</code>. You can scroll down to the caching section to get an idea of what they are, if you're curious.</p> See also: https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Caching</a></p> Otherwise, just rest easy and enjoy blazing fast caching.</p> Restart dnsmasq§</a> </h4> Restart dnsmasq</code>.</p> System</code> -> Startup</code>...</p> Conclusion§</a> </h2> Whoo! This was a long post. I hope this helped you in some way, as it's been a collection of techniques that I've used on my own routers in one way shape or form, and I don't think there's any other place on the internet that has all of this in one place.</p> Go forth, have awesome internet, and prosper!</p> Troublesome iPod 2020-08-16T00:00:00+00:00 In an effort to get away from the shitstorm that is 2020 (COVID-19, major upsets in my personal life, etc), I decided to visit my mom, back in my hometown.</p> </span>Dem boonies§</a> </h3> To frame things a bit, my mom lives a fair ways out from "town". This means that certain amenities that I've come to take for granted aren't available. Things like a truly high speed internet connection (I'm sorry, but DSL does not</em> count any more), cellular reception, a coffee shop filled with cute baristas and college students...</p> I digress.</p> One key point is that the online world is not easily accessible when you take a few steps out of the cityverse. This also means that things like Spotify aren't really usable, especially for my mother, who's as far removed from a "tech wiz" as anyone I know.</p> But she does love her music.</p> And she has loved her iPods. A tiny device with one of the greatest navigation interfaces of all time (seriously, can we bring back the click wheel?), loads of storage space, loads of battery life, and loads of not needing an internet connection.</p> Like too many things in tech, they don't always age well. The iPod Classic my mom had been using for the past few years can barely hold a charge longer than 2-3 hours of playback time. Additionally, the data/charge port on the device^{1</sup> can only do the latter. Plugging it into a laptop yielded zero device recognition, on either Windows or Linux.</p>} So, the old device was kaput. No way to add music, no way to migrate said music to another device, nothing.</p> But, this didn't stop my mom from ordering a "new" iPod. Apparently, you can still</strong> get these things on Amazon^2</sup>.</p> So, there we were with a new iPod to load up with music and a laptop that (very luckily) had pretty much every song my mom had on her old iPod. +1 for coincidental backups.</p> So you want to transfer some files?§</a> </h3> It being 2020, and iPod Classics having ended their official manufacturing run 10 years ago, I thought - "it has</em> to be possible to upload music to these things without iTunes, right?" I mean, we had a Linux laptop that I brought and her Windows laptop for optimal spread of operating system support. Surely there must be some free software that's been written that easily imports music from a folder into the iPod Classic!</p> Boy, was I wrong.</p> I tried:</p> aTunes</a></li> YamiPod</a></li> Manual workarounds like revealing hidden folders in Windows, navigating into the iPod's hidden Music folder, and drag/dropping some files in.</li> A fleet of other software all based around shared libraries used by YamiPod.</li> </ol> But nooooooo. Apple builds such brilliant hardware that something that should be as simple as copying a bunch of MP3 files into a folder isn't even possible. In fact, it's actively disallowed</em>. </p> This isn't really news to anyone. Apple created the greatest MP3 player in the world (and possibly ever), but everyone who got one hated one thing about it - having to use iTunes to manage all of their music. It was slow, it locked down a device to a particular computer, music couldn't be easily shared with friends, etc.</p> All around iTunes was a total regression in user experience, which juxtaposed so very shockingly with the excellent</strong> UX that the iPod Classic offered (I mean that click wheel</em>).</p> And here we are 10 years later, and we're still back at square one - use iTunes.</p> To be fair, there are</em> a stupidly large number of paid spyware</del> apps that claim to handle iPod Classics with zero problem. And I believe them - we as a society have had a very very long time to nail down a 10 year old piece of hardware. But we seem to have given up.</p> Or at least I've given up. </p> We downloaded iTunes, installed it, and I drag/dropped 110GB+ of music onto the device and away we went.</p> Learnings...§</a> </h3> But I learned something. I learned something about myself and Apple. I learned that Apple will actively create consumer unfriendly software. And I learned that I don't want to deal with its bullshit any more. Just give me an MP3 player with an SD card slot and I'm good.</p> Oh look - they do exist</a></p> And of course, it looks like an Apple Watch...</p> Footnotes§</a> </h2> Those old wide + flat connectors from back in the day before Apple made 2-3 more connectors and now everything requires a damn dongle.</li> Although they aren't nearly as cheap as they should be</a>.</li> </ol> (Re)New Beginnings... 2020-07-12T00:00:00+00:00 Jeebus.</p> It's been quite a long time since I last picked up the pen... as it were.</p> This blog has sat in the metaphorical trunk, strapped with a belt, wrapped in a blanket, tucked under the creaky stairs of your old aunt/grandma/weird neighbor's house for a very very long time.</p> </span> Much has changed since I've been away...</p> For one, I'm writing this in the time of Coronavirus (COVID-19), the longest running catastrophe that my fickle mind can remember at my age, and I remember when the Twin Towers fell and the 2008-09 stock market crashed (I was pretty sheltered from that one, still trudging through university at the time, like you do).</p> Coronavirus seems to be the impetus from which so many find new inspiration. I find this stems from a boredom grown from the mind numbing steps one takes, marching through life strapped onto the back of a routine - wakeup, work, workout, eat, sleep, repeat. I could blame this new found desire to write on Coronavirus, and I'd probably be right. But also a little wrong.</p> Aside: Now, I talk of the gift that is boredom from the comfort of my own home. It doesn't take much to recognize my privilege here. Many are struggling to survive, either to provide food for the table, or a place to sleep for the night. A member of my own family survives due to the very helpful program that is unemployment. It's not perfect, but dammit is it a lifeline in this torrential downpour.</p> </blockquote> </blockquote> For another, I've spent over 9 years in the tech industry, working my way through the ranks at Amazon. Far and away a different person than I was in my college days, or just there after.</p> I'd like to think myself more interesting, but I guess you'll be the judge of that.</p> But I digress§</a> </h3> Back to our regularly scheduled programming.</p> </blockquote> </blockquote> I'm restarting this blog.</p> Maybe it's because people in the software industry still seem to have blogs, which convey some startlingly useful information, or solutions to problems that I encounter on the daily, and I'd like to pass it on (my home-room middle school teacher would be so proud).</p> Or maybe it's because social media has molded the web into its own image, where everyone has a site, a profile, a "feed", while possessing none of them. They no longer own the data that they produce. Leave that to Facebook, to Google, Twitter, Snapchat, the list goes on.</p> Honestly, Luke Smith</a> had a point when he argues for setting up your own website. It's a site/profile/feed that you</em> own, you control. It's my own personal space to put whatever I want, and know that I'm not paying for the privilege with anything other than my own cash.</p> Having your own website is the OG social media profile. Most of the world seems to have forgotten that.</p> While preparing this post, I've actually been able to download my LinkedIn data, and I'm working deleting my account as these days pass by. To be honest, it literally never did me any good. I had a job right out of college that I've never left. I haven't had a need to network</em>, to connect myself with as many other like minds, all in the hopes of establishing a game of career hoops and ladders. I've been perfectly fine. </p> When there does come a time when I want a new job, I'll simply start asking. I'll be the ones to reach out to recruiters, demonstrating my skill set. It will be the inverse of what happens today, where multiple times a week I get emails from FAANG recruiters, which I've ignored for... literally years.</p> Who knows... maybe I'll go freelance.</p> Looking forward§</a> </h3> I look forward to more. </p> More sharing of technological breakthroughs, those oh-so-critical "ah hah" moments.</p> Sharing insight on the tech industry, ridiculously good recipes, or other inventive ways to inflate one's ego.</p> Let's hope I stick with it this time... I feel like it has been a long time coming.</p> Headphone Review: PSB M4U 1 vs B&W P7 2014-06-21T00:00:00+00:00 After graduating college, I started working fulltime at Amazon. I went from sprinting from class to class to staying indoors (professionally) for 8+ hours a day. I was forced to spend most of my day in front of a computer, sitting (or standing, thanks to my stand up desk) while I worked.</p> </span> Okay, okay, I'm sure you're wondering why I bring up my change in daily life. I'll cut to the chase. My extended periods indoors in front of a computer practically forced me to develop my taste in music, and eventually good sound.</p> With this migration into the arena of "good sound", I began my journey down the corridors of the audiophiling cabinet (see what I did there?). I bought my first pair of speakers ever, the NHT Absolute Tower speakers</a> . A few months later, I started thinking about where I was listening to music most, as my NHT's are great, but I can't exactly set them up at work.</p> With this came my foray into headphones, and after a few hours of reasearch, I bought the PSB M4U 1's</a>, primarily due to this excellent Wirecutter article</a>. Now, these headphones (at least at the time of this writing) are $300, so not cheap by any measure. However, after about a week of constant use, I noticed a few... particulars that I will detail in a minute.</p> Because of these particulars, I started looking at alternative headphones. I eventually worked my way to the B&W P7s</a>. Now at $400, these are hardly the cheaper option, but they were especially usefull as my local Apple store had a pair on display, so I was able to do a quick comparison before purchasing a pair for futher testing.</p> The Review§</a> </h2> Okay, let's get to the meat of the review. I'm going to go over the differeneces in comfort, build quality, sound quality, and ammenities, and how they applied to me.</p> Comfort§</a> </h3> Right off the bat, I'm going to dive into comfort. Comfort is a big subject for me, as my primary listening environment is 4+ hour sessions at work, so I can't be pulling the headphones off and massaging my ears every 20 minutes.</p> </picture> First off, the PSBs:</p> Pros§</a> </h4> Memory foam ear pads.</li> Excellent head band cushion.</li> Great ear seal.</li> </ul> Cons§</a> </h4> Shallow ear pads.</li> </ul> I know it's not the largest breadth of points, but I want to make one call out about the shallow ear pads. As a blonde elf look-alike, I have nicely pronounced ears (at least I think they are nice...) but nonetheless I have "large" ears. The PSBs are able to completely surround my ears without any problems, and the seal is fantastic, but the shallowness of their ear pads (the distance from the edge of ear pad to the driver on the headband) is problematic for my ears. This shallowness forces my ears to rub, and thus curl, just the slighted bit at the tips, which is irritating over extended listening sessions.</p> This shallowness is the primary reason why I began my search for alternative headphones, and thus the B&Ws:</p> Pros§</a> </h4> Much larger earpad volume</li> </ul> Cons§</a> </h4> Non-memory phone ear pads</li> Stiffer headband cushion</li> </ul> The larger earpad volume is the main reason why I prefer the B&Ws, as I no longer feel like I'm pressing the tips of my ears against a wall every time I put them on. The PSBs have a better headband pad, and cushier ear pads, but if you have larger (or in my case, pointier) ears, you may want to look elsewhere.</p> In summary, the PSBs have a better headband pad, and cushier ear pads, but if you have larger (or in my case, pointier) ears, you may want to look elsewhere. The B&Ws are firmer overall with their cushions, but I think that is simply the nature of using memory foam vs not using memory foam. Still, the B&Ws just fit my pointy-ass ears better.</p> Build Quality§</a> </h3> I have to admit build quality turned out to be more of a care-point for me than I would have thought, so let's break down some differences.</p> Again, we'll start with the PSBs:</p> Pros§</a> </h4> Prime use of leather and metal</li> Two axis earcup pivot</li> Nice case material, kind of a mix of hardshell with soft rubber.</li> </ul> Cons§</a> </h4> Plaassticc, specifically the headband cover and overall exterior.</li> Gut wrenching snap when folding the headphones up, makes me think I'm doing something wrong everytime I fold them up.</li> Cheap zipper on case. Not a huge problem, but you have to align the case just right in order to zip it up smoothly.</li> </ul> All in all, I was actually marginally disappointed with the PSB build quality, but really only because of the plastic exterior. Sure, they look similar to Beats, but the plastic just seems a touch brittle to me, and makes me worry about snapping in extreme cases. Granted, the likelyhood of snapping is pretty much zero, so it's more of a subjective opinion, and by no means hard fact. The interior of the headband is solid, excellent metal connections and hinges, and I like the fact that I can easily yank out the headphone cord for easier folding and storage.</p> On to the B&W's:</p> Pros§</a> </h4> Solid metal and leather, I don't think there is an ounce of plastic on the headphone at all.</li> Large headband cushion, it's much more encompasing of the head than the PSB, but that isn't as large a point as it sounds, as the PSB really only touches the head in one spot, and it's a bitching cushion.</li> Magnetic cushion holders are a nice touch, and hold extremely well. You've gotta give 'em a solid yank to get the ears off.</li> Leather carry case.</li> </ul> Cons§</a> </h4> The metal arms seem marginally thin on the B&W's, which makes one worry about permanent bending or damage.</li> Leather carry case. I know that I listed the case twice in pros/cons, but while the case is well built, I don't believe it actually protects the headphones much, especially in cramped luggage or some other rock/hard place.</li> </ul> The B&W's have some solid build quality, with excellent drivers and top quality leather. The clamping force is certainly strong, which indicated strong metal retention. The carry case definitely leaves something to be desired, at least when compared to the soft-shell of the PSB. The foam ear pads are okay, but I like the memory foam of the PSB's (as mentioned previously). Still, I think it's a part of the stiffer foam on the B&W's that actually allow for a better fit with my extended ears, because if they compressed more I might run the risk of having my ear tips touch the drivers.</p> </picture> Sound Quality§</a> </h3> Both headphones offer excellent sound quality, so you really can't go wrong with either headphone. Still there is a $100 price difference, so let's breakdown the differences that I noticed.</p> The PSB's had excellent clarity, with defined treble, solid midrange, and articulate bass. These headphones lend themselves particularly well to singing/guitar-focused music, such as Johnny Cash's rendition of "Hurt". You can practically see his fingers on the strings in your mind, and his voice comes to life throughout your entire listening session. That being said, I listen to a wide variety of music, and I felt that these phones didn't tune perfectly (but still damn close) to my listening tastes for genres such as trap and dubstep, along with a few hip-hop songs. This is mostly attributed to the more neutral bass response, which some people may like, but I did not.</p> This leads me to the B&W's. The B&W's have a much more... dynamic sound, I feel. I noticed a clear seperation in the instruments</em> in the music I listened too. I could easily focus on the bass track and then switch to the vocals in my mind, hearing each with precision without one over powering the other. I have to say, if you like bass, get the B&W's. They kick the ear drums off your head when compared to the PSB's, which is something I was actually looking for without realizing it. That is, until I found it in the B&W's. I very much enjoyed listening to the clear highs of The Weeknd</em> while also bumping those sick trap beatzzz of Diplo's "Revolution". As soon as I heard the latter track on the B&W's, I just started smiling in the middle of the Apple store.</p> All said and done, the B&W's fit my</strong> listening tastes better. The PSB's are great, and will probably be preferred by some users, as they are more neutral in their presentation. I just preferred the B&W's dynamic sound and sweet bass response. Is it worth the $100 difference? Maybe, as it certainly was for me. Still, you can't go wrong with the PSB's, and they are pretty much the best you can get in the $300 price range.</p> Amenities§</a> </h3> Without a doubt, PSB takes the cake on this one. The inclusion of spare ear pads, cleaning cloth, airplane dual-prong adapter, 1/4" adapter, and secondary output through the alternate audio port makes for a sick package at $300. I do want to take the time to call out the secondary output feature I mentioned. This means that you can listen to your main audio source on your PSB's through the input on one side of the headphone, but then you can also attach a secondary set of headphones to the port on the otherside of the PSB's, allowing another person to easily listen with you. This is great for car/train/airplane trips with my girlfriend, as we can watch a movie in peace, or jam to the same tunes without effecting other people. This is something that should be present on more headphones, and something I was sad to see missing on the B&W's.</p> Don't get me wrong, the B&W's have a carrying case, two cords (one for iPod control, one for "normal" use), 1/4" plug, and the option to plug your cord into either side of the headphone. However, the PSB's have all of this, and more. Plus, the B&W's require the use of a proprietary, angled connector, which means you have to pay B&W everytime you want a new cord. The PSB's have no such limitation</p> Summary§</a> </h3> So, after this (rather verbose) review, I ended up going with the B&W's, but that is mostly because my ears couldn't handle the PSB's form factor and I liked the bass response on the B&W's more. The PSB's have better amentiies, a better price, and better foam materials, but they just didn't fit me properly. If they fit you, great, buy them and enjoy them.</p> If you like possibly</em> better audio quality, especially with respect to bass, and have large, protruding ears, I have to recommend the B&W's instead.</p> Thanks for listening!</p> Getting mac ready to compile Vapoursynth 2014-05-11T00:00:00+00:00 brew install gcc48 brew install python3</p> CC=/usr/local/bin/gcc-4.8 CXX=/usr/local/bin/g++-4.8 PYTHON=python3 ./waf configure build</p> Running IntelliJ with JDK 1.7 and Scala 2.10 on Mac OS X 10.9 Mavericks 2014-01-02T00:00:00+00:00 Since the combination of OS X, Scala, and IntelliJ appear to be a moving target in the development world, I figured I'd go ahead and jot down how I got them up and running on my MacBook Pro. First things first, I used IntelliJ 13 (Community Edition), with Scala 2.10.3, which I installed using Homebrew (I will detail it below). In addition, I'm using the latest Java 7 JDK for Mac OS X from Oracle.</p> </span>Installing Scala§</a> </h2> First things first, use Homebrew to install Scala, including the documentation, like I have detailed below.</p> $ brew install scala --with-docs </span></code></pre> You will likely see a line about where to point IntelliJ in the post install message, similar to this:</p> ==> Caveats </span>To use with IntelliJ, set the Scala home to: </span> /usr/local/opt/scala/idea </span></code></pre> And that's exactly what we are going to do...</p> Installing Java§</a> </h2> Make sure that you have the latest version of the Java JDK installed, I just downloaded the appropriate JDK from Oracle's site</a>.</p> Preparing and Installing IntelliJ§</a> </h2> In order to get IntelliJ to work properly with the latest version of the Java 7 JDK, you need to edit the .app according to the instructions provided by the IntelliJ authors here</a>. Almost there!</p> Starting a Scala Program§</a> </h2> Now, that you have all of the tools installed, you need to start a Scala module in IntelliJ. Start by opening IntelliJ and starting a new Scala Module. Set the Project SDK path to the location of where you installed Java 7. Mine was:</p> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home </span></code></pre> Next, select Set Scala Home</em>, and point it to the location of your Scala install, which was pointed out to us by Homebrew during our earlier install. In my case it was:</p> /usr/local/opt/scala/idea </span></code></pre> IntelliJ should autodetect the necessary libraries and documentation path. Fill out the remainder of your projects information (name, etc), and then click Finish</em> to get started.</p> Installing XBMC 12.2 on an ASRock ION 330HT 2013-06-28T00:00:00+00:00 Here are some notes for an XBMC 12.2 (Frodo) install job that I did for a buddy of mine on his ASRock ION 330HT. We were both a little nervous about performing a full upgrade, but due to smart partitioning, everything upgraded perfectly.</p> Here's what I did...</p> </span>Backup Prior Install§</a> </h2> Using Clonezilla</a>, I made an image of his current install, which was an Ubuntu 10.04 install with XBMC on top of it. I just went ahead and backed up the first 10G partition on his internal hard drive. Everything went smoothly. A small trick for booting something like Clonezilla from a flash drive with the ION 330HT is to have a keyboard plugged in and then mash the F11 key on bootup. You should see a small boot menu appear and then you can select to boot from the corresponding USB stick.</p> Download and Install XBMCbuntu§</a> </h2> Since he only uses his ION 330HT as a media box, he didn't need a full fledged Ubuntu install. So, we decided to go with XBMCbuntu</a>, which is available for download on the main XBMC page. We installed that ISO onto a USB drive using UNetbootin (for Mac! Didn't know they had this now...).</p> Next we booted the ION 330, mashed the F11 key, and started up the live USB. It's important to select the Fixed DPI (120x120)</strong> option if you are installing with an HD TV attached, as otherwise the installer text is way to freakin' small. With the fixed DPI it's totally manageable.</p> We quickly selected our keyboard and timezone options, and reformated the 10G partition he had from his previous install on the internal harddrive. Then, we made sure that the installer pulled down the latest packages and set his user to automatically login on bootup.</p> Then we hit GO!</p> Rejoice!§</a> </h2> After the install finished, we rebooted into our wonderful new XBMC machine. And frankly, it pretty much worked just fine after that with two small caveats:</p> Automount Drives§</a> </h3> We wanted his drives to automatically mount on bootup, so we can always have his media at the ready. We did this by finding each drive's attached UUID using</p> $ sudo blkid </span></code></pre> After we found the partition's UUIDs, we created mount points under /media</em>. For example</p> $ sudo mkdir -p /media/Media </span>$ sudo mkdir -p /media/Media_640G </span>$ sudo mkdir -p /media/FusionHD </span></code></pre> The first two are for his internal drives, and the last one is for an external he has hooked up over eSATA. Then, I edited the /etc/fstab</em> file to include lines like the following for each drive:</p> UUID=<the_uuid> /media/FusionHD ext4 (or ntfs-3g) rw,user,auto,nosuid,nodev 0 0 </span></code></pre> After that, upon reboot, all necessary drives in the fstab file will automount and appear as media locations in XBMC.</p> Fix HDMI Audio Output§</a> </h3> While almost everything worked out of the box on the ION 330HT, including the remote and display, the audio was a little wonky. My buddy has his ION feed out to a Sony receiver over HDMI, and the audio channels were a little screwed up for his 5.1 setup.</p> Luckily for us, this is a solved issue. All that is necessary is to boot into the XBMCbuntu desktop (which you can do by exiting XBMC (not shutdown!), and then selecting "XBMCbuntu" from the login manager's dropdown menu). Once you are in the desktop, open up the asoundrc file located in your home directory:</p> $ vim ~/.asoundrc </span></code></pre> Simply follow the instructions located in the XBMC wiki here</a>. I backed up the asoundrc file and then copy and pasted the settings right into the file. I completely blew away all of the previous contents.</p> Then reboot. Once you login to XBMC again via that login manager from earlier, you should be able to playback your media over a standard receiver with no issues, as long as you make sure to set your Audio Output Device</strong> to hdmi_direct (which we get from that wiki link). * </p> You should be good to go!</p> A few notes...§</a> </h2> First of all, make sure that you go into the ION 330HT's BIOS and give the shared video memory a full 512MB of RAM. This ensures smooth video performance.</p> Also, if you have DPI issues with XBMCbuntu AFTER you have installed it, simply add a line to your /etc/X11/xorg.conf file under the "Monitor" section. There should be a few other entries and another Option</em> entry, so just add this to the end of the Monitor</em> section:</p> Option "DPI" "120 x 120" </span></code></pre> Then logout and log back in. You should now be able to actually read the OS desktop text on a Full HD television.</p> After all of this was said and done, I booted up Clonezilla once more over a USB and made a backup image of the known working configuration. That way he can fallback on it if anything ever screws up.</p> Pick Your CUDA Device at Program Runtime 2013-05-05T00:00:00+00:00 Select your CUDA Device§</a> </h2> While conducting research on my thesis, there were several times where I wanted to test the efforts of my work on different CUDA cards. Luckily, I had access to a machine that contains 4 different CUDA cards, ranging from a GTX 480, to a Tesla C2050, to a Kepler K20Xm (there are two of these). This is all according to nvidia-smi</strong> of course.</p> </span> Now, all of the kernels that I have been writing recently have been only designed to run on a single device (this could very likely change in the future, but not now). Normally, when you want to run a CUDA kernel on a different device, you select the device number programmatically using the cudaSetDevice() API call.</p> Of course, you might be thingking : "But I have a lot of code! I don't want to make a branch in my git repo, edit all of my source files, recompile, and then run from there!" At least, you should be thinking that...</p> Luckily, there is a solution, and it is one that let's you select the devices you want your CUDA kernels to run on whenever you execute</em> your program. Deep in the CUDA Best Practices guide is a little section on the CUDA_VISIBLE_DEVICES</strong> environment variable.</p> By simply setting the CUDA_VISIBLE_DEVICES</strong> variable to your desired device identifier(s), you can change which device your program runs on. This is great for testing kernels across a number of devices rather quickly, without having to recompile every time.</p> Here is an example:</p> $ CUDA_VISIBLE_DEVICES=1 ./myCUDAProgram </span></code></pre> If your kernels DO support multiple devices, you can provide a list like so:</p> $ CUDA_VISIBLE_DEVICES=1,3 ./myCUDAProgram </span></code></pre> Not too shabby, eh? Now use this in your CUDA testing! You can thank me later...</p> Source: CUDA Best Practices Guide</a></p> Profiling CUDA through Python with NVVP 2013-02-09T00:00:00+00:00 For those of you who don't know, I'm currently pursuing my Masters in Computer Science. Part of this undertaking requires me to complete a thesis.</p> </span> For the purposes of my thesis, I decided to implement a framework into the Vapoursynth</a> video filtering project that will allow it to run as many video operations as possible on a CUDA-enabled GPU.</p> In so doing, I've been doing a lot of CUDA kernel writing and profiling recently. The problem, of course, is that Vapoursynth works through a Python script wrapper around a C/C++ core. So, whenever I want to profile my filters, I have to insert their associated calls into a python script, and obtain their timings through it's execution.</p> Of course, this is a less than ideal solution. What I've (finally) been able to do is profile my CUDA filters through their python script files using NVidia's NVVP (NVidia Visual Profiler) program. It took a few tricks to get it working correctly, especially considering that these scripts output video data</em> directly to standard out (stdout) (which NVVP attempts to display, poorly, on the Console tab of the profiler).</p> Let's get to the tricks:</p> Make sure that a hash-bang is at the top of your script. Ex:</p> #!/usr/bin/env python3 </span></code></pre> </li> Make sure that instead of sending out video output to stdout, we send it to /dev/null on Linux, or NUL on Windows. This prevents NVVP from exploding on the massive amount of video data. Ex:</p> with </span>open</span>(os.devnull, </span>'w'</span>) </span>as </span>f: </span> clip.output(f, </span>y4m</span>=</span>True</span>) </span></code></pre> </li> Be sure to make the script executable, otherwise NVVP won't be able to, well, execute it.</p> example@example-desktop:~/src/testing $ chmod +x test.py </span></code></pre> </li> Start a new profiling session in NVVP, and load your target Python script under the File</strong> path, optionally setting the Working Directory</strong> to the same directory as your script, in case you need to load any external files. After that, you're done! Go ahead and profile your application like you would any other CUDA executable. It may not be able to run all ~28 profiling passes, but it will run most, and give you nice execution time line to boot.</p> </li> </ol> So, to sum up, simply make your target Python script executable, being sure to redirect any output to /dev/null instead of stdout if there is a large amount of data, and then run NVVP like you would a normal CUDA program and enjoy the wealth of analysis tools it has to offer.</p> W3 Total Cache and DONOTMINIFY 2013-02-07T00:00:00+00:00 I use W3 Total Cache for all my of my website caching needs and usually it works great. There have been a few problems with the latest release, but that is for another post.</p> </span> One issue that I just experienced recently with one of my sites is that even though minification (or minifying) was turned on, none of my CSS or Javascript files were actually being minified. When I would "View Source", all of my files where in their original, un-minified form.</p> If I scrolled down to the bottom of the source, where W3 Total Cache inserts its cache statistics, I saw something like this:</p> Minified using disk: basic (DONOTMINIFY constant is defined) </span>Page Caching using disk: enhanced (Cookie is rejected) </span>Database Caching using apc (Cookie is rejected) </span>Object Caching 1293/1305 objects using apc </span></code></pre> The DONOTMINIFY statement jumped out at me. After doing some quick research, I discovered this forum topic</a>, which detailed that the problem is actually in my Wordpress SEO plugin, made by Yoast. More specifically, the "Allow tracking of this WordPress installs anonymous data" option that is available under the plugin's Dashboard, under Tracking, was enabled. By disabling</em> this option, minification works like a charm again.</label></p> So, in summary, if you see the DONOTMINIFY notification in your source code, and you have the Wordpress SEO plugin from Yoast, make sure that the anonymous tracking option is turned off.</p> Reading Unicode from raw_input in Python 2012-10-10T00:00:00+00:00 This took me too damn long to figure out in Python, so a quick note.</p> </span> To obtain a unicode string in from the raw_input() function in Python, simply use a line like the following:</p> unicode_string = raw_input("Enter your input").decode('utf8') </span></code></pre> About the only good thing in iOS 6 2012-10-07T00:00:00+00:00 iOS 6 has been all over the news recently with botched promises and new features (culminating in the new Maps application and the removal of a native YouTube app).</p> </span> Granted, the YouTube-written app is LOADS better, but nonetheless, iOS 6 has been... underwhelming. Or lackluster, whatever you want to call it. Still, there is one thing that I like about iOS 6:not having to enter my password to update my apps</strong>. It might seem like a small thing, but it's the small things that really count.</p> Of course, I'm not convinced that not having to enter my password makes up for the debacle of Apple's new Maps app.... ah, whatever.</p> Just some random thoughts, go back about your browsing...</p> Stopping BOINC from running on boot up in Ubuntu 2012-10-07T00:00:00+00:00 Thank you AskUbuntu:http://askubuntu.com/questions/96787/stop-boinc-from-auto-start-on-system-boot</a></p> I like to control when BOINC runs, and there is no easy to access preference in the options dialog, so this little gem is nice to know.</p> Quick Solutions to iOS 6 Issues 2012-09-21T00:00:00+00:00 With the release of iOS 6, a number of my friends and colleagues are jumping on the update bandwagon. I went ahead and did the same thing, but I heard some horror stories about people losing access to their email (those who had Gmail anyways) and the Youtube app disappearing (a "feature" touted in the iOS 6 release party).</p> </span> Luckily, there is a quick solution, and one that should have been used by anybody who has a Gmail account and was/is using the native Mail app for iPhone. The solution? Get the Gmail app. It has much better integration with the full feature set of Gmail. You can pick it up on the App Store</a>.</p> Now, with the removal of the native Youtube app, a lot of people may be angry, scared, or what have you. I beg to differ, as this is actually a good thing. Now Youtube has the ability to update and fix its own app instead of being at the mercy of Apple every time it wants to release a new feature. Again, you can find the Youtube app on the App Store</a>.</p> Two quick solutions to Apple's crippling of Google related features in iOS 6.</p> Beginnings... 2012-09-19T00:00:00+00:00 I started this blog because I knew that I already had the resources available (considering that my other site is fully up and running, adubvideo.net</a>) and sometimes I need a place to put my random thoughts and a few of the things that I learn while adventuring through the precarious world of technology.</p> </span> For those of you who don't know, I'm a Graduate Student at California Polytechnic State University. I've been there five, going on six years, first pursuing my B.S. in Computer Engineering, and now getting my M.S. in Computer Science.</p> Throughout my time there, I've learned a lot and experimented more. One thing is for sure, my experiments lead to more issues and discoveries than any class could ever offer, which is why I plan on posting my results and discoveries to this blog whenever I stumble across them.</p> Until then, good day!</p>

Austin's Thought Basin

Converting a website to an ebook

2022-05-30T00:00:00+00:00

I recently ran through the ups and downs of taking a very well written blog and converting it to an EPUB ebook in order to read it on my Kindle.

Using a few command line applications I was able to produce a rather well formatted EPUB file that I could send to my Kindle using Amazon's Email to Kindle</a> service.

The general principal can be applied to almost any website that renders using HTML on the server side (throws shade at SPAs), so blogs are often perfect for this, or Wikipedia articles, whatever suits your fancy.

Step 1 - Collect your URLs§</a> </h2>
Start by collecting all of the URLs that you want into a single file, with each URL separated by a newline. I called my file urls</code> (original I know).
Example of urls</code> file:
https://austindw.com/harden-openwrt/ https://austindw.com/precompressed-assets-caddy/ </code></pre>Step 2 - Create the book§</a> </h2> Wow, two steps and we're done? Kinda. I wrote up a script to do the rest of the steps in one fell swoop. It requires a few tools be installed, specifically: readability-cli</code> Homepage</a></li> For Arch Linux users, this package is provided in the AUR, so a simple yay -Sa readability-cli</code> should be all you need.</li> </ul> </li> pandoc</code> The macdaddy of document processors as far as I can tell. It's available on most Linux distro's in the package manager.</li> For Arch Linux users, a simple pacman -S pandoc</code> will do the trick.</li> </ul> </li> </ol> Now that we have the required dependencies, let's take a look at the script: #!/bin/sh set -eu # Point this at whatever file your URLs are stored in. urls="urls" # Make the directory where we'll store the clean HTML for each post. mkdir -p posts # Iterate over the URLs, download them and clean them up with the readability-cli # We use a count here to ensure that we organize the output posts in the same order that they are specified # in the input file. This is helpful as you can lay out the full order of your book by just editing the URLs file. count=1 cat "$urls" | while read url do output=$(printf "posts/%03d.html" $count) readable -q --low-confidence force "$url" -o "$output" 2>&1 > /dev/null count=$((count+1)) done # Take all of the posts and put them into a book. pandoc -o TheBook.epub posts/*.html </code></pre> Save, make executable (chmod +x script.sh</code>), and run the script and you'll have a fresh new EPUB named TheBook.epub</code> at the end of it. Open that up in your preferred ebook reader and you'll see a nicely formatted output like the following: </picture> Obviously, you should edit the script to your needs. This post just acts as a guide, demonstrating the power of the readability-cli</code> and pandoc</code>. Note: I used the --low-confidence force</code> option when executing the readable</code> command as I noticed a few posts that failed to extract with the default keep</code> setting. Using force</code> here worked just fine for my use case. Obviously, this may or may not work for your use case, so experiment.

Encoding Snippets 2022-03-12T00:00:00+00:00 I do a lot of encoding work, but I seem to lose track of useful encoding snippets from time to time. This post will act as a central repository for me to organize my snippets. FFMPEG§</a> </h2> Downscale video using spline36§</a> </h3> ffmpeg -i input.mp4 -vf zscale=w=1920:h=-1:filter=spline36 out.mp4 </code></pre> Denoise using hqdn3d or fftdnoiz§</a> </h3> hqdn3d, with a luma_spatial of 6 (default is 4). ffmpeg -i input.mp4 -vf hqdn3d=luma_spatial=6 out.mp4 </code></pre> fftdnoiz, sigma 4, using 1 previous and 1 future frame for temporal filtering ffmpeg -i input.mp4 -vf fftdnoiz=sigma=4:prev=1:next=1 out.mp4 </code></pre> Downscale and denoise§</a> </h3> ffmpeg -i input.mp4 -vf zscale=w=1920:h=-1:filter=spline36,hqdn3d=luma_spatial=6 out.mp4 </code></pre> Output using x264, copying audio§</a> </h3> ffmpeg -i input.mp4 -c:a copy -c:v libx264 -preset slower -tune film -crf 21 out.mp4 </code></pre> Enable video fast start by adding movflags to MP4 file§</a> </h3> ffmpeg -i input.mp4 -movflags +faststart out.mp4 </code></pre> This enables videos to start playing as soon as the first few bytes are sent to the client, instead of requiring the client to download the full video. Process video with Vapoursynth, encode video and copy audio using FFMPEG§</a> </h3> This is useful to handle video processing with Vapoursynth (denoise, deinterlacing, sharpening, etc etc) while still maintaining the audio from the original source. vspipe --arg 'source=./input.mp4' -c y4m script.py - | ffmpeg -i - -i input.mp4 -map 0:v -map 1:a -c:a copy -c:v libx264 -preset fast -tune film -crf 20 out.mp4 </code></pre> The magic is in the -map 0:v -map 1:a -c:a copy</code> line. map</code> is used to select from multiple sources, so in the case we're saying "select video from the first input (raw pipe from Vapoursynth), select audio from the second input (the same file we fed to Vapoursynth), and copy the audio". VSPIPE§</a> </h2> vspipe</code>, used for reading Vapoursynth (Python) files and feeding them to other programs. Feed vspipe output to MPV in a paused state§</a> </h3> vspipe -c y4m script.vpy - | mpv --pause --demuxer-rawvideo-codec=y4m - </code></pre> Feed vspipe to x265§</a> </h3> vspipe -c y4m script.vpy - | x265 --y4m [...] -o out.h265 </code></pre> Feed vspipe to x265 with 10-bit video§</a> </h3> vspipe -c y4m script.vpy - | x265 --y4m --input-depth 10 [...] -o out.h265 </code></pre> Feed vspipe to ffmpeg§</a> </h3> vspipe -c y4m script.vpy - | ffmpeg -i - [...] out.mp4 </code></pre> Pass arg to Vapoursynth script from vspipe§</a> </h3> vspipe --arg "source=./input.mp4" --info script.vpy - </code></pre> The arg takes the place of a python variable, so it works with something like the following script: import vapoursynth as vs core = vs.core clip = core.lsmas.LWLibavSource(source) # Note that 'source' is read from the --arg param # ... do processing here clip.set_output() </code></pre> Note: The --info</code> just quickly validates that the passed arg is legit, and prints out the video statistics. You'll remove it when actually feeding to something like ffmpeg. Hardening OpenWrt 2021-09-26T00:00:00+00:00 For a few reasons I've fallen down the OpenWrt hole again of late. While working on some custom builds, I noticed some areas where I could improve the security of my devices significantly. To be honest, I'm a little surprised that OpenWrt doesn't do some of them by default. We'll walk through various methods that most people can use to harden their OpenWrt install. This guide assumes you know how to either SSH into your router or access the Luci web interface, as well as install packages using opkg</code> or the System -> Software</code> menu. If you don't know how to do these things, simply read the OpenWrt quick start</a>. When installing Luci applications (any package with a name starting with luci-app-</code>), you may need to logout and log back in if the expected menu entries don't show up at first. Restrict SSH access to the local network (LAN)§</a> </h2> First things first, let's lockdown the SSH server (OpenWrt uses "dropbear" by default) so that it only listens on local network. By default, it listens on all interfaces, which includes the WAN/internet interface. This means that anyone on the web could potentially log into your router if they knew your IP address. They'd likely have to brute force your admin password (you did set one right?), unless you locked down the SSH server to use public key authentication. Regardless, I don't want to given the public internet the option of logging into my device. I have no personal need to access my router when I'm away from home, so let's shut it down entirely. Note: The OpenWrt firewall does not open the SSH ports in the WAN zone by default, so the risk is low here. But this extra layer of security never hurts. Method 1 (uci)§</a> </h3> SSH into your router (likely something like ssh root@192.168.1.1</code>)</li> Type the following to configure dropbear to only listen on the local lan:</li> </ol> uci set dropbear.@dropbear[0].Interface="lan" uci commit dropbear service dropbear restart </code></pre> Method 2 (Luci)§</a> </h3> Login to Luci (open your browser and go to http://192.168.1.1</code>, enter your password)</li> Navigate to System -> Administration -> SSH Access</code>.</li> Select LAN</code> from the Interface</code> menu.</li> Click Save and Apply</code>.</li> Go to System -> Startup</code> and click the Restart</code> button next to dropbear.</li> Note: Luci may actually restart dropbear for us, so this last step might be unnecessary.</li> </ol> </picture> Restrict Luci/uhttpd access to the local network, access over HTTPS.§</a> </h2> Just like SSH/dropbear, the Luci web interface (uhttpd server) exposes itself to the public web by default. Assuming you have no need to login to your router when away from home, let's lock it down to only be accessible on the local network. Note: The OpenWrt firewall does not open the HTTP/HTTPS ports in the WAN zone by default, so the risk is low here. But this extra layer of security never hurts. OpenWrt Wiki reference for 'Securing uHTTPd'</a> Method 1 (uci)§</a> </h3> Listen to local LAN IPs and enable HTTP to HTTPS redirects. . /lib/functions/network.sh # Load the list of ipv4/ipv6 addresses into local variables for easily handling. network_get_ipaddrs ip4addrs 'lan' network_get_ipaddrs6 ip6addrs 'lan' uci delete uhttpd.main.listen_http uci delete uhttpd.main.listen_https for ip4 in $ip4addrs; do uci add_list uhttpd.main.listen_http="$ip4:80" uci add_list uhttpd.main.listen_https="$ip4:443" done for ip6 in $ip6addrs; do # Only add the link local/ULA IPV6 address, which starts with an 'f' if [ -n "$(echo "$ip6" | grep '^f')" ]; then echo "$ip6 starts with f" uci add_list uhttpd.main.listen_http="[$ip6]:80" uci add_list uhttpd.main.listen_https="[$ip6]:443" fi done # (Optional) Ensure that we use/redirect to HTTPS always uci set uhttpd.main.redirect_https='1' uci commit uhttpd service uhttpd restart service uhttpd enable </code></pre> Note: This sets up the uhttpd to listen on the link-local/ULA IPV6 address as well. The OpenWrt documentation (linked above) suggests disabling the IPV6 address completely. Do whatever suits your preferences. Listening to the local IPV6 address doesn't seem like a risk to me. Listening to the global IPV6 address would be more of a concern but we're not doing that here. Method 2 (Luci)§</a> </h3> Install the luci-app-uhttpd</code> package.</li> Go to Services -> uHTTPd</code> page.</li> Fill in the HTTP and HTTPS listener sections with IPv4 and IPv6 addresses. You can set the IPv4 address to just 192.168.1.1</code> or whatever you've configured your router's LAN IP to be.</li> Make sure to use brackets to surround the IPv6 addresses and add the port number at the end, like [fd88:38bc:dd39::1]:443</code>.</li> </ol> </li> (Optional) Click the Redirect all HTTP to HTTPS</code> checkbox if you want to use HTTPS with a self-signed certificate.</li> Click Save and Apply</code>.</li> </ol> </picture> BCP38§</a> </h2> This is less about protecting your router and more about preventing your network's contribution to some types of DoS/DDoS attack. For more info I highly recommend the BCP38 info website</a> which explains the root problem quite well. Method 1 (uci)§</a> </h3> Install the bcp38</code> package.</li> Run the following uci</code> commands</li> </ol> uci set bcp38.@bcp38[0].enabled='1' uci set bcp38.@bcp38[0].interface='wan' #Adjust to match your WAN port - may be something like eth0.2 for example. uci commit bcp38 service bcp38 restart service bcp38 enable </code></pre> Method 2 (Luci)§</a> </h3> Install the luci-app-bcp38</code> package.</li> Navigate to Network -> Firewall -> BCP38</code>.</li> Click the checkbox to enable.</li> Select the interfaces that pertains to your WAN (likely labeled wan,wan6</code>).</li> Click Save and Apply</code>.</li> </ol> </picture> banIP§</a> </h2> It's highly recommended to install and configure banIP using Luci as the core configuration is a little verbose when done from the commandline. I say that, but I've been configuring it from the commandline for a while now, so feel free to copy-paste my configuration below - it should work well. Note: This package recommends that your router has at least 128MB of RAM. banIP project page/documentation</a> banIP Support Thread</a> Method 1 (uci)§</a> </h3> Install the banip</code> package.</li> Enable, and configure a core set of blocklists:</li> </ol> uci set banip.global.ban_enabled='1' uci set banip.global.ban_trigger='wan' #Configure to your proper WAN interface, eg. 'eth0.2' or 'wan' uci delete banip.global.ban_sources uci add_list banip.global.ban_sources='darklist' uci add_list banip.global.ban_sources='debl' uci add_list banip.global.ban_sources='drop' uci add_list banip.global.ban_sources='feodo' uci add_list banip.global.ban_sources='firehol1' uci add_list banip.global.ban_sources='greensnow' uci add_list banip.global.ban_sources='iblockads' uci add_list banip.global.ban_sources='iblockspy' uci add_list banip.global.ban_sources='myip' uci add_list banip.global.ban_sources='nixspam' uci add_list banip.global.ban_sources='proxy' uci add_list banip.global.ban_sources='sslbl' uci add_list banip.global.ban_sources='talos' uci add_list banip.global.ban_sources='threat' uci add_list banip.global.ban_sources='tor' uci add_list banip.global.ban_sources='uceprotect1' uci add_list banip.global.ban_sources='yoyo' # Allowlist a few sites that I've seen some blocklists break before. echo "enro.com" >> /etc/banip/banip.whitelist echo "www.reddit.com" >> /etc/banip/banip.whitelist echo "yelp.com" >> /etc/banip/banip.whitelist uci commit banip service banip restart service banip enable </code></pre> This enables banIP with a set of blocklists that I recommend. The iblockads</code> and yoyo</code> lists specifically target ads, and may be redundant depending on your networking setup (like if you're running PiHole or AdguardHome), but they can help regardless. I have seen some false positives come out of iblockads</code> so be sure to use the whitelist functionality if you notice any breakages, or just disable its use completely. Bonus - setup cronjob to refresh blocklists§</a> </h4> I recommend refreshing the blocklists every 24 hours. Here's an example of how to configure a cronjob to run once a day: crontab - <<EOF 15 07 * * * /etc/init.d/banip reload EOF </code></pre> NOTE: This OVERWRITES your existing crontab. If you already have other cronjobs, either edit the crontab manually or add them to this command as appropriate. Method 2 (Luci)§</a> </h3> Install luci-app-banip</code>.</li> Navigate to Services -> banIP</code>. </li> You can leave most settings at default (banIP autodetection is pretty good), but you'll need to both enable banIP and configure blocklists directly.</li> Enable banIP - just check the Enabled</code> box on the General Settings</code> tab.</li> Configure blocklists - go to the Blocklist Sources</code> tab and use the Sources</code> tab to select your desired lists. You can see an example of the lists I personally use in the command used in Method 1 above.</li> Click Save and Apply</code>.</li> </ol> Enabling banIP: </picture> Configuring blocklists: </picture> Bonus - setup cronjob to refresh blocklists§</a> </h4> Just like Method 1, its possible to setup a cronjob to refresh the blocklists. Click Refresh Timer</code> button on the main status page.</li> Set the action to reload</code> (not "refresh", confusing I know...)</li> Pick an hour in the 0-23 time range that works for you.</li> </ol> Here's an example of my own configuration: </picture> Disable or remove UPNP§</a> </h2> If you aren't using UPNP (a service to automatically punch holes in your firewall...), you should disable it completely, and/or remove the package completely. Navigate to Services -> UPnP</code>.</li> Uncheck the Start UPnP and NAT-PMP service</code> checkbox.</li> Save and Apply.</li> (Optional) Navigate to System -> Startup</code> and stop + disable the upnp</code> (or miniupnp</code>) service.</li> (Optional) Remove the package completely using the System -> Software</code> menu or opkg --autoremove remove luci-app-upnp</code></li> </ol> Firewall DROP external connections§</a> </h2> By default the OpenWrt firewall configuration is kinda "noisy" in how it rejects traffic from the global internet. You can see this in action by using Steve Gibson's ShieldsUP tool</a>. Click on the Proceed</code> buton and then use the Common Ports</code> or All Service Ports</code> buttons to scan your whole router. The buttons are kind of hard to see, I admit, but they should be there. Let's configure the firewall to silently drop all external connections instead of gracefully telling them "no thank you". Method 1 (uci)§</a> </h3> # Note: I'M ASSUMING THAT THE LAST ZONE IS THE WAN ZONE. THIS BREAKS IF THE WAN ZONE IS ON A DIFFERENT ZONE NUMBER. # A little safety check - only try and set the input DROP on WAN. Skip/bail if the last interface isn't the WAN. if [ $(uci get firewall.@zone[-1].name) = 'wan' ]; then echo 'Locking down firewall' uci set firewall.@zone[-1].input='DROP' uci set firewall.@zone[-1].forward='DROP' uci commit firewall service firewall restart fi </code></pre> Method 2 (Luci)§</a> </h3> Navigate to Network -> Firewall</code>.</li> Under Zones</code>, set the wan => REJECT</code> zone to drop</code> (instead of the default reject</code>) on both Input</code> and Forward</code> using the associated dropdowns.</li> Click Save and Apply</code>.</li> </ol> </picture> Conclusion§</a> </h2> Hopefully this helps secure your home router just a little bit more. OpenWrt is already significantly more secure than the standard firmware that ships from the factory for most routers, but these tweaks should lock it down to the utmost degree. A few other references that may be helpful: Accessing LuCI web interface securely - OpenWrt wiki</a></li> Secure your routers access - OpenWrt wiki</a></li> </ol> Precompressed Assets with Caddy 2021-05-15T00:00:00+00:00 Serving precompressed assets with Caddy</a> has historically required some configuration gymnastics. A good example can be found in a previous post</a> of mine. That all changes in Caddy 2.4.0</a> Native Precompressed Asset Serving§</a> </h2> As seen in the above-linked 2.4.0 release notes, Caddy added support for serving "precompressed sidecar files". Unfortunately, they fail to point to a more useful section of their documentation that actually describes how to configure Caddy to serve said assets. The real documentation lies under their file_server</code></a> directive. If you want the TL;DR version, here you go: file_server { precompressed br zstd gzip } </code></pre> The above will handle an incoming request for an asset by analyzing the Accept-Encoding</code> header provided by the client (browser) and searching for files ending in .br</code>, .zst</code>, and .gz</code>, respectively. And yes, the order that you specify in the precompressed</code> field does matter, so pick wisely. If you aren't sure, just use what I have above. Once Zstandard has widespread browser adoption (it doesn't yet), and established dictionaries for compressing HTML/CSS/JS, then it will make more sense to prioritize zstd</code> before br</code> (Brotli compression). But we're unfortunately a few years out from then. Note that this feature works wonderfully with file_server</code>'s index</code> attribute. This means that if you use "clean urls" (without an ending index.html</code> for example, like https://austindw.com/precompressed-assets-caddy/</code>), precompressing your index.html</code> files and serving them with this option will work correctly. This required extra configuration to get right using the old way. Before and After§</a> </h2> Just to provide some context on why this is such a nice improvement let's look at the old configuration compared to the new configuration. Before§</a> </h3> example.com { root * /path/to/dir file_server ### Precompression support @brotli { header Accept-Encoding *br* file { try_files {path}.br {path}/index.html.br {path}.html.br } } handle @brotli { header { Content-Encoding br Content-Type text/html } rewrite {http.matchers.file.relative} } @gzip { header Accept-Encoding *gzip* file { try_files {path}.gz {path}/index.html.gz {path}.html.gz } } handle @gzip { header { Content-Encoding gzip Content-Type text/html } rewrite {http.matchers.file.relative} } @html { file path *.html */ } header @html { Content-Type text/html defer } @css { file path *.css } header @css { Content-Type text/css defer } @js { file path *.js } header @js { Content-Type text/javascript defer } @svg { file path *.svg } header @svg { Content-Type image/svg+xml defer } @xml { file path *.xml } header @xml { Content-Type application/xml defer } @json { file path *.json } header @json { Content-Type application/json defer } } </code></pre> After§</a> </h3> example.com { root * /path/to/dir file_server { precompressed br zstd gzip } } </code></pre> Pretty dope improvement, no? That's all for today - go forth and precompress. Load Testing for Fun and Profit 2021-04-04T00:00:00+00:00 I've been doing a lot of load testing at work recently, using an internal load generation and measurement tool. Since I seem to be so obsessed with making this website faster</a>, I decided to run some numbers to get an idea of its true performance. A little context§</a> </h2> Let me start by saying that load testing is an imperfect science. The tests that I'm going to detail today don't properly represent the real-world user experience of someone actually navigating around this website. There are lots of other factors that impact said user's percieved performance on this site, including their browser, my Cache-Control headers, their proximity in the world relative to my server location, the list goes on. But these tests do provide a general idea of this website's max requests (or transactions) per second (TPS), or roughly how many people can access the home page at a time. This not only measures the relative speed at which a user loads my website, but also provides a realistic expectation of how my site will perform if it ever blows up on Hacker News, for example. It should be noted that a better measurement of the perceived user experience speed can be found with a tool like PageSpeed</a>. The tool§</a> </h2> Since I'm not at work, I decided to use a more rudimentary (but still quite good) load testing tool called wrk</a>. That's right, I'm not at work, so I do wrk... eghm. Wrk let's me configure the number of threads, connections, duration, and headers with which to hit my server, Caddy</a>. Since I can specify different headers, I can use the Accept-Encoding</code> header to tune which version of my home page I want Caddy to serve, meaning the raw HTML, a gzip compressed version, or a Brotli compressed version. As explained in this delightful post</a>, wrk's use of threads + connections means that the total number of connections as specified on the commandline are split up evenly across the total number of threads specified on the commandline. For example, if I used 75 connections with 25 threads, this means that each thread would handle responses for 3 connections. In my case, with a little trial and error, I tweaked my connections and threads until I had a stable benchmark that saturated my 100 megabit home internet connection. The tests§</a> </h2> Like I mentioned above, I ran three test runs for each "version" of my home page. One test for the uncompressed HTML, and then two more tests for the gzip and Brotli compressed assets, respectively. I say "version", because I tweaked my static site generator to also produce a minified version of my home page, and then ran three more tests against that. I wish I could leave HTML minification on full time, but alas, a bug in Zola's minification dependency</a> causes my <code></code> and <pre></code> blocks to collapse to a single line. Once a new version of Zola is out with the fixed dependency, I'll definitely be upgrading and activing HTML minification. The results§</a> </h2> Okay, enough talking, let's get to the data. Test Setup§</a> </h3> Download speed of home connection, measured using speedtest</code> CLI: 12.57 MB/s</code></li> Latency, as measured in same speed test: 15.78 ms</code></li> CPU (load test host): Ryzen ThreadRipper 3960x (24c/48t), performance governor activated</code></li> Mem (load test host): 64GB 3600Mhz DDR4</code></li> CPU (web server): Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz</code></li> Mem (web server): 989MB</code></li> All compressed / minified assets are compressed or minified offline, meaning that the server is doing no extra work.</li> </ul> The web server is hosted on a cheap $5.00/month, 1GB/1CPU, 25GB SSD, 1000GB transfer droplet at DigitalOcean. Tests were run for 5 minutes each, late at night on a quiet connection. Commands§</a> </h3> wrk -t 25 -c 75 -d 300s --latency https://austindw.com wrk -t 25 -c 125 -d 300s --latency -H "Accept-Encoding: gzip" https://austindw.com wrk -t 25 -c 125 -d 300s --latency -H "Accept-Encoding: br" https://austindw.com </code></pre> Data§</a> </h3> Metric</th> raw HTML</th> GZIP compressed</th> Brotli compressed</th> minified HTML</th> minified + GZIP</th> minified + Brotli</th></tr></thead> Size (bytes)</td> 16781</td> 3712</td> 3002</td> 11108</td> 3374</td> 2765</td></tr> Threads</td> 25</td> 25</td> 25</td> 25</td> 25</td> 25</td></tr> Connections</td> 75</td> 125</td> 125</td> 125</td> 125</td> 125</td></tr> Requests per sec</td> 752.82</td> 2963.38</td> 3193.88</td> 1130.91</td> 3070.17</td> 3246.17</td></tr> Transfer per sec (MB)</td> 12.22</td> 11.27</td> 9.98 MB</td> 12.23</td> 10.69</td> 9.41</td></tr> P50 (ms)</td> 98.34</td> 40.93</td> 37.61</td> 67.70</td> 38.90</td> 37.39</td></tr> P90 (ms)</td> 114.28</td> 52.35</td> 50.94</td> 74.82</td> 52.63</td> 50.10</td></tr> P99 (ms)</td> 151.74</td> 79.53</td> 75.61</td> 79.71</td> 83.91</td> 70.37</td></tr> Web server CPU 5min load</td> 0.41</td> 0.74</td> 0.87</td> N/A</td> N/A</td> N/A</td></tr> </tbody></table> Takeways§</a> </h2> It should have been pretty obvious, but of course minifying and compressing HTML with Brotli results in the smallest file sizes, and thus the fastest response times. Being able to handle ~3200 users requesting my home page in a single second at $5/month ain't nothin' to sneeze at, that's for sure (and another reason why static site generators are just the best). What is interesting is that the difference between GZIP and Brotli wasn't as significant as I would have expected, but looking at the 5min CPU load and how the "Transfer per sec" dropped as response sizes got smaller hints at why - my server was starting to bottleneck on its single CPU core. To confirm this, I ran another test with a raw text file containing the word "hello", which measured at a total of 6 bytes. The result? 4077.37</code> requests/sec. At first glance, this might seem like a big difference when compared against minified Brotli's 3246</code> request/sec. But considering that the minified + Brotli compressed asset is 2765 bytes, or 460x larger than the 6 byte text file, and both results are within ~800 TPS of each other, it would appear that the CPU on my droplet (which was pegged at 100% pretty much the whole time for both tests) is much more the culprit than the response size or my network speeds. This means that I'd expect even better performance out of the minified + Brotli compressed assets with a beefier CPU/multi-core setup even if all other specs remained the same (network, memory, etc). Summary§</a> </h2> In summary, this website really is fast. Because science says so. One thing I've learned after my 9+ years working in software, it's always important to measure and thus validate your assumptions. Before this, my website always seemed fast, but now I know I can easily handle over 3000 visitors in a single second. This means I've got a lot of headroom on this blog before I'd need to worry about upgrading to a bigger droplet. Was this testing methodolgy perfect? No. Will this web server actually be able to handle 3000 visitors in one second? Maybe. There's a lot that can effect that answer, but these results are very much "good enough", at least for now. This website is (slightly) faster 2021-02-07T00:00:00+00:00 As many of you may be aware, the public internet is a bloated, Javascript-riddled mess. If you don't believe me, I'm not the first one</a> to talk about this</a>. And I certainly won't be the last... You'd think with how much internet speeds have increased in the past 20 years, particularly when compared to the dialup internet that many of us may have grown up with, we'd be loading every website with blinding speeds. But that's just not the case, now is it? There's a multitude of reasons for this, including things like advertising, JavaScript, tracking pixels, the list goes on... Today, I'm going to talk about a much smaller piece of this pie - the actual size of the HTML document you're reading, and how it gets delivered to your browser. Background.§</a> </h2> Let's get some quick background. When you point your browser at my website, you kick off a 3 step proccess. I'm hand waving some things here (like TCP connections and TLS handshakes), so just roll with it if you're more technically inclined. Steps: Browser makes a request to the server, while also specifying what types of content encoding it can accept.</li> The server receives the request, analyzes that list of allowed content encodings, and then finds the requested asset (think an HTML web page or CSS stylesheet).</li> Having found the asset, the server makes sure it can "fit" into the desired content encoding specified by the browser, and may take steps to optimize the target asset for a particular content encoding.</li> </ol> Now, when I say "content encoding" here, I'm really meaning "compression". Specifically, many web servers support serving content using gzip</a> compression. This is an old standard, works pretty well for text data, is relatively fast for a web server to do on the fly, and "just works". Now, it's not guaranteed that you receive a gzip-compressed blob of HTMl for every website your visit. It depends on your browser and the web server configuration. So why is this website "faster"?§</a> </h2> Because I made my compression "more better". Yeah, I said it. More better</del> specifically, I optimized 3 key aspects of how my website assets are compressed: Utilized a better gzip compressor known as Zopfli</a>.</li> Added Brotli</a>, an even more efficient compression algorithm than gzip.</li> Compressed all assets before the web server ever sees them, meaning less work for it to do.</li> </ol> To be clear, I was already using compression via Caddy's encode gzip zstd</code> option, but that requires all compression to be done on the fly (and thus repeated for every duplicate request), doesn't support Brotli, and uses a sub-par gzip compression algorithm (when compared to Zopfli). In summary, I'm increasing existing compression, utilizing more advanced compression, and pre-compressing all of my pertinent text-related assets of this website (things like HTML, CSS, and even SVG images, if I had any...). Great, so how did you do this?§</a> </h2> It all kind of started when I got a little frustrated with my web server, Caddy</a>. Now to be clear, Caddy is an excellent web server. I just wanted to replicate an existing setup I had on my old nginx server where I pre-compressed a number of assets before uploading them. I'd looked at doing this a few times over the past year, but ran into a few roadblocks along the way, including: Older versions of Caddy (v1) made this easy, but this feature was removed in a large rewrite for v2, which is what I use.</li> It was theoretically possible using their new configuration syntax, but no one had fully put the puzzel pieces together yet.</li> </ol> Luckily, if you wait long enough, someone else figures out how to get something done on the internet. Actually, several someones. After doing another web search on handling pre-compressed assets with Caddy, I noticed some forum posts from the past year, and eventually stumbled on this excellent blog post</a>. See the bottom of that blog post for links to the pertinent forum posts. </blockquote> By solving the "serving pre-compressed assets" problem, getting better gzip compression with Zopfli and adding Brotli to the mix was the easy part. Updating Caddy to serve pre-compressed assets.§</a> </h3> Update 2021-05-15: Serving precompressed assets got a LOT easier in Caddy 2.4.0. Read more here</a> </blockquote> This is copied pretty much verbatim from the above blog post, but here is the configuration I added in order to get Caddy to correctly serve pre-compressed Brotli and gzip compressed assets whenever possible. example.com { root * /path/to/dir #I specifically removed the following 'encode' option, since I'm doing compression #myself and Caddy was gzip-ing my PNG and JPEG images with it on (which is wasted CPU effort). #encode gzip zstd ### Precompression support @brotli { header Accept-Encoding *br* file { try_files {path}.br {path}/index.html.br {path}.html.br } } handle @brotli { header { Content-Encoding br Content-Type text/html } rewrite {http.matchers.file.relative} } @gzip { header Accept-Encoding *gzip* file { try_files {path}.gz {path}/index.html.gz {path}.html.gz } } handle @gzip { header { Content-Encoding gzip Content-Type text/html } rewrite {http.matchers.file.relative} } @html { file path *.html */ } header @html { Content-Type text/html defer } @css { file path *.css } header @css { Content-Type text/css defer } @js { file path *.js } header @js { Content-Type text/javascript defer } @svg { file path *.svg } header @svg { Content-Type image/svg+xml defer } @xml { file path *.xml } header @xml { Content-Type application/xml defer } @json { file path *.json } header @json { Content-Type application/json defer } } </code></pre> While long, it's actually not a scary config. It basically tells Caddy to serve pre-compressed assets if they exist, while also fixing the Content-Type</code> header so that we correctly identify *.html</code> files as text/html</code> instead of compressed binary data. Compressing assets using Zopfli§</a> </h3> Now that the web server can correctly serve my assets, I tested a few iterations of the Zopfli before adding it to my web publishing workflow. I already optimize my website's images using advpng</code> and jpegtran</code>, so I just added to my existing pattern. Also, for those who may be confused - Zopfli is basically a "better gzip compressor". It produces files that any standard gzip utility can decompress, but it does so using more advanced algorithms than the standard gzip</code> utility. If you want to get started, here's some useful commands for testing zopfli. zopfli -c index.html > index.html.gz </code></pre> You'll note that I'm using a redirect here (></code>) to produce a new file. By default, zopfli overwrites the existing file, which I want to preserve as a fallback option on my web server in case the user's browser doesn't support compression. So instead, I force zopfli to output to stdout by using the -c</code> flag, and then use shell redirection to create a new file. Do it in parallel§</a> </h4> Here's the fancy fd</code> command that I came up with that will automatically find all of my text assets in my website publishing directory (public/</code>) and compress them using zopfli: fd -e html -e xml -e css -e js -e svg -e txt -e json --exec sh -c 'zopfli -c {} > {}.gz' \; . public/ </code></pre> By default, fd</code> will utilize all of your CPU cores to run whatever command you specify in bulk. This allows me to very quickly compress my entire website. Compressing assets with Brotli§</a> </h3> Brotli produces even smaller files than Zopfli, and has been accepted as an IETF standard</a>, meaning that most web browsers support it by default now. In order to create Brotli files for my assets like I did with Zopfli above, I came up with the following: brotli -Z -f index.html </code></pre> Unlike zopfli, brotli doesn't overwrite its input file, so the above command will produce an index.html.br</code> file by default. For good measure I added -Z</code> to always use max compression, and -f</code> to force overwriting of any existing files. This is useful because I statically generate this site from a set of Markdown files, and I'd rather not have to delete the existing output directory every time just to keep Brotli happy. Do it in parallel, again§</a> </h4> fd -e html -e xml -e css -e js -e svg -e txt -e json --exec brotli -Z -f {} \; . public/ </code></pre> Same style of fd</code> command as above... Results§</a> </h2> For just a quick comparison, I compressed the home page of this blog with Zopfli and Brotli. Here's the number of bytes in four forms: uncompressed, compressed with Brotli, and compressed with Zopfli, and compressed with standard gzip using gzip -9</code> for maximum compression: 17,817 index.html 2,607 index.html.br (85.37% improvement over uncompressed) 3,308 index.html.gz (81.43% improvement over uncompressed) (zopfli) 3,404 index.html.gzip (80.89% improvement over uncompressed) (gzip -9) </code></pre> As you can see, compression in general results in significant savings. Brotli unsurprisingly takes the cake, followed by Zopfli and standard gzip as dead last. That said, the improvement of Zopfli over standard gzip is marginal in this case, but when applied to all text assets across the entire website, you can shave a few percentage points off your web traffic. Conclusion§</a> </h2> So yeah, you're likely reading this via a Brotli-compressed HTML page, or maybe even a Zopfli-compressed HTML page. In addition to the total lack of Javascript, trackers, and any other 3rd party requests, you can enjoy a lighting fast "plain text" experience on this website. Cheers! NetworkManager is bloat 2021-01-31T00:00:00+00:00 Or how I came to love dhcpcd</code> and iwd</code> for all of my networking needs... Background§</a> </h2> I have a weird fascination with running as little software as possible on my machines. For whatever reason, minimizing RAM usage scratches some weird itch of mine. And no, it's not infected. At least, I hope not... Anyways, after watching a video on installing Arch Linux</a> for fun (don't judge me), the presenter Ermanno (who's videos I VERY HIGHLY RECOMMEND) covered a change to the install process that Arch Linux made recently. Or I suppose a more accurate description is a change that Arch Linux made to how to connect to a wireless network when you're installing. Keep in mind that the standard Arch Linux install process doesn't have a GUI or any desktop environment with a graphical installer. There's no comfy icon to click on to configure networking, nothing. Out of curiousity, I started looking into this new replacement sofware - iwd</code>. Once I started falling down that rabbit hole, I discovered just how unnecessary NetworkManager</code> is in my setup. Why this works for me§</a> </h2> Like I said, I run a minimal setup. This means I don't need nor want big fancy GUI's for my tools. To give an example, I run: A tiling window manager (i3, bspwm, sway, river are all on regular rotation...)</li> A CLI or ncurses-UI for connecting to wifi (this was nmtui</code> when I was using NetworkManager</code>).</li> A CLI VPN with no visual feedback or need to graphically configure said VPN.</li> I don't connect to esoteric modems like PPPOE, etc from my laptop or desktop.</li> </ul> With all this in mind, all of the "extra" features (read: bloat) of NetworkManager</code> mean that I'm paying a pretty steep cost in terms of dependencies and memory usage for things I simply don't need. Note that there are some GUI tools for dhcpcd</code> and iwd</code> at this point, so check those out on the Arch Wiki if you're really curious. </blockquote> Arch Linux Wiki entry on iwd</a></li> Arch Linux Wiki entry on dhcpcd</a></li> </ul> How to make the change§</a> </h2> If you want to make the change yourself, it's actually pretty easy. I recommend doing things in this order, because if you can't get iwd/dhcpcd to work to your liking, it's trivial to go back to NetworkManager</code>. Install iwd and dhcpcd (and openresolv)§</a> </h3> We need all three applications. In short, dhcpcd</code> handles the DHCP handshake (and thus giving you an IP address) when a connection is established, like plugging in an ethernet cable or connecting to a wifi network with iwd</code>. As stated iwd</code> handles wireless connections, that's it. Lastly, openresolv</code> (a resolvconf</code> implementation) handles the DNS configuration (mostly the maintenance of your /etc/hosts</code> file). dhcpcd</code> works with openresolv</code> automatically, you can read its documentation</a> for more details. I did mess around with using systemd-resolvd</a> to have one less dependency, but ran into several issues with DNS configuration reliability. Just use openresolv</code> and you'll be much happier. </blockquote> On Arch, I installed all necessary software with pacman like so: # pacman -S iwd dhcpcd openresolv </code></pre> Disable NetworkManager and wpa_supplicant§</a> </h3> Now that you've downloaded the necessary software, you've got to shutdown NetworkManager</code> and wpa_supplicant</code> to prevent conflicts between the two sets of software. # systemctl stop NetworkManager wpa_supplicant </code></pre> Note: Obviously, you'll lose internet at this point. Start iwd and dhcpcd§</a> </h3> With NetworkManager</code> and its wireless sidekick taken care of, spin up iwd and dhcpcd. # systemctl start iwd dhcpcd </code></pre> Wifi setup§</a> </h3> dhcpcd</code> handles all wired connections automatically, so if you're already connected with an ethernet cable you're done. But, if you need to connect to a wireless network (as I'd wager 99.99% of people reading this do...), you'll need to configured iwd</code> to know about your wireless network. To do so you can use the delightful companion for iwd - iwctl</code>. You can run iwctl</code> directly, type help</code>, and then follow the corresponding documentation. Alternatively, you can pass all of your commands directly to iwctl</code>, as I demonstrate at the end of this section. In my case, I simply needed to type the following: $ iwctl [iwd]# device list wlan0 ... </code></pre> I simply wanted to ensure that a wireless device was recognized, and it was. This isn't actually required to connect to an access point. [iwd]# station wlan0 scan </code></pre> Wait a few seconds as iwd looks up all the networks in your area... [iwd]# station wlan0 connect "MyNetworkName" </code></pre> Enter password when prompted and wait ~10 seconds to ensure you don't get an error back. If you do, you likely mistyped your password or your wifi is on the fritz. Note that you can press Tab</code> to autocomplete pretty much all commands, including the wireless name above. </blockquote> [iwd]# quit </code></pre> Exit out of the application. Note that instead of using the iwctl</code> REPL, you can just feed the associated commands directly to iwctl</code>, like so: $ iwctl station wlan0 connect "MyNetworkName" </code></pre> I do like the REPL, as it holds your hand (if necessary), supports tab completion and just feels polished. Use whatever method you prefer. Ensure you've got an IP address§</a> </h3> You may need to wait a few seconds as iwd</code> and dhcpcd</code> do their corresonding thaaangs, but you should receive an IP address that looks somewhat familiar. You can check this with the ip</code> command. $ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 60:e1:86:87:bc:89 brd ff:ff:ff:ff:ff:ff inet 192.168.1.217/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0 valid_lft 33914sec preferred_lft 28514sec inet6 fd22:f6c3:4e66::a7e/128 scope global dynamic noprefixroute valid_lft 35181sec preferred_lft 35181sec inet6 2600:6c52:7b00:55aa::a7e/128 scope global dynamic noprefixroute valid_lft 35181sec preferred_lft 35181sec inet6 fd22:f6c3:4e66:0:dcb8:7af:18a1:8bd9/64 scope global mngtmpaddr noprefixroute valid_lft forever preferred_lft forever inet6 2600:6c52:7b00:55aa:9af:4ba4:ee9b:d32d/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 573699sec preferred_lft 573699sec inet6 fe80::1f0:1103:da4d:ee9b/64 scope link valid_lft forever preferred_lft forever </code></pre> While that might seem like a wall of text, the 192.168.1.217</code> under wlan0</code> is what I'm looking for and tells me that I'm properly connected to my home network. You can confirm network + DNS connectivity with a simple: $ ping austindw.com </code></pre> Or use google.com, whatever you want... </blockquote> If everything looks good, you can make the switch permanent: # systemctl enable dhcpcd iwd # systemctl disable NetworkManager wpa_supplicant </code></pre> Reboot, and you should come up with a perfectly working internet connection. How to revert§</a> </h3> If you run into issues and you want to switch back, simply do: # systemctl disable --now dhcpcd iwd # systemctl enable --now NetworkManager wpa_supplicant </code></pre> This will immediately stop dhcpcd</code> and iwd</code>, then start NetworkManager</code> and wpa_supplicant</code>. With these commands, you'll even be safe to reboot and still use NetworkManager</code>. Results§</a> </h2> "But Austin, I just replaced two networking programs with two more networking programs, how is this an improvement?" </blockquote> Good question - if you measure the RAM usage before and after the switch, you should notice around a ~20MB reduction with the exact same end result. Additionally, my wireless connections were much more stable with iwd</code> than with NetworkManager</code>. I used to have frequent disconnect/reconnect storms when I placed a decent amount of load on my wifi connection. I never quite tracked down the root cause, but the broader stroke of just replacing NetworkManager</code> completely has yielded a much improved networking experience. So I have less resource usage, and an extremely stable wireless network connection. I'm never going back to NetworkManager</code> if I can help it. Hopefully this has helped convert you. Give it a try, and you just might be sold on the iwd</code> + dhcpcd</code> power couple for life. OpenWRT Router Setup 2020-12-06T00:00:00+00:00 Since it's 2020, I have an abundance of free time on weekends, and a buddy of mine kept (rightfully) bugging me to write up a post on my router setup, I finally got around to doing so. Let's dive in on my home router setup, including firmware, QoS, and the beginnings of ad blocking. Prelaunch§</a> </h2> Before I get started, perhaps I should explain why I run a custom router firmware, and why you may want to as well. Security The upside to running a firmware like OpenWRT is that I get much more up-to-date code than what ships with the router. Router manufacturers rarely support their router firmware longer than a year or so, so by using something like OpenWRT I recieve regular updates, which is paramount for network security. Also, by flashing my own firmware, I know exactly what's running on the device, instead of potential malware.</li> If you don't believe me, here's just one example of modern routers being sold on Walmart and Amazon that contain backdoors: Cybernews.com</a></li> </ul> </li> Performance OpenWRT was one of the first operating systems to support bufferbloat mitigation technologies</a> like fq_codel, "cake", etc. It's a long story, but the creators of this cutting edge QoS technology created a custom fork of OpenWRT (CeroWrt</a>) for research purposes. As their research bore fruit, they published their work upstream into the core OpenWRT project. This technology is truly amazing in its ability to reduce or eliminate severe latency on a loaded network connection, and could stand on its own as the sole reason to run OpenWRT.</li> </ul> </li> Ad Blocking While Pihole is all the rage (I prefer, and have contributed to, Adguard Home</a> instead), you can get pretty dang before when it comes to blocking ads across your entire network with some of the facilities available in OpenWRT. Off the cuff, I know of three different methods of blocking ads network wide with OpenWRT, and we'll cover one today.</li> </ul> </li> Because it's fun! Let's face it, if you're reading this blog, you're likely a fellow nerd (said with affection). Hacking away on your own device gives a strong sense of accomplishment and a feeling like you really own something.</li> </ul> </li> </ol> Launch§</a> </h2> Okay, first things first - you'll want to install OpenWRT on your device. There's a few methods for doing this, and I'm just going to refer you to the OpenWRT website, which has a Quick Start</a> guide for taking an existing router with stock firmware and installing OpenWRT. The TL;DR version: Make sure your device is supported by OpenWRT</a>.</li> Boot your router, login to the admin page (generally found at http://192.168.0.1</code> or http://192.168.1.1</code>). If you don't know the username/password (if it prompts you), do an internet search for the brand, as they often use standard credentials like admin/admin</code> or root/admin</code> for the username and password combo. You may be able to find it written on your router near the serial number as well.</li> </ul> </li> Find the firmware upgrade button, and select the *-factory.bin</code> firmware image you presumably downloaded. I have a few routers (don't judge), so using the TP-Link Archer C7 v2</a> as an example, I'd upload the openwrt-19.07.4-ath79-generic-tplink_archer-c7-v2-squashfs-factory.bin</a> as an example.</li> Note: One particular quirk I ran into with this router (funny story actually...) is that the company made a few regionalized models, separate from their global models. These are region locked, and require uploading the correct US</a>/EU</a> version when doing the initial flash. </li> </ul> </li> Wait for the router to reboot (this can take a while, so be sure to give it 5-10 minutes before you assume you've failed).</li> Reset your network (unplug/replug your wired cable, assuming you used one).</li> Login to the admin page at 192.168.1.1</code>. Just type that into your browser and go. If your browser tries to get smart with you and perform a web search with that address instead, look for the little prompt that says "did you actually mean to go to 192.168.1.1?". Yes I did, you cheaky bastard of a browser...</li> </ul> </li> </ol> There's loads of tutorials, Youtube videos, etc about doing this particular song and dance. I recommend you start searching if you want more information or better tutorials. Configuration§</a> </h2> Okay, now that you've got the guts installed, let's get configuring the cool stuff. Root password§</a> </h3> When you initially login, you'll see a prompt with the username root</code> and an empty password box. Simply click Login</code> the first time, since an administrative password hasn't been set. You'll want to set one, for obvious security reasons. You can change it under System</code> -> Administration</code> -> Router Password</code>. </picture> Wireless§</a> </h3> Not all routers actually have wireless radios, but assuming yours does (and you want to use it), you can get started creating your wireless network by navigating to Network</code> -> Wireless</code>. If you're running a modern router, you likely have two radios (one for 802.11n and one for 802.11ac). Simply click the Edit</code> button for each and fill in details like the (E)SSID (your wireless network name), set the Wireless Security Encryption to WPA2-PSK</code> (strong security), and pick a decent password for the Key</code>. </picture> You may want to experiment with the Channel</code> and Width</code> options over time as well. Generally, the greater the Width</code>, the better your speeds will be, but this depends on the router and what the devices on your network support. Generally you'll have better compatibility with a smaller Width</code> (like 20-40 MHz), but you may be sacrificing some speed. TBH, you probably won't even notice... The Channel</code> option is a little more tricky. When in doubt, leave it at default. Maybe try auto</code> (although I've had some routers just straight fail to spin up the radio on auto</code>...). The best approach is to use an app like Wifi Analyzer</a> (Android) to map out the channels being used by your neighbors. You'll want to aim for a less congested channel. Also, channels can overlap a bit (you'll see that in the graphs generated by Wifi Analyzer), so it's recommended to pick something that's ~3 (or more) channels away from another channel, if you can. In general aim for the largest gap in channels, and hope for the best. This is more of a problem on 2.4Ghz networks, since 5Ghz networks have access to a lot more channels. Again, if you're unsure about any of this, simply stick with the defaults for now. You can always change it later. Firewall§</a> </h3> For a little extra security, you can use this trick to "hide" your router from the public internet. To be fair, it's not really hiding, but simply acts as a deterent to anyone running port scans on the open internet. Run a "ShieldsUP" check against your current router configuration, courtesy of Steve Gibson ShieldsUP</a></li> </ul> </li> Navigate to Network</code> -> Firewall</code> -> General Settings</code></li> Under the Zones</code> section, change the reject</code> value to drop</code> for the Input</code> and Forward</code> columns of the wan => REJECT</code> row. See the screenshot below for reference.</li> Run the "ShieldsUP" scan again, which should show that your "equipment actively rejected our UPnP probes!".</li> Rejoice.</li> </ol> </picture> SQM (aka the bufferbloat killer...)§</a> </h3> For this next part, you'll want to install the SQM package. You can either do so from the Web UI, or if you want to go commando-line (heh), then you can SSH to your router via ssh root@192.168.1.1</code>. The password is whatever you set on the Router Password</code> page beforehand. Command line install of SQM§</a> </h4> Nice and easy: # opkg update # opkg install luci-app-sqm </code></pre> Note that I'm installing luci-app-sqm</code>, as that brings in the core SQM dependencies AND a useful widget in the Web UI. If you're hardcore and don't want to use the Web UI at all, then just install the sqm-scripts</code> package. But if you're that hardcore, you probably already knew to do that... why are you reading this blog?? Web UI install of SQM§</a> </h4> Navigate to System</code> -> Software</code>.</li> Using the Filter</code> field, type luci-app-sqm</code>.</li> It should show up as the top result, so click the Install</code> button and wait.</li> It can take a minute, but if the screen seems to have frozen while waiting for the install, chances are you're fine and the install finished, so simply reload the page.</li> </ol> Setting up SQM§</a> </h4> Note: I have a DOCSIS (cable) modem to connect to my ISP (Charter). Depending on your connection, you may want to change a few of these settings. I recommend reading the linked references. After installing the luci-app-sqm</code> package, and rebooting, log back in and go to Network</code> -> SQM QoS</code> Check the box to enable this SQM instance.</li> From the dropdown, select your WAN interface (aka, the interface that connects to the public internet, NOT your local LAN). If you open the list, chances are you'll see an interface with (wan,wan6)</code> tacked on the end. This is what you want.</li> Set the download speed/upload speed accordingly. This will require trial and error, I recommend testing using a good speed test tool (make sure you convert the value to kilobits/s, which is what this tool takes).</li> If you're on Arch Linux/Manjaro/etc, I recommend installing ookla-speedtest-bin</code> from the AUR, and running your speedtest like so: speedtest -u kbps</code></li> You'll want to set these values to someting like 90% of the tested rate to start, but then after doing the later tweaks, you can run tests to bring these closer to 95% or even 99% of rated speeds.</li> For my 100 megabit connection from Charter, I use: 111000</code> for download and 11100</code> for upload on my WRT3200ACM, but I have to tune those down to 110000</code> and 10000</code> respectively on my Archer C7, since it's not powerful enough to handle the higher bandwidth settings without spiking the latency by 200ms in the first few packets.</li> You can test your optimal settings by running a ping (ping 1.1.1.1</code>) in one terminal and then a speedtest (speedtest -u kbps</code>) in another.</li> If the ping spikes significantly while running your speedtest (really noticeable when doing uploads), then you'll need to adjust your settings to more conservative values. If you want an idea of what's bad, simply run this test with SQM off - you'll likely see your ping times pike into the 100's of milliseconds (which is insane).</li> Reference - short version: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm</a></li> Reference - long (and very interesting) version: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details</a></li> </ol> </li> Set the Queue discipline to cake</code>.</li> Set the Queue setup script to piece_of_cake</code>.</li> Check the Show and use dangerous configuration</code> checkbox.</li> Set the Explicit congestion control</code> to ECN</code> for ingress</code>.</li> Set the Explicit congestion control</code> to NOECN</code> for egress</code>.</li> Under the Advanced option...for ingress queueing</code> option, set docsis nat ingress</code></li> Under the Advanced option...for egress queueing</code> option, set docsis nat</code> If your up/down bandwidth is greater than or equal to 1x15, you can experiment with adding ack-filter</code> to your egress options.</li> Reference: https://www.reddit.com/r/HomeNetworking/comments/fvhr4w/psa_sqm_cake_nat_and_bufferbloat_tuning/</a></li> You can read more about these advanced options at the following man page: https://man7.org/linux/man-pages/man8/tc-cake.8.html</a></li> </ol> </li> Both of these are assuming you're using a cable (DOCSIS) modem.</li> IF you aren't using the docsis</code> queue option above THEN do the following, otherwise set the link layer adaptation to none</code> (because you're already adapting for it):</li> Under Link Layer Adaptation, set the Link Layer to Ethernet with overhead</code> if using a cable modem (which I am).</li> Much more information can be found here: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#making_cake_sing_and_dance_on_a_tight_rope_without_a_safety_net_aka_advanced_features</li> Set the Per Packet Overhead to 18 (or 22, seems to be some discrepancy in recommendations).</li> </ol> </picture> Encrypted DNS (with optional ad blocking)§</a> </h3> I'm a big fan of privacy. I'm also a big fan of ad blocking, which generally improves privacy, security, AND performance. One hell of a trifecta. With that in mind, I wanted to give my brother all of these benefits, without the hastle of dealing with adblock lists, special admin pages on the router, etc. To be fair, I do this (via an instance of Adguard Home</a>, installed on my router), as I like to have complete control and reap the performance benefits of having an adblocker running directly on my local router. I'll write a proper blog post about that setup another time... But, this just didn't make sense for my brother's use case. So in order to give my brother the benefits of encrypted DNS while also allowing myself to remotely turn on ad blocking if he desires, I set him up with the following: A profile on my NextDNS</a> account. Even if you aren't using their ad blocking functionality, I think that NextDNS is a fanastic service. They provide high speed DNS servers, encrypted traffic support, optional (!!) metrics/logs, and of course - incredibly comprehensive filtering capabilities, with blocking for everything from ads to malware to social media.</li> Plus, their privacy policy</a> is just incredible and I think sets the standard that other companies should be held against.</li> They're primary service is free for 300,000 queries/month. Because I love their service, I go ahead and throw $2 their way a month. Since I pay for their "Pro" plan, if I spill over 300,000 DNS queries a month, then I don't lose ad blocking support for the remainder of the month.</li> </ol> </li> An instance of dnscrypt-proxy</code> (2), running on the router.</li> </ol> This combination allows for strong DNS traffic encryption, blocking via NextDNS, and local device resolution via the onboard dnsmasq</code> instance that comes with OpenWRT. So, without further ado, let's get started. Installing dnscrypt-proxy 2§</a> </h4> Luckily, this is quite easy these days, ever since they added an official package to the OpenWRT repos. The dnscrypt-proxy wiki</a> has detailed instructions. In short: Update your software lists from either the UI (System</code> -> Software</code> -> Update lists...</code>) or an SSH instance (opkg update</code>).</li> Install dnscrypt-proxy2</code>, just like you did with luci-app-sqm</code> above.</li> Go to Network</code> -> DHCP and DNS</code> -> General Settings</code> -> DNS forwardings</code>, and add 127.0.0.53</code>. This tells the running dnsmasq</code> instance to point to the dnscrypt-proxy</code> instance for DNS resolution, instead of using your ISP's servers.</li> </ol> </li> Save & Apply.</li> </ol> Configuring dnscrypt-proxy 2§</a> </h4> Now, before we restart dnsmasq to get the changes to take effect, let's update dnscrypt-proxy</code> to point to our NextDNS instances. You need to SSH into your router to do so. If you know what SSH is, you probably know how to do it (ssh root@192.168.1.1</code>...) Once you've logged in, you'll want to edit the /etc/dnscrypt-proxy2/dnscrypt-proxy.toml</code> file. Chances are your OpenWRT install has either nano</code> or a stripped down verison of vim</code> installed. Pick your favorite editor. In this case, I used vim</code>: vim /etc/dnscrypt-proxy2/dnscrypt-proxy.toml </code></pre> Point to NextDNS§</a> </h5> Scroll to the very bottom, you'll see a section for Static entries</code>. If you're using NextDNS, login to your account, select the profile you want to use, and then scroll down in the installation instructions, and you'll see an generated config for DNSCrypt support. Here's an example of my own after I updated it to point to my NextDNS profile: ... ######################################## # Static entries # ######################################## ## Optional, local, static list of additional servers ## Mostly useful for testing your own servers. [static] [static.'NextDNS-BroRouter'] stamp = 'sdns://AgEAAAAAAAAACImaBunchOgiBberish...' </code></pre> Scroll back up to the top of the file, and uncomment and update the server_names</code> entry to point to your new NextDNS record. ... server_names = ['NextDNS-BroRouter'] ... </code></pre> You could stop right there. I updated a bit more of the config, out of some old lesson's I'd learned over the years. IPv6§</a> </h5> First, if you have an IPv6 connection, you'll want to able the ipv6_servers</code> option. You can tell if you have IPv6 in OpenWRT by viewing your WAN6 interface - if it's up and sending/recieving packets, you have IPv6 configured on the modem and the router. So, find the ipv6_servers</code> option, and set it to true</code> (it's likely false</code> by default). Blacklist§</a> </h5> Lastly, if you're going to be using NextDNS for ad blocking/domain filtering, I recommend disabling the default blacklist file that dnscrypt-proxy</code> comes with. To do that, find the [blacklist]</code> section. Under that, you should see an entry for blacklist_file</code>. Simply comment it out, like so: [blacklist] ## Path to the file of blocking rules (absolute, or relative to the same directory as the... #blacklist_file = 'blacklist.txt' </code></pre> This will prevent dnscrypt-proxy</code> from doing any of its own filtering on box. While this might sound advantageous at first, if you're using NextDNS to configure your filters, you don't want an extra set of filtering rules running on your router that might conflict with your downstream settings. Trust me, this is confusing as hell when you run into it, took me a little while to figure out why certain websites were't working that should have been. Caching§</a> </h5> You don't have to do anything! Older versions of dnscrypt-proxy</code> ran with less-than-optimal caching settings, but the author conducted some interesting research</a>, found pretty optimal settings, and set those as the defaults in dnscrypt-proxy.toml</code>. You can scroll down to the caching section to get an idea of what they are, if you're curious. See also: https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Caching</a> Otherwise, just rest easy and enjoy blazing fast caching. Restart dnsmasq§</a> </h4> Restart dnsmasq</code>. System</code> -> Startup</code>... Conclusion§</a> </h2> Whoo! This was a long post. I hope this helped you in some way, as it's been a collection of techniques that I've used on my own routers in one way shape or form, and I don't think there's any other place on the internet that has all of this in one place. Go forth, have awesome internet, and prosper! Troublesome iPod 2020-08-16T00:00:00+00:00 In an effort to get away from the shitstorm that is 2020 (COVID-19, major upsets in my personal life, etc), I decided to visit my mom, back in my hometown. Dem boonies§</a> </h3> To frame things a bit, my mom lives a fair ways out from "town". This means that certain amenities that I've come to take for granted aren't available. Things like a truly high speed internet connection (I'm sorry, but DSL does not count any more), cellular reception, a coffee shop filled with cute baristas and college students... I digress. One key point is that the online world is not easily accessible when you take a few steps out of the cityverse. This also means that things like Spotify aren't really usable, especially for my mother, who's as far removed from a "tech wiz" as anyone I know. But she does love her music. And she has loved her iPods. A tiny device with one of the greatest navigation interfaces of all time (seriously, can we bring back the click wheel?), loads of storage space, loads of battery life, and loads of not needing an internet connection. Like too many things in tech, they don't always age well. The iPod Classic my mom had been using for the past few years can barely hold a charge longer than 2-3 hours of playback time. Additionally, the data/charge port on the device^{1 can only do the latter. Plugging it into a laptop yielded zero device recognition, on either Windows or Linux.} So, the old device was kaput. No way to add music, no way to migrate said music to another device, nothing. But, this didn't stop my mom from ordering a "new" iPod. Apparently, you can still get these things on Amazon^2. So, there we were with a new iPod to load up with music and a laptop that (very luckily) had pretty much every song my mom had on her old iPod. +1 for coincidental backups. So you want to transfer some files?§</a> </h3> It being 2020, and iPod Classics having ended their official manufacturing run 10 years ago, I thought - "it has to be possible to upload music to these things without iTunes, right?" I mean, we had a Linux laptop that I brought and her Windows laptop for optimal spread of operating system support. Surely there must be some free software that's been written that easily imports music from a folder into the iPod Classic! Boy, was I wrong. I tried: aTunes</a></li> YamiPod</a></li> Manual workarounds like revealing hidden folders in Windows, navigating into the iPod's hidden Music folder, and drag/dropping some files in.</li> A fleet of other software all based around shared libraries used by YamiPod.</li> </ol> But nooooooo. Apple builds such brilliant hardware that something that should be as simple as copying a bunch of MP3 files into a folder isn't even possible. In fact, it's actively disallowed. This isn't really news to anyone. Apple created the greatest MP3 player in the world (and possibly ever), but everyone who got one hated one thing about it - having to use iTunes to manage all of their music. It was slow, it locked down a device to a particular computer, music couldn't be easily shared with friends, etc. All around iTunes was a total regression in user experience, which juxtaposed so very shockingly with the excellent UX that the iPod Classic offered (I mean that click wheel). And here we are 10 years later, and we're still back at square one - use iTunes. To be fair, there are a stupidly large number of paid spyware</del> apps that claim to handle iPod Classics with zero problem. And I believe them - we as a society have had a very very long time to nail down a 10 year old piece of hardware. But we seem to have given up. Or at least I've given up. We downloaded iTunes, installed it, and I drag/dropped 110GB+ of music onto the device and away we went. Learnings...§</a> </h3> But I learned something. I learned something about myself and Apple. I learned that Apple will actively create consumer unfriendly software. And I learned that I don't want to deal with its bullshit any more. Just give me an MP3 player with an SD card slot and I'm good. Oh look - they do exist</a> And of course, it looks like an Apple Watch... Footnotes§</a> </h2> Those old wide + flat connectors from back in the day before Apple made 2-3 more connectors and now everything requires a damn dongle.</li> Although they aren't nearly as cheap as they should be</a>.</li> </ol> (Re)New Beginnings... 2020-07-12T00:00:00+00:00 Jeebus. It's been quite a long time since I last picked up the pen... as it were. This blog has sat in the metaphorical trunk, strapped with a belt, wrapped in a blanket, tucked under the creaky stairs of your old aunt/grandma/weird neighbor's house for a very very long time. Much has changed since I've been away... For one, I'm writing this in the time of Coronavirus (COVID-19), the longest running catastrophe that my fickle mind can remember at my age, and I remember when the Twin Towers fell and the 2008-09 stock market crashed (I was pretty sheltered from that one, still trudging through university at the time, like you do). Coronavirus seems to be the impetus from which so many find new inspiration. I find this stems from a boredom grown from the mind numbing steps one takes, marching through life strapped onto the back of a routine - wakeup, work, workout, eat, sleep, repeat. I could blame this new found desire to write on Coronavirus, and I'd probably be right. But also a little wrong. Aside: Now, I talk of the gift that is boredom from the comfort of my own home. It doesn't take much to recognize my privilege here. Many are struggling to survive, either to provide food for the table, or a place to sleep for the night. A member of my own family survives due to the very helpful program that is unemployment. It's not perfect, but dammit is it a lifeline in this torrential downpour. </blockquote> </blockquote> For another, I've spent over 9 years in the tech industry, working my way through the ranks at Amazon. Far and away a different person than I was in my college days, or just there after. I'd like to think myself more interesting, but I guess you'll be the judge of that. But I digress§</a> </h3> Back to our regularly scheduled programming. </blockquote> </blockquote> I'm restarting this blog. Maybe it's because people in the software industry still seem to have blogs, which convey some startlingly useful information, or solutions to problems that I encounter on the daily, and I'd like to pass it on (my home-room middle school teacher would be so proud). Or maybe it's because social media has molded the web into its own image, where everyone has a site, a profile, a "feed", while possessing none of them. They no longer own the data that they produce. Leave that to Facebook, to Google, Twitter, Snapchat, the list goes on. Honestly, Luke Smith</a> had a point when he argues for setting up your own website. It's a site/profile/feed that you own, you control. It's my own personal space to put whatever I want, and know that I'm not paying for the privilege with anything other than my own cash. Having your own website is the OG social media profile. Most of the world seems to have forgotten that. While preparing this post, I've actually been able to download my LinkedIn data, and I'm working deleting my account as these days pass by. To be honest, it literally never did me any good. I had a job right out of college that I've never left. I haven't had a need to network, to connect myself with as many other like minds, all in the hopes of establishing a game of career hoops and ladders. I've been perfectly fine. When there does come a time when I want a new job, I'll simply start asking. I'll be the ones to reach out to recruiters, demonstrating my skill set. It will be the inverse of what happens today, where multiple times a week I get emails from FAANG recruiters, which I've ignored for... literally years. Who knows... maybe I'll go freelance. Looking forward§</a> </h3> I look forward to more. More sharing of technological breakthroughs, those oh-so-critical "ah hah" moments. Sharing insight on the tech industry, ridiculously good recipes, or other inventive ways to inflate one's ego. Let's hope I stick with it this time... I feel like it has been a long time coming. Headphone Review: PSB M4U 1 vs B&W P7 2014-06-21T00:00:00+00:00 After graduating college, I started working fulltime at Amazon. I went from sprinting from class to class to staying indoors (professionally) for 8+ hours a day. I was forced to spend most of my day in front of a computer, sitting (or standing, thanks to my stand up desk) while I worked. Okay, okay, I'm sure you're wondering why I bring up my change in daily life. I'll cut to the chase. My extended periods indoors in front of a computer practically forced me to develop my taste in music, and eventually good sound. With this migration into the arena of "good sound", I began my journey down the corridors of the audiophiling cabinet (see what I did there?). I bought my first pair of speakers ever, the NHT Absolute Tower speakers</a> . A few months later, I started thinking about where I was listening to music most, as my NHT's are great, but I can't exactly set them up at work. With this came my foray into headphones, and after a few hours of reasearch, I bought the PSB M4U 1's</a>, primarily due to this excellent Wirecutter article</a>. Now, these headphones (at least at the time of this writing) are $300, so not cheap by any measure. However, after about a week of constant use, I noticed a few... particulars that I will detail in a minute. Because of these particulars, I started looking at alternative headphones. I eventually worked my way to the B&W P7s</a>. Now at $400, these are hardly the cheaper option, but they were especially usefull as my local Apple store had a pair on display, so I was able to do a quick comparison before purchasing a pair for futher testing. The Review§</a> </h2> Okay, let's get to the meat of the review. I'm going to go over the differeneces in comfort, build quality, sound quality, and ammenities, and how they applied to me. Comfort§</a> </h3> Right off the bat, I'm going to dive into comfort. Comfort is a big subject for me, as my primary listening environment is 4+ hour sessions at work, so I can't be pulling the headphones off and massaging my ears every 20 minutes. </picture> First off, the PSBs: Pros§</a> </h4> Memory foam ear pads.</li> Excellent head band cushion.</li> Great ear seal.</li> </ul> Cons§</a> </h4> Shallow ear pads.</li> </ul> I know it's not the largest breadth of points, but I want to make one call out about the shallow ear pads. As a blonde elf look-alike, I have nicely pronounced ears (at least I think they are nice...) but nonetheless I have "large" ears. The PSBs are able to completely surround my ears without any problems, and the seal is fantastic, but the shallowness of their ear pads (the distance from the edge of ear pad to the driver on the headband) is problematic for my ears. This shallowness forces my ears to rub, and thus curl, just the slighted bit at the tips, which is irritating over extended listening sessions. This shallowness is the primary reason why I began my search for alternative headphones, and thus the B&Ws: Pros§</a> </h4> Much larger earpad volume</li> </ul> Cons§</a> </h4> Non-memory phone ear pads</li> Stiffer headband cushion</li> </ul> The larger earpad volume is the main reason why I prefer the B&Ws, as I no longer feel like I'm pressing the tips of my ears against a wall every time I put them on. The PSBs have a better headband pad, and cushier ear pads, but if you have larger (or in my case, pointier) ears, you may want to look elsewhere. In summary, the PSBs have a better headband pad, and cushier ear pads, but if you have larger (or in my case, pointier) ears, you may want to look elsewhere. The B&Ws are firmer overall with their cushions, but I think that is simply the nature of using memory foam vs not using memory foam. Still, the B&Ws just fit my pointy-ass ears better. Build Quality§</a> </h3> I have to admit build quality turned out to be more of a care-point for me than I would have thought, so let's break down some differences. Again, we'll start with the PSBs: Pros§</a> </h4> Prime use of leather and metal</li> Two axis earcup pivot</li> Nice case material, kind of a mix of hardshell with soft rubber.</li> </ul> Cons§</a> </h4> Plaassticc, specifically the headband cover and overall exterior.</li> Gut wrenching snap when folding the headphones up, makes me think I'm doing something wrong everytime I fold them up.</li> Cheap zipper on case. Not a huge problem, but you have to align the case just right in order to zip it up smoothly.</li> </ul> All in all, I was actually marginally disappointed with the PSB build quality, but really only because of the plastic exterior. Sure, they look similar to Beats, but the plastic just seems a touch brittle to me, and makes me worry about snapping in extreme cases. Granted, the likelyhood of snapping is pretty much zero, so it's more of a subjective opinion, and by no means hard fact. The interior of the headband is solid, excellent metal connections and hinges, and I like the fact that I can easily yank out the headphone cord for easier folding and storage. On to the B&W's: Pros§</a> </h4> Solid metal and leather, I don't think there is an ounce of plastic on the headphone at all.</li> Large headband cushion, it's much more encompasing of the head than the PSB, but that isn't as large a point as it sounds, as the PSB really only touches the head in one spot, and it's a bitching cushion.</li> Magnetic cushion holders are a nice touch, and hold extremely well. You've gotta give 'em a solid yank to get the ears off.</li> Leather carry case.</li> </ul> Cons§</a> </h4> The metal arms seem marginally thin on the B&W's, which makes one worry about permanent bending or damage.</li> Leather carry case. I know that I listed the case twice in pros/cons, but while the case is well built, I don't believe it actually protects the headphones much, especially in cramped luggage or some other rock/hard place.</li> </ul> The B&W's have some solid build quality, with excellent drivers and top quality leather. The clamping force is certainly strong, which indicated strong metal retention. The carry case definitely leaves something to be desired, at least when compared to the soft-shell of the PSB. The foam ear pads are okay, but I like the memory foam of the PSB's (as mentioned previously). Still, I think it's a part of the stiffer foam on the B&W's that actually allow for a better fit with my extended ears, because if they compressed more I might run the risk of having my ear tips touch the drivers. </picture> Sound Quality§</a> </h3> Both headphones offer excellent sound quality, so you really can't go wrong with either headphone. Still there is a $100 price difference, so let's breakdown the differences that I noticed. The PSB's had excellent clarity, with defined treble, solid midrange, and articulate bass. These headphones lend themselves particularly well to singing/guitar-focused music, such as Johnny Cash's rendition of "Hurt". You can practically see his fingers on the strings in your mind, and his voice comes to life throughout your entire listening session. That being said, I listen to a wide variety of music, and I felt that these phones didn't tune perfectly (but still damn close) to my listening tastes for genres such as trap and dubstep, along with a few hip-hop songs. This is mostly attributed to the more neutral bass response, which some people may like, but I did not. This leads me to the B&W's. The B&W's have a much more... dynamic sound, I feel. I noticed a clear seperation in the instruments in the music I listened too. I could easily focus on the bass track and then switch to the vocals in my mind, hearing each with precision without one over powering the other. I have to say, if you like bass, get the B&W's. They kick the ear drums off your head when compared to the PSB's, which is something I was actually looking for without realizing it. That is, until I found it in the B&W's. I very much enjoyed listening to the clear highs of The Weeknd while also bumping those sick trap beatzzz of Diplo's "Revolution". As soon as I heard the latter track on the B&W's, I just started smiling in the middle of the Apple store. All said and done, the B&W's fit my listening tastes better. The PSB's are great, and will probably be preferred by some users, as they are more neutral in their presentation. I just preferred the B&W's dynamic sound and sweet bass response. Is it worth the $100 difference? Maybe, as it certainly was for me. Still, you can't go wrong with the PSB's, and they are pretty much the best you can get in the $300 price range. Amenities§</a> </h3> Without a doubt, PSB takes the cake on this one. The inclusion of spare ear pads, cleaning cloth, airplane dual-prong adapter, 1/4" adapter, and secondary output through the alternate audio port makes for a sick package at $300. I do want to take the time to call out the secondary output feature I mentioned. This means that you can listen to your main audio source on your PSB's through the input on one side of the headphone, but then you can also attach a secondary set of headphones to the port on the otherside of the PSB's, allowing another person to easily listen with you. This is great for car/train/airplane trips with my girlfriend, as we can watch a movie in peace, or jam to the same tunes without effecting other people. This is something that should be present on more headphones, and something I was sad to see missing on the B&W's. Don't get me wrong, the B&W's have a carrying case, two cords (one for iPod control, one for "normal" use), 1/4" plug, and the option to plug your cord into either side of the headphone. However, the PSB's have all of this, and more. Plus, the B&W's require the use of a proprietary, angled connector, which means you have to pay B&W everytime you want a new cord. The PSB's have no such limitation Summary§</a> </h3> So, after this (rather verbose) review, I ended up going with the B&W's, but that is mostly because my ears couldn't handle the PSB's form factor and I liked the bass response on the B&W's more. The PSB's have better amentiies, a better price, and better foam materials, but they just didn't fit me properly. If they fit you, great, buy them and enjoy them. If you like possibly better audio quality, especially with respect to bass, and have large, protruding ears, I have to recommend the B&W's instead. Thanks for listening! Getting mac ready to compile Vapoursynth 2014-05-11T00:00:00+00:00 brew install gcc48 brew install python3 CC=/usr/local/bin/gcc-4.8 CXX=/usr/local/bin/g++-4.8 PYTHON=python3 ./waf configure build Running IntelliJ with JDK 1.7 and Scala 2.10 on Mac OS X 10.9 Mavericks 2014-01-02T00:00:00+00:00 Since the combination of OS X, Scala, and IntelliJ appear to be a moving target in the development world, I figured I'd go ahead and jot down how I got them up and running on my MacBook Pro. First things first, I used IntelliJ 13 (Community Edition), with Scala 2.10.3, which I installed using Homebrew (I will detail it below). In addition, I'm using the latest Java 7 JDK for Mac OS X from Oracle. Installing Scala§</a> </h2> First things first, use Homebrew to install Scala, including the documentation, like I have detailed below. $ brew install scala --with-docs </code></pre> You will likely see a line about where to point IntelliJ in the post install message, similar to this: ==> Caveats To use with IntelliJ, set the Scala home to: /usr/local/opt/scala/idea </code></pre> And that's exactly what we are going to do... Installing Java§</a> </h2> Make sure that you have the latest version of the Java JDK installed, I just downloaded the appropriate JDK from Oracle's site</a>. Preparing and Installing IntelliJ§</a> </h2> In order to get IntelliJ to work properly with the latest version of the Java 7 JDK, you need to edit the .app according to the instructions provided by the IntelliJ authors here</a>. Almost there! Starting a Scala Program§</a> </h2> Now, that you have all of the tools installed, you need to start a Scala module in IntelliJ. Start by opening IntelliJ and starting a new Scala Module. Set the *Project SDK *path to the location of where you installed Java 7. Mine was: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home </code></pre> Next, select Set Scala Home, and point it to the location of your Scala install, which was pointed out to us by Homebrew during our earlier install. In my case it was: /usr/local/opt/scala/idea </code></pre> IntelliJ should autodetect the necessary libraries and documentation path. Fill out the remainder of your projects information (name, etc), and then click Finish to get started. Installing XBMC 12.2 on an ASRock ION 330HT 2013-06-28T00:00:00+00:00 Here are some notes for an XBMC 12.2 (Frodo) install job that I did for a buddy of mine on his ASRock ION 330HT. We were both a little nervous about performing a full upgrade, but due to smart partitioning, everything upgraded perfectly. Here's what I did... Backup Prior Install§</a> </h2> Using Clonezilla</a>, I made an image of his current install, which was an Ubuntu 10.04 install with XBMC on top of it. I just went ahead and backed up the first 10G partition on his internal hard drive. Everything went smoothly. A small trick for booting something like Clonezilla from a flash drive with the ION 330HT is to have a keyboard plugged in and then mash the F11 key on bootup. You should see a small boot menu appear and then you can select to boot from the corresponding USB stick. Download and Install XBMCbuntu§</a> </h2> Since he only uses his ION 330HT as a media box, he didn't need a full fledged Ubuntu install. So, we decided to go with XBMCbuntu</a>, which is available for download on the main XBMC page. We installed that ISO onto a USB drive using UNetbootin (for Mac! Didn't know they had this now...). Next we booted the ION 330, mashed the F11 key, and started up the live USB. It's important to select the Fixed DPI (120x120) option if you are installing with an HD TV attached, as otherwise the installer text is way to freakin' small. With the fixed DPI it's totally manageable. We quickly selected our keyboard and timezone options, and reformated the 10G partition he had from his previous install on the internal harddrive. Then, we made sure that the installer pulled down the latest packages and set his user to automatically login on bootup. Then we hit GO! Rejoice!§</a> </h2> After the install finished, we rebooted into our wonderful new XBMC machine. And frankly, it pretty much worked just fine after that with two small caveats: Automount Drives§</a> </h3> We wanted his drives to automatically mount on bootup, so we can always have his media at the ready. We did this by finding each drive's attached UUID using $ sudo blkid </code></pre> After we found the partition's UUIDs, we created mount points under /media. For example $ sudo mkdir -p /media/Media $ sudo mkdir -p /media/Media_640G $ sudo mkdir -p /media/FusionHD </code></pre> The first two are for his internal drives, and the last one is for an external he has hooked up over eSATA. Then, I edited the /etc/fstab file to include lines like the following for each drive: UUID=<the_uuid> /media/FusionHD ext4 (or ntfs-3g) rw,user,auto,nosuid,nodev 0 0 </code></pre> After that, upon reboot, all necessary drives in the fstab file will automount and appear as media locations in XBMC. Fix HDMI Audio Output§</a> </h3> While almost everything worked out of the box on the ION 330HT, including the remote and display, the audio was a little wonky. My buddy has his ION feed out to a Sony receiver over HDMI, and the audio channels were a little screwed up for his 5.1 setup. Luckily for us, this is a solved issue. All that is necessary is to boot into the XBMCbuntu desktop (which you can do by exiting XBMC (not shutdown!), and then selecting "XBMCbuntu" from the login manager's dropdown menu). Once you are in the desktop, open up the asoundrc file located in your home directory: $ vim ~/.asoundrc </code></pre> Simply follow the instructions located in the XBMC wiki here</a>. I backed up the asoundrc file and then copy and pasted the settings right into the file. I completely blew away all of the previous contents. Then reboot. Once you login to XBMC again via that login manager from earlier, you should be able to playback your media over a standard receiver with no issues, as long as you make sure to set your Audio Output Device to *hdmi_direct (which we get from that wiki link). *** ** You should be good to go! A few notes...§</a> </h2> First of all, make sure that you go into the ION 330HT's BIOS and give the shared video memory a full 512MB of RAM. This ensures smooth video performance. Also, if you have DPI issues with XBMCbuntu AFTER you have installed it, simply add a line to your */etc/X11/xorg.conf *file under the "Monitor" section. There should be a few other entries and another Option entry, so just add this to the end of the Monitor section: Option "DPI" "120 x 120" </code></pre> Then logout and log back in. You should now be able to actually read the OS desktop text on a Full HD television. After all of this was said and done, I booted up Clonezilla once more over a USB and made a backup image of the known working configuration. That way he can fallback on it if anything ever screws up. Pick Your CUDA Device at Program Runtime 2013-05-05T00:00:00+00:00 Select your CUDA Device§</a> </h2> While conducting research on my thesis, there were several times where I wanted to test the efforts of my work on different CUDA cards. Luckily, I had access to a machine that contains 4 different CUDA cards, ranging from a GTX 480, to a Tesla C2050, to a Kepler K20Xm (there are two of these). This is all according to nvidia-smi of course. Now, all of the kernels that I have been writing recently have been only designed to run on a single device (this could very likely change in the future, but not now). Normally, when you want to run a CUDA kernel on a different device, you select the device number programmatically using the cudaSetDevice() API call. Of course, you might be thingking : "But I have a lot of code! I don't want to make a branch in my git repo, edit all of my source files, recompile, and then run from there!" At least, you should be thinking that... Luckily, there is a solution, and it is one that let's you select the devices you want your CUDA kernels to run on whenever you execute your program. Deep in the CUDA Best Practices guide is a little section on the CUDA_VISIBLE_DEVICES environment variable. By simply setting the CUDA_VISIBLE_DEVICES variable to your desired device identifier(s), you can change which device your program runs on. This is great for testing kernels across a number of devices rather quickly, without having to recompile every time. Here is an example: $ CUDA_VISIBLE_DEVICES=1 ./myCUDAProgram </code></pre> If your kernels DO support multiple devices, you can provide a list like so: $ CUDA_VISIBLE_DEVICES=1,3 ./myCUDAProgram </code></pre> Not too shabby, eh? Now use this in your CUDA testing! You can thank me later... Source: CUDA Best Practices Guide</a> Profiling CUDA through Python with NVVP 2013-02-09T00:00:00+00:00 For those of you who don't know, I'm currently pursuing my Masters in Computer Science. Part of this undertaking requires me to complete a thesis. For the purposes of my thesis, I decided to implement a framework into the Vapoursynth</a> video filtering project that will allow it to run as many video operations as possible on a CUDA-enabled GPU. In so doing, I've been doing a lot of CUDA kernel writing and profiling recently. The problem, of course, is that Vapoursynth works through a Python script wrapper around a C/C++ core. So, whenever I want to profile my filters, I have to insert their associated calls into a python script, and obtain their timings through it's execution. Of course, this is a less than ideal solution. What I've (finally) been able to do is profile my CUDA filters through their python script files using NVidia's NVVP (NVidia Visual Profiler) program. It took a few tricks to get it working correctly, especially considering that these scripts output video data directly to standard out (stdout) (which NVVP attempts to display, poorly, on the Console tab of the profiler). Let's get to the tricks: Make sure that a hash-bang is at the top of your script. Ex: #!/usr/bin/env python3 </code></pre> </li> Make sure that instead of sending out video output to stdout, we send it to /dev/null on Linux, or NUL on Windows. This prevents NVVP from exploding on the massive amount of video data. Ex: with open(os.devnull, 'w') as f: clip.output(f, y4m=True) </code></pre> </li> Be sure to make the script executable, otherwise NVVP won't be able to, well, execute it. example@example-desktop:~/src/testing $ chmod +x test.py </code></pre> </li> Start a new profiling session in NVVP, and load your target Python script under the File path, optionally setting the Working Directory to the same directory as your script, in case you need to load any external files. After that, you're done! Go ahead and profile your application like you would any other CUDA executable. It may not be able to run all ~28 profiling passes, but it will run most, and give you nice execution time line to boot. </li> </ol> So, to sum up, simply make your target Python script executable, being sure to redirect any output to /dev/null instead of stdout if there is a large amount of data, and then run NVVP like you would a normal CUDA program and enjoy the wealth of analysis tools it has to offer. W3 Total Cache and DONOTMINIFY 2013-02-07T00:00:00+00:00 I use W3 Total Cache for all my of my website caching needs and usually it works great. There have been a few problems with the latest release, but that is for another post. One issue that I just experienced recently with one of my sites is that even though minification (or minifying) was turned on, none of my CSS or Javascript files were actually being minified. When I would "View Source", all of my files where in their original, un-minified form. If I scrolled down to the bottom of the source, where W3 Total Cache inserts its cache statistics, I saw something like this: Minified using disk: basic (DONOTMINIFY constant is defined) Page Caching using disk: enhanced (Cookie is rejected) Database Caching using apc (Cookie is rejected) Object Caching 1293/1305 objects using apc </code></pre> The DONOTMINIFY statement jumped out at me. After doing some quick research, I discovered this forum topic</a>, which detailed that the problem is actually in my Wordpress SEO plugin, made by Yoast. More specifically, the "Allow tracking of this WordPress installs anonymous data" option that is available under the plugin's Dashboard, under Tracking, was enabled. By disabling** **this option, minification works like a charm again.</label> So, in summary, if you see the DONOTMINIFY notification in your source code, and you have the Wordpress SEO plugin from Yoast, make sure that the anonymous tracking option is turned off. Reading Unicode from raw_input in Python 2012-10-10T00:00:00+00:00 This took me too damn long to figure out in Python, so a quick note. To obtain a unicode string in from the raw_input() function in Python, simply use a line like the following: unicode_string = raw_input("Enter your input").decode('utf8') </code></pre> About the only good thing in iOS 6 2012-10-07T00:00:00+00:00 iOS 6 has been all over the news recently with botched promises and new features (culminating in the new Maps application and the removal of a native YouTube app). Granted, the YouTube-written app is LOADS better, but nonetheless, iOS 6 has been... underwhelming. Or lackluster, whatever you want to call it. Still, there is one thing that I like about iOS 6:not having to enter my password to update my apps. It might seem like a small thing, but it's the small things that really count. Of course, I'm not convinced that not having to enter my password makes up for the debacle of Apple's new Maps app.... ah, whatever. Just some random thoughts, go back about your browsing... Stopping BOINC from running on boot up in Ubuntu 2012-10-07T00:00:00+00:00 Thank you AskUbuntu:http://askubuntu.com/questions/96787/stop-boinc-from-auto-start-on-system-boot</a> I like to control when BOINC runs, and there is no easy to access preference in the options dialog, so this little gem is nice to know. Quick Solutions to iOS 6 Issues 2012-09-21T00:00:00+00:00 With the release of iOS 6, a number of my friends and colleagues are jumping on the update bandwagon. I went ahead and did the same thing, but I heard some horror stories about people losing access to their email (those who had Gmail anyways) and the Youtube app disappearing (a "feature" touted in the iOS 6 release party). Luckily, there is a quick solution, and one that should have been used by anybody who has a Gmail account and was/is using the native Mail app for iPhone. The solution? Get the Gmail app. It has much better integration with the full feature set of Gmail. You can pick it up on the App Store</a>. Now, with the removal of the native Youtube app, a lot of people may be angry, scared, or what have you. I beg to differ, as this is actually a good thing. Now Youtube has the ability to update and fix its own app instead of being at the mercy of Apple every time it wants to release a new feature. Again, you can find the Youtube app on the App Store</a>. Two quick solutions to Apple's crippling of Google related features in iOS 6. Beginnings... 2012-09-19T00:00:00+00:00 I started this blog because I knew that I already had the resources available (considering that my other site is fully up and running, adubvideo.net</a>) and sometimes I need a place to put my random thoughts and a few of the things that I learn while adventuring through the precarious world of technology. For those of you who don't know, I'm a Graduate Student at California Polytechnic State University. I've been there five, going on six years, first pursuing my B.S. in Computer Engineering, and now getting my M.S. in Computer Science. Throughout my time there, I've learned a lot and experimented more. One thing is for sure, my experiments lead to more issues and discoveries than any class could ever offer, which is why I plan on posting my results and discoveries to this blog whenever I stumble across them. Until then, good day!

BCP38 §</a> </h2>
This is less about protecting your router and more about preventing your network's contribution to some types of DoS/DDoS attack.</p>
For more info I highly recommend the BCP38 info website</a> which explains the root problem quite well.</p>

After §</a> </h3>
`example.com { </span> root * /path/to/dir </span> </span> file_server { </span> precompressed br zstd gzip </span> } </span>} </span></code></pre> Pretty dope improvement, no?</p> That's all for today - go forth and precompress.</p>`

This website is (slightly) faster

NetworkManager is bloat

OpenWRT Router Setup

Troublesome iPod

Footnotes §</a> </h2>

Those old wide + flat connectors from back in the day before Apple made 2-3 more connectors and now everything requires a damn dongle.</li>
Although they aren't nearly as cheap as they should be</a>.</li> </ol>

(Re)New Beginnings...

Headphone Review: PSB M4U 1 vs B&W P7

Getting mac ready to compile Vapoursynth

Running IntelliJ with JDK 1.7 and Scala 2.10 on Mac OS X 10.9 Mavericks

Installing Java §</a> </h2>
Make sure that you have the latest version of the Java JDK installed, I just downloaded the appropriate JDK from Oracle's site</a>.</p>

Preparing and Installing IntelliJ §</a> </h2>
In order to get IntelliJ to work properly with the latest version of the Java 7 JDK, you need to edit the .app according to the instructions provided by the IntelliJ authors here</a>. Almost there!</p>

Installing XBMC 12.2 on an ASRock ION 330HT

Pick Your CUDA Device at Program Runtime

Profiling CUDA through Python with NVVP

W3 Total Cache and DONOTMINIFY

Reading Unicode from raw_input in Python

About the only good thing in iOS 6

Stopping BOINC from running on boot up in Ubuntu

Quick Solutions to iOS 6 Issues

Beginnings...

Austin's Thought Basin

Converting a website to an ebook

Encoding Snippets

Hardening OpenWrt

Precompressed Assets with Caddy

Load Testing for Fun and Profit

Austin's Thought Basin

Converting a website to an ebook

Encoding Snippets

Hardening OpenWrt

Method 1 (uci)§</a> </h3> SSH into your router (likely something like ssh root@192.168.1.1</code>)</li>

Method 2 (Luci)§</a> </h3> Install the luci-app-uhttpd</code> package.</li> Go to Services -> uHTTPd</code> page.</li>

Method 1 (uci)§</a> </h3> Install the bcp38</code> package.</li>

Method 2 (Luci)§</a> </h3> Install the luci-app-bcp38</code> package.</li> Navigate to Network -> Firewall -> BCP38</code>.</li> Click the checkbox to enable.</li> Select the interfaces that pertains to your WAN (likely labeled wan,wan6</code>).</li>

Method 1 (uci)§</a> </h3> Install the banip</code> package.</li>

Method 2 (Luci)§</a> </h3> Navigate to Network -> Firewall</code>.</li> Under Zones</code>, set the wan => REJECT</code> zone to drop</code> (instead of the default reject</code>) on both Input</code> and Forward</code> using the associated dropdowns.</li>

Precompressed Assets with Caddy

Load Testing for Fun and Profit

This website is (slightly) faster

NetworkManager is bloat

OpenWRT Router Setup

Web UI install of SQM§</a> </h4> Navigate to System</code> -> Software</code>.</li> Using the Filter</code> field, type luci-app-sqm</code>.</li> It should show up as the top result, so click the Install</code> button and wait.</li>

Troublesome iPod

Footnotes§</a> </h2> Those old wide + flat connectors from back in the day before Apple made 2-3 more connectors and now everything requires a damn dongle.</li>

(Re)New Beginnings...

Headphone Review: PSB M4U 1 vs B&W P7

Pros§</a> </h4> Memory foam ear pads.</li> Excellent head band cushion.</li>

Pros§</a> </h4> Much larger earpad volume</li> </ul> Cons§</a> </h4> Non-memory phone ear pads</li>

Cons§</a> </h4> Non-memory phone ear pads</li>

Build Quality§</a> </h3> I have to admit build quality turned out to be more of a care-point for me than I would have thought, so let's break down some differences.</p> Again, we'll start with the PSBs:</p> Pros§</a> </h4> Prime use of leather and metal</li> Two axis earcup pivot</li>

Pros§</a> </h4> Prime use of leather and metal</li> Two axis earcup pivot</li>

Cons§</a> </h4> Plaassticc, specifically the headband cover and overall exterior.</li> Gut wrenching snap when folding the headphones up, makes me think I'm doing something wrong everytime I fold them up.</li>

Cons§</a> </h4> The metal arms seem marginally thin on the B&W's, which makes one worry about permanent bending or damage.</li>

Getting mac ready to compile Vapoursynth

Running IntelliJ with JDK 1.7 and Scala 2.10 on Mac OS X 10.9 Mavericks

Installing XBMC 12.2 on an ASRock ION 330HT

Pick Your CUDA Device at Program Runtime

Profiling CUDA through Python with NVVP

W3 Total Cache and DONOTMINIFY

Reading Unicode from raw_input in Python

About the only good thing in iOS 6

Stopping BOINC from running on boot up in Ubuntu

Quick Solutions to iOS 6 Issues

Beginnings...

Footnotes §</a> </h2>

Those old wide + flat connectors from back in the day before Apple made 2-3 more connectors and now everything requires a damn dongle.</li>
Although they aren't nearly as cheap as they should be</a>.</li> </ol>