Managing fedora toolboxes with ansible (including pyspark installation)

I’ve been using toolbx to manage development toolboxes on Fedora for quite a few years now, but I’ve been a bit lazy about automating the setup of new toolbox containers. I’ve finally got a mostly-working ansible playbook that I can share here. Getting pyspark to work took a bit of trial and error, so this will definitely be useful in the future. Briefly, a toolbox is a mutable container that allows you to install development toolchains - or anything else - in a way that is isolated from the host operating system.

Creating a custom provider for faker with user-specified weights

It wasn’t immediatey obvious how to do this so I thought I’d write it down here. I wanted to create a custom provider for faker that would select from a list of strings. I started out with a Dynamic Provider like this: from faker.providers import DynamicProvider kebab_type_provider = DynamicProvider( provider_name = "kebab_type", elements = ["Doner", "Shish", "Kofte", "Adana"] ) However, I couldn’t find a good way to specify weights for the choices, and as any regular kebab customer will be aware the distribution of these choices is unlikely to be uniform.

Setting up Jfrog Artifactory as a go proxy

In many environments, it isn’t really feasible to take build-time or even run-time dependencies on public repositories such as npmjs, nuget, or maven central,generally due to concerns about security, performance, availability, or all of the above. Historically the story for go has been slightly different than for these other platforms, preferring to fetch dependencies directly from the source - often GitHub - rather than create a centralised registry of go packages.

Iterating over local files with Terraform

This has come up a couple of times in client work, so I thought I’d write down how I go about doing it. This technique can be useful when you have enviroment-specific lists of resources - such as users - that need to be created. Consider a file users.txt: Luke Bo Daisy Jesse This file contains a list of objects we wish to create. In this example they will be of type aws_iam_user, but the same technique can be applied for any kind of resource or provider.

Local PySpark Development on Windows with WSL2, Docker Desktop, and VSCode

Installing Prerequisites I’m not a frequent user of Windows, but I understand getting dependencies installed for local development can sometimes be a bit of a pain. I’m using an Azure VM1, but these instructions should work on a regular Windows 10 installation. Since I’m not a “Windows Insider”, I followed the manual steps here to get WSL installed, then upgrade to WSL2. The steps are reproduced here for convenience: Setting up WSL2 Enable WSL dism.

Configuration as Code using Azure DevOps variable templates

One of the principles of the twelve-factor app methodology is strict separation between code and config, where config means everything that is likely to vary between deployments, and code is everything that doesn’t. Historically, was not a good fit for .NET Framework applications, which relied on tools such as web.config transformation and Slow Cheetah to apply build-time transformations to application configuration files. These transformations are based on environment-specific config files stored alongside the application code in the source repo.

Post-deployment configuration for Azure Web Apps

There are many tutorials on the web that describe more or less automated ways of deploying packaged software to Azure Web Apps. For this particular one I’ve decided to use the popular Content Management System WordPress. What many such applications have in common is that there is an initial installation procedure, consisting of copying files to a web host and configuring database access, followed by a web-based graphical installer that sets up all of the application configuration.

Retrieving forgotten secrets from Azure Pipelines

There are a couple of different ways to store secret variables in an Azure Pipeline. Secrets that are only needed by one pipeline can be created at that scope using the web UI: Creating a pipeline-scoped secret variable Secrets that are used by more than one pipeline can be added to a variable group: Creating a secret variable in a variable group Variable groups can also be linked to an Azure Key Vault.

Updating markdown image references to Hugo figure shortcodes

A short post which might be of use to some, as it took me a while to figure it out. I’ve been making a few changes to this site lately, one of which was to move from having the images remotely hosted in AWS S3 to having them locally in the repo. This was prompted by the availability of the Hugo page bundles feature, which I think was introduced several years ago without me noticing.

Pre-deployment approvals in Azure DevOps multistage pipelines

For users migrating from the “Classic” VSTS/Azure DevOps release experience, it is not entirely obvious how to set up what used to be known as Pre-deployment approvals as part of a multi-stage YAML pipeline. Pre-deployment approvals in a classic release pipeline . The documentation about this is rather unclear, not least because it mixes together concepts from the “Classic” Release Management experience with concepts from the multi-stage YAML experience. There is a clue in the second of these links, which is that manual approvals can be defined on environments .