Celebrating small Wins

Write down, or better yet blog, when you figure out or overcome something cool. Not only will it help you remember what you overcame, if you blog, you may also possibly help someone else with the same issue.

Celebrating small Wins

Before getting into the small win from this week, this post is a bit technical so..


💡    Write down, or better yet blog, when you figure out or overcome something cool. Not only will it help you remember what you overcame, if you blog, you may also possibly help someone else with the same issue. But moreover, when you're stuck on the next challenge, or you get imposter syndrome (I do sometimes), you can look back at your note or blog post and remind yourself that you have overcome things like this before and you’ll overcome them again!

Tech debt… 🤦‍♀️


In almost every IT organization, there is an inherent conflict between Development and IT Operations which creates a downward spiral, resulting in ever-slower time to market for new products and features, reduced quality, increased outages, and, worst of all, an ever-increasing amount of technical debt.

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations.
- Gene Kim, Jez Humble, Patrick Debois, John Willis*

At my current company, from just before the start of 2021, the plan was to move our entire infrastructure from AWS to Azure. The driving force was not tech-related but business-driven. As such, efforts were focused on the way forward in Azure and the existing AWS infrastructure was dormant, to say the least.
However, things change and in this case, prior contingencies that prompted the move were removed and the move to Azure was placed on hold indefinitely.

Now what... Well, before delving into a way forward in AWS, a scheduled PCI audit is here, and low and behold, the servers running Ubuntu needed to be upgraded, like last month...

So it was time to roll up the sleeves, upgrade the base OS from Ubuntu 16.04 LTS to 18.04 LTS.

I should start with this disclaimer...

While Ubuntu 16.04 LTS (Xenial Xerus) is EOL, our systems are protected. We leverage Threat Stack's vulnerability scans and all packages of high severity not patched by Canonical have already been patched internally. We're clean, but to be compliant, we either need to upgrade the OS or buy Canonical’s Extended Security Maintenance (ESM) product.

WARNING: ruby-2.3.0 is past its end of life and is now unsupported.

Yes, our Ruby version is... it's old. Circa 2015 old. There was an attempt to upgrade our Direct-to-consumer (DTC) Application's Ruby version and our ancient version of Rails 4.0.13 just after I started in 2018 by my colleague but that didn't seem to go anywhere and that effort never saw the day of light.

Ubuntu 18.04.5 LTS comes bundled with OpenSSL 1.1.1. Ruby 2.3.0 is only compatible with OpenSSL 1.0.x so I would need to custom build ruby.


Most of our developers opt to develop in VirtualBox virtual machines (VMs). They are used to the VM workflow and using such an old version of Ruby would make developing on their workstation tricky. Docker is the current preferred local development standard, and for good reason... but we sometimes like to empower developer choice.

The first issue I ran into was the following error:

TASK [app_checkout : Install Bundler gem] **************************************
fatal: []: FAILED! => {"changed": true, "cmd": ["/home/vagrant/.rvm/wrappers/ruby-2.3.0@global/gem", "install", "bundler"], "delta": "0:05:00.267413", "end": "2021-06-06 07:26:37.378568", "msg": "non-zero return code", "rc": 2, "start": "2021-06-06 07:21:37.111155", "stderr": "ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:\\n          Unable to download data from [<https://rubygems.org/>](<https://rubygems.org/>) - timed out ([<https://rubygems.org/specs.4.8.gz>](<https://rubygems.org/specs.4.8.gz>))", "stderr_lines": ["ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:", "          Unable to download data from [<https://rubygems.org/>](<https://rubygems.org/>) - timed out ([<https://rubygems.org/specs.4.8.gz>](<https://rubygems.org/specs.4.8.gz>))"], "stdout": "", "stdout_lines": []}

I could ping out of the VM but RubyGems wasn't working... For years my company used a static IP in VirtualBox, via Vagrant + Ansible deployment.  Turns out VirtualBox can't assign IPv6 address to a wireless interface..  and yes, the Vagrantfile is configured for macOS Wi-Fi en0 so when attempting to download RubyGems, which was using the IPv6 address, it fails... This VirtualBox wifi IPv6 issue is from 2009! What year is it?
In my case, assigned the static IP to my wired connection and moved on. The fun begins...

In the Dumps

In my company's DTC codebase, there are no good database seeds such that for me to test the DTC application, I need to dump our Staging database... Not ideal, especially since it's 1.73 GB.

Well, turns out that my initial import into DataGrip was done wrong somehow as our storefront data was missing and the page was 500'ing:

Completed 500 Internal Server Error in 727msActionView::Template::Error (undefined method `[]' for nil:NilClass):
    38: 	product-Ids="<%= passing_Ids.to_json %>"
    39: 	type="<%= productType %>"
    40: 	show-cart="<%= module_data["show_cart"] %>"
    41: 	inventory-show-available="<%= @store_settings['inventory_show_available'] %>"
    42: 	num-limit="<%= module_data['num_limit'] %>"
    43: ></gumm-pair>

I recreated the MySQL dump, followed these instructions and tumbled on.

Ruby 2.3.0 + OpenSSL 1.0

After a few uncommon.mk:203: recipe for target 'build-ext' failed build errors and messing with RUBY_CONFIGURE_OPTS, CONFIGURE_OPTS, CPPFLAGS, and LDFLAGS, I finally found the correct docs: https://github.com/rbenv/ruby-build/wiki#updating-ruby-build, which state:

If an OpenSSL 1.0 package no longer exists for your distro, you can install a shared library from source by downloading a 1.0.x version of the source, unarchiving the files, then running the following from a terminal.

In my case, getting ruby 2.3.0, OpenSSL 1.0 on Ubuntu 18.04 LTS, meant I needed to do the following:

# Download openssl 1.0 and install
wget --output-document /tmp/openssl-1.0.2u.tar.gz https://www.openssl.org/source/old/1.0.2/openssl-1.0.2u.tar.gz

tar zxf /tmp/openssl-1.0.2u.tar.gz --directory=/tmp

cd /tmp/openssl-1.0.2u

./config --prefix=/opt/openssl-1.0 shared
sudo make install

# Install ruby 2.3.0 with rbenv
RUBY_CONFIGURE_OPTS="--with-openssl-dir=/opt/openssl-1.0" rbenv install 2.3.0

Finally, Ruby 2.3.0 with OpenSSL 1.0, but wait, new errors.

Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate


Error during failsafe response: SSL_connect returned=1 errno=0 state=error: certificate verify failed

Turns out that documentation left out one bit.

At first this issue seemed to be related to RubyGems. I downloaded their GlobalSignRootCA_R3.pem CA certificate and put it in the RubyGems ssl_certs directory.

Then the SSL issues occurred when contacting AWS, so I found that the gem I was using could bundle the AWS certs by setting Aws.use_bundled_cert!, which didn't seem like a good solution, but I went with it at first.

Next, I was having the same SSL verify issue but this time with Contentstack, my company's CMS provider... 🙅

After banging my head against the wall 🤦‍♂️, I was running this by a coworker and he suggested I check if the new OpenSSL version had common CA certs, which usually get installed with the ca-certificates package but since this was a custom install, they didn't exist. I needed to manually install the CA certs. So at his recommendation, I symlinked /etc/ssl/certs (location of ca-certificates when the package is installed by apt in Ubuntu) to /opt/openssl-1.0/ssl/certs, the location where openssl 1.0 looks for CA certs.

ln -s /etc/ssl/certs /opt/openssl-1.0/ssl/certs

Modified the ruby-build wiki for other people struggling with this.

I'll be shipping in no time! … but there’s more debt

My frenemy Chef

Remember that Chef code you haven’t touched in a year, well, you need to incorporate the new changes, and things have changed. And how does all this code work again…? I really need to update the docs.

  1. Damn, the chef line cookbook dependency version wasn’t pinned and was literally updated 3 weeks ago with breaking changes.. Fixed.
  2. And this database cookbook dependency's postgresql cookbook dependency just updated two weeks ago, it's not happy... But this dependency is OLD. How are we using it again? Oh... we're not now... Removed.

Jenkins 🔒

Jenkins. Is. Not. Firing. WTH!… oh… yeah.. Looks like Bitbucket changed all their IPs and the Jenkins AWS Security Group (SG) is no longer valid such that the Jenkins webhook wasn’t triggered by Bitbucket… (and the visualping.io alert I had looking at Bitbucket's IP Address  listing page expired! 🤐 ).

But why would Bitbucket's IP change matter?

Jenkins is a security vulnerability haven so keeping it off the Internet, especially with a small team wearing many hats, makes security issues with Jenkins less impactful, but then the process requires fragile practices like IP whitelisting (adding proxy may be a better solution).  Fixed...

And now the Terraform Jenkins job failed, what the… oh… the AWS provider SSL cert changed and terraform's patch version is out of date (it’s still on terraform 0.11 but let’s ignore that for now), have to update that on Jenkins server too. Fixed...
Jenkins Integration, Unit, and Style tests passed!

Yay!! Packer created the AMI correctly, let’s deploy! ..

But there’s more debt

Passenger, Please Be Seated

CodeDeploy deployed but Passenger is not running 🤔 ... oh damn, looks like some things changed in the new version of Passenger on Ubuntu 18.04 LTS, let's remove passenger-dev and add libnginx-mod-http-passenger (why didn’t this come up in dev, oh yeah, using Puma in dev… But why, never mind, focus, look into that later). Fixed

Oh Maria, Maria

New AMI, here we go...
🤬 ️ ‼️
MySQL gem is causing a  segfault!!!

mysql2-0.3.21/lib/mysql2/mysql2.so: [BUG] Segmentation fault at 0x00000000000000

Kill me, seriously WTF!

Okay............ I see... it's related to this MySQL dev gem.
Apparently need to use the MariaDB dev package in 18.04, then install the mysql2 gem. Fine!

  • OUT libmysqlclient-dev
  • IN libmariadbclient-dev


FINALLY 🤩 💥💥💥💥  🙌  🥳

People don’t always  appreciate the things that go on behind the scenes 'in the kitchen'. You might not get recognized for ‘heroic’ efforts to get the food, ahem 🍷, out to the tables but it’s still a win and you need to note these little triumphs.