Code is a liability

Bruno Schaatsbergen on architecture and development in the cloud.

Global KMS keys and secrets in AWS

I absolutely love encryption and secret management in AWS. Don’t get me wrong, it does deserve some love here and there but it’s coming together quite well. In this post I want to give you a brief introduction on how to manage KMS keys and secrets in Secret Manager globally. Multi-region KMS key July last year AWS introduced multi-region keys. A new capability that lets you replicate keys from one region into another....

January 23, 2022 · 4 min

AWS remote backend module

Everytime I have to setup a remote backend in AWS I end up getting slightly more annoyed. I’m either copy pasting code from previous projects (where I manage to improve it after every copy-paste) or I deploy a CloudFormation stack that I forked a while ago from a colleague. It was time to take matters into my owns hands, why not create a Terraform module for this? After all, these remote backends are not rocket science....

January 20, 2022 · 2 min

You’ll run out of money before they run out of space

100 trillion objects That’s how many objects S3 managed on its 15 year anniversary. Quite impressive, right? In this article I aim to shed a bit of light on object storage architecture (very high level) and why cloud providers state that you can store a unlimited amount of objects on their platform. What is object storage? Object storage systems, put data into objects, discrete ‘containers’ that each have a unique identifier, called an object ID (OID)....

January 15, 2022 · 4 min

Delete old SageMaker resources

Imagine creating hundreds of SageMaker resources a day (models, pipelines, etc). You’ll quickly end up with a very polluted environment where searching for a specific resource quickly turns into a tedious thing to do. After running into this problem several times a day it became essential that we sould tidy up. We decided to create a Lambda function that takes care of deleting the SageMaker models using boto3, invoked through a scheduled EventBridge rule....

January 15, 2022 · 2 min

Consistent Hashing

Slicing and dicing data There’s many ways to distribute our data over multiple nodes. Some prefer to partition their data vertically instead of horizontally and vice versa. Lets briefly explore the difference to give some of our readers that are new to these concepts an idea of what we are talking about. Horizontal partitioning When you are partitioning a table horizontally we typically refer to the art of dividing a table into subsets of rows that are stored in seperate nodes....

January 13, 2022 · 16 min