If you’ve looked at Athena in the past and ran away screaming due to the hacks required to manage date partitions (e.g. a Lambda running every day), I have good news.

Now, it takes only one Athena statement to setup a table for your date-partitioned logs, and you are done. No more care and feeding afterward. And best of all, it doesn’t require you to change your S3 paths to have strange syntax like “/year=YYYY/”. See example below for good-old-fashioned ELB logs (the bold text shows the new auto partitioning feature):

CREATE EXTERNAL TABLE default.prod_elb_logs (
timestamp string,
elb_name string…

A 100 line Lambda function runs fine for months, then goes down for two hours, and finally recovers on its own. Cost savings or reliability — pick one.

I recently got an email alert about a certain Lambda having an elevated error %. The key error message in the logs was: “getaddrinfo EMFILE rds.mycompany.com”. No, that is not a DNS server failure — that is Node.js saying it can’t allocate any more file descriptors and is stuck. API Gateway was returning HTTP 502 errors for all requests.

We looked at the source code but found nothing obvious (disclaimer: I don’t…

I had an assignment: copy our operational database to our reporting database every night, then send a status message to Slack. I could use any tool deemed fit. What would be the laziest?

Step 1: Put your configuration and secrets (e.g. database hostname and password) in AWS Parameter Store. It takes a few clicks, if it’s not already there.

Step 2: Make a script using the language of your choice. It should use the AWS SDK to get the configuration from Parameter Store. It should get a list of database table names to copy from an environment variable. …

Recently, I created a toy benchmark in C# / .NET Core (on Linux) that spawns one million async tasks, to test memory overhead and scalability. Shortly thereafter, a kind stranger sent a PR with a Go version. I thought his results were very interesting and decided to run them myself.

Each of the one million tasks runs an infinite loop that does nothing but increment a shared counter variable (atomic Int64) and sleep for one second. The test ends after ~60 million hits are observed, which should thus take ~60 seconds. (It’s approximate because a master thread is polling the…

Karl Pickett

Professional rider of the technology hype-cycle since 1999.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store