On the Blog

Using Apache Airflow and the Snowflake Data Warehouse to ingest Flume S3 data

Do you use Apache Flume to stage event-based log files in Amazon S3 before ingesting them in your database? Have you noticed .tmp files scattered throughout S3? Have you wondered what they are and how to deal with them? This article describes a simple solution to this common problem, using the Apache Airflow workflow manager and the Snowflake Data Warehouse.

Data Integrity Goal

Your goal is to ingest each event exactly once into your analytic database during ETL (extract-transfer-load). You do not want to leave any events behind, nor do you want to ingest any event more than once. Otherwise, your event counts will be wrong. If we assume that any particular event is in exactly one log file, the goal becomes ingesting each log file exactly once. At Sharethrough, we have seen that this data integrity goal cannot be met without dealing with those darn .tmp files.

Read more...

Senior Staff Engineer

So You Want To Build a Keyboard

My journey begins with me already owning two keyboards, one for work and one for home. I was by all means already pushing the boundaries of minimalism. But then I saw The WhiteFox and I knew I simply had to have it in my life. And the only way I could justify getting a third keyboard (and get the...

Read more...

Senior Software Engineer

Increasing Your Happiness in Meetings

Engineers moving into leadership sometimes have a hard time navigating the increased demands for meetings. And when meetings waste your time, or go off the rails, it hurts your team’s productivity and even affect their ability to work together. How can you make meetings better? In her talk, “Three Ways to Grow Your Happiness in Meetings,” Marcy Swenson offers great strategies for making meetings vital and worthwhile.

Read more...

Director of Engineering, Exchange

Embracing Inconsistency

Humans are inconsistent and unpredictable by nature. This can come in the form of conflicting requests from upper management or from engineers who want to change teams only to want to quickly change again. Our job as leaders is to accept that fact and deal with the consequences.

Read more...

Engineering Team Lead, Client
All Posts

Our Values

Our team has a diverse set of backgrounds, experience and interests across several industries and our values bring us together. We know that culture is based on more than a ping pong table and catered lunch. These aren't just the things we wish we were; they’re the values we hold ourselves accountable to and that we're proud to embody every day.

Here's how our values apply to engineering at Sharethrough.

Purpose

Demand and Share the Context of Your Work

We believe teams are most successful when everyone understands where we're headed and why, giving meaning to the work we do every day. During onboarding you'll meet and learn about team at Sharethrough and every day you'll participate in shaping your work by understanding our market and customers. We encourage close collaboration and communication across Sharethrough, so you know the stakeholders of the features you’re building.

Action

Fail Fast and Learn by Doing

We work in two-week iterations and value decision over perfection. We test-drive to facilitate rapid, responsible progress while our Continuous Deployment pipeline pushes every green build to production. If we aren't shipping, we aren't learning and if we break something - that’s okay our automated deployment cycle means we're always rolling forward to success, and we conduct blameless post mortems so we know how to improve as a team without placing blame on individuals.

Optimism

Think Big and Think Ahead

We embrace uncertainty, get excited when an answer isn't obvious and have confidence in our ability find solutions. Some of the most interesting and difficult problems don't always have obvious answers and hey, hard problems is why we're here. We love looking at our past decisions to see how far we’ve come, and reminding ourselves that the team can always tackle the next set of challenges.

Transparency

Self-Awareness and Effective Communication

You'll have a front-row seat into everything Sharethrough at our regular company all-hands. We facilitate transparency in engineering through standups and retros and while we’ve grown to five distinct engineering teams across three cities (we're primarily in SF), we’re constantly communicating and working together to deliver features from product planning through delivery.

Open Positions