When you search for a video on Storyblocks — “redwoods,” “Aspen festival,” or “York skyline” — how does the site know where that is?

Until recently, the answer was: it usually didn’t.

And our customers noticed:

“Finding some of your images is almost impossible because the search engine doesn’t work properly… Many Brazilian cities will never be found directly. I lost hours to get images that should have appeared instantly.”

“Not enough content from Africa.”

Enterprise customers and prospects asked similar questions directly to our support and content teams — Can I get footage from a specific city? From a country? From where we’re expanding next quarter?

The problem wasn’t content.
It was location.

Video metadata is messy. Titles are short. Tags are inconsistent. Locations are abbreviated, misspelled, implied, stuffed, or sometimes just… forgotten.

So we built SB-NERD: Storyblocks’ internal system for Named Entity Recognition + Disambiguation, focused entirely on geographic locations.

Its job is simple to say and hard to do:

• Find location mentions in video titles and tags or infer them
• Figure out what real place they refer to (if any)
• Link them to a known locality with latitude, longitude
• Or label it NO LOCATION or AMBIGUOUS LOCATION

The problem with location at scale

Humans are great at location inference. Machines are not.

Consider these titles:

• “HD San Fran pier timelapse”
• “Zion NP hiking b-roll”
• “Chikago skyline at night”
• “Aspen festival crowd cheers”
• “York city drone shot”

To a human, these are obvious.
To a machine, they’re a minefield.

• Is “Aspen” a city, a tree, or a brand?
• Is “York” New York? York, UK? York, Pennsylvania?
• Is “NP” a location or just noise?
• Is this even a real place — or a studio shoot where location doesn’t matter?

On top of that consider:

The world has a lot of repeated names.
The internet has even more misspellings.

What SB-NERD actually does

At a high level, the pipeline looks like this:

Title + tags → location candidates → best-guess location (or none)

Under the hood, SB-NERD has two main stages:

1. NER — Extract candidate location names
2. NED — Decide which real place (if any) those names refer to

Step 1: Traditional NER (and why it broke)

The obvious starting point was classic Named Entity Recognition or NER:
Run an off-the-shelf model, extract anything labeled LOCATION, and try to match it.

It worked… sometimes.

But Storyblocks metadata isn’t clean prose. It’s short, fragmented, and inconsistent. Traditional NER struggled in two big ways:

1. It missed things we cared about

Abbreviations, slang, and partial references were often ignored:

• “San Fran”
• “SoCal”
• “Zion NP”

2. Extracted text wasn’t matchable

Even when NER returned something like “SF,” (comma included) OR “t atl” that’s not a usable key for a geographic database without a lot of extra logic.

We realized we were asking the wrong question.

Instead of:
“Is this word a location?”

We needed:
“What location should we try to match?”

The shift: using an LLM as a location candidate generator

The breakthrough came when we switched from a state-of-the-art NER model and instead used an LLM as a strict, opinionated candidate guesser.

Its job became:

Given a title and tags, output a location candidate in a format that a database like GeoNames can actually match (e.g. city, state, country)

This worked much better on messy metadata.

The model could:

• Expand abbreviations (“SF” → “San Francisco”)
• Normalize misspellings (“Chikago” → “Chicago”)
• Recognize landmarks (“Golden Gate Bridge”) and record the city that was in.
• And — critically — not force a location when none exists

The key: strict rules + few-shot examples

We got good results only after being very opinionated:

• Precise instructions
• Prefer specific over vague (city > region > country)
• Do not hallucinate
• Output clean, structured results
• Explicitly return NO_LOCATION or AMBIGUOUS for examples like “Best Buy” or if the tags have multiple contradictory locations in them

We iterated by reviewing thousands of examples, finding failure modes, and adding them directly to the prompt as few-shot patterns:

          Input: Title: Niagara Falls Aerial; Tags: ["falls", "niagara", "aerial"]
          Output: City: Niagara Falls; State/Province/Region: Ontario; Country: Canada
        
          Input: Title: Into the Woods of Aspen; Tags: ["aspen", "autumn"]
          Output: no location
        
          Input: Title: Swampy Landscape; Tags: ["swampy", "trees", "york", "new"]
          Output: unknown
        
          Input: Title: Clouds Moving Over Lush Valley Overlooking Na Pali Coast; Tags: ["hawaii", "kauai", "na pali", "island"]
          Output: City: Kauai; State/Province/Region: Hawaii; Country: United States

This step didn’t solve geolocation.
It made the next step possible.

Step 2: Disambiguation with a real-world location database

Once we have a candidate name, we still have the hard question:

Which place is it?

That’s the “D” in SB-NERD.

We anchored everything around GeoNames, an open global database of cities, regions, countries, and landmarks.

GeoNames gives us:

• Canonical names and alternate spellings
• Administrative hierarchy (city → state → country)
• Latitude and longitude
• Population estimates

But it also introduces a classic problem:

Many places share the same name.

Matching strategy: precision first, recall second

We built matching in layers, always preferring correctness over coverage.

Layer 1: Exact matching

Fast, predictable, and surprisingly effective.

To reduce ambiguity, we forced the LLM to return city, state/province, and country whenever possible. That eliminated most “Springfield, USA”-style ambiguity.

There were still a lot of unmatched locations, which, upon examination, were definitely places

Our LLM guess might have said “London, United Kingdom,” but the comparison we made to geonames might say “London, Great Britain”

Layer 2: Fuzzy matching

To handle:

• Common Misspellings (or alternate spellings)
• Accent marks (“Sao Paulo” vs “São Paulo”)
• Formatting differences

We attempted fuzzy matching, but this didn’t work very well. GeoNames’ alternate names field helped a lot here.

We ended up only matching if it was an exact match on a known alternate name in the GeoNames database. This expanded our matches more, but we still had a significant number of missed location matches that a human could have matched.

Layer 3: Embedding similarity (last resort)

For the truly messy cases — where spelling and tokens break down — we used embedding similarity as a fallback.

We kept this layer intentionally conservative.

The rule stayed consistent:

Better to return NO LOCATION than be confidently wrong.

After these 3 steps, we finally were able to confidently tag our library and label exact locations for nearly 1 million videos.

Why this matters & What it unlocks

Location is a core part of a video’s identity.

A misty forest in Oregon is not the same as one in Japan.
A city skyline means very different things depending on where it is.

Accurate geo-tagging unlocks new product value:

• Better search results for place-specific queries
• Natural-language geo search gets much more precise (“videos from coastal Brazil”)
• Geographic diversity in results and collections
• “More from this place” recommendations
• Enterprise workflows tied to regions and markets

It also lets us see the catalog differently.

Once locations are structured, we can visualize coverage, spot gaps, and explore the library on a map (not just a search bar).

And yes, next time you search for “Belo Horizonte,” the right videos should finally show up.

What’s next?

As always, there is more work to do — like expanding our natural language understanding in search, shipping geoquerying on the site, & adding EVEN MORE AI features (not to mention all the other great things Storyblocks’ Data Science team is cooking up).

If that’s work you’re interested in, too, please check out our career page.

Stay tuned for more updates from the Data Science team. And happy searching!

Where was this video shot? was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

We started by asking ourselves: What makes you click?

Put differently, what makes you like a piece of content enough to engage with it? It’s a hard question to quantify, but that’s precisely what we try to do on the Storyblocks’ Data Science team. The more we understand about why users like content, the better we can help them find the content they like.

Long story short: we decided thumbnails matter

We’ve noticed that sometimes our thumbnails don’t always do justice to the amazing videos they represent. A great thumbnail can be the difference between a click and a scroll, and we want to make sure our users are clicking on videos that truly meet their needs. To address this, we’ve embarked on a quest to find the perfect frame — the one that shouts, “Click me!” without actually making any noise.

Side note: what’s a thumbnail?

A thumbnail is the first image you see on our video search page. It’s a single frame from the video meant to represent that video’s content without having to play the video. Here’s what they look like today:

Our old thumbnails just defaulted to the first frame of the video, which could often look like this (car crash):

Or this (cards falling):

Or this (truck camper on a bridge):

Our 4 Step Thumbnail Transformation Journey

Step 1: Understanding User Engagement

First, we needed to understand what makes a thumbnail appealing. Is it the brightness? The action? The presence of smiling faces? We delved into our data to see which thumbnails were getting the most clicks and downloads. This involved a lot of coffee, a lot of data crunching, and a bit of guesswork.

Step 2: Leveraging Machine Learning Magic

With our insights in hand, we turned to the powerful world of machine learning using 3 seperate models. First, We needed a model to help determine which frame of a video is most relevant to the search. We assumed that if a video appears in a search because its title is close to the query, then a frame similar to the title would be the most representative. For this, we used CLIP (Contrastive Language-Image Pre-Training), which helped us align the frames with the titles in a meaningful way.

Rather than score multiple text-to-image similarities as in OpenAI’s example below, we scored multiple image-to-text similarities to get a score per image that said how similar each image was to the title text. This allowed us to say which out of ~15 thumbnails was the most relevant to the title. Check out Huggingface for some code snippets similar to how we used it.

Step 3: Rating the Aesthetic Appeal

Next, we needed to ensure the chosen frame wasn’t just relevant but also visually appealing. For this, we used another model to rate images based on their aesthetic appeal. This model was fine-tuned several years ago to work well on our stock image and video data. We were able to repurpose the existing model. This process was made significantly easier by our previous use of our databricks mlflow model registery and our private python utlities packages. The code looked something like this:

import sb_quality
from sb_quality.predict import get_quality_scores

DELTA_TABLE_NAME = f"{ENV.ds_db}.frame_quality_scores"
#get image urls
loader = build_loader(spark)
stock_items = (loader
                   .get_stock_item_encoding(content_class=CONTENT_CLASS, **image_url_types[CONTENT_CLASS])
                   .filter(F.col('url').isNotNull()))

#only process new images
if spark.catalog.tableExists(DELTA_TABLE_NAME):
  stock_items_processed = spark.read.table(DELTA_TABLE_NAME)
  stock_items = stock_items.join(stock_items_processed, 'url', 'left_anti')

#run our model with our utility function:
scores = get_quality_scores(spark, configs, stock_items)

merge_into_delta(df=scores,
                delta_table_name=DELTA_TABLE_NAME,
                merge_keys=['content_class', 'content_id', 'url'],
                columns_to_partition_by=['content_class'],
                isolate_to_partition={'content_class': CONTENT_CLASS.value})

# read our quality scores in downstream applications & jobs
query_quality = f'select * from {DELTA_TABLE_NAME}'

quality = spark.sql(query_quality)

If you’re curious what that get_quality_scores function is doing under the hood, here’s anothe code snippet to give you a better idea:

def get_quality_scores(spark, configs, stock_items):
    batch_size = configs['BATCH_SIZE']
    num_workers = configs['NUM_WORKERS']

    model_state = get_model_state(configs)

    num_classes = model_state['num_classes']
    base_model = model_state['base_model']
    bc_model_state = spark.sparkContext.broadcast(model_state['state_dict'])

    initialize_with_model_state = build_initialize_with_model_state_func(bc_model_state)

    def predict_batch(urls):
        try:
            with tempfile.TemporaryDirectory() as download_dir:
                files = [
                    download_image(url=url, download_dir=download_dir) for _, url in urls.items()
                ]
                print('number of files to predict = {}'.format(len(files)))

                model, input_size = initialize_with_model_state(num_classes=num_classes, model_name=base_model)
                model.to(device)

                transform = create_data_transform('test', input_size)
                images = ImageDataset(files, transform=transform)
                loader = torch.utils.data.DataLoader(images, batch_size=batch_size, num_workers=num_workers)

                all_predictions = []
                with torch.no_grad():
                    for batch in loader:
                        predictions = list(
                            torch.nn.functional.softmax(model(batch.to(device)), dim=1).cpu().numpy()
                        )
                        for prediction in predictions:
                            all_predictions.append(prediction[1])

        except exceptions.SbQualityError:
            all_predictions = [None for _ in range(len(urls))]

        return pd.Series(all_predictions)

    predict_udf = pandas_udf(FloatType(), PandasUDFType.SCALAR)(predict_batch)
    predictions_df = stock_items.select(col('content_id'), col('url'),
                                        predict_udf('url').alias('quality_score'))

    return predictions_df

Step 4: Bringing It All Together

Combining these two models in a third custom FFN (feed-forward neural network), we developed a system that selects the best thumbnail for each video. The Code looked somthing like this:

# our dataset used the similarity score, aesthetic quality score, & media type
# we scaled our features before feeding them to our model
feature_cols = [
  'quality_score',
  'cosine_similarity', 
  'quality_clip_interaction_term',
  'content_type__motion-background', 
  'content_type__template',
  'content_type__vr360']
X = dataset[feature_cols]

logged_scaler = 'models:/Download_Rate-CLIP-Quality_Model_Standard_Scaler/Production'
loaded_scaler = mlflow.sklearn.load_model(logged_scaler)

X_scaled = loaded_scaler.transform(X)


#final model & download rate prediction: 
logged_model = 'models:/Download_Rate-CLIP-Quality Model/Production'
# Load model as a PyFuncModel.
loaded_model = mlflow.pyfunc.load_model(logged_model)

# Predict on a Pandas DataFrame.
dataset['predicted_download_rate'] = np.exp(loaded_model.predict(pd.DataFrame(X_scaled)).flatten())

The raw features & predicted score looked something like this:

The visual result? Thumbnails that are both representative of the video content and visually engaging. Here’s an example of those videos from before and what they look like now after our thumbnail makeover:

The Impact

Since implementing our new thumbnail selection system, we’ve seen a significant uptick in user engagement. More users are clicking on videos, and the satisfaction with their choices has gone up. It’s a win-win!

That said, Not Every Thumbnail is Perfect

While we’ve seen great overall improvement, there are always exceptions. Sometimes our new thumbnails don’t hit the mark as well as we’d hoped. Despite this, the majority of our updates have been positive, and we are constantly learning and tweaking our models to get better. By the way, what do you think of our new thumbnails? Let us know in the comments below!

What’s Next?

We’re not stopping here. Our team is continually exploring new ways to enhance your experience on Storyblocks.com Whether it’s through better recommendations, smarter search, or even more engaging thumbnails, we’re always working on something exciting. If that’s work you’re interested in, too, please check out our career page.

Stay tuned for more updates from the Data Science team. And as always, happy searching!

Finding the most clickbait-y thumbnail for you was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Storyblocks, we know how to balance work and play. I’m not just talking about letting me go full Utah ski bum while still paying me. As a company, we’ve commited to a structure that helps personal & professional growth.

The perks are great — think unlimited PTO, “No Meeting Wednesdays,” and both summer and winter vacations — but one of the most impactful benefits is our commitment to personal and professional growth through dedicated level up time.

What is Level Up Time?

Level Up Time is our initiative to ensure every team member has dedicated time each sprint to focus on personal growth. Whether it’s taking a course, working on a side project, attending a conference, or even writing blog posts, this time is all about you. And trust me, it pays off for both you and the company.

Courses I’ve Taken: Text Retrieval & Search Engines

One of the courses I dove into was on Text Retrieval and Search Engines. This course was a game-changer, deepening my understanding of how search engines work and how information retrieval systems can be optimized.

How this benefits me:

Enhanced my technical skills in building and optimizing the Storyblocks search service my team is responsible for.

Gave me insights into the latest algorithms and techniques in information retrieval.

How this benefits Storyblocks:

Deepened my knowledge of how search worked on our site (and it’s all over our site).

Enabled me to develop innovative search-related prototypes that can be integrated into our product offerings.

Building Prototypes: From Face Search to Deep Fakes

Level Up Time also allowed me to get hands-on with several exciting projects. Here are a few prototypes I’ve been working on:

1. Face Search: This tool helps users find more faces like the ones they’ve seen in a video. It’s perfect for creators looking for consistent imagery or themes in their projects. I wrote about how we used this to improve a feature on our site here.

2. Follow More Contributors Like This: An intuitive feature that recommends contributors based on user preferences, making it easier for users to discover new content they love. Using a nearest neighbor approach we could recommend a contributor like Space_Motion for users who liked content from a contributor like Albatros

3. Searching Through Embedding Space: Imagine selecting content you like and don’t like, and then finding new content that averages out those preferences. This advanced search technique uses embedding space to deliver highly personalized recommendations.

Our original Search for the word, “Travel”:

Our “Travel” Search after choosing words and videos we don’t like (in this case videos of planes):

4. Deep Fake Video Generator: By inputting an image of a person, this tool creates a deep fake video. It’s a powerful demonstration of AI and machine learning capabilities, opening up new possibilities for video content creation.

Note: To avoid putting fake images on the internet, I won’t be sharing this one in our blog, but it was pretty fun making the head of engineering appear to be doing things that they actually hadn’t.

Attending & Speaking at a Conference

Level Up Time also gives us the opportunity to attend and speak at industry conferences. I recently was given the chance to both attend and lead a workshop at a major conference in our field, Ai4. Storyblocks gave me the time to go do it.

How this benefits me:

Expands my professional network by connecting with industry leaders and peers.

Enhances my public speaking and presentation skills, boosting my confidence.

How this benefits Storyblocks:

Showcases our innovative projects and tools, positioning Storyblocks as a thought leader.

Valuable insights and trends can inform our product development and strategy.

Writing Blog Posts

Level Up Time has also given me the opportunity to share my experiences and knowledge through blog posts like the one you’re reading. Writing about these projects and courses not only helps solidify my learning but also contributes to the broader community by providing insights into our corner of the DS world. It’s also fun 🙂

The Ripple Effect: Personal Growth and Company Success

Taking time for personal growth isn’t just good for me — it’s beneficial for Storyblocks as well. The skills and knowledge I gain directly contribute to our product innovation and improve the user experience. Plus, staying up-to-date with the latest trends and technologies ensures that Storyblocks remains at the forefront of the industry.

So, if you’re considering how to make the most of your time at work, remember that investing in yourself is a win-win. Whether you’re skiing down a Utah slope or diving into a new course, take advantage of every opportunity to level up. Your future self — and your company — will thank you.

If the idea of a company that not only invests in you, but gives you time to invest in yourself is appealing to you, check out our careers page and come work with me!

Level Up Time at Work: Elevate Your Skills and Your Career was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I bring you another story from Storyblocks’ Data Science & Machine Learning Team a.k.a. the custodians of search. This time we build a prototype & have some fun with our friends in User Research, Product, & Data Engineering.

Buckle up, because I’m about to take you on a wild ride through our recent hackathon, where we built a Retrieval-Augmented Generation (RAG) model for customer feedback.

Spoiler alert: it was a total blast, and we even won the hackathon! 🎉

The Databricks Debacle: A Bumpy Start

Our adventure began with high hopes and the Databricks tutorial on this topic. We thought we had everything we needed to deploy a killer LLM chatbot with RAG and Databricks AI Vector Search. We eagerly followed along step by step. Things were looking good until… drumroll please… Unity Catalog decided to play hard to get. (We don’t have it enabled in Storyblocks’ version of Databricks.) 😩

That’s right. No Unity Catalog, no Databricks magic. Our dreams of seamlessly deploying our model were shattered. But we didn’t give up! This was a hackathon, and we were here to win. So, we rolled up our sleeves and pivoted faster than a 2 person Silicon Valley startup.

(In reality: We went to lunch; thought about bailing on the project; then decided to give it another go for funsies.)

Hugging Our Way to Success: Enter Hugging Face

With Databricks out of the picture, we turned our gaze to the wonderful world of Hugging Face. If you haven’t heard of them, they’re like the fairy godmother of open source models. We decided to grab an LLM (Large Language Model) from Hugging Face. Why? Because they have an insane collection of pre-trained models, and they’re super easy to work with.

We chose a model that could handle customer feedback like a pro, and we were off to the races. But wait, we needed a way to search through this feedback quickly and efficiently. Enter FAISS.

Building Our Own FAISS Index: DIY Style

FAISS (Facebook AI Similarity Search) is a library that lets you create fast and efficient similarity searches.

Translation: it’s perfect for our RAG model. We decided to build our own FAISS index with a Python script. Here’s a little snippet to give you an idea of how we did it:

import faiss
import numpy as np
from transformers import AutoTokenizer, AutoModel
# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModel.from_pretrained("distilbert-base-uncased")
# Sample customer feedback data
feedback_data = ["Great service!", "Loved the product.", "Could be better.", "Terrible experience."]
# Tokenize and embed the feedback data
encoded_inputs = tokenizer(feedback_data, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
 embeddings = model(**encoded_inputs).last_hidden_state.mean(dim=1).numpy()
# Create FAISS index
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(embeddings)

The Dream Team: Working Together for 48 Hours

Now, let’s talk about the real MVPs: our team of six. We had a diverse mix of skills, from developers to a PM to a UX researcher. For 48 hours, we worked like a well-oiled machine. There were late nights, countless cups of coffee, and a few moments where we questioned our sanity (read: 1 late night, 2 cups of coffee, and we drank a beer and went to a ball game. TBH it wasn’t that hard lol).

But in the end, it was all worth it.

We divided and conquered: some of us imported the 750k most recent comments from our users into delta, some of us prompt engineered questions to ask the LLM, and some of us built the FAISS index, finetuned the model, & the app that used it.

The components tie together like this:

The Moment of Truth: Presenting Our RAG Model

When it came time to present, we were nervous but excited. Our RAG model could retrieve relevant customer feedback in a flash and generate insightful responses like a charm.

One last curve ball: We mixed up our presentation time! We thought our presentation was at 11 am, when it was really at 10 am, and we had just started making our presentation at 9:55 am. Miraculously, we threw it together in time.

Ultimately, The judges were impressed by the quality of the generated responses and their potential to help inform how we improve our site. Our app was able to sort through many of our users’ comments (~750K), find the most relevant ones for answering questions, and summarize findings with actionable bullets.

Many of those comments could be totally irelevant and looked like this:

1. ayeeeeeeeeeeeeeee

2. I need rare pokemon.

3. Since I’ve started listening to your sounds and music, it gave me a clear view, a vision of happiness. Left my wife and kids, started rollerblading, quit cheese, moved to a forest and live in a tent. Thank You for showing me a way out when I was lost.

The app could churn through all of them and get to something like this in seconds:

1. Upgrade the Content Closet:….

2. Pimp your Search Engine:….

3. Facelift the Interface:….

4. Rethink the Price Tag:….

5. Schmooze & Collaborate:….

6. Niche It Up:….

All these sassy recommendations & more in a matter of seconds!

Please Note: details of the response were removed for the purpose of posting this publicly & keeping our secret sauce secret — looking at you Shutterstock 👀. Readers, you’ll just have to trust me that each of these bullets had a more in-depth summary of what to fix on our site.)

And the Winner Is…

WE WON! That’s right, our scrappy little team took home the grand prize — or at least the hackathon business prize. It was a sweet victory, made even sweeter by the challenges we faced along the way. We proved that with a bit of creativity, teamwork, and a lot of determination, you can hack your way to success.

This was all good fun, and there were plenty of other impressive projects that the wider Storyblocks org came up with in two short days. Speaking of fun, our day-to-day work can be pretty enjoyable. If you like the work you see us doing and want to do it, too, check out our current openings.

Until next time, happy hacking! 🚀

Hacking your way to a RAG Model for Customer Feedback was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Celebrating Wins

Here at Storyblocks, we recently had some major cross-functional accomplishments that required a lot of collaboration and brain power to implement. In the spirit of ‘work hard, play hard’, the Product & Engineering crew wanted to take a moment to celebrate together in recognition of these efforts. Given our remote work setting, we needed a creative way to host a virtual event with an interactive element. I had found a ‘mystery mansion’ template on Gather’s website that I was dying (pun intended) to use, so we decided to organize a virtual murder mystery.

Leaning Into AI

Since we only had a week to create the event, we used AI as much as possible. We generated most of the narrative using ChatGPT, then added a few extra Storyblocks-specific touches. Additionally, most of the text-based clues were drafted with ChatGPT and the ‘photographs’ we used were created using the MidJourney AI tool as well.

Crafting the Experience

1. The Space

Though it’s absolutely possible to create a custom murder mystery space in Gather, we knew that we wanted to use the mystery mansion, which came with built-in rooms and secret passages. We ended up building the narrative and clues around that space. If you want to customize your spatial experience for your own murder mystery event, I recommend doing that first, as the themes and clues can be enriched by the environment your players will be in.

2. The Narrative

We asked ChatGPT ‘can you come up with a murder mystery narrative that takes place in a mansion?’ and it gave us a great multi-paragraph backstory to start with. From there, we asked it to make some alterations, including ‘can you add some unimportant details about the event?’. A few variations of this request gave us a rich backstory with details such as “trays of rare vintage wines and handcrafted cocktails garnished with edible gold leaf” being served at the event prior to the murder. We also added the Shenandoah mountains as the mansion location — a tribute to Northern Virginia, where our home base is located.

3. The Investigator Personas

Depending on the size of your group, you might be able to build out personas for each individual. Our group was too large (~16 people), so we decided to randomly assign people (using ChatGPT of course) to three distinct groups. We asked ChatGPT ‘can you create three distinct groups of investigators?’ and got pretty generic results, so we decided to provide themes that would be easy to put together costumes for. One was a group of recent college grads who wear shirts representing their favorite universities. Another was a group of mediums who wear bright colors. The third was a group of mysterious investigators who wear all black. From there, we tried again, providing the generic themes and requesting unimportant details. ChatGPT provided gems like a pet bat named Midnight for the third group and an “experimental psychedelic rock band” for the second.

4. The Suspects

Again because of our group size, we wanted to limit the number of possible suspects. We asked ChatGPT to give us a list of five suspects with backstories based on the narrative we had already generated, then changed the suspects’ first names to those of leaders involved in the tasks we were celebrating. Once we had a solid draft, we asked ChatGPT to randomly select the murderer from that list.

5. The Rules

We made a rules document outlining basics about the event, including how to interact with objects, the timeline of the event and how to submit guesses. We also noted that there would only be one murderer. Guesses and secondary guesses (in case no one guessed correctly) were submitted by a group-appointed captain using a Google form. We also requested supporting logic for each team’s primary guess, which was a fun detail for our moderators to see. We used the following timeline:

• Intro: ~10 minutes
• Team Discussion + Team Captain Selection: ~5 minutes
• Initial Investigation: 15 minutes
• Team Discussion: ~5 minutes
• Final Investigation: 10 minutes
• Final Discussion + Guess Submission: 10 minutes
• Announcement of Winning Team

6. The Clues

In a traditional game of Clue, timing is a key element: players typically discover clues at a controlled pace. Using Gather as a medium, we had no control over when certain clues were discovered, so we had to account for that with the clues we provided to make identifying the culprit more difficult — but possible. Based on a list of items ChatGPT suggested, we ended up with the following clues:

Eye Witnesses: We came up with 12 eye witnesses (including the suspects) to interact with in Gather. Their avatars were created using this site, then placed in Gather with prompts and notes we primarily created using ChatGPT.

Cell Phone: Missed calls from a suspect could be seen by clicking the phone icon on this phone, which was created using Google Drawings

Custom Gather Objects (created using Krita):

• A pen with a suspect’s company brand was placed at the murder scene
• Blood splatters of various sizes were placed on both ends of a secret passage
• Headphones were placed at the murder scene and, according to an eye witness, belonged to one of the suspects
• A USB on the floor contained a code to unlock a PDF on the victim’s computer
• A diary contained entries mentioning suspicion of one of the suspects
• A surveillance camera, placed in the hallway, showed an image of a suspect leaving the parlor. We edited the image to add text with a timestamp to indicate that this occurred right before the victim was found

Built-in Gather Objects:

• The laptop object provided a locked PDF with suspicious emails
• The note object provided an angry letter from the victim’s daughter
• The document object was used for financial documents, a pawn shop receipt and a ‘grocery list’ that was actually a coded message
• Blueprints linking to a pdf we created with Google Drawings depicted the secret passageways in the space
• A duffel bag placed near one of the suspect included a prompt noting that the bag was full of the victim’s jewels

After generating the content of the text-based clues, we formatted and shared them using Google Drive’s ‘publish to the web’ feature. Players could access these documents by interacting with embedded website objects in Gather. For the locked PDF specifically, we had to drop a link to the PDF in a note object to allow players to interact with the PDF.

Putting it All Together

A few days before the event, two moderators did a walkthrough. Their feedback prompted some slight alterations to the clues to ensure they weren’t too challenging or confusing. This step provided a lot of insight about the difficulty level and flow of the experience.

We created separate Slack channels for each group, as well as an all-participants Slack channel. These channels were used to introduce people to their teammates and team personas ahead of time. They also enabled everyone to communicate when in different Gather rooms during the event.

The day of the murder mystery, we played suspenseful music using Storyblocks’ music library and Gather’s ‘self spotlight’ feature as players gathered in their groups. We then did a brief introduction, a dramatic reading of the narrative, and a run-through of the rules.

Finally, after the players had time to investigate, regroup, then investigate again, we brought everyone together and called for final guesses. In case you were wondering, one group did correctly guess the murderer. More importantly, everyone seemed to have a great time.

Conclusion

Thanks to the powers of AI and Gather, we were able to throw a somewhat complicated event together in a matter of days! When you lean into modern tools, you can really accomplish a lot in a small amount of time.

If you want in on our innovative digital culture, we are hiring!

Who Killed Sir Redshift: How We Used Gather and AI to Throw a Virtual Murder Mystery Party was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

🎧 Step 1: Select Engaging Content
Every few weeks, pick a podcast or video presentation that piques your team’s interest. Explore topics that align with your goals and spark curiosity.

👥 Step 2: Host a Media Meeting
Set up a dedicated meeting to watch or listen to the chosen content as a team. This not only creates a shared experience but also allows for valuable discussions during the session.

📝 Step 3: Collaborative Note-Taking
Encourage everyone to take notes in a shared space during the meeting. This should be informal and relaxed. This serves as a record of individual insights, fostering collaboration and ensuring no ideas slip through the cracks. As a bonus, rate the content to serve as a recommendation for others.

🔄 Step 4: Rotate the Curator’s Role
After each session let different team members take the lead in selecting the next piece of content. This not only keeps things fresh but also provides diverse perspectives.

Benefits of this Approach:

Team Bonding: Shared experiences strengthen team bonds, creating a sense of togetherness even in virtual spaces.

Continuous Learning: Regular exposure to new ideas and perspectives keeps the learning curve steep, fostering a culture of continuous improvement.

Idea Exchange: The collaborative note-taking process encourages open dialogue, leading to the exchange of thoughts and insights.

Let us know how you promote learning in your org! If you would like to join us to get your virtual learn on, we are hiring!

4 Steps to Spark Collaboration & Learning in Your Remote Team was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

• In general, pull requests include tests
• Passing tests is a step in each CICD pipeline
• Our QA team is invested in our automated test coverage

… and some bad:

• No clear strategy on when to write what type of test
• Some apathy towards writing “good” tests (I feel you)
• A lot of legacy tests that are not good, pin us to old technologies, or flake often, which give us negative feelings about our suite as a whole

We’ll name our (very real) Painful Automated Testing Strategy “Patsy”. A good name, because Patsy gets blamed for a poor developer experience, lag in getting code to production, and — for all the lines of code that compose her — she never gives us the coverage and confidence we need. When she’s slow, we’re mad. When she doesn’t catch an issue, we’re mad. When we can’t debug her, we’re mad. Poor Patsy.

Although I’ve seen more bad than good in my career, I’ve developed a few opinions along the way.

Some of them are about deciding to create new tests that are valuable to you… and worrying about the old stuff later. There’s value in simply not making the problem worse.

Others are about how to get test writing off my plate — while still writing good code that we’re confident in, of course.

Test requirements, not implementation

Katy’s simple truths of engineering posit that:

1. The same goal can be accomplished in a million different ways. This is beautiful.
2. Engineers, as protectors of a secret and powerful realm of knowledge, hold the power to do things how we want. This is also beautiful.

I got a little righteous there, but what I’m getting at is: we don’t have to make test writing hard on ourselves. Implementation details change over time and shouldn’t be the subject of our tests. If you find yourself writing a test to confirm a certain function is called with certain arguments, what you’re really doing is adding friction for yourself or another engineer when the target code has to change. Think instead about what would make you confident your code is working as designed. That’s often about outputs rather than inputs.

At a previous job, I worked on a team where our Business Analyst got involved in the testing strategy. Plain english requirements were translated into “it(‘should’)” Cypress test signatures before development started, and the manual QA team even scaffolded them in a PR, so that feature engineers could fill them in throughout development. This had a few benefits:

1. Non-engineers got way more invested in writing complete requirements, including edge cases.
2. Non-engineers were more able to make a priority call when a test broke or failed, because they shared ownership of the test (and the requirements).
3. Engineers, an often mischievous breed, were not left to their own devices / allowed to skip tricky tests.

It was a lot of overhead to get this practice started, but I liked the theory of it. Is it part of your mythical Patsy?

Feed it to the robots

I am so ready for robots to take over test writing. When AI is more able than me to analyze requirements and code and generate tests to cover it… boy howdy.

One thing we’re trying out at Storyblocks is Github Copilot. My copilot AI buddy is getting better every day at using context clues and my personal habits to suggest code snippets. I can definitely see the potential to pass off test-writing responsibility in the future, and focus more on quality and coverage.

Re-evaluate your feelings about snapshot tests

Snapshot testing is an approach to making sure all downstream effects of a change were intended by the developer. By adding a snapshot test, you’re saving an output at a point in time to source control, so that next time you make a change, any impact to that output is clear in a diff. Usually used for DOM elements, they could be used for function outputs or state too.

Say I added a snapshot test for this element, called SideNav.

Here’s an old test we have to confirm the right options are shown in SideNav for a Visitor (as opposed to a Member). It’s ok, but it’s actually a pretty lossy test.

it('should show the right set of drawers for a visitor', async () => {
    renderWrapped(<AssetRailNav />, { initialState }});
    expect(await screen.findByText('Stock')).toBeInTheDocument();
    expect(await screen.findByText('Uploads')).toBeInTheDocument();
    expect(await screen.findByText('Styles')).toBeInTheDocument();
    expect(await screen.findByText('Text')).toBeInTheDocument();
    expect(await screen.findByText('Overlays')).toBeInTheDocument();
    expect(await screen.findByText('Logos')).toBeInTheDocument();
    expect(await screen.findByText('Folders')).toBeInTheDocument();
    expect(screen.queryByText('Record')).not.toBeInTheDocument();
    expect(screen.queryByText('Brands')).not.toBeInTheDocument();
    expect(screen.queryByText('Templates')).not.toBeInTheDocument();
});

It doesn’t make any assertions about the order of SideNav items, which icons go with which labels, or whether the elements are interactive. It also requires manual rewrite if any of the labels change.

Here’s a snapshot test for the same component.

it('should show the right set of drawers for a visitor', async () => {
    renderWrapped(<AssetRailNav />, { initialState }});
    expect(wrapper).toMatchSnapshot();
});

When you run the test, a snapshot is generated, and you add it to git. It holds a lot of information. The items in SideNav are clearly buttons that contain SVGs and labels, in a specific order.

Uh oh, Design wants the items in SideNav to be reordered! Your wish is my command. Make the change in the component, run the test again, and see that the snapshot doesn’t match anymore.

Looks exactly like I intended! Accept the changes, commit, move on with my life.

This example doesn’t capture the main complaints I hear, which are: Too many snapshots have to get regenerated too often! The generated files are huge and hard to read! It adds friction to making a code change!

In my opinion, they’re still super readable if you show restraint when generating snapshots. Stick to the component level, don’t snapshot whole web pages. If you’re well-versed in semantic HTML, the diffs should be readable at that level… and if they’re not, either your code or your component is too complex. They could also illuminate something you didn’t mean to change, or help your teammates review your code better.

Friction is minimal, imo, because visual diffs are the easiest way to understand changes, and updates are as simple as pressing u while the tests are running in watch mode.

There’s lots more to read about in the debate, but if used responsibly, snapshots can be useful for teams weary of writing custom tests.

Choose the test type based on the situation

Think of a software application as an obstacle course. (We are now obstacle course overlords.) Each time we develop a new feature for the app, we have to run the obstacle course again to make sure it’s working as expected, getting sliced by barbed wire and bruised by mechanical arms with boxing gloves that come out of nowhere. We take the fastest and easiest way through, and just get it behind us, every time.

But if we write tests, we can more easily observe that the obstacle course is working. Envision our trustworthy tests like robot assistants. The unit tests are drones with mechanical arms, focusing on specific parts of the course and interacting to get different outcomes. The end-to-end test are cyborgs that run the course for us, while we sit in out comfy chairs. We, the overlords, spend less effort and take on less risk.

One important decision, before test writing starts, is “should it be a unit test, component test, integration test, or end-to-end test?” Here are some questions I ask myself before choosing one:

• Do I want to cover a lot of inputs and verify a lot of outputs? → Probably a unit test. I’ll write it with react testing library and come up with my edge cases ahead of time.
• Do I want to verify different display experiences based on different conditions? → Probably a component test. I’ll make sure I cover both the Visitor and Member experience.
• Would I trust the test result more if it’s based on realistic mock data? → Probably an integration test. I’ll make sure that mock server response is up to date.
• Am I validating a whole user flow? → Probably an end-to-end test. I’ll write it with Cypress and get feedback from the requirements folks.

There are a lot more distinctions you can draw here too, but deciding which type of test before writing will help you write fewer tests in the long run.

Consider smoke tests and sanity tests

Some might say Storyblocks has too many tests! This is a rare problem in the engineering world, where tests are often seen as annoying to write and avoided when possible. But nevertheless, here we are. Our various test suites provide a decent amount of coverage and the main complaint is that they take too long to run.

One approach I’ve seen in the past is splitting tests into a few different levels. The resulting sub-suites can be run in different situations, and even tied to the severity of the situation.

“Sanity tests” are the smallest possible set of trustworthy tests that cover critical functionality. These should be run before deploying code in the situation of a critical bug or outage. Run time should be on the order of seconds, allowing us to get fixes out faster than our normal pipeline.

“Smoke tests” is a term I borrowed from a QA team I worked with, roughly translating to “what does the QA team need to check to green-light this [change|feature|release]?” Related to the “Test requirements, not implementation” section above, this is where those who didn’t build the change have input into how to verify the change with confidence. Ideally, the tests that are identified here stack up and are automated over time to bolster the general test suite.

Invest in SDETs

Taking a moment to shoutout our QA’s, the folks who keep us honest, catch our mistakes with grace, and understand us better than anyone else! You work hard, and we appreciate you.

I’ve always felt weird about engineers writing tests to prove their own code passes. If only I was allowed to grade my own tests in school…

“Software Development Engineer in Test” is a job title that acknowledges that a trustworthy automated test strategy is a full-time job. Without this specialization, tests are always the first thing to punt under time pressure, and the first thing to blame when bugs sneak through.

Let’s hire and grow more SDETs, people! Here at SB, our QA team is awesome and hungry to learn coding to support their (our) goals. A little bit of Cypress can go a long way to improving quality, confidence, and the developer experience. Company leadership can help make this time and training a priority.

If you ever spot the mythical unicorn…

I’m always listening and learning of new ways to make the day-to-day better for software engineers! If you feel good about your testing strategy, please share your lore with others. And if not, know you’re not alone.

Thanks for reading! Storyblocks is building a Talent Community of awesome people like you. You can also reach out in the comments or find me on LinkedIn to connect.

A decade of shade: automated testing from someone who wants the robots to take it already was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is Gather?

Gather is an app that simulates an in-office experience, with the ability to reserve a desk, personalize the space, and interact remotely without Zoom calls. Each employee who opts in at Storyblocks has their own dedicated desk that they are free to decorate using Gather’s item library.

Additionally, employees can create custom objects, using image editor tools such as Gimp or Krita, that can be imported into the app. Gather has a mapmaker feature that allows users to expand the office, create portals to other rooms, and make a space private (only others in the same space can hear) or spotlight a space (everyone in the room can hear a speaker).

Goodbye, Zoom Fatigue

We’ve begun using Gather more frequently for meetings. From our Engineering All-Hands (~40+ attendees) to smaller daily standup meetings, we’ve brought back the practice of convening in a shared space, which leads to more organic discussions, better team building, and an overall feeling of connectedness and community within the workplace.

Celebrating Life Events

When big life events happen, such as birthdays and weddings, we wanted a way to celebrate. Now, we can decorate each others’ desks for any occasion using custom and interactable objects.

Community Spaces

At Storyblocks, we have several active resource groups (BRGs) through which people with shared experiences are able to meet and form community. Out At Storyblocks (OAS), our LGBTIA+ BRG, now has its own meeting space that we recently used to host an OAS luncheon. We also hosted our annual Pride trivia event in Gather this year.

New Hire Resources

For onboarding engineers, we’ve created a room for new hires to hang out in as they set up their environment and complete the onboarding process. This gives them the chance to ask questions and get to know their new teammates in a more organic setting.

Team Building

We meet on Friday afternoons for a remote gaming event that anyone can join. This is a great way to unwind at the end of the week and get to know people throughout the company. Some games we’ve played have included Wavelength, Codenames, Skribbl and Top Four. Gather also has an escape room template that can be used as a fun team building exercise.

Celebrating Individuality

Gather has given us a new way to express ourselves and be goofy. People have come up with so many creative ways to style their avatars and decorate the office space. We’ve even started up a Storyblocks ‘marching band’ group to rally for birthdays and other events.

The ‘Gather Keepers’

A group of our most Gather-obsessed employees has banded together to encourage use of the app and to continuously develop new ways to apply this resource. Gather Fridays is our most recent initiative where we encourage employees to spend Fridays working from their ‘desks’. The Storyblocks Gather team maintains a list of our custom objects, so that anyone in the company can use these items. Our list also contains special requests for wishlist items, such as BRG logos and desk decorations.

A Gather office can change and grow — just like an in-person office setting. We just completed the process of designing a new-and-improved layout alongside the Gather team to facilitate even more collaboration, and we plan to iteratively generate feedback for future improvements.

A Tool For The Times

Gather has introduced new elements of productivity and just-plain-fun to the work day. It’s contributed to employee engagement and collaboration, and has served as a fantastic medium through which we can witness different personalities coming together to build a great product.

Interested in joining our team and getting in on the fun? We are hiring!

How We’ve Leveraged Gather to Enhance Our Remote Culture was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here at Storyblocks, the Data Science & Machine Learning team is always looking to help our customers find or discover content they love. As the custodians of our search, storyboard, and recommendation experiences, we work to improve how we connect our users to our content — whether we’re improving existing experiences or adding new ones. This post will talk about a search problem we’ve recently solved. We’ve used it to improve an existing button on the site, and you can expect other cool new features based on this solution soon!

Let’s say you see a movie and like one of the actors in it. For example, the latest Indiana Jones movie (for the sake of the example, ignore the bad reviews). You’re watching and you really like the main character: Indiana Jones. You’d like to watch other movies he’s in because you think those movies would be good, too. Who plays Indiana Jones? (again for the sake of this bad example, please pretend you don’t know that it’s Harrison Ford).

How do you find this out? You wait until the end of the movie and find out it’s Harrison Ford from the credits (or you google the movie title and click the cast button on the search results page). Once you know the actor’s name, it becomes really easy to see what other movies they’ve been in, so you can go watch those as well.

A similar situation can play out in stock media. Someone sees a video with a person they like and they want to see other videos with that same person. Stock media isn’t Hollywood though. How can you find other videos with the same person?

This post continues below, but here’s the trailer for the Indiana Jones movie if you’re curious:

The problem

Have you ever seen an actor in a movie that you liked and wondered what else they’ve been in? Resources like Imdb, Wikipedia, and Google make it easy to quickly find other movies and tv shows an actor has starred in just by finding their name.

The problem at Storyblocks is that our stock media doesn’t come with a list of credits attached to it (whether that should change is a discussion for another post). This means internally we don’t track which videos have which actors, and worse, we can’t help our users find more videos of the same person when they are looking for them — at least not directly.

Our old work-around needed improvement

While we do have ways for our users to look through other videos from contributors or search through similar footage, it’s not easy to quickly find all the videos an actor has shown up in. We don’t have anything that allows direct face searching even though we know anecdotally that we have a bunch of stock actors that appear in multiple places across our library.

Example Contributor Page:

Example “More Like This”:

Notice how “more like this” only shows videos from the same or a very similar shoot and how the contributor page adds a ton of noise from other videos the contributor has uploaded.

A new face-based approach seems to help

We’ve just released an update to our “more like this” functionality that goes a long way toward solving this problem. We’re now able to find videos with similar faces (in many cases, this leads to finding all or almost all of the other videos a given actor shows up in). Better yet for our users, We now have a direct way to help them when they find an actor they like and want to get other videos with the same person.

This was made possible by clustering face embeddings created for all Storyblocks’ content featuring people. More on that in a moment, but here’s a DAG abstraction:

It worked like this:

1. It started with some research

paperswithcode.com

2. We lucked out. We already had a pipeline that determined which frames have people in them (it made this new task easier)

Imagine a semi-magical Airflow DAG that uses computer vision and NLP on our videos and contributor metadata to find frames of videos with people’s faces in them. It would look something like this:

3. Feed detected faces into a model

4. Create an embedding for each face

Basically this is just a list of numbers that uniquely identify an individual face while also giving an indication of how similar that face is to other faces. Conceptually, this is kind of like DNA or a barcode. How the embedding gets created depends on the approach you take. Google wrote a white paper on their approach here. You could visualize an embedding similar to how it’s done here in the Deepface python package:

5. Cluster the embeddings to get groups of similar faces

Assuming the face embeddings are correct and your clustering algorithm is tuned well, each group should represent very similar if not the same person based on their facial characteristics.

6. Push for it to get on the site as long as it takes to get it on the site

Work with your team & others to get this deployed on the site. This process is kind of hard to describe. Maybe I’ll talk about it in another post.

7. Finally, display the results in the “more like this” button

Here’s that same “more like this” button example from earlier now:

Other examples from our site:

What’s next?

While it’s exciting to enable this functionality, there is still work to do to make this feature better, like improving our face detection and face searching capabilities. This has also unlocked ideas for new content-based recommendations, and we know there are many more exciting embedding-based features we could implement for our site — things like reverse image search, reverse face search, or even reverse object search. Exciting stuff, but those are ideas for yet another blog post.

Thanks for reading! If you made it this far, I added some references at the bottom to help you explore Face Recognition and clustering further. And if you like the work you see us doing and want to do it, too, check out our general interest application.

References:

1. https://towardsdatascience.com/hierarchical-clustering-explained-e59b13846da8
2. https://arxiv.org/pdf/1503.03832.pdf
3. https://arsfutura.com/magazine/face-recognition-with-facenet-and-mtcnn/
4. https://github.com/serengil/deepface

Find More Faces Like This was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

General Flow

1. Employee tries to access an internal application (i.e. admin-service).
2. The employee connects to the internal application url and the request is sent to the Istio service mesh inside our Kubernetes cluster.
3. We have an Istio authorization policy that limits access to the admin service.
4. The authorization policy points uses the custom extension provider, Oauth2-Proxy. Oauth2-Proxy is setup to connect with our identity provider which is Okta.
5. The employee’s request is redirected to Okta for login. On successful authentication it passes through the Istio authorization policy and connects with the admin service.

Technologies Used

• AWS EKS Kubernetes
• Istio
• OAuth2-Proxy
• Okta

We found a surprisingly small number of tutorials when trying to set this up ourselves so here is our quick tutorial.

1. Create an Okta Application for OAuth2-Proxy

This Okta application will allow for access management to your application.

Make sure to add the url to your internal application under the sign-in redirect URI. This will authorize Okta to redirect to your internal app after authenticating an employee.

You can add other internal app urls to this application, but know that any Okta users/groups you add to this Okta application will be able to access every internal application url that is listed here.

If you want to separate who has access to an application (i.e. only the finance team can access this application and not engineering), you will need to create a separate OAuth2-Proxy app in Okta for each application you want to customize access to.

2. Deploy OAuth2-Proxy to Kubernetes

This will be the same process as deploying any other application to Kubernetes; you will just need to add a couple of specific configs. The biggest lesson we learned was that you do NOT need the redirect-url. You will be specifying the redirect URL’s in the Okta application later on. We added clientID, clientSecret, and cookieSecret configs via our secrets management process, but I put it in this yaml excerpt for readability.

Here’s a link to the OAuth2-Proxy and Okta documentation — just ignore the fact that they include the redirect-url in their configuration file.

# Oauth client configuration specifics
oauth2-proxy:
  config:
    clientID: <SEE OKTA OAUTH2-PROXY APP>
    clientSecret: <SEE OKTA OAUTH2-PROXY APP>
    cookieSecret: <SEE OKTA OAUTH2-PROXY APP>
  extraArgs:
    provider: oidc
    oidc-issuer-url: "<COMPANY_OKTA_URL>"
    pass-access-token: true
    skip-provider-button: true
    reverse-proxy: true
    cookie-domain: 
      - <WHITELIST_COOKIE_DOMAINS>
    email-domain: "*"
    whitelist-domain: 
      - <WHITELIST_DOMAINS>
    upstream: static://200
    # redirect url - not needed since you will have it in your Okta application

3. Add OAuth2-Proxy as an extension provider in Istio

Inside your Istio mesh config, you will need to add an extension provider. This will reference the OAuth2-Proxy deployment and will allow us to use OAuth2-Proxy as a CUSTOM action in our Istio Authorization Policy.

# Istio extension provider specifics
istiod:
  meshConfig:
    extensionProviders:
      - name: "oauth2-proxy"
        envoyExtAuthzHttp:
          service: "<KUBERNETES SERVICE URL i.e.oauth2-proxy.default.svc.cluster.local>"
          port: "80"
          includeHeadersInCheck: ["authorization", "cookie"] # headers sent to the oauth2-proxy in the check request.
          headersToUpstreamOnAllow: ["authorization", "path", "x-auth-request-user", "x-auth-request-email", "x-auth-request-access-token"] # headers sent to backend application when request is allowed.
          headersToDownstreamOnDeny: ["content-type", "set-cookie"] # headers sent back to the client when request is denied.

4. Add Istio Authorization policy

This CUSTOM authorization policy allows us to restrict access to the host specified and authenticate via Okta and the OAuth2-Proxy. See the Istio reference document for more info.

# Istio authorization policy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: admin-service-okta-auth-policy
  namespace: admin-service
spec:
  action: CUSTOM
  provider:
    name: oauth2-proxy # reference oauth2-proxy extension provider
  rules:
    - to:
        - operation:
            hosts:
              - <INTERNAL APPLICATION HOST>
  selector:
    matchLabels:
      app.kubernetes.io/instance: <KUBERNETES SERVICE NAME i.e. admin-service>
      app.kubernetes.io/name: <KUBERNETES SERVICE NAME i.e. admin-service>

5. Enjoy secure access to your internal application!

If this work sounds interesting, consider joining our team!

Authenticating internal applications on Kubernetes via Okta, Istio, and OAuth2-Proxy was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.