🇧🇷 Leia este artigo em Português (PT-BR).

As harmless as our component’s .png may seem, when added to the various UI files, it will impact the final size of the game. From the last couple of years working with mobile games and being part of Wildlife's Visual Design team, I have learned that this number greatly influences download metrics. This is because when the application exceeds the file size of 200MB, AppStore and Google Play suggest to the user download it using Wi-Fi, as there may be excessive data consumption. It may not seem like it, but a simple pop-up suggesting you to turn on Wi-Fi can lead to dropouts when downloading a game.

And although 200MB may seem like a super healthy margin, games are operated as services these days, which implies on continuous production of content. As a result, the game will become heavier with every new release. So, in order to be prepared to handle any future issues that may arise, adopting optimization practices from the beginning can save you a lot of time and stress.

A great example of how important is this strategy in the mobile game industry, we Free Fire to help us make a point. In an interview with Harold Teo, published by GamesIndustry, the producer at Garena said this:

“If our game is 4GB, 8GB, or something, people will have to think twice before installing. They need to delete some apps from the phone and sometimes even the full capacity of the phone might not be enough. Even today, one of the team’s priorities is to make the app as light as possible.”

There are several characteristics that can influence how your game is received. The size of the final build, the size, and resolution of the assets, the amount of unused repeated files, as well as many other situations like these which need your attention.

How to optimize your game UI: tools and techniques

Knowing this, I prepared a compilation of optimization techniques and tools that I consider relevant and can benefit games produced in different game engines. However, being specific, I’ll focus on the tools I’ve had most contact with, based on my experience working as a Visual Designer at Wildlife’s Art Studio: Unity and Figma.

Before starting, as much as the suggestions in this article result from the sum of conversations with Graphic Engineers, Art Directors, Engineers, and Game Devs in general, I ask you not to treat it as a rule. Each game has its own context with completely different variables, so always talk to your team before making any decisions that could impact the pipeline.

Do I really need to think about optimization?

Just to give you more context, these are some of the common scenarios I’ve come across:

• UI looks much nicer and more consistent in Figma than the one implemented in the game.
• Asset resolution is lower than planned.
• UI files get cluttered in the project structure.
• Project growth makes maintenance and integration of new components a complex task.

While transitioning between different games and teams, I noticed that many patterns were repeated and maybe you can relate to that, too. Through these patterns, it is possible to see some outputs to improve both the life of the designers and the engineers. It is also important to mention that there is no miracle pill to solve these issues, but a set of solutions developed alongside the team and the stakeholders through the game’s natural post-release process and active communication.

Understanding the solutions

Faced with all the processes, I do not intend to rank from most important to least important. I believe that everyone contributes to the result in a different way. These are the solutions:

• 9-Slicing
• Tint
• Size, resolution, and compression
• Sprite Atlas
• Taxonomy
• Colorspace & Rendering pipeline
• Design review

I will delve into each one along this article, but bear in mind that some might make sense to you and others might not. That said, take some time to reflect on it and decide what is worth testing and what’s not.

9-slicing

For a long time, I worked with this technique but under other names. It’s something that’s been widespread since the early days of the internet, where the biggest use was in creating resizable custom divs that had four corners with a border radius or some other kind of ornament.

This concept is nothing more than a 2D technique that allows you to reuse an image in various sizes without having to prepare multiple assets. You divide your image into 9 slices, where you determine that the corners are fixed and the core is resizable.

Fortunately, in Unity, we have a package that makes this practice trivial. The Sprite Editor! I won’t go into details of how to execute it, because there is a lot of quality material on the internet. Here’s an example if you want to check it out:

Sprite Editor

However, even though it’s trivial, 9-slicing can be super impactful when incorporated into the workflow. Imagine the situation where you would need to export about 4 different assets to produce a responsive card that works in different situations, but now you can get the same result with just 1 if you configure and set the cuts correctly.

In addition to being an excellent development practice, once you learn how it works, it becomes a technical requirement for your Design assets and a new way of thinking. Understanding this concept is one of the first steps to working with UI in games.

Tint

Like 9-slicing, this technique is also revolutionary when it comes to asset reuse and scalability. And surprisingly, it is very easy to understand, apply and see the result.

Imagine the HUD — heads-up display, or screen used to detail the main information and events during a match — of an air battle game like Sky Warriors, where there are several similar icons that are displayed in different colors depending on the chosen team.

In this scenario, if there are ten different icons that need to appear in three colors, we would have a total of 30 files. However, using tint, we will export only 10, but in white. Then we will apply the tint inside Unity with the necessary color through the inspector or via code.

When applied, in addition to making maintenance much easier for everyone on the team, tint can drastically reduce the number of UI files in the project in the long run.

Size, resolution, and compression

This topic in particular is what motivated me to write this article and to start pipeline improvement initiatives with my team. In addition to directly affecting the size of the final build, as I said in the introduction, it plays a key role in the synergy of the other optimizations.

For example: for Unity to compress your file well, and consequently decrease its size through an algorithm, the resolution needs to be a power of 2 (16, 32, 64, 128…). A priori, it seems to be a rule that does not affect your work routine but believe me, the time will come when this detail will demand your attention so often that you will wonder what can be done to optimize the process.

That’s exactly what happened to me. As much as I value the importance of using the power of 2, it has become impossible to always review and export assets with a considerable region in transparency just to hit the right resolution. Hence, in conversation with several people, we noticed something that could alleviate the problem, whose technique I will go into more detail below.

However, despite having other alternatives to solve this problem, working with an 8pt grid in its Design is already a big step towards evolving in this direction.

Sprite Atlas

Ah, the wonderful Sprite Atlas! Why worry about so many assets individually if I can automatically stack them all in a big asset with the power of 2 resolutions and solve all my problems?

Before moving on, allow me to point out: that individual concern for assets is still necessary but in a much healthier way.

Sprite Atlas is an excellent development practice because in addition to relieving micro-management and organizing the project, it also scarily decreases the number of draw calls (name of the process responsible for rendering the assets on your screen), which, in turn, improves performance.

In other words, if your Main Screen requires 25 different files in the normal state, that could mean at least 25 draw calls, assuming they all appear only once. With Atlas running this drops to 1. In technical terms, the amount of allocated memory this can save is directly related to the number of times your asset is rendered on-screen within the same session. Being objective: fewer draw calls, more frames per second (FPS). And in case you didn’t know, performance is gold in game development.

Going further, Atlas ties together all the topics in this article very well. It brings more autonomy to designers and offers the possibility to make tangible the budget for the UI. Let’s suppose you can have an Atlas per feature and, in addition, you have the freedom to have another one that keeps the assets in common use by the game without any compression, being rendered only once. They don’t hurt performance and keep the visual quality high, without that bad low-resolution feeling.

Going back to the budget, you now have a very clear artboard of how many assets you can still insert. If nothing else fits, you need to revisit the old ones and understand if 9-slicing or tint can save you from possible redundancies.

Read more about how to integrate Sprite Atlas into your project:

Sprite Atlas

Taxonomy

For those unfamiliar, Taxonomy is the stage of grouping the contents and actions according to the meaning. It is a term that comes from Biology, where the objective is to organize and classify structures.

In Design, it is used as a way of structuring, naming, organizing, and distributing what has been produced. It can be simple and at the same time very complex. The context influences a lot and most of the time its definitions arise from collective consent, where the team defines good practices emphasizing what should be done and what should be avoided.

Being very objective, when the Designer produces a layout, it is extremely important that the handoff is transparent and easily accessible. In some cases, the Designer himself implements the engine, in others the Tech Artist or the Engineer is responsible for the feature. Anyway, when the assets are exported from Figma, Photoshop, or other software, when arriving at the project, if there is not the least organization, the collapse starts to have a date.

We know that in some urgent scenarios, let’s be honest, it may not be possible to take all the care you want with the names of layers, components, and things as such. However, the ideal is that the Designer always minimally organizes its deliverables and is aware of what has already been imported or not for the project.

That way, when the assets are exported, you avoid duplicate files and make them easier to access later. We know that it is a painful process to integrate, as it requires a lot of responsibility and discipline on both sides, but without the shadow of a doubt, it is of great value for those who are on the team at the moment and for future generations.

There are a few suggestions on how to get started, but as this is not the main goal of this article, I hope the brief introduction to the subject can motivate you to do more research on it. And remember: whatever you apply it to, your team needs to be in agreement with this decision and aware of the adaptations required for the optimization to work.

Color space & rendering pipeline

Of all the suggestions in this publication, this is the one that intrigues me the most. Maybe because it was a completely new subject in my routine. The moment I learned about this topic my view of game development changed a lot.

Artists in the digital world are already familiar with the term Color Space, but within the gaming universe, it can be quite different. This subject can be extremely complex, so I will only stick to the details that impact the result of the Designer’s work.

Basically, Linear and Gamma are the most relevant Color space in digital production. The rendering pipeline is a separate process that determines how the graphics engine will work with the game’s colors, materials, shaders, and graphical information. To connect the two terms, the Rendering Pipeline directly influences which Color Space will be used.

The decision to use one form or another is related to the game’s art direction, technical limitations, team knowledge, and studio requirements. While one allows for more realistic graphics, for example, the other can provide greater performance. The reasons are many.

And what does this have to do with UI?

Probably the software the Designer works on is exporting the assets in the Gamma pattern. Hence, when imported into a project using Linear, the fidelity can drop a lot.

There are several ways to solve this problem. For example, it is possible:

• Create a camera that renders the UI in gamma and then adds it to the game’s render in overlay format.
• Use a custom shader on UI elements to correct the mathematical variance calculated automatically by the engine.
• If you are producing your layouts in Photoshop, it is possible to export directly in the engine’s color space (with the exception of Figma which is limited to Gamma).

As I work with Figma, I noticed this frightening fidelity difference, and with the help of my team, we realized that the biggest anomalies arise from the difficulty of converting Unity in situations where there is a mix of assets. For example, you have a background image of a card and above it, you have another image with a gradient of a color that you use to highlight. And this gradient alpha goes from 100% to 0%. The anomaly occurs from the moment it is at 99% and needs to blend its color with the background in real-time.

So one way to soften the impact is to avoid having gradients fade to transparent. Instead, fade the gradient to the color you will actually use for the background. This way the color blend will already be rasterized in the image and Unity will not need to calculate the blend result.

I know it’s a complex topic to understand and delicate to explain, but I thought it was important to share, because, at least in Unity, there is no definitive solution. And in case you go through this situation, remember the suggestions above and don’t despair.

If you want to read more about it, check out the link below:

3D scene need Linear but UI need Gamma

Design review

Finishing in harmony with all the previous topics, this one will be pretty straight to the point. If you, as a Designer, want your UI to be of high quality during implementation, you need to be involved in the entire process. When I say this, I refer to all production steps.

Don’t just stick to Figma. Participate in definition and implementation! See how engineers understand their deliverables, how they export, how they import into the project, and, above all, try to understand how to implement your UI in the engine you work with. That way, whenever necessary, you can make a second pass to refine what only the trained eyes of a Designer can see.

In some studios the UI Designer doesn’t need to implement anything, in others it takes care of all the steps. Regardless of how it works where you work, keep in mind that learning new skills that touch your area of ​​mastery will always improve you as a professional.

Conclusion

Why should you care about optimizing mobile game UI?

• Being a designer/artist who understands the basics of optimization can keep you from coming across a number of routine mishaps.
• Including some of the steps mentioned above in your workflow will help you become a more seasoned professional concerned with the impact of your work.
• Optimizing from the start is the best way to avoid problems.
• Understanding these issues is critical to achieving a high level of visual quality in your game.
• Optimization practices, in general, improve the lives of game developers and players. It brings scalability and performance.

After all, optimizing is an eternal learning process. Even though I share all these themes with you, I’m constantly talking to my team to understand how we can improve.

And more important than optimizing is being aware of what you want to improve. In order for you to be able to sell these ideas to your team, it is necessary to make it clear what problems will be solved, what will be the short and long-term gains, and the size of the impact on the roadmap. Having that mapped out, the trend is only getting better!

The solutions will always vary according to the context in which your game is inserted. All the suggestions cited here are based on my experiences, so it’s important to analyze what is the biggest source of problems in your UI pipeline today.

If you’ve made it this far, I hope this material has contributed to your professional growth in some way. And if you liked it, feel free to check out my other articles or to reach out to suggest new topics, chat, follow my work or even tell me if you managed to apply any of the techniques I mentioned.

Before leaving, a special thanks to mentors, coworkers, and friends who contributed directly or indirectly to the publication of this material: Ingrid Bugarin, Om Tandom, Allan Camargo, Patrick Sava, Musa Sayyed, Huila Gomes e Vitor Canuto.

Thanks and hope you enjoyed!

– Stéfano B. Girardelli.

Read more

UI, UX & visual designers behind great games

References

• Unity User Manual 2020.3 (LTS)
• How Garena Free Fire plans to stand out from the battle royale crowd

Why should you care about optimizing UI for mobile games? was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article aims to briefly explore how Machine Learning Engineers (or MLEs) can bring more value to the whole ML ecosystem at Wildlife, helping our teams increase their ML-related productivity and reduce technical debt.

In our current scenario, Data Engineers (DEs), Data Scientists (DSs), and MLEs work closely, which could lead to a confusing work dynamic. When we reduce the scope to focus on the two most similar roles, we end up with MLEs and DSs using a very similar tech stack. At a high level, we are talking about Science and Engineering.

While DS need to deeply understand Math, Statistics, and Probability Theory, the MLEs are tasked to build deployable ML projects at scale using Software Engineering best practices — but of course, it’s not a rule, it’s just a matter of focus!

Role of MLEs

When we talk about the competitive advantage of certain technical roles, some may consider the factors and attributes that allow a team to consistently outperform in a given context. With this in mind, the MLEs are the most competent team to take ML models that were created by DSs and make them scalable, optimal, serviceable, while making sure they perform well in production.

In a few words, MLEs turn a predictive promise into a reliable and efficient piece of software. For us, there are three main paths to make a process more reliable and efficient:

(i) Automate it so that it’s easier and cheaper to be maintained;

(ii) Improve the process itself making it intrinsically better. For instance: Generating value by transforming the process to gain performance, time or reliability; and

(iii) Simplicity, most of the times ML systems are very complex and hard to maintain, so keep it simple.

In the next paragraphs, we deep dive into how MLEs can use their knowledge to improve ML processes and tasks, and discuss the ongoing costs associated with real-world ML systems and their technical debts.

MLEs and the ML Lifecycle

A typical ML project can raise many technical challenges by just putting a single model into production. Algorithmic complexity, scalability, latency, validation, and monitoring are examples of common concerns that can turn ML-based systems hard and costly to maintain in the long run. Ideally, we would design and build a whole pipeline to automate these blocks and solve every engineering problem effectively, however, in the real world things are more complicated.

Our aim is to deliver business value as fast as possible. In our view, MLEs can play a major role by helping teams grow sustainably. To the best of our knowledge, while DSs concentrate their efforts on creating as much impact as possible using data (first) and ML techniques (second), MLEs can collaborate with them by providing support, tools, and infrastructure to automate the lifecycle of the ML models.

When looking at the ML lifecycle, we can’t help noticing the skills that MLEs can bring to other teams and projects to improve their productivity. We work to create reliable and efficient infrastructures and a set of tools to enhance ML-related tasks. By deep diving, we are able to identify three main stages, which are common to most ML projects at Wildlife:

• In the first stage, we are interested in the experiment itself, creating new models and features to generate as much impact as possible using data.
  Additionally, MLEs can focus on reducing the time of experimentation creating a reliable experiment environment, either by frameworks, model researchers, or libraries;
• In the second stage, we focus on putting the models into production in an automatic triggering setup. Note that this stage connects the Training Code and Serving Code steps, it tells us that the experiment code should be adjusted for production. i.e. the model is validated and we should be ready to write the serving code, test it, and do the deployment. Here, MLEs must consider the potential technical debts, however, it’s important to focus on implementing the best code practice and structure to the ML scenario, improving the reproducibility of models and predictions, governance, and regulatory compliance;
• Lastly, in the third stage, we are interested in serving the models in a reliable infrastructure and monitoring all aspects related to the model performance and execution. Nevertheless, note that even in previous stages, DSs, MLEs, and Software Engineers (SEs) should communicate and collaborate to define artifacts/metrics and other model specifications, but also to create log diagnostics to make sure that every ML-component is performing at the expected level and triggers alerts in case of anomalies or model drifts.

In summary, we understand that MLEs can bring more value to teams and projects by focusing on the following aspects of each ML-lifecycle stage:

Stage 1

• Data modeling and evaluation for ML task;
• Provide tools to create scalable and effective training runs;
• Provide tools to facilitate the experimentation;

Stage 2

• Automate processes to avoid manual errors, omissions, or oversights;
• Create ML tools that wrap the models to make them production-ready, reliably and traceable;
• ML tools to create an easier translation from experimentation code to production code with few extra steps, e.g. boilerplates and abstractions;
• Design and build ML pipelines managed by a CI/CD, or a scheduler such as Airflow;
• Provide automatic code linters and automated tests to improve code quality and reliability;

Stage 3

• Implement the serving code effectively;
• Create tests to ensure that the validated models are behaving as expected from both execution and performance aspects;
• Provide tools to monitor the models, e.g. the input feature vector and misbehaviors;
• Adapt ML runs to different infrastructures, e.g. Kubernetes and Spark clusters;
• Provide an automated deployment strategy that saves model artifacts and wrappers.

Hidden Technical Debt in ML

Since only a tiny fraction of real-world ML systems are composed of ML code (as shown in Figure 2), it is not hard ending up with technical debt all over the place. This can become particularly bothersome when dealing with legacy systems. Additionally, spending time to rebuild the necessary setup to have a ML in production for each squad can considerably slow down the throughput of a company, once squad members would have less time to work on their core activities.

By deep-diving into an example, we can make this scenario even harder. Imagine that a squad had been developing — in a part-time effort — their own deployment solution for two months, but they had to deprioritize both feature and model validation tasks during this period. This sort of effort, at a company level, could be dramatically reduced if we invested in having a unified platform that handles all this additional complexity.

Nowadays, we can be on top of that, due to the fact that the MLE organization is a compound of (i) MLEs working within squads and (ii) a platform team.

Another huge opportunity for improvement is by having MLEs working on ML-based squads as early as possible. By focusing on MLOps adoption, they can help mitigate technical debt and, in the long run, increase squad results. In case of late arrival, these MLEs usually have to spend a lot of time refactoring code and making migrations. In our experience, these types of activities can negatively impact squads’ value delivery and also cause both burnout and frustration for those involved.

Key Takeaways

• MLEs can bring value to the whole ML ecosystem by helping teams increase their ML-related productivity and reduce technical debt, but also by helping teams grow sustainably.
• MLEs should collaborate with DSs, DEs, and SEs in all stages of the ML lifecycle to provide support, tools, and infrastructure that improve ML-related tasks, especially in the mid-long terms.
• ML systems can rapidly accumulate technical debt, once they combine the maintenance challenges of traditional software and ML-specific issues. MLEs can help teams and projects be on top of that.

Originally published at https://medium.com on December 13, 2021.

Machine Learning Engineers at Wildlife Studios was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

If we can’t experiment, how can we know whether a business action had the expected impact? We will present you with a real case and its solution using an innovative library for causal analysis.

If there’s only one option, how can we know it is the right one for us? It seems more like a philosophical debate, but actually, there are many recurrent situations in which a company needs to answer these kinds of questions. Today, I’ll present you a real case in which data science helped us understand if we made a good decision without running an experiment.

Wildlife Studios is the first video game company I’ve ever worked for. It’s fun and challenging — not only because of the magic of the games but also the magic of advertising. Have you ever thought about all the things that happen behind the advertising videos that appear when you play a mobile game? Many things happen, and they are exciting.

What is an SDK?

SDKs (Software Development Kit) are packages of code that fulfill particular parts of an app’s functionality. In general, in the mobile industry, effectively reaching users with an ad relies on a specific SDK responsible for connecting the app with the advertising market. Part of the action and my work occurs when this software needs to be improved.

For that, a new version is created, tested, and finally deployed on production. But even after testing it many times, we may still wonder whether the latest version is indeed better than the old one. Even when we are sure it is better, we still want to know how much better it is.

A/B Testing and SDKs

Usually, when this type of question arises, data science answers them through experimentation, for example, designing an A/B test. In this case, we could release the new SDK for some randomly chosen users (treatment group) and keep the rest of them with the old version (control group). Then we would compare our performance in the new SDK group versus the old SDK group.

But in the app world, we can’t show different SDK versions for different users. Here’s when things get complicated because the A/B test is unfeasible.

You can argue that several versions can coexist at the same time because some users still have the old version, and you would be right. But it would be a mistake to simply compare the group of users who have the new version with those who have older versions. It’s easy to imagine a situation in which both groups aren’t comparable. For example, a user whose device has enough free storage space is more likely to have the latest version, while in the opposite case, may not. So, the direct comparison could introduce a bias in our analysis.

How can we measure the releasing impact of a new SDK version?

The idea is to estimate what would have happened if we hadn’t changed the SDK version. There is no way to answer that question without proposing a counterfactual. A counterfactual is a situation that could have happened but did not really happen.

In this way, what has actually happened will be our treatment group, whereas the counterfactual will represent our control group. Then we can compare both cases and draw conclusions about the effect of the version change.

A more technical approach

Maybe, the most straightforward reasoning is to suppose that nothing would have changed if we had not changed anything. Let’s imagine that we are in a case where our metric of interest is always approximately within the same values. As we can see in the following diagram, it would be reasonable to think that the metric would have kept its value in the absence of changes.

But now imagine that the metric had a positive trend over time. It would be a mistake to propose a counterfactual that simply replicates what happened in the previous period. As we can see in the following example, the comparison with the counterfactual would overestimate the impact of our new version.

So to improve our estimate of the counterfactual situation, we should consider those pre-existing elements that somehow governed the evolution of the metric. These would be, mainly, the trend, the cycles, and the seasonality of this series. A counterfactual that includes these elements would give us a reasonable basis for measuring the impact of our intervention.

However, it is not that simple because although we are now in a much more comparable situation, unexpected factors may affect our inference. For example, there can be abrupt but perhaps temporary drops in some big buyers (adnets).

In that case, something that can help us is to analyze what happens with other games which don’t use our new version yet and build a more robust counterfactual in the face of those unexpected events.

R, Causal Impact and the Google team

Now we need a method capable of incorporating all the necessary elements to build a good counterfactual. An essential part of the task is learning to predict the time series. For that, we must capture characteristics such as trends, seasonality, and cycles. But also, we have to choose which of all the other games we will use to complete our counterfactual scenario. The Google team proposed an exquisite way to solve all these challenges at once. Thanks to their work, we have available both an article with the academic explanation and an R package called CausalImpact.

Proposed Solution: The Bayesian Approach

The solution proposes to use a structural model of time series within a Bayesian approach. Don’t be scared. The important thing is this type of model will allow us to express each of the components of the counterfactual.

I won’t tell you how to get trend and seasonality, which you can learn in any time series analysis course. I’m going to jump straight to the most interesting part of the estimation, which is how we will use the information that comes from the other games.

The best scenario that we could find is a game or a group of games that behave in a very similar way to the game we are trying to analyze. But actually, we see many games that may have a little or null correlation with the time series that we are interested in modeling and a few games with a considerable correlation. So, the art will be in finding the group of games with a strong correlation.

How does the package tackle this difficulty? Using a Bayesian variable selection technique called spike-and-slab, we will determine to what degree we should include a certain contemporary covariate (a game) as a predictor of the time series. It allows us to decide which games to use and what weight to give them. And best of all, since this variables selection technique is helpful in scenarios where the number of predictors exceeds the number of observations, we can test as many games as we want!

The results!

Finally, how were the results? Robust in most cases. In the following graph, we can see how the launch of the new SDK had a neutral impact on this particular game. You can see this by comparing the actual metric (solid black line) with the counterfactual (dotted blue line) after the new SDK release (first vertical dotted grey line). For privacy reasons, the absolute values are eliminated, but we can see that the margin of error is relatively low.

We also had a good experience in other games, in several ways. Not only showing the positive impact of the SDK release but also predicting outliers like the one marked by the red circle. In particular, the drop in the following graph was related to a generalized situation after the presidential elections in the United States. Points go to having used this technique that includes other games as predictors!

Implementation in R

The best of all is that the implementation does not require significant effort. A simple demonstration is that no more than ten lines of code are needed, as shown below.

# Import the library

library(CausalImpact)

# Load the data frame. Each column represents the time series for each of the games, ordered in ascending order by time. It is important to mention that the first column is the game to be evaluated and the rest of the columns are the other games that will be used as covariates.

df <- SparkR::sql(“SELECT * FROM temp_df”) %>% SparkR::collect() %>% as_tibble

# Delimit the training period of the model (pre.period) and the counterfactual period (post.period)

pre.period <- c(1, 60) # We use the first 60 days for training

post.period <- c(61, 75) # We evaluate the impact on the following 25 days

# Fit the model (parameterization is optional)

impact <- CausalImpact(df, pre.period, post.period, model.args = list(nseasons = 7))

# Evaluate the model

plot(impact) # Visual inspection of the fit

summary(impact) # Numerical summary of main statistics and model fit

summary(impact, “report”) # Written summary of main statistics and model fit

Conclusions

The results were very robust and allowed us to quantify the impact of generating a completely renewed version of the SDK. The possibility of using other games as predictors was vital. And finally, quickly obtaining results was advantageous as the Causal Impact package is quite intuitive and requires little effort to implement.

Causal Impact applied to the mobile games industry was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written in collaboration with Carlos Sarraute.

We at Wildlife understand that even though our business is, quite literally, “all fun and games”, the “business” part is still a key aspect of our lives here. As you’re probably aware by now, our company is data-driven. We collect and interpret all kinds of information to develop new, innovative ways to make our products lucrative, while at the same time keeping them free and open to consumers and monetizing them through digital advertising.

That’s where we come in.

You see, when we say we’re Wildlife’s “Data Scientists”, people tend to mix up some things: even among those who have some understanding of data work, it is not rare for them to think of us as “Data Engineers” — the difference between both things actually is much clearer if you’re part of our daily routine.

But to avoid getting too technical about it, we can say that, in a nutshell: “Data Engineers” are awesome at directing collected and verified data to where it needs to go — our data lake. But after that, we “Scientists” collect the raw material from a data lake and analyze what it is and what it means. It’s a bit of a two-way work, where engineers and scientists depend on each other, but even though we are close, we’re not quite the same.

In a sense, we find meaning to the data, establishing patterns and values that other teams (every team!) can benefit from. And that difference is important here: on average, we receive 18 TB per day of info (as in “terabytes”) of information from players all over the world.

From that raw data, we work to generate conclusions about the players: not only their behavior but also what drives them to download our games, how much they talk about them with their friends and acquaintances, and every other bit of data that tells us how good our games are to them.

Handling raw data and getting insights

Carlos Sarraute — Sr. Data Scientist Manager and co-author of this article — and I, work specifically on the Ads Monetization team, which is responsible for analyzing all that raw data and creating innovative ways to promote our games through social channels and other platforms that allow for advertising features.

And to handle all that, we need a sizable structure to house all of this data, which is why we use cloud-based solutions from Amazon Web Services for hosting, but that’s not all.

We divide our projects/analysis/modeling work into clusters for each data science team to work with it. A “cluster” is “a set of computation resources and configurations on which you run data engineering, data science, and data analytics workloads".

We have the marketing cluster, game development cluster, legal cluster… that are used by data scientists to offer insights to other teams. Every data science squad has its own space from which to access curated partial data so they can improve, in a more certain manner, their business practices.

A practical application: what we do

Let’s give an example of how we operate: let’s say that a certain number of Brazilian players are skipping our ads more often. We will never know that you, specifically, did not wait the mandatory 15 or 30 seconds from the ad piece’s running time before skipping it, but we know that a user did it, and what really matters to us is that a group of people that have similar details to this user, normally behave this way. So we can look at the entire package of data and determine why a handful of people are doing that, and plan our actions accordingly.

In this particular example, we can surmise that that particular ad piece has been running for a long time, so we know it’s about time to create a new, more fun piece — one that can lead you to download our games or play it more often due to a holiday campaign or something of the sort. That interpretation is then sent from us to the design and advertising teams, along with recommendations driven from our models, so they can come up with something new, which we’ll then use to run new ads in our back channels.

Collaboration with other teams

That interaction with the other teams is also paramount to us: one level of it is the work we execute with other data science teams. We share common knowledge and expertise between every squad, and then every squad works that information on its own vertical — Carlos and I, for instance, are in Ads Monetization, so we are the team that generates the variables that we use in our advertising, choosing which ad to display where.

With the information of those variables, the models we train with, and the conclusions that they generate, we work closely with the people that work on events inside our games, such as holiday season special items and such.

And there are also data science teams dedicated to benefit the player. The fact that we have ads allows us to distribute our games for free, so in a lot of our products, seeing ads is something beneficial — you choose to see an ad, you get an in-game benefit.

There is something that maybe not everyone thinks about, as well, and that’s the fact that whatever benefits the gamer, also benefits us: the gaming experience, for instance, is something that tells us when to show an ad for a game, to not spam the user with banners.

Different backgrounds, same mission

Currently, we have 57 data scientists worldwide — 11 of them here in Buenos Aires. One of the things that I love about our team is that we are eclectic about our skills. I started in Industrial Engineering and Carlos is a mathematician. In the squad, we also have a Ph.D. in statistics and one in physics… So we have a lot of diverse backgrounds, which helps us as a team, because we cover a lot of ground and knowledge, hence being able to tackle new problems with different mindsets.

Sure, we ask that candidates know the basics of data science and machine learning. We have a very thorough interview process where we evaluate all of that. But we also take into consideration the curiosity and research skills that you may bring to solve problems on the go. We’re building a world-class team.

Besides the background that we come from, I think that what is very clear in our team is that we have broad previous experience in the field. It becomes clear that you’re talking with people who know how to work with data.

So, as we said at the start of this article: data is essential to us. We have a lot of fun working with it, and it allows us to, in a specific manner, know more about you — and Wildlife always thrives on understanding its users a little bit more every day. We’re doers, and we’re glad to be like that all the time.

This article was inspired by a talk we presented in Nerdearla — the biggest IT event in Argentina — 2020 edition. If you want to dive a little deeper into our work, check out the video! (Autotranslations available)

Data Science at Wildlife: A deep dive into how our team works was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

There isn't a perfect CI/CD solution —and this shouldn't come up as a secret if you have experience on the matter.

There are several use cases and different maturity levels of the software development lifecycle amongst developers. A quick research about CI/CD can be overwhelming, as there are several solutions and different approaches for it, each one with its own advantages and disadvantages.

At Wildlife, our developers are responsible for the full Software Development Lifecycle, from building several products, to configure their own CI/CD pipelines, deploy the applications, and to be on-call for what they built.

Having ownership of the full SDLC helped us achieve our goal of continuing to grow without a co-dependent relationship between teams, and it has worked pretty well for several years.

As the company started to grow, developers started creating solutions to share between them, simplifying the steps needed to deploy their applications. Development teams started using tools to simplify their deployments, but we had different levels of Kubernetes knowledge amongst teams.

When you start to grow fast

This led us to have a few different deployments tools over the years, like the DEIS PaaS for some teams that wanted more abstraction and Jenkins for others that wanted more control over their Kubernetes deployments.

When our SRE team was constituted, we had developers shipping code with DEIS, Jenkins, Gitlab pipelines, Heroku, and even manually. Although the solutions worked great for a long time, we started having a few problems. Jenkins had several custom plugins that blocked us from upgrading it. DEIS was discontinued by its community and couldn’t cover every aspect you would like on a CI/CD solution.

Running applications worked fine, as the pipelines were already solid and properly configured, but we started seeing difficulties from developers to set up new pipelines. This was a red flag for us, as we started to see constant increases in the initial time to set up new applications.

At that time, we had a few Kubernetes clusters and some deployments on EC2 instances, but we were creating more and more Kubernetes clusters, migrating applications out of EC2 instances and integrations were tough. At first, we thought about how we could be the least disruptive, as learning a new process involves a learning curve that could slow us down.

We also researched how to update our current solutions, which, each time, sounded more difficult, as we ran really old versions of them. This brought us to the whiteboard and generated the following questions:

• Is it worth it updating our CI/CD tools? How much time would it cost us?
• How can we avoid developers having to write complicated Jenkinsfiles? Can we create some predefined templates?
• Should we create a DEIS fork and start maintaining it by ourselves, as it was not maintained anymore by its original owners?
• How can we manage integrations with Kubernetes clusters?
• How can we create a predictable path to deploying new applications? How can we migrate old pipelines to this new standard?

Those questions led us to one of our core values:

We innovate with research

Having many doubts and no answers made us start researching how other companies tackled this issue. We started studying the code of our Jenkins plugins, the code of DEIS, and research new tools that could be implemented.

We noticed that upgrading Jenkins in-place was impossible, which would have led us to have to create a new Jenkins deployment from scratch and rewrite every pipeline to new standards, which would yield an immeasurable amount of work. So it made sense to start to explore other options as well.

On the SRE team, we rely heavily on creating Proofs of Concept, writing internal Call-For-Comments, and, after deciding on an approach, writing Architecture Decision Records. This research period led us to create several PoC, generate multiple discussions, and present multiple resources, alongside reports of their usage.

At that time, we had researched the following solutions:

• ArgoCD
• Jenkins/Jenkins X
• Hepfy (active Deis Workflow fork)
• FluxCD
• Gitlab Pipelines with Kubernetes Integration
• Gitlab Auto DevOps

After the research, we liked the GitOps approach and had to decide between FluxCD and ArgoCD. The GitOps principles are greatly summarised by WeaveCloud:

• The entire system described declaratively
• The canonical desired system state versioned in Git
• Approved changes that can be automatically applied to the system
• Software agents to ensure correctness and alert on divergence

At Wildlife, we strongly rely on feedback from our peers so, with both PoCs running, we reached a few developers to see which tool they felt was easier, setting up a few pipelines for their applications.

After this brief validation, we decided that FluxCD was better for our use case, increasing the developer’s experience whenever shipping new code.

How Flux Works

The Flux GitOps approach relies on having a declarative representation of your Kubernetes manifests inside a Git repository. Flux will then sync with that repository, generate the manifests, and apply them to the desired namespace. This flow is represented below:

Instead of interacting with Kubernetes, developers are able to push code to a Git repository, validate the manifests with a pipeline and then merge it to the desired branch, which Flux would be watching. Whenever the changes are merged, Flux applies the new configuration. Developers still need to set up a pipeline, to test, lint, and build the application, then push the images to the respective Docker repository.

One thing that we noticed quickly was that Flux wouldn’t scale up if it was a single deployment, as it would have to sync with thousands of git repositories, generate the manifests for that applications and apply them.

To solve this we decided to use Flux in a Multi-tenant approach, as having thousands of resources being managed by the same pod could increase sync times between the Git repos and what’s actually deployed on the Kubernetes clusters.

This approach has a centralized Flux daemon, responsible for deploying multiple other Flux daemons on the namespace of the applications, those namespace-scoped daemons would then manage only the application resources, not a cluster-wide set of resources. The architecture could be exemplified as below, extracted from the Flux documentation:

This also enabled us to create namespace bound policies, where the team’s Flux daemons could only manage resources inside that namespace. With cluster-wide resources being managed by the cluster-admin Flux.

Another advantage of this approach is that it allowed us to monitor only the cluster-admin Flux, with the teams monitoring their own stack, which reduced the alert noise of the SRE team.

The default Flux behavior is to watch a Git repository, pointing to a path and branch, reading all manifests declared on the repository, and applying them with a simple kubectl apply -f -.

This behavior could be overridden using a custom file called .flux.yaml where you can specify the actions that Flux should execute in order to generate the manifests, instead of simply applying the existing one. The apply process remains the same, but we can use this action approach to use tools like Kustomize to generate manifests on the fly, instead of relying on a plain static existing one. A simple example, extracted from the Flux docs, looks like below:

version: 1 # must be `1`
patchUpdated:
  generators:
  - command: kustomize build .
  patchFile: flux-patch.yaml

A more detailed explanation can be found here. This generator.command yaml key can be used to specify any application for Flux to run and generate the manifest, in our case, that would be Helm and Kustomize.

There is no perfect solution

One thing we found when using Flux, is that you need to set up a Deploy Key in your application repositories, so it can clone and push commits to it. Those keys are generated whenever the namespaced Flux was created, and it could be a manual step, which is okay for a few namespaces, but it does not scale up for more than a hundred.

Our approach was to develop the Gitlab Flux Controller, which is a Kubernetes Controller that searches for Flux deployments keys and pushes them to the respective repositories.

We also rely heavily on Helm and Kustomize for templating our Kubernetes applications. We like to offer developers tools that they are used to, and FluxCD had support for both. For managing Helm Releases, Flux implemented a CRD called Helm Operator that manages Helm releases, which could further decrease the complexity for developers, but there was a catch.

Whenever you sync a Helm deployment using the Helm Operator, the only resource that Flux manages is the CRD itself. This implies manual changes on other Kubernetes resources not being overridden, which could reflect in the declared Git state drifting from the actual running configuration.

Having that in mind, we decided to use Helm template to generate the manifests, then use Flux to apply the generated manifests as plain text. This would ensure that the resources are always the same as declared on Git.

When trying to use Helm template as a generator command for Flux, we noticed that helm wasn’t present on the Flux image, so we built a custom Flux image and added the Helm binary to it. Our Flux configuration files started to become big and confusing, as each new namespace required a new generator command entry, to understand that, let’s see an example of a Cluster admin Flux repository:

So, as you can imagine, our .flux.yaml file started looking like this:

Which doesn’t seem like a good approach at all. Besides that, we also had to manage the Namespace creation, which wasn’t included in the default chart.

To handle that, we decided to create a helper program that was able to discover several helm charts and render them without further configuration, also handling the Namespace creation.

We created helm-generate, which uses the Helm SDK to handle manifest creation using Helm functionalities Having this tool helped us to simplify our Flux configurations to be as simple as below:

With those details set up, we then started looking into how to monitor Flux and found the Fluxcloud project, that sends Slack messages whenever some resource is updated. After integrating Fluxcloud with Slack, we decided to create an exporter to send events to Datadog, our monitoring tool of choice here at Wildlife, but after sending a PR and waiting for several days, we discovered that the project was abandoned.

We contacted the Flux community and after some back and forth conversations, they assigned us as the new official maintainer, as our fork was evolving the project and continuing its development.

The new official CI/CD tool: Gitlab Pipelines

After having a great Flux workflow, we moved on to fixing our CI/CD problem. There were many different options running on the company and we needed to push a single one for our backend applications. Gitlab is one of the most used CI/CD tools today, offering multiple resources and even native Kubernetes integrations.

Standardizing pipelines can be really hard as different programming languages and solutions require different configurations. In order to help developers build and standardize their pipelines, we’ve provided a set of external pipelines that can be included as an external file. Instead of creating a complex Job to login to ECR, for example, developers could include our template job with something like below:

You can also notice the usage of make for the build step. One approach for having a standard on your pipelines is the usage of Makefiles with predefined steps. Those predefined steps help us translate complicated steps to simple commands as make lint ,make cover , make tests , for example, enabling us to create predefined stages and Jobs, that only rely on Makefiles.

Flux Automatic Updates and Gitlab

One of the greatest features of Flux is its ability to automatically update images by scanning the docker repositories and updating images based on specific rules. This is done by specifying an annotation to enable the automation: fluxcd.io/automated: true , this enables the default behavior of updating images based on timestamp, the newest always get updated. You could also apply some constraints, like regexes, glob rules, or semver rules.

For our pipelines, developers mostly use two approaches: having a stag/master branches, or using semver to version production images.

For the first approach, whenever something is merged to staging/master, the docker image is built and tagged, then Flux notices the change on the repository and applies the changes.

For the second, the production image is only built whenever a developer pushes a SemVer tag, creating a Git release, and usually we use the automation constraints to ensure that only minors and patches are updated automatically, for example, with the annotation: fluxcd.io/tag.container: semver:~1 , which applies any version within the major 1.

How to abstract Helm

One of the things we also decided to do was to create a generic helm chart that could deploy a simple stateless application. This chart was then versioned and available for all developers.

Having good documentation and a set of good examples helped developers with no knowledge of Kubernetes to deploy applications by simply filling a values.yaml file, instead of needing to understand what is a Deployment, for example. This is important in a company that has several developers that don't interact with backend systems, as enables them to focus on their relevant work, instead of losing weeks trying to deploy an application, as Kubernetes has a harsh learning curve.

What's next?

Reaching a point where developers can deploy an application from scratch with only a few MR hasn’t been easy, as tools usually don’t abstract enough of the needed steps. Building several tools, and making customizations helped us make the process less painful and easier to predict, as we can now know what to expect from any pipeline.

Removing complicated Jenkinsfile, using Makefiles to create templated pipelines, and using FluxCD + Generic app chart have drastically improved our developer’s experience. Although we migrated several applications to this new model, we haven’t managed to migrate every single application to it and there is still much to do:

• Migrate every remaining app to the new GitlabCI + FluxCD approach.
• Enable FluxCD integration with Flagger for Blue/Green and Canary deployments
• Migrate Flux v1 to the GitOps ToolKit

Soon we'lI have some news about those next steps — and I could bring you updates in a new article. For now, I hope this one can help you make better decisions to deploy your applications. Good luck!

How we improved our CI/CD pipelines over the years was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deploying in modern days can be challenging, given the complexity of systems built in microservices structure. But thanks to some deployment strategies, you can do that with much more confidence.

In this article, I’ll show you how to use the canary deployment pattern. Despite the heartbreaking stories that made this little bird notorious for being a type of "informant", in software engineering, canary symbolizes substantial risk reduction. Let's find out why!

What are canary deployments?

They are a way to release updates to a shifting percentage of the total user base. What’s alluring about that is that when new code isn’t really ready to reach production, due to undetected bugs during the testing phase, performance issues, or other factors, it’ll only affect a small percentage of the users before being detected and rolled back.

There are for sure many ways to do canary releases, even more so because there are many ways to deploy and run applications. Since at Wildlife we’re heavy Kubernetes users, I’ll show you two approaches to do them on software running over Kubernetes. Let's dig in!

A simplistic approach

A typical request flow

Typically, to make an HTTP/HTTPS API available to external users we set up an ingress controller and define routing rules to send traffic to specific services.

So, the bare minimum we need to confirm with the diagram above is something like the snippets of YAML below.

All resources are named “your-api”, and the ingress will route all requests for “api.example.com” to the “http” port in the service also called “your-api”.

The only thing the deployment needs so that the service can point to the expected pods is the “app” label.

With manifests like the ones above we can have a backend in place with a fixed number of replicas running. One natural extra step is using a Horizontal Pod Autoscaler.

When we approach deployments like the shown flow, we put ourselves in an all or nothing kind of situation. Where we have a mix of versions only during the Kubernetes rollout of new pods, which can be very short.

So, if we find a problem in production after that transitional period, we’ll have to rollback after identifying the issue and having potentially negatively affected 100% of the live user base.

The canary ingress

To prevent exactly this situation regarding code changes, we can deploy applications to an ever-increasing percentage of requests until it covers all of them.

We’ll call the flow above “stable” and create a new one for “canary” releases. They’ll be incredibly similar but point to different resources in the chain. In our case, “stable” will keep naming all resources “your-api”, and “canary” will call them “your-api-canary”.

The differences between this ingress manifest and the one from before are in the annotations. In this case, we’re using the NGINX ingress, letting Kubernetes know that this ingress is canary and that 5% of the requests are expected to go to the service “your-api-canary”.

That’s all that’s needed to route a specific percentage of requests to another Kubernetes service, that can point to another deployment, with a different version of the same application, for example.

You’ll hardly be satisfied with just that. Here are some questions one might ask:

• How to automatically shift percentages until all traffic is reaching a new version?
• How to automatically cancel a canary release if it’s not production-ready?

One way to tackle both issues is by creating new stages in your continuous delivery pipelines. The first thing to do can be defining targets like 5%, 25%, 50%, and 100% and manually playing some or each of them.

When we reach 100% of traffic on canary, this means that the stable flow should switch to the canary version and canary is now at 0%.

There’s also Flagger, a Kubernetes operator that helps on canary deployments.

Flagger

Instead of creating the manifests we’ve already seen, we could achieve the same result by installing Flagger on the cluster and creating a “canary” manifest.

Here’s the official documentation on how to use Flagger with NGINX. The cool thing about using this operator instead of doing your own, as in the previous section, is that it has several integrations out of the box. You can set, for example:

• Acceptance tests;
• Load tests;
• Watch Prometheus’ metrics to use in deployment strategies;
• Strategy for automatic percentage shift;
• Strategy to rollback.

Final words

I hope that at this point you’re aware of how simple it is to avoid all or nothing situations when releasing new code to production — using a community-supported operator or in a “DIY fashion”.

Canary deployment in Kubernetes: how to use the pattern was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In many developer teams, being the Technical Writer is like being the bassist in a band. There are some very remarkable bassists out there, but mostly you are the person who didn’t get to play guitar.

It’s common to see teams that assign their documentation to the “less busy” member of the team, or to the newer members, so they can learn while they document. Other teams make the most knowledgeable person among them do the documentation, which creates better documentation but is sometimes viewed as punishment, to get the extra work as a consequence of doing a great job.

Throughout my career, I’ve seen many kinds of teams, and many types of Technical Writers. The truth is I haven’t seen one way of working to be necessarily better than the other, but there is one thing all creators of good documentation have in common: They focus on solving problems.

So let’s dig into this concept, shall we?

How we solve problems by making information easier to access

If you google “Knowledge Hierarchy”, you will find the “Knowledge Cycle” multiple times.

Data > Information > Knowledge > Wisdom

Everyone in a knowledge management team knows it by heart, but not everyone has seen data become wisdom in real life, because documentation is often considered a secondary task that needs to be done, and more often than not, few people use it.

The subjectivity of knowledge makes it hard to grasp. I won’t give you a full chapter on theories, but I can tell you what my experience has taught me about each phase in practice. I’ll break it down to explain how these theories can become a practice.

Step one: Data

Nowadays, everything can be considered Data. Even “no data available” is data. So this will depend on what your team has, and what they want to achieve. It will vary every time. Don’t waste too much time looking for databases. If there was enough data to do a Technical Writer’s job, there would be a bot doing that job.

A technical writer’s data is most likely hidden in daily operations. So start by getting some clarity on why your work is necessary for your team. Are they onboarding new people? Are senior developers leaving the team? Is the team struggling with a particular tool?

Identify the problem, and identify what already exists
Often, this means looking at OKRs and SLAs. What is your team trying to achieve, and what’s standing in their way? The person who requests a document will not always give you this. It’s not their job. A Technical Writer completes the task. A knowledge professional’s job is finding the way in which the task contributes to the team’s goals. Even before getting the information for a document, identify how this document can make their work better.

Step two: Information

This is usually the request Technical Writers get: “I want this information documented”. The truth is, Information is Data rearranged and analyzed to have a meaning. To give meaning to the information, you need to understand who is going to read this document and give them context. What do they already know? What do they ignore? When and why will they need to know this?

This is why many teams use language professionals as their technical writers, more than developers. If this is your case, get technical. Learn code basics, and understand what your engineers need. If you’re a developer, get into UX. Understanding the audience is key to making your information valuable for them.

The information a Technical Writer is given is not to be copied and pasted. It is to be analyzed and restructured for the people who will read the document. Sometimes this means turning one document into four or merging six documents into one. This is why a robot can’t do a technical writer’s job. The source of the information is never the user. Identify the audience of your documentation.

Documentation must be designed so that the people who read it understand what the people who need you to create it want. By this time, you should be well aware of the problem you are trying to solve. It’s like getting your final test the night before. Study the information with that mindset. Solve the test as you read the information you get.

Step three: Knowledge

Now the hard part begins. The biggest difference between information and knowledge is that knowledge solves problems. Often. It’s common to ask why the team is not following, or even reading the documentation. If this happens in your team, read your document thinking “What’s in it for them?”. “They save time” is a common answer. If your document saves 5 minutes in reading but requires six searches to find, then it’s not working. They don’t have to make it easy for you. It’s your document that should make it easier for them. It’s part of your job to make sure they have it handy on a few clicks when necessary.

Whether documentation was made by an engineer on the team or a Technical Writer, the document will be better if it was designed for the user of the information to save time, reduce cost, or in general, add value to their work. Documentation should be related to the main issues faced daily by the people using this information.

It should not get in the way of their objectives. Forcing engineers to read documents, or comment on them is a terrible practice. Voluntary use is your best feedback. They have to WANT to use your document because it makes their job faster, easier, and better.

If the document is about coding standards to follow, it should be easy to read, handy, and highlight the most commonly used and relevant guidelines, and have a quick cheat sheet on the ones that are less common, and easily overlooked or forgotten. We should never force people to use the knowledge we create. Knowledge use is feedback. If your documents are not being used, or not being found, then they’re not solving problems. Title them well, tag them appropriately, and make sure they have relevant information that is easy to scan, without having to read 3 paragraphs before you reach a solution.

Step four: Wisdom

This is the final stage of maturity. It means the use of the knowledge created is helping create better knowledge. It means you solve bigger problems. This is when your work is adding value that is appreciated and enriched by the documentation owners and users.

Wisdom is what you get when other teams trust you with their information because you offer solutions that adapt to their needs.

So how do we get there as a team?

When I joined Wildlife, I got to meet their small Technical Writing team. I heard the word around was that the engineers were not valuing the work of the Technical Writers very highly, and they wanted to change that.

They helped me explore the Wiki (the company go-to source of information), and I have to say that, even though some documents were very good, they were also very hard to find.

Being used to work for very big companies, with multiple clients, and support teams, I was used to seeing messy knowledge bases. This was not one of the worst I’d seen. It could use some organizing, but in general, this wiki contained good information and well-defined spaces.

There was a lot of useful information, but it was mostly hard to find (sometimes even for people within that team) and even worse, easy to confuse with outdated information. There also was little use of tags and naming conventions, and I couldn’t find one template that was practical for general use.

Each one of the Technical Writers — in their own way — were trying to bring a little extra to the teams they were working with, and I thought maybe, they just needed the right guidance to get there.

Knowing this, I wanted that “extra” effort to be focused on what adds value for the engineers. I began working with each one of them on templates and standardized structures, so what is working for one team or squad can be useful for all Wilders.

When an engineer adds a new document, I want it to be as easy as clicking a button that creates a “fill in the blanks” document that they pour their knowledge into. When they make a search, I want the right document to appear, and the title of such document to be very clear on its content.

We have a long way to go, but I see a future where every team can get more than a document when they make a request from our team, I want every team to get guidelines, standards, and structures that are functional, scalable, findable, and in general, a tool to grow without worrying about the mess.

We want to empower every team to think big, move fast, and stay committed to the work they’re best at. We’ll do what we are best at, so that when you get there, your wiki is waiting for you, clean, organized, and ready for your next challenge.

From Data to Wisdom: Why our Technical Writing team focuses on problem-solving was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Wildlife is one of the 10 largest mobile gaming companies in the world. But that didn't happen overnight. Our portfolio of over 60 games engages billions of players around the world — and that's possible thanks to incredible teams that focus on making Ads Technology.

Why do we care about advertising?

Let’s imagine that you have nailed it: you have the ultimate mobile game that will become viral and will have tons of downloads. Now, is it a business? Well, it could be… if you monetize it!

How to make money out of mobile games

You could think of a game as a system that acquires users (usually at a cost) and generates either money or churn:

To monetize a game you will most likely (a) sell your game in the store or (b) make it free and monetize it inside the app. Let’s talk about this later case: you could offer in-app purchases (ie: swords and gems), subscriptions (ie: pay recurrently and get goodies), or show ads.

But again, why caring about ads for mobile games?

This is a mega-competitive industry and a large part of the gaming audience is offered/expects to get the fun for free (this is so true that before being Wildlife Studios, our studio was named Fun Games For Free!). So ads are powerful because they are a way to monetize this large audience of people who is not willing to cash out money for it.

OK! I want to display ads. Now what?

Now the fun part starts! You have an iOS or Android application and you have selected certain spots where ads can be shown. It’s time to go to the SDK supermarket and choose between Admob, Facebook, Mopub, AdColony, Applovin, Ironsource, Applovin, UnityAds, Rubicon, Vungle, or…

You are starting to see it right? It’s not so easy to choose — and also very inefficient. Choosing a single network will allow you to monetize users for a certain region, demographics, and ad type but will totally fail to do so for others. Just to give an example, Facebook you could be very effective monetizing a 35-year old person from California but might just not have anything good for a 14-year old kid in Singapore which could be much better served by, let’s say, Ironsource.

This is why people use a mediation SDK which allows you to create a “waterfall” of networks, trying to maximize revenue.

A waterfall would look like this:

Here we introduced a key term on this business:

• eCPM: this roughly represents how much money you will get every thousand impressions
• Fill rate: the percentage of times you ask for an ad and you get one

As you can see, things start to get funnier, now you have a business to optimize. 😱

Ads Tech at Wildlife

As I've mentioned earlier, at Wildlife we have a portfolio of over 60 games that accumulate more than 2 billion downloads, so we built our own technology to efficiently monetize our games and operate the ads stack.

Here is a diagram of how this looks like:

It's worth mentioning that we have a world-wide business that never sleeps: when people in the US have gone to sleep, Singapore kicks in, and then Europe… The fun runs 24x7!

How mediation works

The mediation mission is to be “one SDK to rule them all”. It has to provide a simple, uniform, low-footprint, never-crashing service for games to monetize every opportunity. Behind the scenes, it implements the core waterfall caching strategies, integrates +20 SDKs from the networks we partner with, takes care of analytics processing, and A/B testing.

One more thing: it also has a rendering engine of its own. This is because our mediation is integrated with an in-house world-class Exchange service that executes real-time auctions on a set of OpenRTB partners. Some of these partners take care of rendering their ads and some delegate that task to our SDK. Check the next section to know more about RTB.

This is a critical part of our ads stack that requires a team of highly skilled developers. We currently write different versions of our SDK for iOS (Swift), Android (Kotlin), and Unity. For efficient communication between our SDK and our backends -normally implemented on Go- we favor the use of gRPC and protocol buffers.

In the latest times, we have been also becoming fans of reactive programming so we refactored a core part of our iOS SDK with extensive use of RxSwift (we have different languages and platforms but the same coding principles). Finally, testing a lib that will integrate into so many games is crucial and we take our testing culture seriously. Besides unit, integration, and UI tests, we have an extensive infrastructure for having test modes and crafted configurations for test automation.

Real-Time Bidding explained

The ad business is evolving into what’s called Header Bidding, Real-Time Bidding, or just Bidding. In this world, instead of going through each network in a waterfall fashion, we run a parallel server-to-server auction and select the winner using either a first-price or second-price algorithm. Here’s an example:

Here things start to become more “backend-ish”. If you love Golang you’ll find yourself at home. We care a lot about our technology and we can proudly say that there’s little legacy around: we’re completely cloud-based, our backend stack is based on Kubernetes, Terraform, Golang, gRPC, data pipelines run on managed Kafka, Spark, Flink, Airflow and if you keep looking around you’ll get into Redis, Postgres, Kibana, Datadog and that kind of stuff.

Ads Ops

Operating an ads business is challenging. We have highly-skilled ops and biz dev team and we support their operation with tools that allow them to manage, analyze, and report what happens on each game.

Our UI for services is built on React and GraphQL. We have extensive RBAC (role-based access control) needs so we have partnered with authentication providers and implemented OAuth across the whole infrastructure. Internally, everything is based on REST micro-services that allows each part of the company (for example, User Acquisition) to seamlessly integrate with each other.

Optimization

Last but not least… the inner circle of magic. This topic would need an article as extensive as this one and we can recommend you read our articles here at Wildlife's Tech Blog to learn more about what we do in Data Science. What we can say at this time is that, as we have seen before, there are a bunch of parameters and complexities to tune in order to have the best performance.

For example: is it better to have a long and dense waterfall at the risk of timing out and losing an opportunity or is it better to cluster smartly around some price points? Does your answer hold the same for any device in any location in the world? Those kinds of things are the problems that the Data Science team tackles in the context of ads optimization.

And sometimes the problem is just about the product. For example: is 100% fill rate the target? No! It would be very easy to have high fill rates by spamming the user with ads. That’s why we limit ourselves to how many ads we show per session. And what’s the optimal point? Is it something to A/B test? Well, there you go, more to optimize. 😀

Join the team!

I hope you enjoyed reading about what we do as much as we enjoy doing it! After all, having fun is part of our core business, and that's why we want to build games that will be remembered for generations.

We have opened our Buenos Aires office last year (photo below) and we are hiring! We have open positions for Senior and Jr. Managers, Mobile, Backend, Fullstack, and Data Engineers.

Are you interested? 🚀 Reach out here => http://lifeiswild.co/careers

See you soon!

Ads Technology for mobile gaming explained was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Go is a modern programming language and as such was designed with incredible testing support out of the gate. Project and file organization combined with official tooling and a lean but powerful (and also official) testing library: that's all you need to know to get started.

Let's start from the top: the go test command

Go’s standard library already has everything you need and there’s a go test command included in its binary that, by default, picks up all test files in a given path and executes all test cases.

Anatomy

Test files are expected to be located within the same package of the code they are testing. It doesn’t have to be like this, but this colocation is a well-liked and followed a pattern. For one, it makes it easy to know where tests for any given package are, instead of relying on a single test or specs directory without any enforceable structure.

There’s also the possibility of customizing the setup and teardown for all tests in each package, which is a handy feature when creating integration tests that depend on starting and shutting down servers or connections, for example.

Each function that starts with the prefix Test and receives a single *testing.T argument is a different test case. As we’ll see further, multiple test cases for the same function are encouraged to be colocated in a tabular structure.

Related to this structure, it’s possible to run the tests for an entire project, tests for a single package, or even tests that match a specific name pattern.

Table testing

One way to approach testing in Go is by creating a new function for every test case. But rare are the situations where you’re comfortable with testing a single (input, output) tuple.

For that reason, a well used and recommended pattern to test functions is through tables where you describe several (input, output) tuples and check all of them in the same loop.

To paint a proper picture, imagine yourself creating a function that takes a time value and returns a human-readable indication of time passed since right now. What are some test cases for it? Here’s some I could think of:

A good case for table testing is the ease of adding and checking new behavior. If you’re into TDD, it feels organic to write your function in an iterative way, you could add the barebones return “now” function and keep adding new time units, seconds, minutes, days, etc.

On the other hand, all test cases rely on the same inner loop code. This can become especially challenging to maintain with complex integration test cases where side effects are expected for some table rows but not others.

White box vs Black box testing

You saw the tests — behold the code:

There’s a public function called Since and that’s that. When we rely only on the public interface of a package on our tests, that’s what we call black box testing. Black box in the sense that we see nothing behind the public interface.

To guarantee that you’re really in a black box, Go allows you to have one extra package name under the same directory, that’s the same name of the real package with the _test suffix.

One problem when testing things that depend on time is that the test cases might not always pass, depending on how much time it took in each run. In our case, if we had many, many cases in the table, it could be that time.Now() was already delayed by one second or more when the loop reached the last one.

So, to illustrate what a white box test is, let us create a private function that receives a second argument:

Now both files belong to the same package, and we have the guarantee that we’re always comparing to a controlled now value. White box testing has no assurances that the tester isn’t playing with the private, or inner workings, of the test target, for good or bad. In some situations, it might be very handy or even desirable, as in the example above.

In most cases, however, you should stick with black box testing. They are the only ones that can guarantee the public interface, the one your clients will use, has the expected behavior independently of knowing, and even mutating, private state.

Unit vs Integration testing

The example we had so far is a good representation of what a unit test is. It depends only on the software code base itself. However, it’s really common for software to engage in external communications and talk with other software.

It’s very possible that you’ll write gRPC/HTTP calls, queries to database systems, and other things of this sort. One way of taking these code paths into account when testing is through mocks.

A good way to encapsulate code that makes external calls in Go is by using interfaces. Then you could have an actual implementation that talks with the real thing and either write a mock yourself or use an automatic tool like go mock. This way you can still rely on unit tests for the rest of the code but at the cost of not knowing whether the actual integration would work as expected.

It’s important to note that when you rely extensively on mocking interfaces you’re not testing the actual integration, this is in reality an effective bypass of it. So mocks are good to create unit tests in addition to integration ones, especially when checking many scenarios in integration tests prove too cumbersome.

Tags

Integration tests have incredible value, and you should prefer them almost always. Unit testing is relevant when specific bits of business logic are highly complex and testing them separate requires substantially less preparation than through integration.

So as the last piece of interesting information for this article I would like to mention tags. Go provides a simple and efficient mechanism of tagging code files in order to execute them only when one wants.

Since the file above has the integration tag in it, it’ll only run through go test if we include the flag -tags=integration. This way, only unit tests, that require nothing but the code to execute, run by default, due to having no tags associated.

Wrap up

Now you know the most important bits there is to know about testing in Go, but this is still far from all there is. Testing is an interesting craft with many colors, and much of it isn’t specific to Go. Give it a try!

Testing Golang code: our approach at Wildlife was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most endearing things about what we do at Wildlife Studios is the way we promote our games. You see, we are a data-driven company that aims to get better and better and offer new experiences to our players. And much of that comes from how we advertise all the amazing things we create: don’t get me wrong, our games are top-notch, but nothing would come out of them if we didn’t get the word out and make people know about them, right?

Think about this: if our games already reached more than 2 billion downloads, our playable ads could have reached much more.

That’s where I come in: alongside my team, I develop “playable ads”, a form of digital advertising that isn’t really new, but we got to work on a whole new level for us. And the most interesting part of this story: when I joined Wildlife almost two years ago, I didn’t even know what playable ads were. What represents a major force towards our external communication efforts today, literally started with an epiphany from one of our top executives, but we’ll get to that in a moment.

My job at Wildlife started at college: company representatives came into my classroom and introduced themselves, the company, and its portfolio, and gave us a test for skill assessment. At first, they told me I could test for a game developer position, but then I found out there was this other marketing-related position that required JavaScript knowledge.

Despite having little experience in that particular language, I thought I could learn more once I got into the company. I did pretty well on that test after all and was later contacted by our Human Resources department for an interview. When asked, I made it clear that I knew how to code and was learning new, general stuff at college, but there were minor gaps that I felt I should be upfront about. Specifically, I didn’t know how to work with Javascript frameworks.

I honestly thought I was done for, but much to my surprise, I got a call a couple of days later and got the job. Back then, I had a general idea of what I would be doing: programming. After I joined Wildlife, I learned about the details, and the work I would be doing with “playable ads.” I use quotes here because, much as you’re probably feeling right now, I had no idea what that meant, apart from the obvious — that they were some kind of advertising (I mean, it’s right there in the name).

So… Is that a game inside an ad or an ad inside a game?

Anyway, playable ads: basically, they’re a digital form of advertising inserted in the context of gaming. You can think of them as “the short experience of particular gameplay”, regarding the game we would love you to download on your phone. As I said earlier, it isn’t really something new, but we’re probably the first company in Brazil (and probably one of the firsts to use playables in 3D with advanced tech worldwide) to use them with data and style.

What do I mean by that? Well, a successful playables needs to stand out in a crowd, looking good and getting our user excited about the content but, unlike the game itself (which has technical limitations, especially for user hardware, but nevertheless more flexible than playables), the ad needs to be light.

In the early days, we did some pretty bad playables, which was expected, because we had technical restraints and we lacked experience. We used to create a kind of static only advertising piece. But those will only get you so far and, soon and surely enough, we started to feel like we needed to do something different… So we decided to go bold and invest in better playables.

So we studied many (seriously: many!) playable ads to see what we could create based on our viewing experience. As consumers, this allowed us a unique view of what we wanted. I’m sure you have seen them by now.

We want to create the experience, not just show you an ad in order to sell you something — it has to be fun. Hundreds of millions of people have already seen them!

The tech part of it is where the real charm is at like I said, playable ads have to be light. This comprises not only its duration (unlike inside games where people play for several hours on playables they just interact a couple of minutes) but also its file size, processing, memory consumption, and user experience.

At Wildlife, our playable ads can be as small as a floppy disk, and under that, we have it all: 3D character; scenarios, animations, interactive elements (your controls for playing the ad), as well as all the links and buttons that redirect you to the game’s page on the app store so you can download it.

Empowering your skills through hands-on work

One of the main aspects of our technology is efficiency. For those who are not well versed in programming, picture this: your character needs to move from one side of the screen to the other. The character walks on a floor that must be continually visible in the ad, right?

A regular programmer would create a huge amount of tiles as a first step, presuming that the player would never move so far away, until it reached the bottom of the screen, or draw tiles as the player moves, but leaving the “old” tiles back there, useless. Problem: it takes a lot of space and increases memory consumption.

A good programmer would draw the first three to five floor tiles and, as the character walks by, the previous tiles become useless, being destroyed (deleted) in the process and new tiles are created to make the floor look continuous. That helps save up memory usage, but it raises the demand for processing, since deleting something is quite expensive in terms of processing cost.

How do you think a Wildlife programmer would solve that?

Most frameworks would not allow us to edit our ads at its core programming level. We have to think of delivering all of this in a way that does not clog up your device’s performance. By the average data, most smartphones have 2 GB to 4 GB of RAM memory available and, while the game we’re advertising can and probably will use most if not all of it when launched, the ad most certainly needs this kind of restraint.

So our proprietary tool helps us make sure that the whole ad will flow efficiently. It’s the Wildlife way.

Which leads to the relevance of a tool we developed within our team is called “miniplay”. Basically, it’s an engine used for game development, like Unity, Phaser, and other ones publicly available. The difference, however, is that miniplay is the kind of resource that we developed “on the go”, meaning that, whenever we hit an obstacle that common tools did not provide means to solve — or did solve but not in a way we would like to–, we went on and created our own solutions by hand. In the end, miniplay became the engine that puts it all together in one package, and we feel it fits our needs much better than any option out there, although we studied external solutions as well.

Working on your skills

Wildlife requires a lot in terms of programming knowledge whenever hiring a new contributor to our team, but we don’t look for a specific language — instead, we’re looking for people very good at data structures, algorithms, and complexity analysis. Taking myself as an example, the company usually asks that you know the general do’s and don’ts about coding. Web-based development, game development, and digital art are all part of the extras you can bring with you.

The reason behind this is that our department grew so much, and so organically, that our hands-on approach allowed us to create a multitude of documentation that you’ll not just study about, but use in your day to day routine. For example: if you need to animate a scenery with, say, raining, tools like Unity and Phaser have pre-made resources for that: you just select one, insert it in your playable project and see what looks best.

It’s all very simple. Is it bad? Of course not! Everyone loves simple and easy. But it did stop us from growing, creating much of a comfort zone that did not allow us to tinker and mess with core coding the way we wanted too.

That led us to create our own tools. And the way we did it was the best possible: we went full hands-on, made a lot of mistakes, and created and reviewed a lot of documentation. We spent months trying to find a framework that worked for us and not one that made us work for it. It’s a demanding process, I won’t lie, but as I said in the beginning, I didn’t consider myself an expert, and now I can deliver a simple playable ad project really fast, in a programming language I didn’t appreciate very much back then. That goes for you too. At Wildlife, you’ll evolve.

So, as I said, we developed all of our resources internally. They may not be as easy to use as the more readily-available solutions out there, true, but they will empower you to edit your work so deep to a level you’ll be coding and programming what Picasso was to painting.

Sometimes during a playable development, we’ve had people asking us to send a playable ad file so they can kill some time playing it, which is proof that we’re going in the right direction.

Now, I know you may be asking yourself: “who wants to work specifically in playable ads?”. I used to ask the same thing, but the reasons became clear.

Our job entails creativity, programming knowledge, and self-adaptation and improvement. There are a ton of challenges, but there’s also a sense of partnership that brings our team together and helps us overcome them.

Wanna join our team? Check out our open positions.

Playable Ads: how we use them to generate heavy buzz for our games was originally published in Wildlife Studios Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.