Cracking the Code: Showcase Quest Snafu

Are Challenge Rewards truly random?

As many of you already know, there was a significant issue with the distribution of Challenge reward Moments from the latest Throwdowns Showcase Quest yesterday: the serial numbers were not distributed in a truly random fashion. While Top Shot has since publicly addressed the mistake and made amends with Collectors who participated in the Quest, there’s plenty of interesting data to dive into before we can fully grasp the situation. Were these users hand-picked by Top Shot? Has this less-than-random distribution occurred in the past? Let’s dive in.

What Went Wrong

In Top Shot’s public statement, they mentioned that there was an issue with the latest distribution of Quest reward Moments, specifically that the randomized list of recipients was re-sorted by an Internal ID tied to each account. But to what extent were the rewards distributed in a non-random fashion? To better understand the impact of this bug, let’s visualize how the reward Moments were distributed to Collectors. Thanks to amazing data from LiveToken (one of my favorite tools for sleuthing through TopShot Data), we can sort all 809 Collectors by the date they joined Top Shot and compare this to the serial number they received.

Right away we can see that there is some clear connection between when Collectors joined the platform and the serial number they received. The trend is an interesting one, and I think it’s easier to see what’s going on if we break-up the chart into three distinct sections as shown below:

Each of these sections have some unique characteristics with regards to how random the serial numbers were distributed:

• Section A: Never random; all Collectors in this section received the lowest serial numbers.

• Section B: Occasionally random; most of these early Collectors received the lowest remaining serial numbers, but some received a seemingly random Moment.

• Section C: Exclusively random; join date has no impact on serial number, and everything is entirely random.

This makes me wonder: what on the Top Shot platform has followed a similar pattern? What was something that you could never do for a period of time, then something you could occasionally do, and then something you could never do. What about the method for creating your Top Shot account? In the early days, it was only done by providing Dapper with an Email address, next you could create an Account through Google, and eventually this became the only way to create an account.

Funny enough, each Collector’s authentication method is public information. Head to any Collector’s account, view the page source, and you’ll find something like

"dapperID":"auth0|..."

which indicates an authentication via Dapper, while others explicitly mention Google.

Let’s identify how each account was authenticated and re-draw our scatter plot:

Well there’s our answer! Dapper mentioned that the list of recipients was accidentally sorted by their Internal User ID, and this ID likely looks very different, depending on whether or not the user authenticated their account through Dapper or Google. Nerd stuff aside, it explains the seemingly unexplainable scatter chart.

Isolated Incident?

To understand if this is genuinely the first time that Challenge reward Moments have been distributed in a non-random fashion, let’s take a look at the Showcase Quest for Ben Simmons. Using the same approach as before, let’s see if there’s a relationship between when Collectors joined Top Shot and the reward serial number.

Now that is truly randomized, and I think we can all put our pitchforks down. I can hear it already, “but what about regular Challenges, not just Quests?”. How about the recent Playoff Challenge for Anthony Davis? Let’s take a look.

So it’s clear to me that the recent Throwdowns Quest was an isolated incident, at least with regards to the theory that Top Shot OG’s receive better serial numbers than newer members. How about from Challenge to Challenge? Do some Collectors seemingly have better or worse luck than others when it comes to Challenge rewards?

To answer this question, we can choose two recent Challenges from the 2021 Playoff Set such as Paul George and Giannis Antetokounmpo. Next, we can look for Collectors that completed both Challenges, and compare the serial numbers they received for each.

Again, these results look random. If there was a tendency for Collectors to receive similar serial numbers from Challenge to Challenge, we would see far more points along the diagonal. Sure, there are a handful of examples where Collectors received very similar serial numbers, but we can’t cherry-pick one or two questionable examples and turn a blind eye to the remainder of the dataset.

Final Thoughts

Now that the dust has settled, we can take a moment to zoom out and look at things from a higher-level. Personally, this incident highlights the centralized nature of Top Shot. While we’re still collecting NFT’s, there is a considerable amount of information that remains private. The Top Shot smart contracts around buying and selling Moments on the marketplace have been made public, yet many of the internal mechanisms remain behind closed doors. How are we randomly assigned positions in pack drop queues? How are Moments randomly inserted into packs? Today we simply have to trust Dapper Labs, something that can be more difficult when errors like yesterday occur. Fortunately, as we’ve seen, we have all the data we could ever want to look at past events and evaluate their fairness.

That’s all for this week! As always, I hope you enjoyed this article. Want to support the newsletter? You can help with the late nights and early mornings by buying me a coffee ☕️

Disclaimer: None of the above constitutes professional and/or financial advice. All opinions expressed on Minted Moment are from the personal research and experience of the author, and are intended for educational purposes only.