Science of Skillscape

Why Skillscape?

One of the most common responses folks hear when they ask, “How do I get better at climbing?” is, “Climb more!” With Skillscape, we sought to start answering the inevitable follow-up question . . . “Climb what!?”

Much of climbing progression is influenced by the appropriateness of a given challenge relative to the individual climber. When we are talking about how a climb challenges the climber, it isn’t always the grade of the given climb that defines how challenging it is. A climb can be challenging for many reasons and not all of those reasons are reflected in the grade of a specific climb. In many cases climbs that challenge areas of weakness (both technically and physically) in a given climber are graded “easier” than the climber would typically anticipate. Climbing is a complex sport and knowing what your personal strengths and weaknesses are and how to challenge them is difficult and constantly changing.

It has been my experience that climbers usually experience more significant improvements in their climbing ability when they are consistently presented with challenges just outside of their comfort zone. As their comfort zone begins to expand, new assessments and new challenges are necessary. A good coach builds a relationship with an athlete and knows how, when, and to what extent to push that comfort zone to promote improved performance. However, not everyone has access to a coach. There are many places without readily accessible outdoor climbing or indoor climbing facilities. Even if the climbing itself is accessible, it can often be difficult to find a coach. Even then, coaching can be prohibitively expensive. Skillscape is our attempt to make personalized coaching more broadly accessible through an AI, data-driven virtual coach. That being said, Skillscape is not a 1:1 replacement for a human coach. What it is, is a very significant improvement over climbing without a coach and also a tool for coaches to work more effectively with remote clients. 

This is the tip of the iceberg and we look forward to continuing to develop this technology and further improve the climbing experience. 

How do I train with Skillscape?

Skillscape helps you identify concrete training goals, recommends specific climbs to practice those skills, and helps you monitor your progress toward achieving those goals. You can start using Skillscape when you have 10 climbs logged but the more climbs you log the more accurate and stable your profile becomes. We recommend logging at least 50 distinct boulders for Skillscape to have an accurate picture of you as a climber. 

Your Skillscape profile reflects your relative strengths and weaknesses based on what you climb most. It updates in real time with every climb you log (although not every climb may lead to a visual change). Since everything on Skillscape is relative to your personal strengths and weaknesses as you improve in one dimension, another dimension will become (in relative terms) a weakness. For example, if you first log a powerful, jumpy climb, your skill profile will shift slightly in that direction. If you then log a different style of climb, say a more static climb on small crimps, your skill profile will shift yet again slightly in this other direction. So the skills you see as acquired on Skillscape will always represent your current, relative strengths and weaknesses. These relative strengths and weaknesses can fluctuate, especially if you don’t have many logged climbs overall. It can take many climbs to lock in a skill as a “permanent” strength. One great strategy to “lock in” a skill a bit more is to also work on adjacent skills as represented in Skillscape.

NOTE: It is perfectly normal to acquire a skill and then later on it flips back to unacquired. Especially when you first acquire a skill it is not “locked in” as a permanent strength of yours yet and so when you log other climbs, it can revert back to unacquired.

Here is the important part, however. Under the hood, Skillscape knows and remembers your precise achievement level (in terms of difficulty rating) for each individual skill. Each new climb you log counts toward your profile, even if it does not change the acquired status of a skill. So even though one of your skills may have flipped back to unacquired, internally you may have pushed your achievement level for that specific skill a bit higher. As a result, every new recommendation for that skill—even if it is unacquired—will be a little bit more challenging and if you do acquire the skill later on, you will have acquired it at a slightly higher level. That means that Skillscape keeps track of your entire profile: even if your current strengths and weaknesses change, Skillscape always knows your achievement level for each skill.

This means that if you follow Skillscape recommendations, you are not going in circles in your training, even if a specific skill flips to “un-acquired” temporarily. Every single climb recommended by Skillscape will push you to climb harder and harder. For example, a V4.05 strength in one skill, may become a V4.06 weakness as you push other skills further, and then a V4.07 strength again and so on.

Skillscape: Layout and Difficulty

Our patent-pending Skillscape algorithm uncovers the structure of climbing skills and helps answer key questions about climbing training.

  • What are the most important climbing skills? 
    • Impact on climbing training: Generally speaking, for efficient training, we want to focus on the most useful skills as opposed to niche skills that are less useful (i.e., skills that transfer less to other skills). 
  • What is a natural progression? 
    • Impact on climbing training: To advance in our training we need to expand our comfort zone. But what is the right amount? Given where a climber is now, what is the next skill they should train to acquire or improve? This helps answer the question: “What should I focus on in my training today/tomorrow/next month?”
  • How can I measure and manage my progress?
    • Impact on climbing training: We want to progress upward (in skill difficulty) rather than backward or sideways. The ability to measure and monitor your workout is key to progress. Ideally, we want to rely on objective data to measure and monitor (a) what skills we train and (b) how difficult those skills are.

Skillscape is based on the analysis of 3.0 million moves and over 23 thousand climbs.

Skillscape is entirely data- and machine learning-driven. Nothing in the app is manually coded or labeled — everything comes directly out of the data. In Skillscape, skills that are directly connected by a line share similar aspects. The length of the line corresponds to how similar skills are (i.e., skills connected by a shorter line are more similar; skills connected by a longer line are less similar). Skills that are at the center of the network and have more connections with other skills are more useful. Those skills transfer well to other skills. We performed community detection using Louvain clustering [X] on the Skillscape graph to identify clusters of similar skills. Those are the four clusters shown in the figure below. That is: Skillscape visibly reveals a hierarchy of four levels of complementary skills corresponding to Novice, Intermediate, Advanced, and Expert-levels. Those four clusters were identified purely based on data and are not manually assigned by us. The graph clearly reveals a hierarchical structure: most Novice skills are connected with other Novice skills. A few Novice skills are connected with the next hierarchy level (Intermediate), but links even further up the hierarchy (e.g., to Advanced skills) are rare or entirely absent. This suggests that in climbing, it is very difficult to progress in huge leaps without filling in all the intermediate steps as well. This is a natural insight that most climbers and coaches probably share. Overall, skills follow a pattern from less complex, less physical movement patterns in the bottom left corner to more complex and more physical movement patterns in the top right.

One interesting thing to note about the Advanced cluster: the Advanced cluster is much less dense and lines are much longer than in the other clusters. This suggests that transitioning through the set of Advanced skills is more difficult, requiring more time / training / effort than transitioning through the other clusters. This corresponds with the experience of many climbers that transitioning from V0 to V4 feels easier than transitioning from V4 to V6.

Validation of Skillscape Technology

We use advanced machine learning, specifically neural networks [X], as the basis of the virtual coaching that Skillscape provides in the Tension Climbing App. Skillscape is a data- and AI-driven framework that provides adaptive (i.e., dynamic over time), fully personalized training for climbing. This technology enables the Tension Training App to recommend individualized exercise programs—act as a virtual coach—based on the match between dynamically changing profiles of the climber and the demands of a climb. For example, we use this technology to create personalized “Limit Boulders” and “Volume Boulders” recommendations. The climbs recommended to you in those lists are chosen from the entire database in a way such that these climbs should suit your style. That is, these boulders should work well to train power and power endurance without encountering many technical skill challenges relative to the user’s individual profile. We performed three tests to validate the algorithm we have developed. 

First, we test if the fit between those climbs and the climber’s training profile (based on past climbs logged in the app) is predictive of the number of attempts the climber’s recorded requiring to complete a climb successfully. Number of attempts is a natural measure of how difficult a climb feels. We find that the match between a climber’s skill profile and the climb has significant predictive power of the number of attempts that are required to complete the climb successfully ( ρ = 0.78 [p < 0.001] ). That is, if the algorithm predicts that a climb should suit a given climber, the climber will be able to do the climb in fewer attempts (on average), than if the algorithm predicts that a climb would not suit a climber’s skill profile. [Note: in this test, number of attempts is not used as input during the training of the machine learning model.] 

Second, we test if the algorithm is able to pick up on meaningful attributes of climbs. For example, climbs that are predicted to be similar should be of similar difficulty. Below we show a plot of a 2-dimensional UMAP projection [X] of climbs clustered together by similarity and color code each climb by its consensus difficulty rating. As you can see, climbs of similar difficulty cluster together very nicely. This suggests that when the algorithm identifies climbs as being similar, they are also of similar difficulty. [Note: climb difficulty is not used to train the machine learning model.]

Third, we conducted a small study in which we measured the heart rate of climbers while climbing different climbs on the Tension Board. To test whether the machine learning model predicts meaningful physiological responses, we collected heart rate data (HRmax and HRmean) during the completion of n = 64 climbs, from 4 participants across 6 sessions. Below is example data from one of those sessions of one athlete. Heart rate was recorded using an Apple Watch Series 4 and validated in one session using secondary measures from a Polar H7 chest heart rate sensor. 

Within individual and workout we find that climbs that are farther away from a participant’s training profile result in significantly higher HRmean (hierarchical linear model with workout-level fixed effects; standardized coefficient = 3.40 [p = 0.006]). That is: a climb that suits a climber’s profile less (one standard deviation difference) will result in 3.4 beats per minute higher average heart rate (HRmean). This preliminary data suggests that the data-driven machine learning framework we have developed captures relevant physiological responses of what it means if a climb “feels” easy or hard.

In summary, predictions by our machine learning framework correspond to important physiological responses (heart rate), perceived difficulty (number of required attempts), and can reliably group climbs of similar difficulty.

Skillscape Helps You Diversify Your Climbing and Improve Faster

You have probably heard that climbers have a tendency to fall into a rut and do the same thing over and over again, which can lead to training plateaus. This is a central theme in Dave MacLeod’s book “9 out of 10 Climbers Make the Same Mistakes” [X] (Chapter 1 “Creature of habit”), the Power Company Podcast [X] (e.g., episode on over learning), and Lattice [X]. We designed the Skillscape system to help you learn new skills and increase the difficulty of skills you are already familiar with. We use some of the early data to check if Skillscape helps climbers achieve this goal. 

We perform a basic analysis to answer whether the climbs recommended by Skillscape are actually more diverse than the climbs climbers would otherwise do. Just because recommending diverse climbs was a goal during algorithm design, doesn’t necessarily mean the algorithm succeeds in accomplishing this. We use a measure called the Gini coefficient [X] to answer the question of whether Skillscape helps climbers diversify their training programs. The Gini coefficient is a measure of inequality: a Gini value of 1 indicates complete inequality and everyone completes the same climb; a Gini value of 0 indicates complete equality and every climb gets an equal number of ascents. Among regular users, a few popular climbs get a lot of ascents and inequality is high (Gini = 0.71). Among Skillscape users, inequality is much lower (Gini = 0.57). That is, Skillscape users distribute their ascents over a much broader range of climbs (rather than concentrating it among a few popular climbs). We can also show this relationship overall climbs (see figure below which shows a Lorenz Curve [X]). You can see that the line for Skillscape users is closer to the diagonal which would represent a perfectly equal distribution (all climbs receive the same number of ascents).

One possible critique of this analysis is that Skillscape users are ex-ante different and that this analysis could suffer from selection bias. Two key points stand out. Maybe Skillscape users (1) use the Tension Board more than non-Skillscape users and simply climbing more leads to more diversity in their ascents; (2) are more interested in systematic training and hence already aware of the importance of climbing not only climbs that suit their strengths. One possible solution to this could be to make a before & after comparison. However, given that many people experience significant changes in their climbing due to the pandemic and the climbing season this wouldn’t necessarily lead to a good apples-to-apples comparison either. Furthermore, Skillscape isn’t the only aspect that changed with the release of the new app and hence patterns in ascents may have changed in other ways as well. We think a more robust analysis that gets at actual causal differences is our second analysis outlined below.

This critique notwithstanding, we have some evidence that following Skillscape recommendations leads to a more diverse “diet” of climbs and more diverse training. 

Do Skillscape-based recommendations–and the increased diversity that comes with them–help you to get better over time? We perform an analysis of the performance improvements over time (learning rate) and test whether this learning rate [X] is significantly different depending on which feature of the App was used to pick this climb. That is, instead of comparing between the cohort of people who use Skillscape and those who don’t, we compare within users: a given user may log some ascents based on the Climb List feature, and others based on Skillscape, Limit Session, Volume Session, and so on. This analysis suggests that climbs recommended by the Skillscape AI help users progress significantly faster than climbs the users select themselves using the Climb List feature. 

We run this analysis in a very detailed way, looking at the contribution to a climber’s growth of each individual ascent, and where in the Tension Training App this ascent originated from. This is relative to the baseline of browsing the climb list which is shown in the figure with 0% improvement. Since this analysis is driven by differences “within a user” it does not suffer from the limitations of the previous analysis that Skillscape users might be ex-ante different. Such an analysis controls for any observed and unobserved differences that may exist between Skillscape users and non-Skillscape users [X]. Note that this analysis also controls for the total experience–total number of ascents including those before Skillscape was launched–that a user has climbing on the Tension Board. This means that this analysis also controls for the fact that the rate of improvement of climbers may be different for climbers who climb more. [Technical note: all regression coefficients are significant at p < 0.005 with the following exceptions: Friends Feed: p = 0.07; Own creation = NS; Previously sent boulder = NS; Logbook of others = NS]

This analysis shows that climbers have a hard time picking climbs that are outside their comfort zone, potentially limiting their overall improvement. Features in the App that give climbers outside input into what climbs to do (the machine learning recommendations for Limit & Volume Session and Skillscape, the curated Classics and Placement playlists, and the Friends Feed) help people improve much faster. Doing climbs you have already done (logbook) and climbs you set yourself actually have a negative effect on progression. This underlines how difficult it is to set hard boulders for yourself. 

Where do we go from here?

Though not a replacement for a human coach, Skillscape effectively bridges the gap between not having a coach at all and paying upwards of $60-$90 per session to work with a qualified coach. As we continue to develop Skillscape and the sophisticated technology behind it, we expect further improvements, utility, and results. The Tension Board is just the beginning. 

Get the App