User Interface Telemetry: Motivation, Goals, Methods
11/2015
Steve Henty
[UPDATE 7/2018: added ReactJS section to Existing Tools]
Outline
Executive Summary
Motivation
Goals
Constraints
Payload
Collection
Analysis
Epilogue: Existing Tools
Executive Summary
The desirable characteristics of a UI Telemetry system for web apps are presented and discussed. The primary motivation is to discover and illuminate UX-related behaviors of the UI and application workflows, as well as performance-related issues around page load and UI rendering. Sales conversion is specifically not a focus, as it is already best-served by existing web analytics and is most appropriate for web sites, while less applicable to web apps.
Proposed metrics fall into three categories,
- Client characteristics
- Workflow characteristics
- UI characteristics
Each category contains some metrics that can be derived directly from the telemetry mechanism itself, and some that are better supplied explicitly by the application as part of instrumentation.
Telemetry payloads are intentionally kept lightweight and general, consisting primarily of event timestamps and select event markers: lightweight to limit the overhead caused by telemetry, and general to allow maximum analytic flexibility without re-instrumenting the UI.
Analysis is focused on post-hoc queries of the collected data to reconstruct time series or histograms that can illuminate the following kinds of inquiries, to cite a few,
- What are the typical paths through the application?
- Where are there lags in activity during a workflow?
- How are workflows handed-off or coordinated among different actors?
- Which UI elements are involved in completing a workflow?
- Which UI elements are rarely used?
- Where do loading or rendering latencies occur?
- Do typical workflow paths vary by client? By project type (OSS vs. comercial)? By license? By release? By time of day/week/month/year?
Motivation
Why measure the UI, and what should be measured?
Sales Conversion
Measuring conversion rates on web sites is probably the most commonly thought of reason for instrumenting web pages. While counting unique visitors and tracking bounce rates can be important for consumer-facing web sites, such metrics are generally less interesting and even misleading for web applications. This document is not focused on Sales Conversion, although it’s important to note the bias of most existing UI telemetry tools is to Sales Conversion. This bias will be referenced as necessary, especially in the Existing Tools section.
Front-end / UI Performance
Front-end performance is another commonly thought of reason for instrumenting some portion of the UI. The typical focus is on load times and latencies at various points in the request-response-render cycle. Performance is one aspect of UI telemetry this document hopes to address, but is not the only, nor even the primary aspect.
UX Understanding
Understanding how the user experience unfolds in a web application isn’t the go-to reason most think of for collecting front-end metrics. However, it is the primary aspect of UI instrumentation this document hopes to elucidate. This is not to say that the other aspects are unimportant or inconsequential. Given the historical focus on Sales Conversion and Front-end / UI Performance, those aspects represent largely solved problems. UX Understanding, on the other hand, likely requires more consideration as to whether existing tools and methods are sufficient, or whether they need to be customized or replaced.
Goals
The overarching goals for UI Telemetry as it is conceived in this document are,
- discover workflow paths (UX)
- discover workflow interruptions (UX)
- discover UI control-surface usage (UX)
- discover multi-actor workflow dynamics (UX)
- discover network latency (Performance)
- discover UI latency (Performance)
It’s important to note the term ‘discover’ in each of the above goals. Metrics in isolation typically do not, and perhaps cannot isolate the actual cause of the measurement. This is especially true for UX-related metrics.
Say we ‘discover’ as a result of workflow path metrics that a typical path through an application is Page1-Page8-Page2-Page3. The expected workflow based on the application design may be something different, say Page1-Page4-Page7-Page3. At this point, the most we can say is the typical path is one we didn’t design. It would take further investigation to determine the reason for the variance, which could be for any number of reasons, to cite just a few,
- the designed path is broken, and the typical path is a workaround
- the designed path is non-obvious, and the typical path is a user discovery
- the designed path is deficient, and the typical path is the workaround
- the typical path represents a previously unknown workflow requirement
- the typical path reflects externalities the designed path doesn’t support (interruptions, data availability, etc.)
In short, honest goals for UI telemetry are focused on discovering appropriate questions to ask, not the answers themselves.
Constraints
The ideal UI telemetry would offer unlimited information for zero increase in overhead. Since this is obviously not achievable in practice, trade-offs will be made. What follows is a discussion of constraints that seek a point in trade-off space between information capacity vs. overhead.
Low network traffic — the telemetry stream should be either sparse over time, or consist of small payloads, or both.
Low server load — the telemetry stream should be tuned for fast handling at the collection end-point, probably focused on caching the raw data quickly, and deferring more complex handling, collating, analysis to an offline process.
Instrument Once, Query Forever— any instrumentation required should either be automatic, and/or sufficiently general in order to avoid subsequent effort to re-instrument the code in response to new performance or UX investigations.
Custom Analysis — the resulting data stream should be accessible for post-hoc, custom analysis regardless of the mechanism used to collect and store the data.
No endpoint lock-in — neither the telemetry data format, nor the collection mechanisms at the client or server should dictate the use of a specific storage or analysis endpoint. Endpoints should be swappable or simultaneous according to current needs.
Payload
The following specification of telemetry payload seeks to limit the payload to the barest essentials to keep it lightweight and general while allowing maximum ‘minability’ during post-hoc analysis. Certain data is expected to come ‘for free’ as part of typical UI telemetry offerings, such as Google Analytics, derived from the telemetry mechanism itself. Other data is peculiar to the application, and would be better sent as payload data explicitly.
Get Steve Henty’s stories in your inbox
Join Medium for free to get updates from this writer.
Note: Any of these data points can / should be anonymized for privacy without losing distinctness for workflow traceability
Client characteristics
Telemetry-derived
- user agent (browser)
- OS
- geocode
- session uniqueness (IP, MAC, etc.)
- etc…
App-specific
- user id
- organization id
- license model
- app version
Workflow characteristics
Telemetry-derived
- delay between page requests
- event timestamp
App-specific
- Entity ID — whatever sufficiently identifies the entity being acted upon during the workflow. This becomes especially important for mapping workflow collaboration among actors who might be from different organizations.
UI characteristics
Telemetry-derived
- page requested
- request round-trip latency
App-specific
- UI element type
- UI element id
- render latency
Collection
For performance reasons, processing at the collection endpoint probably proceeds in different phases: online and offline. I don’t claim to have deep knowledge in this area, but the following is a scenario described to me by a systems architect.
Online
This phase just needs to capture the incoming telemetry as efficiently as possible. If using an external analytics service, like Google Analytics, this phase is already handled. Otherwise, for a custom endpoint some kind of fast, in-memory buffer is probably sufficient. A ring buffer would allow for continual collection of telemetry without overflowing the storage system. It is then up to the second Offline phase to collect the buffer contents to avoid losing data. Sizing the Online capacity is a function of storage cost vs. desired backlog of data upon activating the Offline phase.
Offline
This phase pulls data from the Online phase as needed. If no telemetry data is needed, the offline phase can simply go dormant and avoid incurring storage and processing overhead for data that will not be used. Any time telemetry data is needed, the Offline phase can be activated, and start requesting data from the Online phase, storing it in any appropriate format necessary for post-hoc analysis.
Analysis
Given the payload definitions above — essentially a stream of timestamped events with associated event markers — then two kinds of data can be reconstructed with quite a bit of flexibility.
Time-series
Stringing the events together in series, filtered by various combinations of event markers, can produce these kinds of inquiries,
UX
- Typical paths through the application — by client organization, time of day/week/month/year, project type, license, etc.
- Unexpected / non-designed workflows
- Workflow timing / interruptions
- UI control surfaces used to advance workflow
Performance
- Load latencies
- Render latencies
Histogram
Counting the events, filtered by various combinations of event markers and time-periods, can produce these kinds of inquiries,
UX
- UI element usage frequency — by client organization, etc.
- Feature usage — client organization, etc.
Performance
- Which resources incur latencies — by time of day/week/month/year, etc.
Epilogue: Existing Tools
This is an [obviously] non-exhaustive survey of UI telemetry tools. They are simply the tools I have investigated to-date.
Google Analytics
Google Analytics (GA) is ubiquitous, and should be considered as a possible endpoint. The analytics are biased to sales conversion and bounce rates, which can be misleading when applied to a web application. To cite one example, a workflow interruption in a web application could be evidenced by getting partway through an expected path followed by a long, over-threshold delay, or an unexpected page request. GA’s native analytics would likely characterize this sequence as a ‘bounce’ or failed conversion, both of which are nonsensical in the context of a workflow interruption.
This sales bias may be alleviated or overcome by using GA’s ability to define and collect custom metrics, assuming the data can be extracted for analysis by other means.
ReactJS
redux-saga
While not a purpose-built telemetry framework, the use of sagas is obvious for managing the telemetry stream back to the collection source.
redux-state-history
While not a purpose-built telemetry framework, the state change history has obvious use in creating ‘paths through the application’, especially if a reducer-enhancer is also used to inject telemetry-related metadata into the state.
AngularJS
Angulartics
Angulartics is an AngularJS $http service interceptor that can send telemetry data to any number of endpoints simultaneously, including the big analytics engines like Google Analytics. Custom endpoints can be accommodated as well by writing a custom service plugin. Instrumentation is handled directly in the HTML markup via Angular directives.
Angulartics is a very capable, lightweight, customizable measurement system for Angular.
