virtual reality

VIRTUAL REALITY: Enterprise applications of VR prove we are on track for a $200 billion mixed commerce market by 2025

We stand by our position that mixed reality seems to be headed more towards large, enterprise use-cases like city planning, construction, low skilled worker on-site instruction for utilities or manufacturers, and the military. Yet among young consumers, the behavior of buying digital goods in video games, and the associated monetization of content from video games using channels like eSports continues to be a powerful secular trend. Billions of revenue are generated by free games that only sell cosmetic in-game objects. See, as proof points, the fast growth of Twitch users and the $1B+ in revenue Fortnite made from microtransactions. Last week, Facebook doubled down on the former enterprise-centric use case for mixed reality -- announcing its Oculus device-management subscription for enterprise users. The subscription will cost $180 per device per year and promises "a dedicated software suite offering device setup and management tools, enterprise-grade service and support, and a new user experience customized for business use cases" (see here). Evidently, companies deploying mixed reality solutions generally see better customer retention, satisfaction and operating metrics. Take VR surgical training platform OssoVR -- who claim to have witnessed a 230% improvement in performance by surgeons training in VR. Whilst Walmart admits to VR training boosting employee confidence, retention, and overall training test scores by 10-15%. And let's never forget the VR training platform for cooks in fast-food giant KFC's Chicken Mastery program -- the nightmare-sh and BioShock-esque “escape room” replete with narration from an omnipresent, mildly demonic Colonel Sanders. Apart from giving trainees a mild post-traumatic stress disorder, the training platform (on average) reduced instruction time by 60%.

In financial services much of the framework-setting falls to a centralized function, whether that's a Chief Investment Officer creating portfolios or a more decentralised one i.e., branch or advisor office role assisting in the task of consolidating accounts, or discussing mortgage finance options. Yet realistic presence and emotional resonance, via a truly immersive experience, still matter. Facebook Reality Labs, recently announced, that it's working on this -- bringing full-body avatars to its Oculus experience. Will this allow us to emotionally connect with others in a virtual setting or merely remind us that virtual worlds have no place for such complexity? Either way it's important to note that in our latest payments report we estimated the install base for AR/VR active devices to reach 1 billion by 2025, fueling a revenue pool for mixed commerce of $200 billion at the same time. Seemingly, we are on track.


Source: Oculus for Business, KFC Virtual Training Room (Youtube), Facebook Full body VR (via CNN)

GAFA: Snap seeks to colonize the web one Tinder profile at a time

From camera-mounted sunglasses that failed to be welcomed in any social setting, to Bitmoji's creepy cartoon depictions of reality, it seems like Snapchat's parent company - Snap has tried it all to stay relevant. Especially, when Zuckerberg's army of clones - boasting an impressive 1 billion daily story users vs Snap's 186 million, threatens the story-based core social media model of the app,. Well, it seems Snap has yet to be snapped. At its recent partner summit, the social media company announced its launching StoryKit - a plan to allow apps like Tinder the ability to embed Snapchat stories into their app. The incentive being enhanced engagement and security for the partnering app, whilst Snap additionally benefit from the data they gather from users using their native camera. Quite the colonization strategy you might say. Then again, with the day-to-day data privacy-exploitation headlines streaming from Facebook HQ, it's a no-brainer that advertisers, content creators, and businesses alike are looking to alternatives such as Snap to save them from being victims in the Facebook apocalypse. 


Source: Statista (Social Media Stories) (Storywars)

2019 FINTECH PREDICTION: Government and Enterprise Platforming, led by AI and Mixed Reality

Source: Images from Pexels,     2019 Keystone Predictions Deck

Source: Images from Pexels, 2019 Keystone Predictions Deck

Over the last decade, consumer tech has undergone a cycle of platform building, user aggregation, data mining, and value extraction, resulting in GAFA monopolies. Exhaustion with Facebook and the adjacent issues of privacy and radicalization, in our view, will lead to problems building new splintered consumer attention platforms for AI, AR/VR and other new media ground up.  This implies that consumer platforms based on new technologies will be much more long-tail oriented, serving niche markets with very strong fit. Communities may be passionate, but smaller.

Enterprise tech lags retail adoption by, give or take, 5 years. Similar platforming has not fully penetrated on the enterprise side -- Salesforce is not yet the AI monopoly we should all fear, and Open Banking is barely a fizzle. Therefore, we expect increasing data transparency, aggregation and monetization to occur in enterprise underwritten by venture capital investors. As an example, augmented reality adoption and economics will be driven primarily by municipalities, utilities, large industrial manufacturers, and the military. Similarly, artificial intelligence at scale (and its meeker cousin Robotic Process Automation) are to be directed largely at the workflows and manufacturing processes of large corporates. Dont' get us wrong -- consumer AI is extremely important -- but within Financial Services, the scope for this in the corporate world is even larger.

The corollary is that the pricing pressure that started in consumer Fintech -- roboadvice (150 bps to 25 bps) or in remittance (600 bps to 10 bps) -- will spill over into B2B banking, money movement, insurance, treasury management and product manufacturing. An inevitable outcome is pressure on profit margins as prices equilibriate. For those companies that are able to re-design operations using a digital chassis, they will be able to compete on the margin with Fintech unicorns. Those that are not should exit, or retreat into more bespoke, relationship-driven business lines. 


Here's what we said would matter in the past year year:

Let’s go out on a limb, with that limb being a 3D rendered object in virtual reality. We think there’s a storm brewing in digital goods spilling out into our real world (think Crypto Kitties), and physical goods becoming virtual (think Ikea). Machine vision combined with Whole Foods, Amazon’s augmented reality app, and the iPhone X signals to us that a new type of commerce is emerging. Symptoms like the dominance of eSports and the popularity of sponsored SnapChat filters will only increase, and lead to new purchasing and payments experiences. Financial companies will miss this completely.

How did we do? Not so great with the timing of the theme. While we continue to think that augmented reality, machine vision and edge computing will be combined by Amazon, Alibaba and other retail tech giants into digital shopping experiences in a physical space, this certainly has not happened yet. Tests for a cashier-less shopping experience are happening, as is the gradual but certain adoption of mixed reality on iPhones and Android devices, but we have not seen a consumer tipping point. The $125 million funding of Trax by Warburg Pincus is a start.

If anything, mixed reality seems to be headed more towards large, enterprise use-cases like city planning, construction, low skilled worker on-site instruction for utilities or manufacturers, and the military. However among young consumers, the behavior of buying digital goods in video games, and the associated monetization of content from video games using channels like eSports continues to be a powerful secular trend. Billion of revenue are generated by free games that only sell cosmetic in-game objects. See as proof points the fast growth of Twitch users and the $1B+ in revenue Fortnite made from microtransactions. In addition to being trained to value imaginary objects, they are also being trained to use virtual currency issued by brands.


Source: 2018 Keystone Predictions Deck, Trax via Bloomberg, Twitch data via SuperData/Nielsen

VIRTUAL REALITY: Headset shipments grow to almost 2 million per quarter

Simple, feel good news. IDC updated its mixed reality headset shipment tracker, and people are buying more devices. Relative to a year ago, sales went up by about 10%, and global shipments are nearly 2 million per quarter. Headsets that don't have a screen (like Samsung's Gear VR or Google cardboard) are becoming less popular, while intrerest in standalone headsets that come with a screen and a processor seem to be growing. Facebook's Oculus Go would be an example (as an aside, the thought of Facebook knowing what we look at in a VR environment seems inevitable). The rest -- or about half of all the shipped devices -- are those you plug into a computer or a Playstation.

Dedicated Augmented Reality hardware is doing much worse in the retail market, shipping a couple of dozen thousand for even the best biggest brand. We think this is due to (1) every Apple and Android phone manufactured from now on being an AR device, and (2) folks waiting for Magic Leap and the next gen Microsoft Hololens. Further, as we had explored previously, Microsoft just secured a $480 million HoloLens contract with the Untied States military. It's likely that some of these early technologies will fail to be attention platforms, but succeed at being government or enterprise technology. 

To bring it back into financial services, we recently attended Fintech Connect to moderate a panel on artificial intelligence. There, we came across digital consultancy Softserve, which built a fun prototype for the conference simulating a payment experience using hand gestures within a Magic Leap environment. Payment menus appeared in the view, and a camera that read hand gestures could understand whether you were confirming a transaction. This suggests different ideas -- from building a virtual checkout located physically next to a purchased good with a rendered interface, to the sale of virtual goods in a physical environment. And the folks at Magic Leap are willing to pay developers up to $500,000 per app to fill up its barren app store.


Source: IDC Tracker (release), Verge (US Army contract), Game Daily (Magic Leap), Hollywood Reporter (Infographic)

AUGMENTED REALITY: More productive enterprise use AR for field work

Interesting study for an enterprise usecase of augmented reality -- productivity in the field service management industry. Think about telecom technicians installing cable, mobile doctors or nurses in healthcare, engineers deployed to sites in industry, or other service providers that travel to location. The study found that companies deploying AR solutions -- like (1) remote video and analysis of the site by experts, with work carried out by lower skilled labor, or (2) AR-based on site training modules -- saw better customer retention, satisfaction and operating metrics. Imagine coming to a complex piece of industrial equipment, and seeing three dimensional notes and explanations on how things should work.

There is a potential parallel to financial services front offices. Much of the framework-setting in finance falls to a centralized function, whether that's a Chief Investment Officer creating portfolios or a Chief Data Scientist creating a loan underwriting model. Yet physical presence and emotional resonance still matter. Perhaps the human relationship is managed in person, but relevant financial stats and metrics about the relationship are physically tied or annotated to particular geo locations, or to the emotions of the customer. Clients are less and less likely to come to a branch or advisor office, spending time instead in their phones. But can the front office come to them, with a CIO in their AR glasses? Or, could machine vision be used to assess client net worth? If Truata can scan a windshield to figure out insurance damages, could a machine estimate car or house value in real-time net worth?

While certainly not an exact match, we can see the future in some analogous applications. Below you'll see concepts of some of the apps being developed for headsets like the Magic Leap. For example, AT&T is bringing DirecTV to AR, with up to 4 different channels streaming at once, which is how communication with a colleague can happen. Or, it could be done through a render, like we see in Avatar Chat -- remember that VRChat has millions of users. As a last point, Adobe has entered the game with a professional creative suite in augmented reality, bringing what is currently a novelty toy one step closer to being a platform.


Source: Aberdeen Study; Techcrunch (Adobe), Upload VR (Avatar Chat), Variety (AT&T)

AUGMENTED REALITY: Government and Military Use Will Drive Magic Leap, HoloLens Adoption.

Last week, we spent a bunch of time talking about how consumer VR as a standalone platform is not turning out to be as good as iTunes, the iPhone, YouTube or the Web. One problem was the form factor, another problem was the lack of pirated content -- though games and adult content will slowly address this. This week, we want to point to IoT (Internet of Things) and Augmented Reality (AR). Do these themes have a reason for being and are they an opportunity for a major retooling of our interaction with technology? Here, we think the answer is a stronger Yes. But this is due to a surprising reason -- government and military use.

The Web was popularized through consumer use and now powers our digital selves. But it was brought to life and initial use as ARPANET in the 1960s through funding by the US Department of Defense. Imagine unlimited funding with life and death use cases by a nationally-embedded client base. This is also what the Chinese government is doing in relation to AI, blockchain and quantum computing, and get to the meat. First, Bloomberg reported that AR companies Magic Leap and Microsoft's HoloLens are bidding on a $500 million augmented reality Army project. The order is for 100,000 headsets which would run the Integrated Visual Augmentation System, overlaying intelligence on the physical world. These would be used for both training as well as in live combat. The manufacture of these types of devices would create an economic base on which consumer versions could be created, as well as condition a whole generation that using AR headsets is normal.

Another data point supporting this idea is the investment by local government entities (e.g., UK councils) in digital twins of their neighborhoods for urban planning. In particular, Liverpool is running a £3.5 million IoT program that combines the rollout of a 5G network with innovative health and social care services for residents. Of the 11 proofs of concept in place, examples include video connection between vulnerable people at home and their pharmacy, AR maps that bridge physical distance and combat social isolation, and sensors that monitor whether older adults are dehydrated. Similarly, earlier this year, Bournemouth was mapped into 3D, incorporating 30 different data sets, also as part of planning the 5G network. These live 3D maps, which could then be projected into the real world via AR devices, are a social good and should be part of centralized infrastructure. This in turn can further move the needle in consumer adoption and market maturity.


Source: Magic Leap (BloombergNext RealityDaily Mail), UK Authority (LiverpoolBournemouth), Wikipedia (ARPANET

VIRTUAL REALITY: VR Failing as Platform as Headset Shipments Stall.

Virtual Reality isn't working out as a platform. Below, you'll see the declining popularity scores of the major headsets being sold on Amazon, as scraped by Thinknum. Samsung's Gear headset slipped from the top 100 after November 2016, Facebook's Oculus by July 2018, HTC Vive in June of 2018 and Sony Playstation in April of 2018. IDC also just updated their headset tracker and shipments are down across device types. Headsets that insert smartphones (e.g., Samsung, or Google Cardboard) have declined from 1 million in 2Q 2017 to less than 500,000 a year later. Tethered headsets (those that connect to PCs and devices) fell 37% year over year.

The only category that did grow is standalone headsets, which is a completely independent device for which you need no other peripherals. Oculus Go and Xiaomi Mi VR are examples. And while there might be several dozen million devices installed out there, the technology is not showing the hardware growth of the iPhone, or the software growth of a Whatsapp. So what can we take away from this stalling? First, VR is meant to be a platform war for tech companies, including a dedicated app store, video and gaming content, and the other benefits of owning the customer. However, to be a platform, you need to capture an audience. And to capture an audience, you need to have spectacular differentiated design and proprietary content which forces adoption. The design part is still in wild flux, looking for a form factor people like. 

On content -- today's platforms bootstrapped themselves off existing media. Youtube started out as a place to park bootleg movies. iTunes grew on being able to play pirated music and sync it to your iPod. Facebook gave a home to college kids' digital photos. VR does not have this luxury, because the making of VR content remains far outside the capabilities of the average user. Nor is there a large library of interesting things to digitize. So the bootstrapping is much more difficult, because not only do you have to create all the content from scratch, but you also have to teach your clients about a new experience, which suggests the audience is not built in. That's not to say that the payoff for winning VR is small, but in the short term it is primarily a novelty gaming system.


Source: Thinknum (VR ranks), IDC (Shipments)

PAYMENTS: Instagram Payments = Tip of AR/VR Iceberg

Source: Instagram

Source: Instagram

Social media and tech companies have been adding native payments into their apps for years, and nothing in the West has yet come to resemble WeChat's payments success. Venmo has the "texting money" behavior on lock, despite competing with a native feature of iOS. Banks are still in the growth mode for Zelle, a bank-pipe for messaging money. Facebook payments, and Amazon Pay, and the credit card networks' Buy Now button are all competing for our payments share of wallet (!) on the web.

So why do we think Instagram adding payments to its app is interesting? For the same reason that we think Snapchat adding a store can be compelling. Instagram is a way for many Millennials to consume brands and lifestyle. It is a platform of creators and influencers that broker fashion and retail. Platforms (like Amazon) are unlike individual products in that they can flex to include many different products and services once the use-case is proven. And influencer marketing, powered by propaganda bots, AI, and other growth hacking, is only becoming more important as a trend for generations that grew up on YouTube and eSports. This will become true in time for financial services companies.

Source: Instagram

Source: Instagram

The other side of this coin is Augmented Reality and AI. Using machine vision, Facebook / Instagram can extract products from images posted on its network. Through integrated native payments, Instagram can become the platform where commerce happens, rather than linking out to third party sites. On top of this, Facebook's Oculus Go is an upcoming $200 virtual-reality head set meant to bring VR to the masses. Building Instagram into VR and AR, similar to how Snapchat is experimenting with both the medium and its associated hardware, would allow the company to open up a new commerce category. If this category catches on, Facebook / Instagram will have a meaningful moat that even Google and Amazon cannot match, because it will have (1) the social graph that can drive commerce, (2) the AI talent to build real-time image and product recognition, and (3) the customer's device to interact with this platform. 

Source: Oculus

Source: Oculus

VIRTUAL REALITY: 12 Million AR/VR Devices via Pop Culture

Despite all the talk of mixed reality hardware leading us towards augmented commerce, it still feels like nobody has a AR/VR headset. Will this be an actual platform shift, like mobile phones, or a dud like 3D films? First, the numbers. Last year, about 8 million headsets shipped to consumers, with 12 million expected in 2018. These include a variety of quite different devices — screenless viewers into which you plug in a phone (good for 360 video, but laggy), stand-alone headsets (VR rendering hardware and software in a single package), tethered headsets (plugged into a desktop for rendering horsepower). We are also on the verge of seeing more augmented reality devices, like Magic Leap and HoloLens, that have semi transparent lenses and render virtual objects in the real world, as well as wireless headsets for full VR.

The developer ecosystem is also moving well along. Google has released its developer kit a while ago, turning Android devices into AR units. It now has 85 apps, of which several enable commerce. See EbayOverstockWayfairIKEA, and the Food Network. Google is also opening up its Maps API to help catalyze the development of location based AR apps (think PokémonGo). Microsoft’s HoloLens has done the same in 2016, targeting industrial applications, like architecture and construction. And Magic Leap is opening up its hardware for developers now, promising a high end augmented reality experience — the least they could do after over $2 billion in private funding. And in the crypto world, projects like Bubbled* are exploring augmented reality land titling, to keep vagrants trying to catch some rendered critter out of your backyard.

It’s hard to catalyze a change in human behavior. If you do it, and then own some dimension of the ecosystem along which you can take economic rents (e.g., hardware or capital or data), the outcome is a multi-billion dollar honeypot. Thus HTC, Facebook, Google, PlayStation and others are all throwing billions into the sacrificial fire. But getting people to change how they pay for things, or what currency they use, or what data they share is immensely hard. 

Which is why, we think, there’s the beginnings of a media content wave that’s meant to normalize mixed reality hardware. See for example the blockbuster film “Ready Player One”, where the main character’s life is dreary in the real world, but full of potential in the virtual one. Or the teen show called “Kiss Me First”, where the main characters struggle with social media, identity and the requisite drama in part through adventure in a virtual world. If iconic cultural experiences tell us that mixed reality is normal and here to stay, well you get it. You might not care, but your kids will tell you to buy it.

VIRTUAL REALITY: Getting Used to Mixed Reality

Virtual reality is still missing its killer app, though VRChat is showing some real potential with 3 million downloads and 7,000 daily users. The app is an open environment where users can render both their world and their avatars. Think about a rudimentary version of Ready Player One that looks like Second Life. The app has had success for three reasons: (1) user generated content and thus endless variety, (2) the ability to use it even without VR on a regular desktop computer, and (3) video streaming of the app on popular video site Twitch, with nearly as many people watching the the virtual world as are actually in it. 

Source:  Steam / RoadtoVR

VR games are essentially behavioral training for augmented reality commerce. If users build and value objects and experiences in a virtual world, users will value them when overlaid on the physical world. Think about how video games from the 1980s and 90s became the blueprint for gamified mobile interfaces in the 2010s (see Mary Meeker's thesis on this here, pp 103-155). And we already see this happening. One sign is the planned entry of Magic Leap into retail. Another sign is fashion brand Chanel investing into tech company Farfetch (which had already raised $400 million from Asian fintech Chanel is explicitly not interested in distributing through mass online retail, but are moving towards creating highly personalized augmented reality shopping experiences. 

Weaving together some crypto projects in the space can also help us see ahead. AR glasses manufacturer Lucyd has partnered with Gaze Coin, so that interactions with objects rendered in AR can be monetized. Similarly, there's a partnership with gaming network Gizer and algorithmic advertiser Combined with a rights-management overlay like Bubbled, you get a coherent integration of services that replicate digital advertising and commerce infrastructure in the physical world. Because in reality, we search for things not by typing or speaking, but by looking.

Source: Lucyd

Source: Lucyd

VIRTUAL REALITY: Misunderstanding Augmented Reality as a Gimmick

Source: Google, Amazon (smart mirror), Lucyd Lens

Source: Google, Amazon (smart mirror), Lucyd Lens

Financial services companies are failing to engage with virtual and augmented reality (VR/AR) in a meaningful way. But that does not mean AR is not moving to the mainstream. The latest symptom of this is Google's commitment to building AR into the web browser. A user will be able to come across an object on the web, which can then be taken out and rendered within the living environment of the user. Three dimensional objects within the web browser so far have not taken off due to bandwidth and processing limitations, but that will change over the coming years. See of example, Punk Office, a company that is using 3D scanning of people's bodies to help clothing retailers map products onto their bodies. It is a small jump from this to Amazon's smart mirror, which can then connect the digital avatar to its retail catalog. All this physical / digital melding powered by machine vision of course.

The way financial companies have engaged with the medium so far is to make small games or media experiences. For example, Ally Bank created a game to catch dollars flying around in your living room during the Super Bowl. Citi and Wells, among others, have created VR experiences (think concerts, immersive videos) that do nothing but advertise in a next-gen medium. The Financial Brand wrote a great post on the ways banks are using this tech last year, and the examples are roughly consistent: (1) various marketing experiences, (2) virtual offices and branches, (3) touring real estate investments, and (4) taking 2D trading and wealth interfaces and rendering them in 3D. 

This is fundamentally wrong. Sprinkling 3D on a complicated financial product, like trading, or sending people into a weird interaction with your virtual banking branch does not make the banking experience simpler or better for the customer. A simple mobile app or chatbot will do. What we need is not financial products rendered in gimmicky 3D, but to add finance features to digital objects and their attributes. This is why Augmented Reality retail makes sense. People will make some purchases natively in AR, and new payments experiences will support this activity. Similarly, this is why the VR crypto economy can make sense. From the financing of virtual land, to the monetization of attention, to the overlay of financial actions over the physical world in smart devices, incumbents need to think broader about this opportunity.

Source: Google, Amazon (smart mirror), Lucyd Lens

Source: Google, Amazon (smart mirror), Lucyd Lens

Source:Amazon (smart mirror)

Source:Amazon (smart mirror)

VIRTUAL REALITY: Google and Amazon Augmenting Humans

Source: Magic Leap

Source: Magic Leap

Here is your futurist palette cleanser. At the world's largest consumer electronics show, augmented reality and virtual assistants dominated this year. It's like the tech companies are living in a completely different world than the rest of us.

First the data points on virtual reality. A pair of augmented reality glasses called Vuzix Blade are attempting to do again what Google Glass and Snap's Spectacles failed to do, i.e., matter at all. But, this time *may* be different. Augmented Reality is more mature, is plugged into various operating systems, and has developers building cool apps. Vuzix is working on an interface layer on top of the real world, which could theoretically power a payment experience. Another example is Magic Leap, a developer of a mixed reality headset that raised $500 million last October, and finally revealed its Lightwear product end of last year. The developer kit is out there, and one of the advertised uses is AR commerce (see the promotional image attached). And as a last data point, HTC Vive now has a wireless adapter. So you no longer need to have wires sticking out of your head to use VR -- just beam it over please!

Did we mention that the Vuzix Blade comes built in with Amazon Alexa? So expect to see people talking to their smart glasses, which solves a major user interface challenge for something you cannot touch. Google is heavily invested into this trend as well. One of the biggest platform battles today is around bring artificially intelligent assistants into the mainstream, a battle that Amazon and Google have only begun. To that end, Google is embedding its AI assistant into cars (Android Auto) and smart displays that will likely be embedded into refrigerators and other willing appliances. So finally, we will be able to talk about our finances to the car, and ask the toaster about investments.

But let's generalize just a bit more. What we are seeing is the rise of the augmented human. Whereas computing previously lived mostly in our sense of touch (keyboard, phone), it is now moving to our sight (AR/VR) and our speech (assistants). Yes, everyone looks like silly cyborgs in the stock photos, but understand that this is just the beginning and we are in the awkward teenager phase of such augmentation. Further, as our senses are digitized, their function will be tokenized and used as a medium of exchange. Projects like Brave and Gazecoin point the way.

Source: Vuzix, Magic Leap, Google

Source: Vuzix, Magic Leap, Google



We were awestruck by two projects. The first allows sketches of objects to become rendered images using generative neural networks. We've shared similar versions of this idea -- from Google's open source library of 3D rendered models to 3D gestures that map onto a space of virtual objects -- but this particular application shows how simple it is to go from concept to realistic (ish) environment. Yes, it's still ugly and messy, but for how long?  

The second project does an even more impressive trick. It takes the visual environments rendered in the 3D bubbles (or "360 video") of Google Maps and generates background sound for the environment. Note that this isn't the actual recorded sound, but a neural network hallucinated auditory experience that is correlated to the image mathematically. Listen to the video for full effect.

The melding of physical and digital spaces requires steps like this to become scalable and repeatable. We believe that once this type of technology is polished around the edges, augmented reality experiences and commerce will become profound.

PAYMENTS: Alibaba's Virtual Reality Commerce

Source: CNN, Alibaba

Source: CNN, Alibaba

 If you want to look at retail and Fintech innovation, no better place to learn from than China. In a consumerism celebration called "Singles Day", Alibaba just sold $25 billion worth of merchandise. Last year, the company premiered a platform called Buy+ that rendered a mall in virtual reality and allowed customers to purchase items digitally. This year, the firm reportedly created an augmented reality game like Pokemon Go, where the users interacted with digital characters that led to physical shopping. Cute apps like that are already available in US app stores as well.

The Buy+ VR platform did not reappear this year, however. And despite the hype about the medium as an empathy machine and digital overlay on the physical world, there are still very high barriers to its adoption. See this account of a journalist trying to spend a day working in Microsoft's mixed reality operating system. The experience is clunky, physically uncomfortable, disorienting and tiring. Our current computer and mobile equipment is far better optimized for tasks, while mixed reality is still in its "toy" phase. Or see this view that even the entertainment VR industry will fail, because the expectations gap is too strong between the current hardware and marketing slogans. This stuff is not ready yet.

But, we know that companies like Alibaba and Tencent have some of the largest user bases and data sets in the world. Even in Fintech, payments subsidiary Ant Financial has 450 million users, more than the entire population of the United States. While Amazon's efforts in mixed reality are just beginning, understanding where the more experimental Asian tech giants are going in this space will be an early indicator of what is possible.

PAYMENTS: Amazon's Augmented Reality Commerce

It seems insufficient to talk about augmented reality, and much more powerful to show it, so enjoy the images below. We are seeing the first iterations of thinking about AR commerce taking shape, with Amazon joining Ikea in deploying a product Preview (not Review) app that takes a rendered version of an Amazon product and places it in the shopper's home. One image below shows a chair being placed into a carpeted room. Other examples in the linked video show vases, kitchenware, and Amazon Alexa being projected around the house.

Commerce matters for Fintech because of payments and financing -- how will we pay for things in this future? Will we still use the iPhone or something like Magic Leap? The other two images show the power of artificial intelligence when it intersects with AR. In the first image, a neural network is used to "transfer" Van Gogh's artistic style on a rendered portrait. Think about how this could be applied to a customized bank experience, creating environments targeting the appropriate demographic with the right aesthetic. Popping gradients and futuristic designs for Millennials, marble and mahogany for Boomers?

The last image uses a neural network interface to build objects in a virtual world. You can see the user creating a house by drawing an icon of a house and placing it on the ground. The AI part comes into play mapping the house drawing, which would be different each time a user creates it, and the rendered object that appears. You can think of this as a freeform gesture that beckons a product. If AR is built into all operating systems and mapped to commerce via Amazon, anything you want is just a flicker of the hand away.  Already, Google is building out the rendered library.

Source: Amazon

Source: Amazon

Source:  Sam Snider-Held

INSURANCE: Insurtech Virtual Reality vs Machine Learning

Source: Farmers, Fortune, Tractable

Source: Farmers, Fortune, Tractable

For something even more futuristic, check out this news of Farmers Insurance using virtual reality to train its property claims representatives for 500 scenarios of damages and customer interactions. The course is about 15 minutes long, rendered in video game engine Unity with randomly generated layouts, can be watched by other reps and managers on a big screen, and leads to a performance assessment on completion. 

This year, 50 reps will be trained using the simulation, growing to hundreds or thousands in the years to come. You can see the hyper-realistic rendering in the linked image. Other enterprise examples of using VR to simulate human experiences and create learning outcomes have popped up in medicine, such as surgical training, and education

What's curious, however, is that we are using machine simulations to enable human learning. At the same time, we are using real world imagery to facilitate machine learning, for essentially the same job. See for example this article, which describes how 70% of auto damage claims could be analyzed and estimated by machine vision by insurtech Tractable. Race between the AIs and the transhumanists in on!

VIRTUAL REALITY: Apple's Augmented Reality and $1 Billion for HTC

AR/VR is such a divisive topic. 50% of you will think what we are saying is obvious and inevitable. 50% of you will think that we are crazy and on the moon. In a recent conversation with a portfolio manager of a multi-billion dollar fund, we heard "Maybe in 10 years my kids will shop for things in Augmented Reality". And yet, here we are, today, with Apple's ARkit powering live apps that you can download that deliver the future without much fanfare.

This is a case of show, not tell. In the pictures below you can see (1) how Ikea is selling furniture in a live AR app on iOS today, (2) how a neural network can be trained to extract a 3D reconstruction of a face from a single photo image, and (3) a virtual reality space built for meetings and social networking by VR Chat, which just raised $4mm from HTC. Can you connect the dots to see that within just a few years the line between the physical world and the rendered one will be gone, and that commerce, payments, bank branches, and virtual assistants will all live in a world access to which will be tolled by Apple, Google, Amazon or Facebook (or Alibaba/Tencent)?

If these data points are not enough, consider that Google is about to spend $1.1 billion on a slice of HTC, the phone manufacturer. Why? To own the capability to make hardware that competes in the Artificial Intelligence / Augmented Reality age. In order to deliver these futuristic capabilities, chips have to be architected in way that optimizes for neural networks (see NVIDIA and Intel) and massive graphic rendering, and cameras need to support volumetric recording and facial recognition. That is a hardware game. One in which financial companies will be price takers, not price setters.

Source:  VR Chat

Source: VR Chat

VIRTUAL REALITY: BNP Paribas Jumps into VR

Source: BNP Paribas

Source: BNP Paribas

While there are reports of millions of virtual and augmented reality hardware units being shipped by Google, Vive, and Sony, most mainstream consumers have not yet adopted the technology. And certainly, few people are asking for financial services to intersect with virtual or augmented reality. But that is about to change, as technology firms push to make VR/AR ubiquitous, jump-starting another platform battle. Developments like Pokemon-Go and Snapchat Spectacles have given us a preview of the future, but now it's time to apply these ideas in finance.

BNP Paribas is taking the VR bet, and has shared several concepts in a new video showing potential client experiences. Examples include (1) real estate brokerage apartment viewing, (2) bank transaction initiation in VR, and (3) partnerships with festivals and venues focused on the technology. The announcement comes on the heels of a payments demo by Worldpay and efforts by bothCiti and Wells.   

While nobody knows how this theme will develop, it is likely that commerce will continue to move from web to mobile, from mobile to augmented reality, and from augmented reality to virtual reality. Systems built by virtual world developers will need to interact with payments and banking APIs as the world evolves around us.

PAYMENTS: Worldpay Demos Virtual Reality Payments

Source: VentureBeat, World Pay

Source: VentureBeat, World Pay

We are experiencing a platform shift -- perhaps not as quick as the introduction of the mobile phone, but certainly equally as fundamental in the longer term to how people behave and communicate. Media, retail and financial services intersect in this new paradigm in ways that are fairly distinct from our current world. When companies build virtualenvironments and goods, and they are treated as physicalenvironments and goods, will a bank be the right way to store value to purchase them? Or would it be the the system that renders that virtual world? If you think this is ridiculous, remember that more people now watch blockbuster eSports finals than the NBA finals.

Commerce in this augmented or virtual reality needs payments, and Worldpay developed a prototype for a virtual shopping experience. The user can choose either contactless payment, where a transaction below GBP 30.00 needs nothing other than selecting your virtual card, or to enter their pin to authenticate. It's skeuomorphic design -- remember when the iPhone rendered all of its notepads and buttons as if they were physical objects? Similarly, this experience literally mirrors having a physical credit card with a 3D-rendered credit card, to be swiped at a 3D point of sale terminal. Weird. 

Maybe this approach will help get people comfortable with the idea of using virtual cards in virtual worlds, or maybe it will give value to virtual goods more intuitively. But it's also very odd. When a payment can happen in an instant through biometric authentication in your device (eyes anyone?), why would we create the inconvenience of handling a physical object? See for a counterexample how Ant Financial will allow people to pay in virtual reality by merely nodding their head. Which experience has less friction?