From camera-mounted sunglasses that failed to be welcomed in any social setting, to Bitmoji's creepy cartoon depictions of reality, it seems like Snapchat's parent company - Snap has tried it all to stay relevant. Especially, when Zuckerberg's army of clones - boasting an impressive 1 billion daily story users vs Snap's 186 million, threatens the story-based core social media model of the app,. Well, it seems Snap has yet to be snapped. At its recent partner summit, the social media company announced its launching StoryKit - a plan to allow apps like Tinder the ability to embed Snapchat stories into their app. The incentive being enhanced engagement and security for the partnering app, whilst Snap additionally benefit from the data they gather from users using their native camera. Quite the colonization strategy you might say. Then again, with the day-to-day data privacy-exploitation headlines streaming from Facebook HQ, it's a no-brainer that advertisers, content creators, and businesses alike are looking to alternatives such as Snap to save them from being victims in the Facebook apocalypse.
The digitization of the human animal continues unopposed, with symptoms all over. Chinese firm Megvii, maker of software Face++ that has catalyzed 5,000 arrests since 2016 by the Ministry of Public Security, is looking for an $800 million IPO. The other champion of public/private surveillance, Facebook, is working a virtual reality angle. The company is improving the technology used to model rendered avatars of human faces, which can then be displayed across virtual environments. Using multi-camera rigs and hours of facial movement footage, Facebook is building neural networks that learn how to translate realistic facial muscle movement into models. The Wired article linked below is worth exploring for the videos alone, and the uncannily realistic motion these animation possess.
One of our recurring points is that frontier technologies -- AI, AR/VR, blockchain, and IoT -- appear disparate now, but are intricately connected. Take for example the new feature from Google called YouTube Stories. Similar to SnapChat and Instagram, video creators can apply 3D augmented reality overlays to their faces. While this technology looks like virtual reality rendering, it is primarily a machine vision (i.e., AI) problem to anchor rendered objects to a human face realistically. To do this Google provides a developer library called ARCore, not to be confused with Apple's ARkit. Human video avatars can be further extended and customized with code -- the twenty first century version of personal branding.
Another take on the same issue comes from generative adversarial neural networks (GANs). We've discussed before how hyper-realistic images and videos can be faked by a model where one algorithm creates images and another accepts or rejects them as sufficiently realistic, with repeated evolutionary turns at this problem. Highlighted below is a recent software release from NVIDIA, where a drawing of simple shapes and lines is rendered by a GAN into what appears to be a hyper-realistic photo of a landscape. We can imagine a similar approach being applied to the output generated by Facebook's avatars, which still border on creepy, to ground the outcome in reality. Little details, like a reflection of a cloud on water, are hallucinated by GANs automatically, based on massive underlying visual data. Expect these digital worlds to become increasingly indistinguishable from reality, and to spend way more time living in them for the years to come.
First you take a traditional physical industry, and make it digital. Walmart turns to Amazon. Taxis turn to Ubers. Next, you take the digital environment -- online shopping, expense management software, maps and navigation -- and re-instantiate it back into the physical world. This is how you get weird results like augmented commerce, where retail locations of physical stuff grow augmented reality overlays to create omni-channel data tracking for a company's AI. Take for example Walgreens rolling out Cooler Screens digital windows for its shopping venues. The monitors replace fridge doors, displaying products in an idealized state, with (potentially dynamic) digital prices prominently designed. You are interacting with an app, or maybe a website, on a door behind which lies the ice-cream you want to buy. Let's repeat that. A website is in front of you, an ice-cream is an inch behind the website, the website watches you with cameras, records your reactions, advertises things at you, and sends everything to the cloud. Enjoy your online in-store experience!
Or let's take transportation. There are the digital upstarts, arbitraging a phone's GPS to deliver mobility with greater precision than a human transaction. From Waymo, Ofo, Lyft, Uber and Lime littering our phones with icons of summonable critters, to manufacturers like Citroen creating mobile-app connected vehicles like the Ami One, transport is mobile and on-demand. So what's the next meta game? Check out CityMapper, a mere-mortal mapping application focused on beating Google and Apple at giving directions for city travel. The app is not original, but well executed. It charts out public, private and pedestrian modes of getting from here to there with time estimates, and does so locally on a device, which means no internet connection required. After acquiring a userbase for aggregated directions, they are now launching aggregated transportation through a subscription offer called Pass. This physical card costs £30 per week, and includes public transportation, bikes, and ride-sharing, with loyalty points on top. Here is an instantiated financial products that sits on top of abstracted digital infrastructure.
Another Silicon Valley favorite is fintech start-up Brex. It provides a corporate credit card for small business, which consolidates spending and expenses across the entire organization and leverages existing corporate spending behavior to offer higher credit limits. It's never been easier to give WeWork employees their own spending account, and track just how much Starbucks they drink. The interesting thing about Brex isn't that it's a card -- banks know how to issue credit to businesses, despite what the startup may tell you. The interesting thing is that the expense management software for the business owner is the primary proposition (we think), leveraging modern data aggregation into expense management and credit permissioning. The accounting industry got digitized (e.g., Wave and Quicken), and now is instantiating itself back in our physical world through a smart card and financial product. This opportunity to bridge software into the physical world with finance, and payments in particular, is an area we are are thrilled to see develop further.
Over the last decade, consumer tech has undergone a cycle of platform building, user aggregation, data mining, and value extraction, resulting in GAFA monopolies. Exhaustion with Facebook and the adjacent issues of privacy and radicalization, in our view, will lead to problems building new splintered consumer attention platforms for AI, AR/VR and other new media ground up. This implies that consumer platforms based on new technologies will be much more long-tail oriented, serving niche markets with very strong fit. Communities may be passionate, but smaller.
Enterprise tech lags retail adoption by, give or take, 5 years. Similar platforming has not fully penetrated on the enterprise side -- Salesforce is not yet the AI monopoly we should all fear, and Open Banking is barely a fizzle. Therefore, we expect increasing data transparency, aggregation and monetization to occur in enterprise underwritten by venture capital investors. As an example, augmented reality adoption and economics will be driven primarily by municipalities, utilities, large industrial manufacturers, and the military. Similarly, artificial intelligence at scale (and its meeker cousin Robotic Process Automation) are to be directed largely at the workflows and manufacturing processes of large corporates. Dont' get us wrong -- consumer AI is extremely important -- but within Financial Services, the scope for this in the corporate world is even larger.
The corollary is that the pricing pressure that started in consumer Fintech -- roboadvice (150 bps to 25 bps) or in remittance (600 bps to 10 bps) -- will spill over into B2B banking, money movement, insurance, treasury management and product manufacturing. An inevitable outcome is pressure on profit margins as prices equilibriate. For those companies that are able to re-design operations using a digital chassis, they will be able to compete on the margin with Fintech unicorns. Those that are not should exit, or retreat into more bespoke, relationship-driven business lines.
Simple, feel good news. IDC updated its mixed reality headset shipment tracker, and people are buying more devices. Relative to a year ago, sales went up by about 10%, and global shipments are nearly 2 million per quarter. Headsets that don't have a screen (like Samsung's Gear VR or Google cardboard) are becoming less popular, while intrerest in standalone headsets that come with a screen and a processor seem to be growing. Facebook's Oculus Go would be an example (as an aside, the thought of Facebook knowing what we look at in a VR environment seems inevitable). The rest -- or about half of all the shipped devices -- are those you plug into a computer or a Playstation.
Dedicated Augmented Reality hardware is doing much worse in the retail market, shipping a couple of dozen thousand for even the best biggest brand. We think this is due to (1) every Apple and Android phone manufactured from now on being an AR device, and (2) folks waiting for Magic Leap and the next gen Microsoft Hololens. Further, as we had explored previously, Microsoft just secured a $480 million HoloLens contract with the Untied States military. It's likely that some of these early technologies will fail to be attention platforms, but succeed at being government or enterprise technology.
To bring it back into financial services, we recently attended Fintech Connect to moderate a panel on artificial intelligence. There, we came across digital consultancy Softserve, which built a fun prototype for the conference simulating a payment experience using hand gestures within a Magic Leap environment. Payment menus appeared in the view, and a camera that read hand gestures could understand whether you were confirming a transaction. This suggests different ideas -- from building a virtual checkout located physically next to a purchased good with a rendered interface, to the sale of virtual goods in a physical environment. And the folks at Magic Leap are willing to pay developers up to $500,000 per app to fill up its barren app store.
Let's paint the progression. Finn.ai, a chatbot company that integrates banking services into Facebook Messenger and other chat channels, launches Bolt for the Bank of Montreal. The project took 10 months to private label and deploy to BMO's 12 million customers, covering 250 questions at launch. As the app gathers more information about what customers ask, its usefulness grows and it becomes an increasingly relevant channel for customers to ask their financial institution informational questions.
By the end of 2017, 40 million smart speakers were installed worldwide, with 2018 projected to land at a 100 million install base. People are getting more than one instantiated smart assistant -- littering their home with several Echo dots. And, reportedly, Alexa lives in 3,000 others types of smart-home devices -- giving this bot army 45,000 skills, from Spotify to financial conversations. Google and Apple are working to catch up to these numbers, releasing eerily realistic robo-conversationalists like Google Duplex that can answer telemarketing calls and book hair salon appointments. And maybe cancel your financial subscriptions. What's perhaps most important is that an answer to a voice query is 40-80% fulfilled by the "featured snippet" at the top of a search engine's list, according to Martech. That means no more long tail of any kind, full stop.
Eventually, it is no longer enough for our avatars to be disembodied functions powered by a retail product recommendation engine. From virtual worlds and into augmented reality, agents will take on hyper-realistic rendered physical bodies. A recent Intel whitepaper describes how machine learning is now being combined with a modeling of animal and human bodies under a physics simulator to quickly and realistically build CGI for games and movies. Instead of an artist using intuition to draw the perfect frame, machines build skeletons with physical properties, connect bones with digital ligaments, fire up virtual muscles, and package the final version in the species of choice. Machine learning algorithms add realistic, generalized movement to the equation. They can also add speech, appearance and function. And maybe even help the UBS chief economist look a little bit more lifelike!
Interesting study for an enterprise usecase of augmented reality -- productivity in the field service management industry. Think about telecom technicians installing cable, mobile doctors or nurses in healthcare, engineers deployed to sites in industry, or other service providers that travel to location. The study found that companies deploying AR solutions -- like (1) remote video and analysis of the site by experts, with work carried out by lower skilled labor, or (2) AR-based on site training modules -- saw better customer retention, satisfaction and operating metrics. Imagine coming to a complex piece of industrial equipment, and seeing three dimensional notes and explanations on how things should work.
There is a potential parallel to financial services front offices. Much of the framework-setting in finance falls to a centralized function, whether that's a Chief Investment Officer creating portfolios or a Chief Data Scientist creating a loan underwriting model. Yet physical presence and emotional resonance still matter. Perhaps the human relationship is managed in person, but relevant financial stats and metrics about the relationship are physically tied or annotated to particular geo locations, or to the emotions of the customer. Clients are less and less likely to come to a branch or advisor office, spending time instead in their phones. But can the front office come to them, with a CIO in their AR glasses? Or, could machine vision be used to assess client net worth? If Truata can scan a windshield to figure out insurance damages, could a machine estimate car or house value in real-time net worth?
While certainly not an exact match, we can see the future in some analogous applications. Below you'll see concepts of some of the apps being developed for headsets like the Magic Leap. For example, AT&T is bringing DirecTV to AR, with up to 4 different channels streaming at once, which is how communication with a colleague can happen. Or, it could be done through a render, like we see in Avatar Chat -- remember that VRChat has millions of users. As a last point, Adobe has entered the game with a professional creative suite in augmented reality, bringing what is currently a novelty toy one step closer to being a platform.
Last week, we spent a bunch of time talking about how consumer VR as a standalone platform is not turning out to be as good as iTunes, the iPhone, YouTube or the Web. One problem was the form factor, another problem was the lack of pirated content -- though games and adult content will slowly address this. This week, we want to point to IoT (Internet of Things) and Augmented Reality (AR). Do these themes have a reason for being and are they an opportunity for a major retooling of our interaction with technology? Here, we think the answer is a stronger Yes. But this is due to a surprising reason -- government and military use.
The Web was popularized through consumer use and now powers our digital selves. But it was brought to life and initial use as ARPANET in the 1960s through funding by the US Department of Defense. Imagine unlimited funding with life and death use cases by a nationally-embedded client base. This is also what the Chinese government is doing in relation to AI, blockchain and quantum computing, and get to the meat. First, Bloomberg reported that AR companies Magic Leap and Microsoft's HoloLens are bidding on a $500 million augmented reality Army project. The order is for 100,000 headsets which would run the Integrated Visual Augmentation System, overlaying intelligence on the physical world. These would be used for both training as well as in live combat. The manufacture of these types of devices would create an economic base on which consumer versions could be created, as well as condition a whole generation that using AR headsets is normal.
Another data point supporting this idea is the investment by local government entities (e.g., UK councils) in digital twins of their neighborhoods for urban planning. In particular, Liverpool is running a £3.5 million IoT program that combines the rollout of a 5G network with innovative health and social care services for residents. Of the 11 proofs of concept in place, examples include video connection between vulnerable people at home and their pharmacy, AR maps that bridge physical distance and combat social isolation, and sensors that monitor whether older adults are dehydrated. Similarly, earlier this year, Bournemouth was mapped into 3D, incorporating 30 different data sets, also as part of planning the 5G network. These live 3D maps, which could then be projected into the real world via AR devices, are a social good and should be part of centralized infrastructure. This in turn can further move the needle in consumer adoption and market maturity.
Augmented commerce continues to develop as a theme in two directions: (1) digitizing the physical world through machine vision and (2) embodying the digital world through Augmented Reality. On the former, a few examples are Bossa Nova Robotics inventory robots deployed in 50 Walmarts, AI guardsman using machine learning on video feeds to spot shoplifters, and Amazon harnessing computer vision to bypass the need for checkouts in its stores.
As another example, Singapore-based Trax just raised $125 million (in addition to a previously raised $140 million) to do exactly that. The company owns a powerful image recognition engine for various retail products, which allows retailers to create digital versions of its physical shelf-space. At first, pictures of the shelves were taken by employees with phone or tablet devices. But this can be done with cameras or robots today. Downstream from the image recognition are data and analytics, which managers can use to run the business in real time, across all locations. Just like e-commerce, which provides data that is instantaneous and real-time, the physical environment can become similarly measured and optimized.
Another version of this wizardry is Baidu’s vending machine concept (see video link below). The user interacts with a voice interface, which projects a menu onto the glass window in the front of the machine. This interface was also touch sensitive, so a user could select the good to buy and confirm by tapping the screen, with payment enabled through facial recognition. While cool, this process was still inefficient compared to pressing a few buttons on the side of the machine. But, over time, we expect to see such smart interfaces spill out further into the world. Which leads us back to finance. Is there a version of a bank branch that benefits from such technology?
Let's keep it light-hearted. Here are things we did not know about Pokemon Go, the augmented reality game. It has seen over 800 million downloads, and generated over $1.2 billion in revenue since inception. Niantic, the firm that makes the game, has raised $200 million to build a Harry Potter version of the same. And the game has just run through a partnership with several cities including Akron, Ohio; Charlotte, North Carolina; Philadelphia, Pennsylvania; and San Jose, California. See the link below in the Sources for more detail -- we found the study fascinating.
Niantic and the cities created joint events that drove people outdoors and on adventures through physical communities by sprinkling scarce digital objects along the way. Like Pacman eating dots or bunnies following a trail of nibbles, people were guided along a video game narrative to participate in battles and events. In San Jose, 35,000 people showed up and spent $450k; in Philly, 10,000 had a more bespoke experience along historic landmarks. In some cases, players could modify the game in progress. What is also notable is that the most effective way to get community engagement was not to send invites to individual players, but to networks, which then self-organized and came together. It's a version of migrating a web-based forum or community to the physical world.
So what, you say? Ok: (1) 22 million virtual and augmented reality headsets will be sold this year -- still a far cry from Whatsapp adoption, but laying the groundwork for mass adoption nonetheless, (2) software platforms like Metaverse let any person or SME program their own AR apps, which can be epxerienced on the major mobile platforms, (3) blockchain companies have laid down the groundwork for scarce digital goods owned and maintained by a community, (4) bank retail branches and physical shopping are both facing pressure from the migration online. Massively multiplayer augmented reality games could be used to bootstrap crypto economies, financial services engagement, or new types of commerce. Or at least, to watch the World Cup on your tabletop.
Under our augmented commerce theme, we believe that mixed reality, AI and other digital enhancements will leave the browser and become anchored in the physical world through hardware and software pushed by companies like Apple and Amazon. Such a transition will be as transformative to retail commerce as the web and the browser were to e-commerce. The United States is showing glimmers of this through incremental symptoms. For example, Walmart and its competitors are implementing an IBM blockchain solution called Food Trust, which acts as a ledger for each food item across the entire supply chain, and cuts down identification time from 6 days to 2.2 seconds. Or look at DHL employing augmented reality headsets to help employees identify required packages in a warehouse.
But Chinese tech companies are many steps ahead. A great Axios article focused our attention on an Alibaba initiative called "New Retail". A mom and pop store can pay $6,000 for a digital renovation, and a $620 annual membership fee which locks the store into the Alibaba ecosystem. The digital tools include a heat sensor to track foot traffic, an AI-backed app and the Alipay payments system, and the entire Alibaba delivery and fulfillment infrastructure. The store becomes a physical endpoint for Alibaba's e-commerce platform. JD and Tencent are doing the same.
A chain called RT-Mart updated 400 of its stores in this manner. Here are the bells and whistles: (1) one-hour arrival on e-commerce orders in a 3km radius to the physical store location, (2) orders through a branded app are fulfilled by inventory from local physical stores and carried by conveyor belts to packing/delivery areas, or all the way to a customer's home, (3) in-store physical e-commerce kiosks with payments using QR codes, and (4) red envelope coupons that are gifted within Alipay can be redeemed for physical goods. Will customers adopt these solutions? Well, 500 million already have the needed app on their phone and are trained in digital payments behavior. Maybe China will save the American mall! And as we've been saying about Amazon, technology and finance are mere enabling features of the commerce happening in the system.
There is no greater gift to geeks like us than Mary Meeker's Internet Trends report. If you haven't seen the 2018 version yet, what are you waiting for? Time to read 300 slides in 30 minutes. The key takeaway we remember from last year is the consumerization of the enterprise, and that much of the user interfaces of today's mobile and web apps are directly inspired by 1980s video games. Not surprising, since many of the people building companies today and making design decisions grew up together with the maturation of the video game industry. This year's report has a great section on China and sovereign investment on edge technologies, but lets leave that rabbit hole for another time since we covered that last week.
In this entry, we want to highlight takeaways around payments. The story that comes through the deck is that digital commerce and digital payments are intertwined, and both are rising. E-commerce has grown to $450 billion per year, up from $180 billion in 2010. As share of total retail sales, E-commerce has doubled from 6% to 13%. In terms of everyday transactions, consumers claim to have only paid for items in retail stores 40% of the time, with digital channels comprising the other 60%. This category lumps together everything from P2P payments like Venmo (7%), to money movement through messenger apps (7%! Really!?), to shopping through smart home devices (3%). While this survey sentiment doesn't yet line up with actual retail dollars, it's fascinating to see consumers using so many emerging payment mechanisms.
So we are experiencing this broad digitization of commerce, with E-commerce living in the web and in our mobile apps. But the world is on the verge of another shift, which is augmenting the physical space with embedded digital commerce. Meeker highlights retail stores that use traffic heatmaps to optimize layout, and smart devices in shoes that measure in-store sales conversion (from fitting to purchase). We add the technology of Angus.ai, which is similar to what is likely under the hood at Amazon's stores, and at some point will be at Whole Foods. A machine vision algorithm can watch product stocks, measure sales in real time, and send workers to replenish them. Physical is becoming digital, which we believe have a deep implication on the types of payments companies that succeed. Not point of sale, but point of intent.
If you're PayPal, maybe it's frustrating to see people talking about the future of money and not mean you. But it would be a mistake to count them out of the game. The company has been expanding expertly across the Fintech ecosystem, with the latest data point being a $2.2 billion acquisition of iZettle. iZettle is a European version of point-of-sale dongle manufacturer Square, processes $6 billion in payment volume and accepts Google/Apple pay. The success of this model is a consequence of the adoption of mobile phones and tablets -- meaning that smart devices are cheap, everywhere, and can accept payments. Thus the (slow) power shift from dedicated hardware, to payments as a feature within tech ecosystems.
Two directions to think about. First, payments is a great entry into a broader fintech business model. When looking at Chinese messaging company WeChat, we see the world's largest messenger user base, tech payments app, and money market fund. Associated with such a messenger is a large data set of conversation and commerce -- which powers lending and investing. The same playbook could happen in the West. For PayPal, such logic would drive the partnership with micro-investing app Acorns and the acquisition of social payments app Venmo (and developer focused Braintree). Lending is the next logical step -- for analogy, look at Square's merchant lending business or Goldman's success with Marcus and its expansion into Europe.
The second direction is augmented commerce. As AR/VR penetrate the world with devices and change how people shop, we think it will be important to own a hardware asset that can adapt to the opportunity. iZettle gives PayPal an option for success in the next wave of tech transformation. Of course, it will be complicated to rationalize capability, especially in relation to Google and Apple pay. Looking at the Venmo acquisition, PayPal is now stripping out its web functionality and limiting Venmo to the mobile app -- there is no sense in having two separate web standards. And on retail commerce, there is still the question of crypto. Companies like Basepay (support Ethereum as currency at 11 million retail location) and Revolut (prepaid card that can index to crypto and be swiped at point of sale) are building a bridge that PayPal is yet to cross.
Social media and tech companies have been adding native payments into their apps for years, and nothing in the West has yet come to resemble WeChat's payments success. Venmo has the "texting money" behavior on lock, despite competing with a native feature of iOS. Banks are still in the growth mode for Zelle, a bank-pipe for messaging money. Facebook payments, and Amazon Pay, and the credit card networks' Buy Now button are all competing for our payments share of wallet (!) on the web.
So why do we think Instagram adding payments to its app is interesting? For the same reason that we think Snapchat adding a store can be compelling. Instagram is a way for many Millennials to consume brands and lifestyle. It is a platform of creators and influencers that broker fashion and retail. Platforms (like Amazon) are unlike individual products in that they can flex to include many different products and services once the use-case is proven. And influencer marketing, powered by propaganda bots, AI, and other growth hacking, is only becoming more important as a trend for generations that grew up on YouTube and eSports. This will become true in time for financial services companies.
The other side of this coin is Augmented Reality and AI. Using machine vision, Facebook / Instagram can extract products from images posted on its network. Through integrated native payments, Instagram can become the platform where commerce happens, rather than linking out to third party sites. On top of this, Facebook's Oculus Go is an upcoming $200 virtual-reality head set meant to bring VR to the masses. Building Instagram into VR and AR, similar to how Snapchat is experimenting with both the medium and its associated hardware, would allow the company to open up a new commerce category. If this category catches on, Facebook / Instagram will have a meaningful moat that even Google and Amazon cannot match, because it will have (1) the social graph that can drive commerce, (2) the AI talent to build real-time image and product recognition, and (3) the customer's device to interact with this platform.
One of our key early trends for 2018 is the shifting nature of retail commerce. The themes of augmented reality, mobile device adoption, and artificial intelligence are all nipping away at how people make purchasing decisions, and how they implement those decisions in the real world. Further, as tech retailers like Amazon.com get into banking, money and life get meshed together at the expense of niche financial product manufacturers (like regional banks or financial advisors).
Back in 2012, British grocery store giant Tesco used South Korea as a test bed for a virtual grocery store in subway stations and bus stops in downtown Seoul. Images of grocery-stocked shelves were printed onto walls allowing for consumers to use their mobile devices to scan the QR-code attached to each item, checkout via the mobile app, pay, and have their items delivered to their doors all while waiting for transportation. The product/market fit is between consumer lifestyle (busy commuter), payment mechanism (QR code enabled digital wallet), and available technology (mobile device).
Missing a platform shift matters. For example, our colleague Craig Maurer at Autonomous highlighted the importance of a new initiative from Visa and Mastercard to create a single online shipping "Buy Button", supported by standards built into the web browser. Had such a technology existed in, say 1998, we may not have seen PayPal grab the branding, technology and business opportunity to become the default payment mechanism for the web. Today, this "Buy Now" space is threatened not only by tech firms and payments companies, but also by coins like Bitcoin and Ripple. Whichever provides the easiest tokenization and comes standard with the browsing experience is likely to win in the long run. So who is doing this best for Augmented Reality? Seriously, write to us and tell us!
What if consumers look at shopping completely differently in ten years? Think of the Starbucks example -- it's not a coffee shop, but a third place. Today, many consumers are using brick-and-mortar locations to browse options, and then later buy those options cheaper online. One interesting solution to this is B8ta: a brick-and-mortar retail presence for new brands that don’t have an offline presence of their own. Consumers go to the store, play around with early products, and provide feedback, which then is sent to the manufacturer who pays for the data. All of a sudden, the physical world is not about product delivery, but about browsing-driven engagement with digital fulfillment. Who owns the "Buy Now" button then?
Despite all the talk of mixed reality hardware leading us towards augmented commerce, it still feels like nobody has a AR/VR headset. Will this be an actual platform shift, like mobile phones, or a dud like 3D films? First, the numbers. Last year, about 8 million headsets shipped to consumers, with 12 million expected in 2018. These include a variety of quite different devices — screenless viewers into which you plug in a phone (good for 360 video, but laggy), stand-alone headsets (VR rendering hardware and software in a single package), tethered headsets (plugged into a desktop for rendering horsepower). We are also on the verge of seeing more augmented reality devices, like Magic Leap and HoloLens, that have semi transparent lenses and render virtual objects in the real world, as well as wireless headsets for full VR.
The developer ecosystem is also moving well along. Google has released its developer kit a while ago, turning Android devices into AR units. It now has 85 apps, of which several enable commerce. See Ebay, Overstock, Wayfair, IKEA, and the Food Network. Google is also opening up its Maps API to help catalyze the development of location based AR apps (think PokémonGo). Microsoft’s HoloLens has done the same in 2016, targeting industrial applications, like architecture and construction. And Magic Leap is opening up its hardware for developers now, promising a high end augmented reality experience — the least they could do after over $2 billion in private funding. And in the crypto world, projects like Bubbled* are exploring augmented reality land titling, to keep vagrants trying to catch some rendered critter out of your backyard.
It’s hard to catalyze a change in human behavior. If you do it, and then own some dimension of the ecosystem along which you can take economic rents (e.g., hardware or capital or data), the outcome is a multi-billion dollar honeypot. Thus HTC, Facebook, Google, PlayStation and others are all throwing billions into the sacrificial fire. But getting people to change how they pay for things, or what currency they use, or what data they share is immensely hard.
Which is why, we think, there’s the beginnings of a media content wave that’s meant to normalize mixed reality hardware. See for example the blockbuster film “Ready Player One”, where the main character’s life is dreary in the real world, but full of potential in the virtual one. Or the teen show called “Kiss Me First”, where the main characters struggle with social media, identity and the requisite drama in part through adventure in a virtual world. If iconic cultural experiences tell us that mixed reality is normal and here to stay, well you get it. You might not care, but your kids will tell you to buy it.
Virtual reality is still missing its killer app, though VRChat is showing some real potential with 3 million downloads and 7,000 daily users. The app is an open environment where users can render both their world and their avatars. Think about a rudimentary version of Ready Player One that looks like Second Life. The app has had success for three reasons: (1) user generated content and thus endless variety, (2) the ability to use it even without VR on a regular desktop computer, and (3) video streaming of the app on popular video site Twitch, with nearly as many people watching the the virtual world as are actually in it.
VR games are essentially behavioral training for augmented reality commerce. If users build and value objects and experiences in a virtual world, users will value them when overlaid on the physical world. Think about how video games from the 1980s and 90s became the blueprint for gamified mobile interfaces in the 2010s (see Mary Meeker's thesis on this here, pp 103-155). And we already see this happening. One sign is the planned entry of Magic Leap into retail. Another sign is fashion brand Chanel investing into tech company Farfetch (which had already raised $400 million from Asian fintech JD.com). Chanel is explicitly not interested in distributing through mass online retail, but are moving towards creating highly personalized augmented reality shopping experiences.
Weaving together some crypto projects in the space can also help us see ahead. AR glasses manufacturer Lucyd has partnered with Gaze Coin, so that interactions with objects rendered in AR can be monetized. Similarly, there's a partnership with gaming network Gizer and algorithmic advertiser Advir.co. Combined with a rights-management overlay like Bubbled, you get a coherent integration of services that replicate digital advertising and commerce infrastructure in the physical world. Because in reality, we search for things not by typing or speaking, but by looking.
Financial services companies are failing to engage with virtual and augmented reality (VR/AR) in a meaningful way. But that does not mean AR is not moving to the mainstream. The latest symptom of this is Google's commitment to building AR into the web browser. A user will be able to come across an object on the web, which can then be taken out and rendered within the living environment of the user. Three dimensional objects within the web browser so far have not taken off due to bandwidth and processing limitations, but that will change over the coming years. See of example, Punk Office, a company that is using 3D scanning of people's bodies to help clothing retailers map products onto their bodies. It is a small jump from this to Amazon's smart mirror, which can then connect the digital avatar to its retail catalog. All this physical / digital melding powered by machine vision of course.
The way financial companies have engaged with the medium so far is to make small games or media experiences. For example, Ally Bank created a game to catch dollars flying around in your living room during the Super Bowl. Citi and Wells, among others, have created VR experiences (think concerts, immersive videos) that do nothing but advertise in a next-gen medium. The Financial Brand wrote a great post on the ways banks are using this tech last year, and the examples are roughly consistent: (1) various marketing experiences, (2) virtual offices and branches, (3) touring real estate investments, and (4) taking 2D trading and wealth interfaces and rendering them in 3D.
This is fundamentally wrong. Sprinkling 3D on a complicated financial product, like trading, or sending people into a weird interaction with your virtual banking branch does not make the banking experience simpler or better for the customer. A simple mobile app or chatbot will do. What we need is not financial products rendered in gimmicky 3D, but to add finance features to digital objects and their attributes. This is why Augmented Reality retail makes sense. People will make some purchases natively in AR, and new payments experiences will support this activity. Similarly, this is why the VR crypto economy can make sense. From the financing of virtual land, to the monetization of attention, to the overlay of financial actions over the physical world in smart devices, incumbents need to think broader about this opportunity.
Here is your futurist palette cleanser. At the world's largest consumer electronics show, augmented reality and virtual assistants dominated this year. It's like the tech companies are living in a completely different world than the rest of us.
First the data points on virtual reality. A pair of augmented reality glasses called Vuzix Blade are attempting to do again what Google Glass and Snap's Spectacles failed to do, i.e., matter at all. But, this time *may* be different. Augmented Reality is more mature, is plugged into various operating systems, and has developers building cool apps. Vuzix is working on an interface layer on top of the real world, which could theoretically power a payment experience. Another example is Magic Leap, a developer of a mixed reality headset that raised $500 million last October, and finally revealed its Lightwear product end of last year. The developer kit is out there, and one of the advertised uses is AR commerce (see the promotional image attached). And as a last data point, HTC Vive now has a wireless adapter. So you no longer need to have wires sticking out of your head to use VR -- just beam it over please!
Did we mention that the Vuzix Blade comes built in with Amazon Alexa? So expect to see people talking to their smart glasses, which solves a major user interface challenge for something you cannot touch. Google is heavily invested into this trend as well. One of the biggest platform battles today is around bring artificially intelligent assistants into the mainstream, a battle that Amazon and Google have only begun. To that end, Google is embedding its AI assistant into cars (Android Auto) and smart displays that will likely be embedded into refrigerators and other willing appliances. So finally, we will be able to talk about our finances to the car, and ask the toaster about investments.
But let's generalize just a bit more. What we are seeing is the rise of the augmented human. Whereas computing previously lived mostly in our sense of touch (keyboard, phone), it is now moving to our sight (AR/VR) and our speech (assistants). Yes, everyone looks like silly cyborgs in the stock photos, but understand that this is just the beginning and we are in the awkward teenager phase of such augmentation. Further, as our senses are digitized, their function will be tokenized and used as a medium of exchange. Projects like Brave and Gazecoin point the way.
We were awestruck by two projects. The first allows sketches of objects to become rendered images using generative neural networks. We've shared similar versions of this idea -- from Google's open source library of 3D rendered models to 3D gestures that map onto a space of virtual objects -- but this particular application shows how simple it is to go from concept to realistic (ish) environment. Yes, it's still ugly and messy, but for how long?
The second project does an even more impressive trick. It takes the visual environments rendered in the 3D bubbles (or "360 video") of Google Maps and generates background sound for the environment. Note that this isn't the actual recorded sound, but a neural network hallucinated auditory experience that is correlated to the image mathematically. Listen to the video for full effect.
The melding of physical and digital spaces requires steps like this to become scalable and repeatable. We believe that once this type of technology is polished around the edges, augmented reality experiences and commerce will become profound.