RealityCapture

Virtual Puzzle Game Puzzling Places

Puzzling Places is a VR jigsaw puzzle game created with photogrammetry RealityCapture scans where players puzzle, piece by piece, hyper-realistic miniatures from around the world. The core concept and gameplay is quite simple which makes it familiar and extremely fun for people of all ages. Coming soon to Oculus Quest and PlayStation VR!

Variety of models in Puzzling Places by Alexandre González Rivas, Gerpho3D, & Azad B.

Azad Balabanian from Realities.io , creators of Puzzling Placesshares the story behind Puzzling Places, his knowledge and plans for the future.

Tell us more about the Puzzling Places and where the idea came from.

The purpose of the project was to create a go to app when people want a calm and meditative VR game to play, an alternative to all the shooters and horror games out there. Everybody knows how to puzzle, which makes Puzzling Places a game that can be played by anyone. It’s also a fantastic way to see, hear, and learn about new places from around the world in a fun and engaging way, to playfully interact with scanned content in VR and AR.

The origin story of the game is a result of a total accident, a true "bug turned into a feature" moment! One of the steps of our Photogrammetry optimizations is to separate large Photogrammetry scans into equally-sized "chunks" to support rendering optimizations like frustum and occlusion culling.

Level of difficulty in Puzzling Places. Scan by poiformat

One day in 2018, while importing one of our processed Photogrammetry worlds into a game engine, the chunks of the scans appeared out of order due to an error in the importer, and were jumbled up across the scene. Shahriar Shahrabi, the Graphics Programmer/Game Designer on our team joked, "what if we ship it like this and have people just puzzle the pieces back together instead?". And so, the idea was born! Over the next week or so, Shahriar developed a SteamVR prototype to play and evaluate the idea. Lo and behold, it ended up being extremely fun!

More about the Technical Site:

Although the concept of the game is simple, technically, it is anything but!

Assuming most people reading the RealityCapture blog are aware of the basics of Photogrammetry, we won't reiterate the introductory details. Our Photogrammetry datasets are typically a combination of terrestrial DSLR's (Sony A7rii's) and aerial drones (DJI Mavic 2 Pros, DJI Inspire) processed with RealityCapture. Occasionally, we also deploy LIDAR for certain locations that Photogrammetry would not be suited for.

We often capture locations with a specific lighting in mind which is baked-in in the albedo texture, similar to Photography, which can often result in stunning photorealism that is stunning to see in VR. If the model's texture lacks certain contrast or is somewhat bland, however, we like to bake CG lighting into the albedo texture using Blender which can create very realistic results.

Tsuruga Castle by Fuji


Tsuruga Castle 3D model - top view

It's worth noting that many of the puzzles that we've created recently were in fact captured years ago with all the RAW data stored and archived. One of the wonderful things about raw imagery and Photogrammetry is the ability to re-process datasets with new tools and know-how to achieve better results! Some of the more recent updates from RealityCapture such as "color correction" have been great at producing more consistent texture blends in complicated Photogrammetry projects, both in color as well as exposure.

Apart from using our own scans, puzzles for Puzzling Places are also made with wonderful scans from some of our Photogrammetry partners, including Gerpho3D, who capture stunning locations around France from their plane. As most of the world is still inaccessible due to travel restrictions, we're focusing heavily on creating partnerships with Photogrammetry professionals who capture interesting locations we can turn into puzzles.

More about Workflow:

Photogrammetry

  1. Capture photos
  2. Processing RAW photos to JPEG
  3. Processing Photogrammetry with RealityCapture (align, mesh reconstruction, texture)
  4. Import mesh to Mesh Mixer or Blender for mesh fixing
  5. Fix any texture issues in Blender (this becomes the “master mesh” of the puzzle scan)

Tatev Monastery by Realities.io

Puzzle Processing

  1. Crop the border of master mesh to create an interesting, finished, puzzle with clean edges using Blender.
  2. Render up to 5 different viewpoints of the puzzle to include in the game as helpful hints to the players using Blender.
  3. Process the master mesh with our Puzzle-making pipeline, which simplifies the mesh, segments it into however many puzzle pieces that we’d like (for each puzzle, we make 50, 100, 200, and 300 piece variants), unwraps the pieces, and bakes them; We do quite a bit of optimizations here to reduce the amount of draw calls in-game.
  4. Import the results into Unreal Engine and to set up a “puzzle scene”.
  5. Create and add audio regions to the puzzle, which start to play once a player has finished puzzling various regions of the puzzle; Each puzzle ends up having its own customized audio environment.
  6. Finalize the puzzle and playtest!

Plans for the future:

Our plan with Puzzling Places is to launch the game in Fall and continue developing new puzzles and gameplay features for years to come.

Mont St. Michel 3D model by Gerpho3D

We'd love to talk with Photogrammetry teams that would be interested in contributing their scans to be turned into puzzles! We're interested in having people pitch us either singular scanned locations or a set of scans centered around a location or theme (such as a "Slovakian Puzzle Pack” featuring many of its fantastic locations).

Market square Gliwice by poiformat

High-quality mesh exports from RealityCapture projects are exactly what we need to turn a scan into a beautiful miniature to puzzle together in VR. If you have a scan you think might make for a good puzzle, please contact us at contact@realities.io. Thank you RealityCapture for creating your fantastic Photogrammetry processing software. We have been frequent users of RC from the very start and are excited to see your future with Epic and Unreal Engine to come!

About Realities.io:

Realities was established in 2016 with the goal of bringing the real world into VR using Photogrammetry. With the onset of consumer PS VR headsets like the Oculus Rift and the HTC Vive, we launched our first app - Realities - which showcased our Photogrammetry scanning, processing, and optimizing for VR. It also became a platform for us to experiment with non-linear storytelling and documentary style VR experiences that go beyond 360 videos (which was the most popular framework for VR films at the time). With it, we created interactive experiences like America Deserta, Cologne Cathedral, Terres d'Exploration, and glückauf!