> If I'm to be honest, after compiling and writing this entry I'm actually in a state of disbelief about how much we've managed to get done in such a short space of time.
That is the magic of having and utilising the power of free and open-source software. Imagine not having three.js, QGIS, PostGIS etc.
Gorgeous work and a highly interesting "how we are doing it" insight. Thanks! Please more!
You're entirely right. What I've learnt most about this project is that by utilising and combining free data and software you can quite quickly knock up something amazing. However, free and open stuff doesn't fix performance and all the little detailed issues that took up most of our time this past month.
I'm glad you like it, we'll make sure to write up more as we progress.
> Unfortunately, in our efforts for global domination we quickly discovered the performance limitations of the project as it stood, as well as my personal knowledge with Three.js and WebGL. In short, the expanded version of London ran at a horrible 20–40fps depending on the system you viewed it on.
You got a lot farther than me when I tried Three.js before hitting performance problems. I was trying to make a WebGL 3d warehouse visualization app: http://chir.ag/projects/wh/ and stopped because it was too slow once I loaded 1000 bins. I also never quite figured how to handle rotation/panning nicely (src: http://chir.ag/projects/wh/main.js). I came up with a shorthand http://chir.ag/projects/wh/bins/wh.txt that expands to http://chir.ag/projects/wh/bins/bins.php - I was going to make it an easy to use app where you could describe how the warehouse was organized with 20-30 lines of bin#s and positions and let the system generate the rest. Then it could parse APIs containing various properties per bin (full/empty, inventory amount, frequency of usage, days-since-last-used etc.) Clicking on any bin would open up a panel with more information and you'd have filtering/highlighting capability for the whole warehouse.
I made good progress initially but got busy with life and never got around to solving the too-many-bins problem. I still don't know how to improve the performance. Even when entire objects are outside the view, they still take up resources and time. I see you mentioned LOD support in Three.js - no idea if it would help me or not because each object is already as simple as it can be - just a box.
If someone wants to hack on this and make it functional, I'll help however I can.
Any chance you could put that up / a version of it up on jsFiddle or something so I can edit it online and see how I can help? There's a few techniques that I could play with. 1000 objects should be silky smooth…
Thanks so much! Everyone is welcome to share/extend this. If the 3D part works better, I can make a JS function to convert: 1-WH-[001:011]{A:D}, 30,{0},[20] into:
The data is in the format bin:[x,y,z,sx,sy,sz,color] - the color will obviously be dependent on the data. Position and size are fixed in most warehouse simulations.
Absolutely! Give me a few hours and I'll have it up. Thanks for the offer. Also it ended up being 6000 objects because a box is 6 planes. I wanted to reduce that to only visible planes but because boxes could be partially translucent depending on some property and be selectable when clicked, it would look weird if inside sides were missing.
Cool, I'll keep an eye open for the link. Even 6,000 objects would be fine, especially if they're cuboids. There are demos with 120,000 cubes running at 40fps+, or even 500,000 cubes if you want to use fancy buffer geometry techniques.
Not sure if this is feasible for you, unless you are comfortable with getting your hands dirty, but I decided to forgo the use of a 3rd party library and write the WebGL code myself for my latest project. Other folks mentioned geometry merging, and for me, that has been the single biggest performance booster on my current project. We were hitting a wall with some models that had 30,000 individual objects (several million verts, 10 million indices).
But since we were writing the WebGL ourselves, I just added a bit of code to batch up all the objects into as big a bucket as WebGL could handle (up to 65k verts per draw call), thereby reducing the number of draw calls (drawElements). For our larger models, we went from 1100 ms per frame, down to 1-5ms per frame. We've done other things as well, but none had so large an impact as merging the geometry into larger chunks.
But I can't speak to doing that in Three.js, since I didn't use it. But you should easily be able to handle the geometry you've got at 60 fps.
You should be hired by the uk.gov team to work on this full-time.
This, combined with various statistical information (crime, traffic, other stuff from the ONS) would be a huge boon to anyone using it and the interface makes it beginner friendly.
We'd love to talk to the data.gov.uk guys about working together, though I have no contacts there atm.
We do plan to visualise huge quantities of data, including the datasets you mentioned. Most of what we've been using so far is from the 2011 Census, as well as the Indices of Multiple Deprivation.
Traffic too, we're looking into using the live data from TFL to do some cool stuff.
This is exactly the sort of thing the whole "Show us a better way" site was supposed to encourage. I took a stab at some stuff using various sources but didn't have enough time (Edit: or skill, or access to data sources).
While I appreciate some of the game related stuff this looks most promising for allowing people to easy see statistical information at-a-glance, data journalism etc.
The NapTAN datbase would be an awesome addition that should be relatively easy to add.
Regarding the data.gov.uk team. They do have a twitter account and website, hit them, hit them hard. Even if they can only smooth the way for data access (Like the building heights) that would be awesome.
Thanks for the advice, we'll definitely follow up on it! And I agree, the data side of things is very interesting and is something we're keen to explore as a separate entity to the playful/art aspect.
How did you implement your data path from getting GIS data (shapefiles I presume) to PostGIS then GeoJSON and finally into ThreeJS?
What about combining the statistical data with the ___location?
And the heatmap/choropleth, is that something Three.js can do?
I have no knowledge of developing 3D stuff, but I have been looking into using D3 with canvas rendering just for this kind of visualizations. I'm playing with multidimensional data as stacked 3D cubes. So, getting something like this would really bootstrap my project to another level. Any thoughts about open sourcing?
In all, this looks very useful and expendable. The sky is the limit. Literally, when is sky coming? ;)
> The sky is the limit. Literally, when is sky coming? ;)
Definitely one of my favourite comments about the project so far! =p
The rough and ready approach we took to transforming data was to import the shapefiles into PostGIS using pgShapeLoader, doing any manipulation in PostGIS and exporting straight out to GeoJSON using the ST_ToGeoJSON method in PostGIS, or using the GeoJSON options in ogr2ogr. From there it's a simple case of importing the JSON into Three.js and constructing the objects from the vertex coordinates.
Re: Combining statistical data with the geographic data, we can do that with GeoJSON and you can use ogr2ogr to include related data about each ___location when exporting to GeoJSON.
The choropleth map is something we constructed from scratch. In reality, it's nothing more than a bunch of shapes in Three.js that each have a different material colour – it's very simple.
D3.js is a fantastic place to start. In fact, that's exactly where we started with all this. We actually use the Geo module within D3 for a few parts of ViziCities.
We haven't decided yet about opening the full project but we'll certainly be documenting and releasing aspects of it (most of the juicy bits). We'll announce that via the @ViziCities Twitter account.
There was a WebGL 3D flying simulator on HN few weeks back which used the sky and light changes to great effect. Perhaps something like morning rush hour traffic against evening or rainy days againts sunny.
I'll definetely have to delve more into PostGIS. Three.js should follow when I get a better graps of D3. I think there was an example on how to generate a satellite 3D like map from GeoJSON at Mike Bostocks example site.
I am working on something similar with the prospect of getting CannonJS physics to work so that one can walk the city (in my case procedurally generated): https://twitter.com/dirkk/status/309024330115842049
Very cool, thanks for sharing it! The multi-device thing is something we're actually keen to explore; we basically want to replicate the Liquid Galaxy thing that Google did.
We haven't yet worked out what we're doing re: opening the project and collaboration but we're open to talk more. Mind reaching out to us via [email protected]?
I've been thinking about this a lot lately -- What's most interesting to me is the implications for urban data modeling and prediction when the end user has full control of how the data is used. Remember that article a few weeks back about the fellow who used the new SimCity to model his town's traffic problems? Imagine if you had full control over the model and can tweak the city at any level you want, not just what the game UI allows you to adjust (e.g. feeding in data from a government API)? Urban planners could potentially get a lot of useful information from gamers playing around with variables in their city.
Another interesting idea could be turning a city into a sort of MMORPG where each user has control over a smaller section of the city (instead of the whole thing). If the user only pays attention to optimizing a small part of the city, interesting patterns could emerge in the overall model.
> Another interesting idea could be turning a city into a sort of MMORPG where each user has control over a smaller section of the city [...]
There are indeed a number of ways this could make for an interesting multiplayer experience. I'd particularly prefer it if it was fully P2P and networks would emerge by people offering and making connections. Oh one can dream.
That would certainly be kinda cool. My recent background is in HTML5 games (I used to work at Mozilla) and I'd love take what we've done so far with ViziCities and see how a game would look.
It's somewhat mixed, but there is a good amount of ambient stuff in there as well. (Also note that almost all of it - except for the one remix - is CC licensed.)
The Ordnance Survey doesn't even hold height data on all buildings in the UK. They've started a few isolated projects but nothing as a deliverable dataset yet. I don't think there's a US equivalent despite your link to NYC Open Data.
You might want to look to Germany and CityGML. You might discover some town datasets that are complete in height and geometry detail.
If you happen on a LIDAR data source you've then got to tidy up against the buildings but you will run into errors caused by the time spent between the two data capture points. There's SRTM but i'm sure it'll be too coarse for your needs. I'm sure there's a replacement to SRTM but the name escapes me.
There's LandXplorer - now owned by Autodesk which will generate models from oblique aerial photogrammetry. There's also CityEngine that'll create fake cities based upon road networks.
Thanks for the reply Andy, appreciated. We're actually talking to Ordnance Survey atm so hopefully we can work something out. Building height data for the UK (that's available outside higher education) is a pain to get hold of so I'm hoping we can work something out.
I'll definitely look into the CityGML stuff, it sounds great.
LIDAR is certainly something we're keen to explore further. If anything, it'll get us a much more accurate estimation of height, at least better than an educated guess or via SRTM (which is way to low detail and can't be relied on for buildings).
The CityEngine stuff is amazing. I've not actually played with it myself yet but I've been having a good look at it during my research. Would be cool to have a go on it at some point.
Wow, very interesting, this tickles my fancy... Imagine something like this but with an airfield, you could model the airfield and every airplane landing and getting up in real-time.
Thanks George. I replied on Twitter but worth saying here also…
I saw Pigeon Sim via Andy @ CASA the other day, it looks like awesome fun! We'd also like to play with Leap Motion and create something similar to the Google Liquid Galaxy stuff.
Quite keen to work with CASA, I'm sure there is a tonne that we can help each other with. I told Andy about the project a while back but that was before we had anything visual. I'll try pinging Andy again, unless you have another suggestion?
Excellent! Love the blog post. And ViziCities already looks beautiful. I can only imagine how wonderful it's going to be once a lot more vectors (population, vehicles, weather, energy, etc) are added in.
Would a simulation component be added in? Cause if it is, It'll be nothing short of awesome.
We've been thinking a little about simulation and, although not likely for the first release, it's something we'd love to explore in the future. It's not a huge step to go from our basic AI to a full-on simulated game-type world.
This is a terrific blog post. Excellent project and excellent sharing. I feel like we have a lot of games set in real world cities to look forward to. Also: autonomous vehicle pathing?
Thanks! The vehicle path side of things is something I'd love to explore in more detail at a later date. Right now it's all rather basic but it could certainly be a whole lot more interesting.
This is amazing! It was cool to see the process of turning data points into a 3d city.. Combine this with the Occulus Rift and the Singularity won't be that far away :)
Thanks! It's definitely easier when you're standing on the backs of free software (like PostGIS and Three.js) but the 3D aspect has certainly been tricky.
That is the magic of having and utilising the power of free and open-source software. Imagine not having three.js, QGIS, PostGIS etc.
Gorgeous work and a highly interesting "how we are doing it" insight. Thanks! Please more!