foss

We are getting a final head-count this week to plan for food arrangements. If you are planning to attend, add your name to the list https://wiki.osgeo.org/wiki/FOSS4G_2017_Code_Sprint#Registered_Attendees. If you are unable to add your name to the list, feel free to send Regina an email at lr@pcorp.us with your name and projects you are interested in so I can add you to the list. Looking forward to hanging out with folks interested in PostgreSQL and PostGIS development.

District Hall is a gorgeous community space. Check out the District Hall View http://bit.ly/2f61J8c

So PostgreSQL 10beta1 came out recently as Holly mentioned. When first mention of beta hits, things start getting serious for me. I have to make sure that PostGIS compiles against said distribution to make sure eager testers aren't held back.

As with other releases, PostGIS didn't compile against the new PostgreSQL version without some nurturing. We've still got one regress failure, but at least PostGIS 2.4 now compiles cleanly against PostgreSQL 10beta1. I'm hoping that we can release PostGIS 2.4.0 just in time for PostgreSQL 10 planned release in September so I don't have to backport PostgreSQL 10 patches I made to lower PostGIS versions.

For PostGIS 2.4 the main focus will be cleaning up the parallel work so that all aggregate functions can enjoy use of parallel optimization. This is even more important with PostgreSQL 10 now that more kinds of queries can benefit from parallelization work. I'm also hoping to focus more energy on the raster side of things.

Leo's pgRouting : a Crash Course video made it thru great. Better than mine. Leo doesn't believe in slides, so this is all live demo stuff. The data he used in the video is part of our code/data download for pgRouting: A Practical Guide.

If you are in Boston or wanna visit Boston for a FOSS4G conference, please join our effort in bringing the FOSS4G 2017 international conference to Boston.

For details check out Help Bring FOSS4G to Boston. Since we are Bostonians we'll be submitting talks and workshops for PostGIS and pgRouting if the event is hosted in Boston.

This past Monday we gave an introductory workshop on PostGIS. Our slides are here: PostGIS tutorial. Overall the event was very educational. I did have a butterfly feeling through out since we were the last to speak.

Talk slides

Talks were good, but most of the topics were not new to me so harder to appreciate. The talk I thought was the best was given by Steve Gifford on Geospatial on Mobile. Two things I liked about this talk was it was a topic I didn't know much about (developing native 3D globe map apps for iOS and Android) and Steve was a very funny speaker. That kind of funniness that looks unplanned and natural. I wish it had a bit more Android content in it though. The full list of talks and associated slides below.

• The State of the Geospatial Industry
• Using Application Templates to Jumpstart Mapping Sites
• Running A Scalable OSGeo Stack
• The NoSQL Geo Landscape
• Geospatial on Mobile

Workshops

The workshops I think were the best part of the event. I particularly enjoyed Andrew Hill's CartoDB talk (PostGIS without the pain but with all the fun) mostly because I haven't used CartoDb so first I've seen it in action. Being able to drag and drop a zip file from your local computer or a url of data from some site and have a table ready to query I thought was pretty cool. You could also schedule it to check for updates if it was a url to a shapefile zip or somethng. Of course being able to write raw PostGIS spatial queries and a map app in 5 minutes was pretty slick.

Ours came after, and unfortunately I think it was pretty dry and too technical for most GIS folks. Plus we hadn't told people to download the files before hand so next to impossible to follow along. We should have called it PostGIS with the pain and on a shoestring budget.

• Intro to QGIS
• Making maps for the web: Introduction to CartoDB
• Introduction to PostGIS

We'll be giving a PostGIS Introduction tutorial at AvidGeo Conference: LocationTech Tour Boston which will be held 8 December 2014 from 8:30 AM to 4:30 PM at:

Hack / Reduce
275 Third St
Cambridge, MA, 02142

Tickets are $40 for admission.

In addition to our PostGIS Intro tutorial, there is a great line-up of talks from visionaries such as Adena Schutzberg, talking: State of the Geospatial Industry and Raj Singh on the topic of The NoSQL Geo Landscape. Also check out other tutorials on CartoDb and QGIS.

In a previous article Waiting for PostGIS 2.1 ST_Resize we demonstrated a new raster function ST_Resize that is available in upcoming 2.1. In this article we'll talk about another function called ST_Tile, which similar to ST_Resize can work with both georeferenced and non-georeferenced rasters. As the name suggests, ST_Tile can be used to cut up rasters right in the database, similar to what the raster2pgsql -t Tilesize commandline switch does when you import. In this article we'll demonstrate the misuse of ST_Tile. ST_Tile can also be used for tiling out of db rasters and what it does when it does that is not to actually tile, but instead mark off the areas where it would cut if it were to actually cut.

Another thing I'd like to mention here which can not be summed up in a single function is that a lot of work has gone into PostGIS 2.1 in improving the performance and robustness of out of database rasters (rasters stored externally but queried as if they were inside the database), and more is planned before 2.1 hits the street. All this work has made me reconsider where out dbs play a role and how their speed profiles compare to in-db rasters. We'll demonstrate the speed profile differences in this article for ST_Tile as well for this small sampling.

If you are not afraid of chewing the fat a little and you are on Windows, there are always fresh windows binaries packaged with all these goodies you can get from: Winnie's fresh baked windows PostGIS binaries corner.

OpenStreetMap by Jonathan Bennett and published by Packt Publishing is a book that outlines what OpenStreetMap is and how to navigate its various offerings and terminology. Perhaps more importantly, it provides the information you need to become an active contributor to this community mapping project and how to get help beyond what the book offers.

First a brief note about OpenStreetMap. OpenStreetMap is an open source community driven mapping project that comprises data, tools to edit the data, and tiling services and other services to view and reuse the data. It is built on PostgreSQL, our favorite relational database. Unlike other tile services such as Google Maps and Microsoft Bing, the tools and various backup mirrors available allow you to download all the data for your region and setup your own customized tiling service in house. This is actually the main reason we are interested in using it. Many of our clients have networks where because of firewalls, portability, general security concerns, need for further zoom in options for their area of interest, or just need to hide/add more layers to the tiles than what OpenStreetMap or other tile services provide out of the box, they need to be in control of the whole process.

We apologize for this short-notice. If you happen to be in the Cambridge / Boston area this week, are a big FOSS4G, Web Mapping, or overall GIS fan, you won't want to miss this event happening Friday and Saturday May 6th-7th at Harvard.

Geospatial Collaboration: New Common Ground

Harvard in cooperation with CGA are holding a conference this year May 6th-7th, 2011. Registration is Free and you can register at Geospatial Collaboration: New Common Ground Conference Registration.

There are lots of exciting sessions lined up. Chris Holmes of OpenGeo will be giving the Keynote address.

Lots of sessions lined up on a wide array of GIS topics. A flavor of topics to give you a sense of what is in store:

• Community-based mapping
• Examples of OpenStreetMap use
• Land Managment
• GeoDjango, MongoDB, and Solr in one breath
• Open Access to Web Mapping Services
• Opengeoportal
• GIS at MIT
• Open Geo Sofware
• Mashable Boston
• Using Balloons in GIS

Check the event schedule for more details. Event Schedule

PostGIS in Action book sale at Registration Booth

We'll be selling some copies of our book at the registration counter. We'll be selling it for a little less than Amazon or Manning. You may even be able to see us in person, and if you want us to dirty your book with our signature, we'd be happy too. Well Regina will be happy too, Leo still has to overcome his hangup of defacing books with ink.

PostGIS 1.4 out soon

PostGIS 1.4 will be out soon, which will be good because it feels like forever we've had this release baking in the oven. The key changes are as follows: Which many are detailed in New in PostGIS 1.4

• ST_IsValidReason() -- requires GEOS 3.1 -- will tell you why a geometry is not valid. Its a complement to ST_IsValid() which has existed for as long as I can remember
• Improvements in all the aggregate functions ST_Collect, ST_Union, ST_Accum to use new array logic that can better handle collecting of many geometries. You won't notice this benefit much until you start collecting and unioning 1000s of geometries
• Name change of hidden st*garray functions to be exposed as ST_Collect(geometry[]), ST_Union(geometry[]) ..etc
• Cascade Aggregate Union -- requires GEOS 3.1 -- need I say more?
• Prepared geometry - required GEOS 3.1
• Better debugging facilities and cleaner code base

Sadly ST_DumpPoints did not make it into 1.4, and we seem to have jumped to 2.0 thinking. Where is 1.5? 2.0 seems offly ambitious (geodetic, better 3d, WKT Raster, restructure to allow support for more types, improved curve) that I feel we really need at least a 1.5 to cushion the path. There are a lot of things we have left on the plate such as ST_DumpPoints, Steve Frost's Tiger Geocoder upgrade for new tiger 2007/2008 format, minor enhancements to distance speed and functionality and dumper that don't require major restructure that I feel those should just come first and that need only be a 3-6 month incubation. Given our new policy of not introducing new functions in minor releases, it would seem almost necessary to have at least a 1.5

We started to upgrade all our old mapfiles to the new 5.2 standard because first its about time and now that we are starting to do a non-trivial amount of Mapserver consulting, we felt it important to keep informed.

In doing so we started to develop a cheat sheet to help guide us on what's possible in Mapserver Mapfile 5.2. The cheat sheet can be found here -- keep in mind its still a work in progress so please refrain from laughing. In the middle of making this -- all the mapserver document links changed. I guess Mapserver documentors were busy too. I must say the new documenation is much better than the old. It has some nice examples all nicely color coded. So I want to thank Steve Lime, Jeff Mckenna, Jean-François Doyon and the others in Mapserver doc group for a job well done.

In mapping this cheat sheet out, the first thing slapping me on the side of the head is Mapserver really needs an XML defined schema. Yes I know I sound like a hypocrite and one who has been smoking XSLT a tad too long for my own good. Yes I was on the side of the group of people who thought What is it with these XML idiot lovers. What is to love in such a monstrously verbose unforgiving format? Mapfile format is wonderful, easy to read, fast to parse, and write THANK YOU very much.

Now it appears I am switching camps. Wait a minute though - the truth is I could care less if the Mapserver engine can understand XML. I just want to write my map file in XML and am happy to XSLT it to a format Mapserver can deal with. Why? Do I think XML is a beautiful language? On the contrary -- I find the format stinks and I think it is inefficient in many use-cases. It is in fact the most abused language if there ever was one.

Our main frustration with XML is that in our line of work we do a lot of data loading, data scrubbing and so forth, and we are usually on the side of the fence where some goof-ball gives you some blob of XML with weird characters and other junk and just stuffs it in a CDATA tag and thinks that will solve all issues or where some guy just took OO UML 101 and he has spent a great deal of time over-thinking his data model when time would have been better spent getting loaded at the nearest bar. He has this wonderfully elaborate 30-level portable XML data file for his simple list of jobs complete with 200-pages describing every facet of it for you to digest.

XML is bulky, unforgiving, and easily abused so why is it useful? The beauty of XML and XML XSDs is that a lot of software and development tools support them.

• If I had an XSD for mapserver, my editor would immediately tell me what values are valid for each type that takes a fixed number of constants
• My editor would immediately tell me when I am stuffing something in a layer that is not valid
• I can XSLT my way to a standard mapserver file or god forbid some sort of tree thingy with links to documentation of that element. How many times have you looked at the mapserver documentation and thought? Can I stuff this object in a layer group? Have no clue - the documentation doesn't really tell me that unless the writer happened to remember and even then there is way too much reading involved for my 5-minute attention span to deal with.
• Nowadays XSLT joins the group of SQL as an embeddable language. Can you think of a modern language that can't deal with XML/XSLT? In PHP/.NET which we program in predominately, both have XSLT classes that can take any xsl file and xml file and transform. My code editor has a built-in XSL processor.
• Mapserver mapfile format just lends itself to that - think nested groups, groups that can go in some groups but not others or can have at most x number of item z in it. Mapserver is far from a simple structure with simple rules, so demands something designed for that kind of thing.
• I tend to think it would help the documentation effort immensely too, because right now as the documentation stands, there is not enough cross-referencing going on or an easy way to visualize the natural flow of the map file. An XSLT of an XSD would solve that problem nicely I think. Take for example LABEL. Do you have any clue what elements a label can be stuffed in if you were just starting out?

So to me -- or at least with my understanding of things XML -> MapFile -- fairly trivial, XSD fairly trivial but torturous exercise, MapFile --> XML not so trivial but less torturous exercise. I'm perfectly content with an XML -> MapFile route and am actually thinking about creating such a thing for 5.2 if someone hasn't done it already, if only I could ever find the time, but if things keep going as they are, I may just have to do that if nothing but to keep my sanity.

A few people have been asking us what are the pros and cons of using SQL Server 2008 Spatial and PostGIS and as a Windows user, why would you still consider using PostGIS. Rather than simply providing some hand-waving saying "well if you just care about displaying data, then use whatever you feel comfortable with, but if you want to do real intensive sophisticated spatial analysis and geometric processing without having to purchase a bunch of expensive software, then PostGIS is probably better for you. Hell why must you think in either or propositions - just use both using the strengths of each.", we have tried really hard to quantify the similarities and differences between the 2 and to boot - we have also added in MySQL.

Our analysis can be found at Cross Compare SQL Server 2008 Spatial, PostgreSQL/PostGIS 1.3-1.4, MySQL 5-6

If you have any comments, suggestions of additions, things you felt we got wrong, then please don't hesitate to comment and we'll try to update our survey.

I was reading James Fee's blog recently on Open Source on the beach at Waikiki and as usual James does a good job of stirring up thinking.

One thing I found interesting about the comments was the perception a lot of people have that you choose to understand one tool or another and if you choose open source for one thing, then you are somehow taking away time to learn more useful proprietary techniques. I have found this speculation to be for the most part untrue.

For example I consider myself to be fundamentally a Microsoft SQL Server expert and fairly competent in Microsoft technologies. When I started to get involved in Open source software - such as PostgreSQL, MySQL, Linux, Mapserver, PHP, etc. I discovered something very startling. These Open source folks really care about standards.. Understanding standards is more important than understanding how to use a set piece of software because standards are more aligned with trends in technology.

For example, A long time ago, I used to think COALESCE and NULLIF were functions developed by the PostgreSQL folks. One day forgetting which database I was in, I accidentally used these in SQL Server. And guess what? SQL Server knew what they meant. Similarly SQL concepts like INTERSECT and EXCEPT existed in PostgreSQL long before SQL Server introduced it into SQL Server 2005. So when SQL Server 2005 came out, gosh darn it I already knew how to use these tools. The same holds true for PostGIS by the way. You will find that PostGIS does things the OGC standards way so when SQL Server 2008 finally comes out with OGC Spatial support, I suspect people using PostGIS and similar tools will be way ahead of the curve.

Other example - because of my exposure to open source tools such as PHP, Linux, PostgreSQL, I realized early on that Microsoft's Active Directory was nothing more than a Light-weight Directory Access Protocol (LDAP) service by another name. This had some very interesting consequences. For example I realized that it was much easier to query Microsoft Active Directory with PHP than it was even using Microsoft.NET Framework.

In the GIS world, because of my exposure to Mapserver and other open gis tools, I knew what ESRI's WFSConnector and WMSConnector were for and how to test them long before my GIS brethren knew what those terms even stood for. I knew the foundations of these and found myself explaining the concepts of web services etc. to my GIS friends.

So the point is this - if you are spending your time simply learning how to use proprietary tools instead of really trying to understand the fundamentals that drive these tools, you have totally missed the boat and you will be forever playing catchup.

I have found that Open source really enforces understanding fundamentals more than proprietary does and that is a very good thing because that knowledge has a longer shelf-life.