GDAL is about to drop support for VS2013 and earlier, please vote

As you may have noticed from recent emails on the gdal-dev list, support for C++11 in VS 2013 is only partial and kind a bit of a pain to deal with.

The GDAL team created a poll to probe the community how much support for VS 2013 is seen as needed for GDAL 2.3

Please cast your vote (doesn't require any account, open to any user / developer / whatever party with at least some interest in GDAL ) at

http://www.easypolls.net/poll.html?p=5a399ccfe4b04ebe73ddb454

Voting opened until end of this week

Thanks

OGR MSSQL bulk insert support added in GDAL 2.1

With the release of GDAL 2.1.0 the new bulk insert support has been added to the OGR MSSQL Spatial driver. Using the bulk insert option the upload is approx. 20 times faster comparing to the earlier driver versions, which is especially useful when uploading large data tables to MSSQL server.

The bulk insert option involved the need to implement the write support for the native SqlGeometry/SqlGeography serialization formats, which in turn eliminates the need of the server side geometry conversions from WKB and WKT causing large amount of memory pressure at the server and may cause issues like this one.

The implemented bulk insert support requires the SQL Server Native Client 10+ driver to be installed on the system. This is installed along with a SQL server setup or available from a separate package provided by Microsoft. This requirement is a bit unpleasant for the binary distribution providers, because the driver should link against the corresponding sqlncli dll. Therefore we added support to build the MSSQL spatial driver as a plugin dll (ogr_MSSQLSpatial.dll). The gdal-core msi installers (development, daily) at GISInternals have been modified to include the bulk insert enabled MSSQL driver as an optional component (not installed by default). If you enable this component, make sure that the SQL Server Native Client 11 driver is also installed on your the system, otherwise ogr2ogr will likely fail to start.

By installing this driver, the bulk insert option is enabled by default when using MSSQLSpatial as an ogr2ogr target database, however we can disable bulk insert by using the MSSQLSPATIAL_USE_BCP=FALSE config option which turns the upload to a set of "INSERT INTO" operations as usual, but that option continues to use the native geometry/geography serialization formats. We could also select the earlier WKT/WKB upload by using the UPLOAD_GEOM_FORMAT layer creation option. For more information about the config and layer creation options supported by the driver please refer to the MSSQL Spatial driver information page.

The new GISIntenals site (providing Windows binary packages) is about to be released

GISInternals is an online system for creating daily built binary packages for the GDAL and MapServer projects. The anchestor of this system has been provided to be the Windows buildslaves for the osgeo buildbot in 2007. The build system in the current form (providing downloadable packages) has been set up in 2009. As of this time, the system has been continuously improved by adding more and more packages to make the life easier for the users and developers of these open source projects. During this time the number of the visitors and the amount of the downloads continues to grow, and the site has been visited from more than 160 countries around the world.

In 2014 December, I'm about to release a new version with several changes in the design and the architecture. The functionality has been divided into 3 portions: the GUI frontend, the file servers (php and asp.net backends) and the the build server. This approach will provide to increase the reliability and the availability of the site by the chance of using multiple build servers (upload agents) in the future.

If you would like to preview the upcoming version visit download.gisinternals.com.

There are several upcoming changes are planned to be released in December including support for the newer compiles and additions in the packages (like MapCache and the new GDAL drivers).

IntraMaps Roam first release - Windows QGIS Data Collection

Digital Mapping Solutions, and Nathan Woodrow, would like to announce the first version of IntraMaps Roam - a simple, but flexible, Windows based data collection application built using Python and QGIS.

IntraMaps Roam (or Roam for short) is a standalone, fully bundled, Python application that was created to do data collection with a QGIS backend. For those of you who have seen my QMap project Nathan started a year or so ago you can consider this a reincarnation of that project. Most of the code has been reworked and using the QGIS libs gave him full flexibility in layout.

The release page can be found at https://github.com/DMS-Aus/Roam/releases/tag/v2.0 and the wiki with all the information to get started at: https://github.com/DMS-Aus/Roam/wiki You can also take a look at the FAQ for the common questions: https://github.com/DMS-Aus/Roam/wiki/FAQ

Roam has been a great exercise in using and bundling QGIS libs with a Python application.

As Roam is based on PyQt and QGIS it is under the GPL2 license. Pull requests are welcome.

Links:

• https://github.com/DMS-Aus/Roam/releases/tag/v2.0
• https://github.com/DMS-Aus/Roam/wiki
• https://github.com/DMS-Aus/Roam/wiki/FAQ

Happy mapping!

Announcing MapServer MapManager 1.0

Digital Mapping Solutions along with Tamas Szekeres is pleased to announce the open source release of MapServer MapManager.
MapServer MapManager is an easy to use Windows desktop GUI application (based on the MapScript C# API and GDAL C# API) that allows you to easily create and configure great looking MapServer map files. The application provides a set of dialogues for setting the various properties in the map configuration and renders the map directly on the screen allowing panning and zooming of the displayed area by using the mouse. The MapManager user interface provides common operations for selecting the features and inspecting the attributes for setting up labelling and changing the style and symbology. The created map configuration can then be saved into a map file which can be used by various MapServer CGI/MapScript applications. Mapserver MapManager also allows users to save the map images in various output formats supported by MapServer.

MapServer MapManager is provided with GPLv3 license.
The build SDK from gisinternals can be used to compile MapManager. For more information about building MapManager please refer to the Readme file.
Windows installers are provided in the following download locations. Certain GDAL drivers are provided as separate installers due to their custom licensing terms. When installing the GDAL driver plugins you should select the MapManager install location as the target directory.
Installers based on the current MapServer release (6.4.1)
MapServer MapManager
GDAL ECW driver
GDAL FileGDB driver
GDAL MrSID driver
GDAL Oracle driver

Installers based on the current MapServer stable branch (6.4) (compiled daily)
MapServer MapManager
GDAL ECW driver
GDAL FileGDB driver
GDAL MrSID driver
GDAL Oracle driver

For more information about using MapManager please refer to the online help.

Mapserver daily builds has been added to osgeo4w

The build server at http://www.gisinternals.com/sdk/ has now been configured to provide daily built packages (x86 and x64) from the latest MapServer development and stable branches.

In addition to the mapserver core, the python and the java packages are also provided.

In the osgeo4w package selection page the following packages can be selected:

1. mapserver,mapserver-python,mapserver-java -> corresponds to the latest release (these packages are being built manually as soon as a new release is issued)
2. mapserver-dev,mapserver-python-dev,mapserver-java-dev -> corresponds to the latest master branch
3. mapserver-stable-dev,mapserver-python-stable-dev,mapserver-java-stable-dev -> corresponds to the latest stable branch

When using the -dev and -stable-dev packages You should re-run the osgeo4w installer regularly to be up to date with the recent builds

MapServer SVG symbol support

Windows builds at gisinternals are now provided with SVG symbol support in MapServer. For more information about this feature see this article SVG Symbol support in MapServer.

Issue tracker for gisinternals.com

The build system at www.gisinternals.com has been added to github. Reporting issues or enhancement requests related to the build system can now be posted to the github issue tracker.

www.gisinternals.com is down

GisInternals support site will not be available until Monday due to hardware maintenance.

QGIS MS SQL provider has been added

Thanks to the support of Digital Mapping Solutions, Nathan Woodrow and I have now added the new native MS SQL provider in the QGIS mainstream. The new provider now supports full featured read-write access to the MS SQL databases. By using the dragdrop support in the browser window you can easily push layers up to the database. The driver creates fully registered layers (by adding entries to the spatial_ref_sys and geometry_columns metadata tables) which are compatible with the GDAL MSSQL driver as well (to take advantage of the author being the same :-). In the heat of the work we have also enabled the dragdrop upload capability for the PostGIS provider.
We encourage you to give it a try and let us know if you find issues, by adding an entry to the QGIS issue tracker.

www.gisinternals.com is not available

The gisinternals support site providing Windows builds for GDAL and Mapserver will not be available this weekend due to the reconfiguration of the local area network at the server. Hope it will work again on Sunday at 20:00 CET.

www.gisinternals.com is down

The GDAL and Mapserver daily build package provider site www.gisinternals.com/sdk is currently not available due to server relocation. I hope it will be online again by the end of this weekend.

Windows SDK support site has been relocated

Looks like my account on dyndns has been lost :-(  Since I'm getting tired to update the account in every month, I've already registered a domain to point to this location which is:

http://www.gisinternals.com/sdk/

If you have any links or bookmarks to point to the support site then update them to refer to this new location. The old URL is considered to be deprecated from now.

GDAL installers for multiple Python versions

After a bit of tweaking in SWIG and Python distutils the installers for GDAL are now available targeting Python 2.6/2.7 and 3.1 (x86 and x64) at http://vbkto.dyndns.org/sdk/

Hmmm... I must mention that quietly, there haven't been made an effort to fix the problem descibed in this post with the recent Python versions. Probably I'm the only one who's trying to compile anything with MSVC2010? I doubt it. Looking forward to the next version...

Windows binaries for GDAL 1.8.0 and MapServer 5.6.6

As you may already know both GDAL and MapServer had a new stable release in the recent days and I've updated http://vbkto.dyndns.org/sdk/ to provide the Windows binaries according to these versions.
If you prefer using the latest stable releases (GDAL 1.8.0 and MapServer 5.6.6) then choose the 'release' packages, but you may also download the most recent daily builds of the stable branches (always containing all the subsequent changes since the last release) by choosing the 'stable' packages.
The development versions of GDAL and MapServer are also available for testing by using the 'development' packages.
 I've also added Windows installers for the GDAL components providing to install the GDAL command line tools and the python bindings easily. Installers for MapServer with IIS registration and PHP support will also be available shortly.

Daily built binary packages for MapServer with OpenSSL support

SSL support has been added to the daily built binaries which provides the option to access a secure WMS/WFS server on Windows. Here are the key steps to configure this option in your Mapserver/Mapscript installation:

1. Configure the mapfile with one or more WMS/WFS client layer

This will require to configure a layer with 'CONNECTIONTYPE WMS' or 'CONNECTIONTYPE WFS' and set up your connection information according to the following fragment:

LAYER
    CONNECTION "https://www.secureservice.com/service?"
    CONNECTIONTYPE WMS
    METADATA
      "wms_srs"    "EPSG:4326"
      "wms_name"    "NAME"
      "wms_format"    "image/png"
      "wms_server_version"    "1.1.1"
      "wms_auth_username" "username"
      "wms_auth_password" "password"
  END
    TYPE RASTER
    ...
  END

2. Make sure to install all the required files in your deployment

For the SSL support you require to install libeay32.dll and ssleay32.dll along with libcurl.dll in the run-time configuration. Without having OpenSSL added to the builds you got the following error:

msHTTPExecuteRequests(): HTTP request error. HTTP: request failed with curl error code 1 (Protocol https not supported or disabled in libcurl)

3. Set up the cerificate bundle along with your installation.

In short it will require to set up the CURL_CA_BUNDLE environment variable to point to the location of the .crt file. You can find this file (curl-ca-bundle.crt) in the /bin/curl folder of the binary packages mentioned above. The file contains the most recent version of http://curl.haxx.se/ca/cacert.pem (at the time of the nightly build).

Note: CURL_CA_BUNDLE can be set up system wide (in My Computer->Properties), however you may prefer to apply this environment setting only for the process hosting the mapserver libraries so as not to cause side effects for other applications using libcurl and openSSL. The main issue here is that most of the libararies use getenv to retrieve the environment setting, which operates only on the data structures accessible to the run-time library (msvcrt) and not on the environment "segment" created for the process by the operating system. In this regard the libraries work only on a snapshot of the variables that have been set during the process startup. In this regard you may choose to set up the environment before the process has been started (by using a startup script) or use _putenv to set up the environment variable at run time. I will be trying to discuss this topic in more detail in a subsequent post.

You may check whether your WMS/WFS server is working with the standalone installation of curl.exe in the binary packages and open an SDK command prompt by using SDKShell.bat supported with the packages. In the command prompt you may use a valid url to your service, like:

 curl "https://username:password@www.secureservice.com/service?..."

At this point if you get valid data response you may omit the following step.

4. Extract the certificate entry from the server

In the previous step if you get the following error, your remote server probably use a self-signed SSL certificate and the server certificate is not included in your CA bundle file.

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

To get the remote server certificate you may use openssl.exe in the command prompt (also included in the daily buillt packages) according to the folowing example (you may probably use the default SSL port:443 in most cases):

  openssl s_client -connect www.secureservice.com:443

Copy all from "-----BEGIN CERTIFICATE-----" tag to "-----END CERTIFICATE-----" tag. Paste it at the end of the curl-ca-bundle.crt file, and repeat #3 to make sure the certificate data is now working.

OGR MS SQL 2008 Spatial driver

I've now added the initial revision of the OGR MS SQL Spatial driver to the GDAL repository. For further information please refer to the driver information page.
Working binaries (which always contain all the recent fixes) can be downloaded from the daily build system available at: http://vbkto.dyndns.org/sdk/

Since I didn't have the chance to test with all kind of the fancy spatial data types any feedback on the potential issues are graciously accepted.

Upon the experiences gathered when writing this driver, I'm also planning to add support for accessing the native SqlGeometry/SqlGeography serialization format in the MapServer sql2008 driver which appears to provide a significant increment in the performance of the data retrieval.

Python distutils and manifest files (Visual Studio 2010)

Visual Studio 2005 introduced a new deployment model for Windows client applications based on isolated applications and side-by-side assemblies. The redistributable libraries (such as MFC, ATL, CRT) have been rebuilt as shared side-by-side assemblies and installed in the native assembly cache, WinSxS folder in Windows. This has provided a common location for particular versions of the dll-s and the applications had to declare their dependencies by using a manifest file. The manifest file is an XML file which contain references to the dependent dlls located in WinSxS, these files had to be deployed along with every dll or executable which depend on the redistributables. Without having the manifest (either by a separate file or embedded in the executable as a resource) the application generates an error.
Though the purpose of this change was to simplify deployment, the result was probably the opposite. As a result Microsoft changed deployment requirements in Visual C++ 2010. As of Visual Studio 2010 Microsoft has backed out the manifest generation and no manifests are generated by default with the VC2010 builds. There's no requirement to embed manifests in the executables anymore and CRT-s need not be installed in WinSxS. All you need to do is copy the VC++ dependent DLLs to the application folder and run.

This change may however cause issues for those build systems (like python distutils) which don't expect that the manifest may eventually be missing and stop the compilation in this condition. I don't see any change in this regard in the python SVN trunk either. Fortunately to prevent from this issue we could easily alter the corresponding file (/Lib/distutils/msvc9compiler.py) to something like:

   # embed the manifest
   # XXX - this is somewhat fragile - if mt.exe fails, distutils
   # will still consider the DLL up-to-date, but it will not have a
   # manifest. Maybe we should link to a temp file? OTOH, that
   # implies a build environment error that shouldn't go undetected.
   if os.path.isfile(temp_manifest):
        mfid = 1 if target_desc == CCompiler.EXECUTABLE else 2
        out_arg = '-outputresource:%s;%s' % (output_filename, mfid)
        try:
             self.spawn(['mt.exe', '-nologo', '-manifest',
                  temp_manifest, out_arg])
        except DistutilsExecError as msg:
              raise LinkError(msg)
        else:
              log.debug("skipping %s (up-to-date)", output_filename)

Which may work even if the deployment model will frequently be altered by Microsoft from versions to versions ;-)

Visual Studio 2010 (Win32/Win64) packages are now available

After some investigation of a couple of issues, I've finished to set up daily build packages compiled with Visual Studio 2010. Just as for the previous packages I've compiled all the dependecies with Visual Studio 2010 to prevent from potential memory corruption issues as described in my earlier post: Redistribution of the shared C runtime component in Visual C++

The packages can be downoaded from the usual location: http://vbkto.dyndns.org/sdk

In addition to the binary packages the build SDKs for VS2010 are also available (containing the libs and headers)

Although the builds seem to be in good condition (the auto tests are running correctly) don't hesitate to contact me if any further issues encountered.

Note on the MrSID driver: Currently there are no available VS2010 versions of the LizardTech GeoExpress Raster DSDK, therefore the MrSID plugin is not included in the Visual Studio 2010 packages. However you might want to copy (bin\lti_dsdk_dll.dll and bin\gdal\plugins\gdal_MrSID.dll) from the corresponding VS2008 packages in order to have MrSID support switched on.

MapServer 5.6.2 released

After fixing a few important issues found in 5.6.0 and 5.6.1 the MapServer team has announced the release of MapServer version 5.6.2.
Please refer to the history page for the related changes.
The binary distributions listed in the download page should be updated with the new release in the next few days.