Reading Inspire XML with FME

@SafeSoftware asked, if there’s any chance I’ll write this blog post also in English: http://gissiajapaikkatietoa.wordpress.com/2011/11/16/mmln-kuntarajat-fmella/. There’s a very good chance you can read it in English, since I’m now writing it in English. Finnish doesn’t seem to be the universal language yet, thus I understand @SafeSoftware’s question.

To begin with, according to Safe Software, FME 2012 can read XML documents in Inspire schemas out of the box, since FME 2012 contains Inspire schema files (from Annex I). This would mean that reading XML documents in Inspire schemas and writing to different formats is easier than before. This needs testing!

Downloading data

I’m testing the new functionality with open data from the National Land Survey of Finland.

I download municipality boundaries in XML from http://www.maanmittauslaitos.fi/ilmaisetaineistot.

Reading data

I choose INSPIRE GML as the format and municipality boundaries as the dataset with a scale of 1:100 000.

Inspire GML-luku
Next thing to do is to check parameters. I choose Dataset Only as the setting for Show Feature Types from:

As a result, I get two feature types, AdministrativeBoundary ja AdministrativeUnit. Complex properties are mapped now as nested attributes. For example following XML fragment

<inspireId>

<Identifier>

<localId>au13572990</localId>

<namespace>FI.NLS.AU100</namespace>

</Identifier>

</inspireId>

will form two attributes inspireId.Identifier.localId and inspireId.Identifier.namespace.

AdministravieBoundary ja AdministrativeUnit

Writing data

Let’s continue the test by writing administrative units and boundaries to different formats, e.g. MapInfo TAB, ESRI Shape, KML, PDF, DWG. I built the workspace and defined three parameters, with which I can direct the transformation of the data. The parameters are: Source file(s), Destination folder and Destination data format. The end-user doesn’t need to do anything else than define the values for the parameters and click OK.

Inspecting results

After running the workspace, we can inspect the results in FME Data Inspector or Viewer. For the first run, I chose MapInfo TAB as the destination data format, and the result has correct geometries and attributes:

Kuntarajat MapInfo TAB:ina.

Other formats that I tested were KML, PDF ja DWG, which worked without problems.

Reading XML data in Inspire schemas is now very handy with FME.

Leave a comment

Filed under Inspire

Real time bus visualization

Some of you might know that you can publish real time data to FME Server as a KML Network Link, and then visualize data in Google Earth. Safe Software (www.safe.com) has this really great real time taxi demo:

It generates taxi locations with a Python script, and FME Server is used to stream the locations and visualize them in Google Earth.

While I was fiddling with the demo, I throught, could this be used for visualizing buses, which get real-time passenger information, for example current number of passenger. So, I downloaded the demo Safe provides, and started modyfying it.

Modifications included creating 3D features from each bus location, and extruding each feature by number of passengers. Also, the number of passengers is visible on top of each feature with the bus number, for example, 111 | 30, where 111 is bus number, and 30 is the number of passengers.

The bus data is real time data, and is updated in every second. FME provides a really easy way to handle real time data, and publish it to applications such as Google Earth.

1 Comment

Filed under FME

FME 2011 launched!

I’m a week late with my blog post about FME 2011, but it doesn’t make FME 2011 shine any less. During the week after the release, I’ve been trying to understand, how Safe Software does it. FME 2011 is a very big release amongst earlier releases. It takes spatial data crunching to a whole new level. It’s easier, faster and has a lot new transformers and supported formats.

As I’ve found when reading blog or magazine posts about the launch, lots of them mention FME 2011 point cloud support to be one of the most important new enhancements. I agree. Lidar is now going to be mainstream or is already. This means very good times for FME. Read more here, http://www.directionsmag.com/articles/a-q-a-on-fme-2011/151615, how Dale Lutz describes the new point cloud support. He says point clouds are “misbehaving rasters” actually. He says adding raster support point helped them in adding point cloud support.

I’m also very happy for the Rest support for FME Server. Now it’s much easier to integrate FME Server to already existing web services.

Rest of the enhancements can be found here: http://www.safe.com/highlight/fme2011/whats-great/

Last but not least, FME 2011 World Tour starts from Vancouver and tours around the globe in 25+ cities. The events are free! That is indeed very generous. More information: http://www.safe.com/FME2011WorldTour

P.S. Now it’s time to start spamming Safe Software with new enhancement requirements.. 😉

2 Comments

Filed under FME

Smallworld Data Validation with FME

Hello, I built an FME process to validate couple of error types in Smallworld database. The errors that need fixing are:

  • Lines that don’t connect (I had some cable network in my database).

 

Cable lines that don't connect.

 

  • Intersections, that should have an object marking the intersection (In this case I had joints in the cable intersections, but some were missing).

 

A joint is missing from an intersection.

 

I read the cable and joint data to FME, validate it, find errors and report back to Smallworld. FME transformers that are really handy in inspecting geometry errors are

  • Snapper
  • LengthCalculator
  • Tester
  • Intersector
  • ListElementCounter
  • PointOnPointOverlayer

There are lots of transformers which could be used too, but I found those ones suitable for this validation case.

Here’s what the result looks like in Smallworld:

 

The red flag indicates cables that don't connect, and the orange flag indicates a missing joint.

 

This process is quite handy to validate invalid geometries and missing features. Data is read to FME from Smallworld, validated, and written back to Smallworld with just one click. After that the user fixes the features. Of course FME could also do the fixing, but here FME is working as a validation and reporting tool.

Leave a comment

Filed under Smallworld

From Smallworld to 3D PDF

I’ve worked with Smallworld for several years now. One of the useful ways to visualize all kinds of spatial data is utilizing 3D PDF. PDF itself is widely known format, thus why not use 3D PDF for visualizing data derived from Smallworld database. First I built some electricity network in Smallworld (Poles and cables), then forests around them and a few buildings. Poles, forests and buildings had height attributes, so that helped to transform them from 2D to 3D. Also I had a DEM from the area, so the task was to transform all the features to 3D and place them on the DEM. Also, a raster was added on the DEM.

The main challenge

The main challenge was how to transform data from 2D to 3D world. FME has a couple of useful transformers for constructing and transforming 3D objects: Extruder and SurfaceDraper.

Extruder stretches your 2D features to 3D by a specified height or a vector.

SurfaceDraper places your 2D features on an elevation model. You should use SurfaceDraper first and then after that Extruder.

An other challenge were to make the cables hang. It worked well with a bunch of transformers. The result of the hanging cables can be seen in the picture.

Here is a picture of the result, where you can see the cable, forests and buildings in the left.

Here is another picture of the result 3D PDF.

From Smallworld to 3D PDF with FME

1 Comment

Filed under 3D

FME Scandinavian User Conference 2010

Over 110 FME enthusiasts, lots of presentations, great conversations with FME users and Safe guys, FME 2011, good food and of course So you think you can FME. That’s what happened in Malmö, in FME SUC 2010. Thank you Sweco, thank you Safe, and other organizations and participants!

Ken and Dale did a remarkable job with presenting FME Server, CAD <-> GIS use case, and FME 2011. Ken and Dale gave separate presentations, when previously Don&Dale have given the main presentations together. I guess some hyperventilation was reduced in this way. Ken pointed out some top challenges people face when moving between CAD and GIS worlds. Styles, valid geometry, and valid attributes. It was also interesting to hear that when Safe went to this one GIS fair, they had CAD and GIS words on their posters for the first time, people were stopping by their booth a lot more than previously. CAD to GIS is still one of the main largest needs for data transformation. And FME does CAD and GIS transformation well!

FME 2011 is going to be awesome! http://www.fmepedia.com/index.php/Whats_Great I have already used the beta, but I didn’t really know all the new stuff, that Safe’s been working on. FME 2011 has inspection point functionality. You can set inspection points to any link, and really see what features go to the inspection point. It’s really handy to use for debugging. It has a window, where you will see the feature, and below it has the feature attributes. You can even set rules. For example, if you know that certain type of features are sensitive for crashing, you can make the inspector stop to first one of those type of features, and you can check, why it crashes. Clever!

I’ll try to post more of the conference later, when I get to see the presentations again. I lost my notebook there, and I need to through the presentations again, and hopefully I then recall the visions and ideas that I got in the conference. Stay tuned.

3 Comments

Filed under FME

Finnish FME User Day 2010

We had our first FME User Day yesterday, and I realized, that Finnish FME users are almost getting ahead of me in FME knowledge. It’s frightening but at the same time challenging. Many users were very intrigued by XMLTemplater and all the other new XML / GML functionality in FME. I guess one of the top problems is, how to obtain complex XML from various data sources, when the source schema and the XML schema change. XMLTemplater and SchemaMapper really make handling XML and GML much easier.

It’s not no to XML anymore!

Some were also interested in reading WFS, and filtering the WFS data in FME. All in all, there were a lot of discussion, and the users are looking forward for the next Finnish FME User Day 2011!

Thank you very much Safe Software and Dean for your commitment to the user day!

Leave a comment

Filed under FME

Inspire: Part II


Having written about the basics of the Inspire initiative, it’s time to write more about the means by which Inspire will be implemented. Before that go through this short review of the last post:

Inpire in short:

  • Inspire defines several spatial data themes
  • Governmental Agencies will provide their data that belong to a theme
  • The data is provided with metadata
  • Online services are needed for data acquisition

Services

There are several services that are defined:

  • Registry Service: Provides the data specifications and their data models of spatial datasets from Download Services for users to browse.
  • Discovery Service :Enables the search of spatial datasets and services with metadata. Also enables viewing metadata.
  • View Service: Enables viewing spatial datasets on monitor, and several map view functionality.
  • Download Service :Enables download spatial datasets and their subsets to your computer.
  • Transformation Service :Enables transformation of the spatial datasets to Inspire defined datamodels. First it will apply to coordinate transformation from national coordinate systems to European ETRS89 coordinate systems.
  • Invoke Service: Enables invoking the web services.

The services work as interface services, which means that agencies are not required to create client applications, just the interfaces that the client applications use. In the following picture you can see the services, metadata, spatial data sets, service bus and applications.


Standards

  • ISO: ISO 19101:2002 Reference model, …
  • OGC: GML, WFS, WMS, …

Not too hard, wasn’t it? Part II was mainly for shortly presenting the web services. In the following parts of this series, you’ll see more of the services. It’s actually my favorite component of Inspire.

Inspire defines several web services: Registry, Discovery, View, Download, Transformation and Invoke services.

Leave a comment

Filed under SDI

Inspire: Part I

Inspire Directive's Weird Logo

This post begins my Inspire series. The series contains general information, what is Inspire, and when it happens. I won’t go too deep into details and the law text of the Inspire directive, which could easily make my few subscribers press the unsubscribe button.

Introduction

As some hawk-eyed readers might have noticed, Inspire is a directive meaning that it is a legislation for which EU members are bound.  It is also an initiative to create a European SDI. It aims for

  1. better coordination between government agencies,
  2. more effective usage of spatial data sets,
  3. and diverse services for citizens.

The directive itself  can be emphasized in following keywords: interoperable spatial datasets, exchange of spatial data services, their joint use availability in various levels of governing and industries. If we bring the keywords together in one sentence, it might look like this:

“The directive’s aim is to enable exchange, joint use, and availability of spatial data sets, and services associated with this information.”

Spatial Data sets

There are 42 spatial data themes to which the directive is applied, and they belong to 3 Annexes. I won’t list them here, you can find them in the sources.

Whew! Time to have a deep breath, yes, just like that. This is needed before going to metadata.

Metadata

The agencies that provide the spatial data sets should provide their metadata and attach it to the services. Metadata includes descriptions about the data the agencies make available. For metadata enthusiasts, you might want to check out this document: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2008:326:0012:01:EN:HTML

End

This is the end of the part. Part II will tell you, what are the methods to implement this, and what a moon (what an earth is too often used, so I use moon instead) are the services, which should provide the 42 data themes?! Stay tuned.

Sources

http://www.agi.org.uk/SITE/UPLOAD/DOCUMENT/policy/inspire_intro.pdf

http://www.paikkatietoikkuna.fi

http://www.inspire-geoportal.eu

1 Comment

Filed under SDI

FMEDays 2010

Being a FME user, I just had to travel to Münster, Germany, where FMEDays 2010 was held. After arriving to Düsseldorf, I took a wrong train to Minden, which is not apparently Münster. After totally 5 hours of train travelling I found Münster, which should be about 1 hour 30 minutes from Düsseldorf. Does anyone else, who’s working with GIS, has a lousy homing instinct?

After all, I found Münster. Thank you Safe Software and con terra for a great user conference! I won’t forget all the resellers and users either: Thank you for all the conversations we had. I spilled couple of times coffee, first one happened when I heard the secret word “WFS” and my coffee fell all the way from 3rd floor down to the con terra reception on a poor con terra employee. Again, my apologies, if you’re reading this blog. Anyway, was the conference worth it? I give you 3 reasons, why it was worth it:

  1. FME 2010
  2. FME Users
  3. Future

1. FME 2010 belongs to one of the most biggest releases Safe Software has ever had. Just look at its XML, raster, 3D capabilities and performance. I’m really excited about replacing all those XSLT scripts with a transformer called XMLTemplater. 3D capabilities include raster draping, this one is from Dimitris 3D Place:World DEM

I’ve used FME 2010 since the beginning of 2009, and I’m really impressed of the new functionality such as adding transformers just by typing in the canvas, and the workspace search.

2. FME Community has quite excellent and smart users, if you look at what they’ve accomplished with FME. Here are just a few examples, that I saw:

  • Augmented Reality, Vicrea. FME was used to create 2D and 3D objects (City center) for augmented reality application on Iphone.
  • Building a 3D City Model with FME, HNIT BALTIC. In contrast with earlier CAD to 3D solutions, FME was used to build a 3D city model from GIS datasets. The 3D city model was also published to web, so you could have a virtual walk inside the model!
  • Lithuanian SDI, GIS-Centras: FME Server was utilized in providing download and transformation services for users.
  • Contractor Portal, Dottedeyes: If you’re looking for a solution which combines open source components with a transformation services software, you have to see this presentation.

Remember that FME users are not just FME users, they come from several different product, database, format and coordinate system environments. Just to mention a few, some are enthusiastic ESRI products users, some work closely with databases, and some are CAD software users. They still gather together because of FME, and they have fun together!

3. Safe Software seems to love XML, since FME’s XML support seem to increase in every FME release. They’ve also made a very good progress with metadata support, and that is a tale which is just beginning. Which makes me write a postit note that I should start studying metadata more seriously. Don & Dale show is also one thing to mention. If you don’t have any other reason to attend FME User Conferences, at least do it because of Don & Dale. They’re the rock stars of GIS world!

Now I just have to process all the ideas I got from the presentations and discussions, it’s just hard to cope with the post-conference-melancholy. Coming back to normal weekdays is always tough after great conferences. 😉

3 Comments

Filed under FME