DEM Creation using QGIS Tutorial

[WrinkleysRule]

DEM Creation using QGIS Tutorial

For this tutorial you will need the following programmes and items

Google Earth Pro ---     https://www.google.com/earth/about/versions/#earth-pro

QGIS                       ---    https://qgis.org/download/

Overlay.zip                ---    Attached at the end of this tutorial

Download Google Earth Pro and QGIS (may take a while as QGIS is approx 1.2Gb)

Unzip the Overlay.zip and place the Overlay folder into you map root folder

Create Farm Marker

This section details how to create a marker that outlines your proposed farm area for use in QGIS and is basically the same as first part of the previous tutorial, DEMs and Borders the Easy Way.

Once both programmes are installed open Google Earth Pro –

Select the Temporary Places icon and right click any where in the Places panel,

Select- Add –Folder name this folder with your map name (this is to keep all your files in one folder).

You may need to drag drop any newly created files into this folder.

Ensure the Terrain Icon in the Layers panel is deselected

Navigate to your proposed farm location

Create a vertical and horizontal 2.1km line using the ruler tool centred roughly at the middle of your proposed area

Select the Add image Overlay icon in the Toolbar (green markers will appear) and browse to the Overlay folder location and Select Area.png, rename to 2k Area don’t close the Image Overlay pop up at this time.

You will see a red square with green markers, adjust the edge markers (T shape) to fit the horizontal and vertical lines previously made this will give you a 2.1k square which you can move over the location to finalise the area by means of the centre cross or rotate using the diamond (if you move any of the corner markers you will distort the square and have to reset the edges). Once you are happy with the position click OK.

If you wish to move it again right click on the 2k Marker in the Places panel and select Properties this will bring the adjustment markers back.

At this point mark and name the area using the Add Placement tool as when adding the Lidar Data/SRTM Google Earth zooms right out so the Markers may not be visible for you to centre the cursor on to scale the relevant Geo.tif

Should you wish to create a border around your 2k area then, using the ruler tool create 1k lines away from the edge of the 2k area. 1k chosen for performance reasons but if you want a bigger border just increase the distance.

Select the Add image Overlay icon in the Toolbar and browse to the Area.png location and select it, rename to 4k Border and repeat the alignment detailed above.

In the Places Panel right click the 2k Area icon and Save Place As  .kml (in a suitably named  folder) do the same for the 4k Border Area if needed.

Obviously if creating a larger map change the markers to an appropriate value.

The Line Measures can now be deselected leaving just the two markers

Finally save the My Places  -- File – Save – Save My Places

 

Creating the DEM

Different Countries will have different methods of downloading data so check the relevant Sites methods.

For this tutorial I will be using UK Lidar Data available from -https://environment.data.gov.uk/survey

Open in browser zoom in to your location Select Draw polygon, Draw polygon round the farm area (right click move to next point until your shape is complete double click to finish)

Select - Get Available Tiles

Select Product --LIDAR Composite DTM --- This produces a ‘terrain only image’, DSM will produce all buildings trees etc image.

Select Year -- Latest year currently 2022

Select Resolution – 1m  - higher 2m resolution will have smaller file size

1m resolution files will have a file size of approx 80kb per tile.

Select  - Download All and save to a suitable folder (the one you saved the 2k/4k Area .kml’s)

Create another folder in that folder and name it Your Farm Name Tifs

Unzip the downloaded files and copy paste the DTM_1m.tif images into this folder.

QGIS

I will be using QGIS 3.40.0 throughout this section later versions may become available so version numbers may change.

All the programmes required for QGIS should in a folder on your desktop after installing QGIS 3.40.0

In the folder QGIS 3.40.0 select QGIS Desktop 3.40.0

In the Browser panel check to see if you have a Google Earth overlay in the XYZ Tiles icon ( this overlay will only be used as a reference).

If you do not have the Google Earth Satellite overlay then right click the XYZ Tiles icon and select

In the Pop up panel enter a suitable name (eg Google Satellite) and in the URL panel copy paste the following

http://www.google.cn/maps/vt?lyrs=s@189&gl=cn&x={x}&y={y}&z={z}

  the rest of the settings can be left as is and click OK.

   

 

Should you prefer or want to add others then do the same as above here are some

Open Street Map

https://tile.openstreetmap.org/{z}/{x}/{y}

Bing Maps

http://ecn.t3.tiles.virtualearth.net/tiles/a{q}.jpeg?g=1

Google Satellite Hybrid

https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}

Other map tiles can be found here, bear in mind some may require an API key.

https://www.geohowtos.com/howtos/xyz-tiles/xyz-tiles-qgis

In QGIS double click on the Map Tile you want to use, it will open with a complete map of the world, don’t bother zooming in yet.

Input Lidar/SRTM Data

The following method is the same for both Lidar and SRTM data in this instance I am using Lidar data

Drag and drop the Lidar tif image(s) (if more than one select all) into the QGIS main screen they/it will open in the Layers Panel, if more than one then they will be listed separately in the Layers Panel.

If you get a Select Transformation pop up as shown below

when adding data Select  -- Cancel this will keep the original CRS of the data otherwise distortion of the data will occur.

Right click on any of the layers and Select-  Zoom to Layer, this will zoom to the layer image with the Map Tile reference layer underneath.

If only one Layer (Lidar tif image) skip the next section

Multiple Layers

If more than one layer Select --Raster -- Miscellaneous --Build Virtual Raster

At the end of the Input Layers box select the 3 dots icon this will bring up another pop up with a list of all the layers in your Layers panel

Select the Lidar tif image(s) then Run

A new Layer will appear named Virtual with all the selected Layers merged into one image with the correct brightness/contrast

If you want to save this Virtual file for future reference, in the Layers Panel right click on the Virtual entry and Select - Export -- Save As

In the 'Save Raster Layer as' pop up

Select –Output Mode -RawData

Select the three dots at the end of the filename box to open a file browser navigate to where you want to save the file add file name then Select Save and in the Save Raster Layer pop up select OK.

     

 

Clip the image tif to the 2k (4k) size

If necessary right click the new Virtual (or Single) Layer and select Move to Top this will place the image above the Map Tile layer

Or middle mouse button select and drag to the top of the Layers panel.

Drag the 2k Area.kml saved earlier into QGIS and in the pop up Select Add Layers depending on the .kml/kmz used there may be two Layers one will have the marker colour the other will be the marker shape file it is the shape file that is required for the next step, if both shown in the Layers Panel deselect the one with the three colour bands.

Select – Raster – Extraction –Clip Raster by Extent

In the pop up panel

Input Layer - in the drop down Select  the Virtual (single) Layer

Clipping Extent --- Select the arrow on the far right, in the dropdown –Calculate from Layer and select the 2K Area shape file

Clipped Extent--- Select the Arrow far right and Save to File, browse to where you want to save this file, name the file and Save as Type *.tif

All other settings can be left as is

Select-- Run

The clipped image will not be aligned to your marker but be slightly rotated this is because the marker has a different CRS to the Virtual Layer.

This seems to only  affect data in the EPSG:27700 - OSGB36/ British National Grid format.

 

If someone a bit more QGIS savy can show how to correct this I would appreciate it. I have tried re-projecting all Layers to both the British National grid

and the default Google earth EPSG:3857 but without success and as stated previously converting from EPSG:2700 to EPSG:3857 creates artefacts in the image.

Right click the new 2k clipped layer and Select – Export – Save As --in the Save Raster Layer as.. popup

Select Output mode – Raw Data

File Name – Select three dots icon and navigate to where you want to save the file – name and save

Extent(Current:Layer) –Calculate from – in the Layer drop down Select the 2k clipped layer

Select -- OK

This .tif image will be required if creating overlays, Google Earth/and ones used for positioning (Grass/Asphalt etc).

Clip Raster by Extent ref image

You can check the new rotation alignment of the clipped area by reducing the Opacity of the layer by right clicking the layer and Select-  Properties - Transparency and adjust the transparency slider to suit (transparency will not occur until after OK is selected) and Select OK.

You will have to reset the opacity back to 100% before continuing.

 

Create the 2048 x 2048 16 bit Greyscale Dem

With the new 2k clipped layer selected, Select – Raster—Conversion – Translate (Convert Format)

Input Layer – Clipped 2k Area

Assign a specified No Data Value to output bands (optional) -- setting done by Additional command line parameter i.e.  ‘-a_nodata None ‘

Additional command Line Parameters

-a_nodata None  - This will set any blank/transparent data to black

-scale Min Max 0 65535 (Min, Max Lowest ,Highest value in the virtual Layer, 0 65535 means the whole greyscale range between 0 and 65535 will be used to represent the height values)

-outsize 2048 2048  -- this is the output image size

e.g  -a_nodata None -scale 0 264 0 65535 -outsize 2048 2048    – do not forget the – at the beginning of each command else you will get Error messages

Output Data Type – Select UInt16 in the dropdown

Converted – Select icon on the far right and in the dropdown –Save to File

Browse to where you want to save the file, Name and Save as Type PNG files (*png)

This file can then be copy/pasted into your map data folder and renamed as dem.png without any further modification.

Resultant Dem images and Giants Editor images for both the DTM and DSM data

The Grid lines in the images appear to be being caused by Giants Editor 10.0.3 (don’t show with same image in GE 9.0.6)

Whilst there is more detail in the DSM version it also means more smoothing/flattening of the terrain when placing items

 

Creating the Overlay.(Google Earth and an Image editor required)

This is just for the Google Earth overlay as in the new Giants Editor it is possible to overlay the image on to the terrain using the terrain decal setting in the Shapes Tab.

This method is for the EPSG:27700 - OSGB36/ British National Grid Lidar data -SRTM data may not require any adjustment so skip to Saving Image

Ensure Terrain is deselected in the Layers Panel

Drag drop the previously saved 2k clipped .tif into Google Earth

Select the 2k Area in the Places Panel and right click copy/paste it into the same folder rename to 2k Overlay.

Right click the 2k Overlay and Select - Properties and adjust the rotation/alignment to match the 2k clipped.tif  (I find it easier to do the rotation adjustment via the Location settings in the Google Earth –Edit Image Overlay panel which allows for finer rotation movements)

Once aligned Select- OK in the Edit Image Overlay pop up.

Marker Rotated and aligned to the 2k clipped.tif

Saving Image.

Select the Save Image Icon and zoom in to the 2k area so the marker top/bottom close to map image edges.

Deselect the 2k clipped icon in the Places Panel, if your marker has any rotation you can either adjust it in Google Earth or later in the Image Editor. (I tend to align the top marker line with the border edge)

Select - Map Options and deselect all options.

Select – Resolution Maximum and Save image in the pop up name and save in a suitable location – this will be basis for the Google Earth Satellite Overlay image.

Marker aligned to edge of Google Earth border

 

Image Editor

Any Image Editor will suffice I will be using Photoshop (crimbo present) otherwise I would still be using my free legacy version of Photoshop CS2 which does almost everything needed for Giants Editor/Farming Simulator creations with just a couple of free plugins.

https://www.techspot.com/downloads/3689-adobe-photoshop-cs2.html

Drag/drop the  Google Earth image into the image Editor

Duplicate the Google Earth image and move to the top of the Layer list, the original Background Image can now be deleted.

Cropping the image for the overlay, there are two ways of doing this

The easiest  is by using the cropping tool and adjust it so it is just inside the marker edges then crop and resize the image to the largest Height/Width value.

The second is by creating an overlay that fits inside the marker and using this to crop the image, for this you will need to have the rulers set to pixels.

Zoom in to the left hand marker edge and note the pixel vale about 20 pixels from the edge of the marker do the same for the left hand marker edge subtract the lowest from the highest and create a new image of this size and fill with colour (any), copy/paste into the Google Earth image and adjust to be inside the marker edges, when satisfied select the new image with the Magic Wand tool and select Image - Crop.

Save the cropped Google Earth Layer as GE_ Overlay.png and place it in the Overlay folder (replacing original), keep the cropped size as this is just used as a guide, you will get size errors in the log but these can be ignored.

Once the Ge_Overlay.png image and dem.png are placed in their relevant folders open Giants Editor and import the overlay.i3d into the Scenegraph adjust the height of the overlay to match the terrain height, the following image shows the result (dem is the DTM version)

Note: The image is aligned to all the terrains 'lumps/bumps' roads/cliffs etc

Should you wish to create additional overlays/markers using dirt/grass asphalt weight textures or border meshes then please refer to the Overlays section in the DEMs and Borders the Easy Way tutorial

 

 

Overlay.zip

[Eff518]

Thanks so much for this guide, or at least the bit about how to export a proper 16-bit greyscale png, took forever to find that information, when googling I mostly saw a lot of people just accepting that they couldn't get better than 8-bit resolution. And when I tried doing the conversion myself I just got spikes for some reason, but your parameters worked, although I'm not sure what's actually different.

 

My original lidar data was in a point cloud format, so had to do some extra steps to process it and convert it to a raster format (which was pretty nice, to be fair, because I could filter out vegetation and houses). If anyone needs help with that sort of thing I can maybe help (depending on how your data is set up).

[WrinkleysRule]

The spikes could be caused by enlargement of the dem in QGIS caused by the point cloud data.

There is a problem in the gdal translation between different data types and CRS's so it might have been bettwer to source something other that Point Cloud data.

As I mentioned in the tutorial I had the same sort of 'terracing' when changeing CRS.

[zacHB]

Incredibly helpful tutorial, just one question:

On 12/30/2024 at 2:04 PM, WrinkleysRule said:

-scale Min Max 0 65535 (Min, Max Lowest ,Highest value in the virtual Layer

Are you able to please clarify exactly where I can find these Min/Max values for my raster? I see you've used 0 and 264, I just put in 0 and 255 to start with and the results seem good but I'm wondering if I can be more accurate.

Thanks

[WrinkleysRule]

The Min/Max value can be found in the Layers Panel -- Band 1 (Grey )  has a min and max figure, this is the min/max height in your raster.

In the tutorial the min/max figures are taken from the VRT raster not the Clipped version as this was the full version of the dem.

You can always check to see if the heights are correct by using the method detailed in pics 9 and 10 in the Dems and borders tutorial using Google Earth and GE

[zacHB]

Got it working and learnt some things which might help some people in future:

As it turns out the heights seem to be correct for me when i use Min/Max values of 0/255, which I guess is the maximum possible range in the Layers Panel greyscale band?

For me using the min/max values shown in the raster layer instead caused the min/max values in the resulting PNG to be 0 and 65535 (black and white), which means that the height variation in Giants Editor is exaggerated (to 256m I imagine), which is why I was confused.

I may have misunderstood something but it seems to me that for accurate terrain in GE the min max should always be set to 0 255 (or 256 maybe) but I don't know how that would work if the actual height in your map varies more than 256m.

This is with QGIS 3.44.3 and the latest Giants Editor version for FS22 (not FS25), in case that makes any difference.

[WrinkleysRule]

The 0 - 65535 value refers to the number of shades of black and white in the final dem and has nothing to do with the miin max values.

Regarding height values there are two main options,

One is to adjust the height in the i3d == Terrain transform group --- heightScale="255"  (can cause problems above 355 setting)

The other is :-

If the difference in min/max  height values is greater than 255m for example i.e 37 to 560m = 523m to keep the same ratio between the min and max heights  it’s just a case of  bit of maths

i.e. min 37 max 560 – ratio  560/37 = 15.13

max hgt in GE 255/15.13  = 17

so new min/max will be min 17m,  max  255m

If the max height is above 255 you will have to adjust the heightScale value in the map i3d to the higher value (not recommended but can be a max of 355m otherwise distortion will occur) and adjust the max height figure  in the above example to suit

A minimum height of 10m is advisable  as GE does not allow any terraforming below 0m so no rivers or holes can be dug
After adjusting the min/max you may find that when you check the various heights against Google Earth and  in GE there will be a discrepancy in the readings this is due the different way each does its own height calculations but the difference between two points should be approximately the same i.e.
Google Earth point 1. 325m, point 2. 150,  
GE point.1 250, point2. 75m
diff between points 1 and 2= 175m.