4 How often do you use LSA SAF products? RegularlyOccasionallyNever used, but planto start
5 Session 2 Summary Obtaining LSA SAF products General Product InformationDocumentationFile structureQuick Visualization using HDFViewGIS tools for visualization and analysisQuantum GISGRASS GISILWISTool for georeferencing LSA SAF productsUsing programming tools – Python
6 Obtaining LSA SAF products Near Real Time Dissemination via EUMETCastDissemination on-demandProducts are available for registered usersRegistration is free of charge, but may take up to 3 working days for approvalDownloads use the FTP protocolSeveral download methods according to desired temporal extent and download settings
7 How do you obtain LSA SAF products? EUMETCastLSA SAF(website, FTP, helpdesk)Other?
9 Obtaining LSA SAF products Dissemination optionsOrdering methodsRandom slot orderingThe user can select individual time slots for several productsAutomatic disseminationLong series orderingOrdering of long time series of products.Requires human intervention and may take up to one weekDissemination methodsLSA SAF FTP serverThe order is placed in the LandSAF server for the user to downloadUser's personal FTP serverLSA SAF's system transfers the order directly to the user's FTP server
10 Product Documentation Product User Manual (PUM)Main documentation sourcePhysical phenomenaAlgorithm basisOutput data specificationsEtc.Product Output Format (POF)Details concerning description of metadata, file naming formats, etc.Validation Report (VR)Algorithm Theoretical Basis Document (ATBD)
11 Product file structure LSA SAF products use the HDF5 formatHierarchical Data Format (HDF)Groups – Containers for datasets and other groupsDatasets – Multidimensional arrays of a homogeneous typeAttributes – Metadata on groups and datasetsHDF files:Are self-describing. Complex data relationships and dependenciesCan contain binary dataSupport n-dimensional datasetsEach element in the dataset may itself be a complex object
12 Product file structure LSA SAF HDF5 files:A common set of attributes on the root groupgeneral informationA dataset for the main product (LST, DSLF, etc)Additional datasets for cell metadata (Quality flags, etc.)Each dataset may have:Specific attributes (Scaling factor, missing value, etc.)Different dimensions (Most datasets are 2D)Different data types (Most datasets are 16bit signed integers)
13 Quick visualization of products HDFView softwareJava-based HDF ViewerInspection of a HDF5's structure and dataVisualize data in spreadsheet or image modeBasic statistics, line plotting and histogram generationExport to image formats
17 When visualizing products take note: HDFViewWhen visualizing products take note:Cell values are unscaledThe 'missing value' values have no physical meaningProduct is not georeferenced (line, column coordinates only)
18 GIS tools for visualization and analysis Quantum GISUser friendly interfaceMany plugins created by the communityGRASS GISMany raster analysis modulesPython scriptingILWISDirect import of LSASAF products through the GEONETCast toolboxOther tools (ArcGIS, Mapinfo, SAGA, etc.)
19 Open source multi-platform GIS software User-friendly interface Quantum GISOpen source multi-platform GIS softwareUser-friendly interfaceReads many input formatsRaster files, Vector files, Web services, etcOn-the-fly reprojection for vector and raster dataLarge and friendly community of users and developers
20 Quantum GISIntegration of different data sources
21 Quantum GISQuick inspection of relationships between data
22 Geographic Resources Analysis Support System GRASS GISGeographic Resources Analysis Support SystemOpen source multi-platform GIS softwareHundreds of modules for raster and vector processingReads many input formatsCan read HDF5 directly (won't be georeferenced)It is preferable to georeference the HDF5 file first
24 GRASS GIS Raster calculator Allows for computing a combination of logical and algebraic expressionsExample: Calculate the arithmetic mean LST values for a specific day (using hourly sample)24 raster files each with the LSA SAF LST product for the hourly timeslotsApproach #1: Using only cells where there is enough data for the whole day(In GRASS, any math performed with NULL cells always results in a NULL value for these cellsApproach #2: Using all the cells where there is at least one value in the day
27 GRASS GIS Most commands are available in two interfaces: Graphical – Good for visualization and exploratory analysisCommand-line – Nice for automating repetitive tasks
28 Are you used to creating small scripts for automating tasks? YesNot reallyI might start using themIn the future...
29 GRASS GIS Python scripting example: Automatically import and rename 191 LST files for a given dayGRASS (world4326):~ > python>>> # example importing all the LST geotiffs into GRASS>>> import grass.script as grass>>> from glob import glob>>> import os>>> import datetime as dt>>>>>> for path in glob('/home/geo2/test_data/lsasaf/modificados/georefs/*LST*.tif'):... fName = os.path.basename(path)... nameFields=fName.split('_')... product, area = nameFields[3:5]... timeslot = dt.datetime.strptime(nameFields, '%Y%m%d%H%M')... newName = '_'.join((product, area, timeslot.strftime('%Y_%m_%d_%H_%M')))... grass.run_command('r.in.gdal', flags='oe', overwrite=True, input=path, output=newName)
30 Integrated Land and Water Information System Open Source GIS software ILWISIntegrated Land and Water Information SystemOpen Source GIS softwareAvailable only for MS Windows (as of now)Imports LSA SAF products, using the GEONETCast ToolboxSeveral modules for raster and vector processing
31 GEONETCast Toolbox GEONETCast Toolbox Available as a toolbox for ILWIS Available as a toolbox for ILWISDeveloped by ITC – University of TwenteEnables easy access to and management of GEONETCast datavarious satellite and environmental data and/or resulting productsTailored for users running GEONETCast reception stations
32 ILWIS/GEONETCast Installation Importing LSA SAF products Follow the GEONETCast Toolbox's user manualRead section 1.1 and 1.2 CAREFULLYImporting LSA SAF productsBeware of file names:Files obtained through GEONETCast (EUMETCast)S-LSA_-HDF5_LSASAF_MSG_LST_Euro_ bz2Everything will work OKFiles obtained through LSA SAF's on-demand disseminationHDF5_LSASAF_MSG_LST_Euro_ bz2GEONETCast Toolbox expects the 'S-LSA_-' prefix and will throw errorsSolutions:Uncompress → rename → recompress (time consuming and boring)Modify the GEONETCast's batch import scriptsContact LSA SAF's helpdesk and they'll send you the modified scripts
33 ILWIS/GEONETCastModular user interface, integrating the GEONETCast Toolbox
34 ILWIS/GEONETCastUse native NGP projection of the LSA SAF products
35 Have you used these GIS tools before? Yes, some of themNoI might start using themIn the future...
36 LSA SAF Product Coordinate System Normalized Geostationary Projection (NGP)Projection of the underlying MSG LRIT source dataImportant parameters for MSG satellite:Satellite height above ellipsoid: mCentral meridian: 0ºDocumented on EUMETSAT's LRIT and HRIT Global Specification
37 LSA SAF Product Coordinate System LSA SAF products hold coordinate system information in the attributes of the root groupBut HDF5 files have no standard way of defining coordinate systemsIt's hard to automatically import LSA SAF products into a Geographical Information System (GIS) software AND have the data correctly georeferenced
38 Visualization of LSA SAF products GIS tools are nice forvisualization, analysismap making, etc.HDFView is nicefor a quick inspectionMost GIS tools cannotrecognize LSA SAFcoordinate informationSome GIS toolscan open HDF5 directlyBUT ...The files will not beprojectedIt won't be possible tooverlay different dataUNLESS …
39 LSASAF Georeferencer Tool A graphical tool for georeferencing LSA SAF productsStill in-development, but already usableOpen developmentSource code hosted on a public repositoryhttps://github.com/ricardogsilva/LSASAF-georeferencerUses open source toolsPython, GDAL, Proj, Qt4Multi-platform
40 LSASAF Georeferencer Tool Convert LSASAF HDF5 files to GeoTiffGeoTiff is a Tiff based interchange format for georeferenced raster imageryReproject from the NGP projection to other coordinate systemsApply scaling factor and missing value information to the output fileOutput GeoTiff files are suitable for further processing in GIS softwareProcess multiple files simultaneouslyAlso has a CLI interface, useful for scripting
41 LSASAF Georeferencer Tool Graphical Window usage exampleSelect the file(s) to processLoad file informationChoose the dataset(s) to be processedChoose the output projectionChoose the output directoryProcess the files
42 LSASAF Georeferencer Tool Command line usage example# display usage information./georef_hdf5.py -h# process all the LST files in ~/input_dir./georef_hdf5.py -x -o ~/my_output_dir ~/input_dir/*LST*
43 Guess who is developing the LSA SAF Georeferencer Tool? Ricardo Silva(me) :)Some other dude...Contributions, bug reports, etc. are very much welcome!
44 Programming tools – Python Python(x, y)A (free) scientific and engineering development software fornumerical computations,data analysisdata visualizationBundles Python and a big set of extra modules in a single packageIncludes modules for manipulating HDF5 files
45 Programming tools – Python Some useful python modules for working with LSA SAF products:NumPyThe fundamental package needed for scientific computing with PythonProvides an N-dimensional array object and specialized data types and functionsScipyA library built to work with NumPy arraysProvides efficient numerical routines for interpolation, statistics, linear algebra, etc.Matplotlib2D plotting library which produces publication quality figuresPyTablesPackage for managing HDF5 filesDesigned to efficiently and easily cope with extremely large amounts of dataGdalA translator library for raster geospatial data formats.Has read/write functionality for many raster formats (HDF4/5, GeoTiff, GRIB, netCDF, etc.)Allows assigning georeferencing information
46 Programming tools – Python Example interactive session for manipulating a LST productpython>>> import tables>>> import numpy as np>>> import matplotlib.pyplot as plt>>> product = tables.openFile('path_to_HDF5_file')# list attributes of the root group>>> product.root._v_attrs# list attributes of the LST dataset>>> product.root.LST.attrs# import the LST dataset into a 32bit float numpy array>>> lst = np.array(product.root.LST, dtype=np.float32)# convert missing values to nan and apply scaling factor>>> lst[lst == product.root.LST.attrs['MISSING_VALUE']] = np.nan>>> lst = lst / product.root.LST.attrs['SCALING_FACTOR']# plot the data>>> plt.imshow(lst, interpolation='nearest')>>> plt.colorbar()>>> plt.show()
47 Programming tools – Python The script's output:
48 END of session 2 Thank you for attending! Enjoy the rest of the LSA SAF week
Your consent to our cookies if you continue to use this website.