A apresentação está carregando. Por favor, espere

A apresentação está carregando. Por favor, espere

Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal Tel.: (351)

Apresentações semelhantes


Apresentação em tema: "Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal Tel.: (351)"— Transcrição da apresentação:

1 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: MONITORING WEATHER CLIMATE SEISMIC ACTIVITY TO A SUSTAINABLE DEVELOPMENT AND A SAFER WORLD EUMeTrain LSA SAF Week Ricardo Silva, IM Session 2 – Messing around with the data

2 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Who am I? Ricardo Silva

3 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Where do you come from?

4 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: How often do you use LSA SAF products? Occasionally Never used, but plan to start Regularly

5 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Session 2 Summary Obtaining LSA SAF products General Product Information Documentation File structure Quick Visualization using HDFView GIS tools for visualization and analysis Quantum GIS GRASS GIS ILWIS Tool for georeferencing LSA SAF products Using programming tools – Python

6 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Obtaining LSA SAF products Near Real Time Dissemination via EUMETCast Dissemination on-demand Products are available for registered users Registration is free of charge, but may take up to 3 working days for approval Downloads use the FTP protocol Several download methods according to desired temporal extent and download settings

7 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: How do you obtain LSA SAF products? LSA SAF (website, FTP, helpdesk) Other? EUMETCast

8 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL:

9 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Obtaining LSA SAF products Dissemination options Ordering methods Random slot ordering The user can select individual time slots for several products Automatic dissemination Long series ordering Ordering of long time series of products. Requires human intervention and may take up to one week Dissemination methods LSA SAF FTP server The order is placed in the LandSAF server for the user to download User's personal FTP server LSA SAF's system transfers the order directly to the user's FTP server

10 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Product Documentation Product User Manual (PUM) Main documentation source Physical phenomena Algorithm basis Output data specifications Etc. Product Output Format (POF) Details concerning description of metadata, file naming formats, etc. Validation Report (VR) Algorithm Theoretical Basis Document (ATBD)

11 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Product file structure LSA SAF products use the HDF5 format Hierarchical Data Format (HDF) Groups – Containers for datasets and other groups Datasets – Multidimensional arrays of a homogeneous type Attributes – Metadata on groups and datasets HDF files: Are self-describing. Complex data relationships and dependencies Can contain binary data Support n-dimensional datasets Each element in the dataset may itself be a complex object

12 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Product file structure LSA SAF HDF5 files: A common set of attributes on the root group general information A dataset for the main product (LST, DSLF, etc) Additional datasets for cell metadata (Quality flags, etc.) Each dataset may have: Specific attributes (Scaling factor, missing value, etc.) Different dimensions (Most datasets are 2D) Different data types (Most datasets are 16bit signed integers)

13 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Quick visualization of products HDFView software Java-based HDF Viewer Inspection of a HDF5's structure and data Visualize data in spreadsheet or image mode Basic statistics, line plotting and histogram generation Export to image formats

14 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Have you used HDFView before? No HDFWhat? Yes

15 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: HDFView 1. Datasets 2.Attributes 3.Table View 4.Table options 5.Image view 6.Image options 7.Pixel values

16 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: HDFView Inspecting file and dataset attributes

17 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: HDFView When visualizing products take note: 1.Cell values are unscaled 2.The 'missing value' values have no physical meaning 3.Product is not georeferenced (line, column coordinates only)

18 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GIS tools for visualization and analysis Quantum GIS User friendly interface Many plugins created by the community GRASS GIS Many raster analysis modules Python scripting ILWIS Direct import of LSASAF products through the GEONETCast toolbox Other tools (ArcGIS, Mapinfo, SAGA, etc.)

19 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Quantum GIS Open source multi-platform GIS software User-friendly interface Reads many input formats Raster files, Vector files, Web services, etc On-the-fly reprojection for vector and raster data Large and friendly community of users and developers

20 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Quantum GIS Integration of different data sources

21 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Quantum GIS Quick inspection of relationships between data

22 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GRASS GIS Geographic Resources Analysis Support System Open source multi-platform GIS software Hundreds of modules for raster and vector processing Reads many input formats Can read HDF5 directly (won't be georeferenced) It is preferable to georeference the HDF5 file first

23 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GRASS GIS Modular and flexible user interface

24 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Raster calculator Allows for computing a combination of logical and algebraic expressions Example: Calculate the arithmetic mean LST values for a specific day (using hourly sample) 24 raster files each with the LSA SAF LST product for the hourly timeslots GRASS GIS Approach #1: Using only cells where there is enough data for the whole day (In GRASS, any math performed with NULL cells always results in a NULL value for these cells Approach #2: Using all the cells where there is at least one value in the day

25 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GRASS GIS Approach #1

26 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GRASS GIS Approach #2

27 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GRASS GIS Most commands are available in two interfaces: Graphical – Good for visualization and exploratory analysis Command-line – Nice for automating repetitive tasks

28 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Are you used to creating small scripts for automating tasks? Not really I might start using them In the future... Yes

29 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Python scripting example: Automatically import and rename 191 LST files for a given day GRASS GIS GRASS (world4326):~ > python >>> # example importing all the LST geotiffs into GRASS >>> import grass.script as grass >>> from glob import glob >>> import os >>> import datetime as dt >>> >>> for path in glob('/home/geo2/test_data/lsasaf/modificados/georefs/*LST*.tif'):... fName = os.path.basename(path)... nameFields=fName.split('_')... product, area = nameFields[3:5]... timeslot = dt.datetime.strptime(nameFields[5], '%Y%m%d%H%M')... newName = '_'.join((product, area, timeslot.strftime('%Y_%m_%d_%H_%M')))... grass.run_command('r.in.gdal', flags='oe', overwrite=True, input=path, output=newName)

30 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: ILWIS Integrated Land and Water Information System Open Source GIS software Available only for MS Windows (as of now) Imports LSA SAF products, using the GEONETCast Toolbox Several modules for raster and vector processing

31 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: GEONETCast Toolbox Available as a toolbox for ILWIS Developed by ITC – University of Twente Enables easy access to and management of GEONETCast data various satellite and environmental data and/or resulting products Tailored for users running GEONETCast reception stations

32 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: ILWIS/GEONETCast Installation Follow the GEONETCast Toolbox's user manual Read section 1.1 and 1.2 CAREFULLY Importing LSA SAF products Beware of file names: Files obtained through GEONETCast (EUMETCast) S-LSA_-HDF5_LSASAF_MSG_LST_Euro_ bz2 Everything will work OK Files obtained through LSA SAF's on-demand dissemination HDF5_LSASAF_MSG_LST_Euro_ bz2 GEONETCast Toolbox expects the 'S-LSA_-' prefix and will throw errors Solutions: Uncompress rename recompress (time consuming and boring) Modify the GEONETCast's batch import scripts Contact LSA SAF's helpdesk and they'll send you the modified scripts

33 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: ILWIS/GEONETCast Modular user interface, integrating the GEONETCast Toolbox

34 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: ILWIS/GEONETCast Use native NGP projection of the LSA SAF products

35 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Have you used these GIS tools before? No I might start using them In the future... Yes, some of them

36 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: LSA SAF Product Coordinate System Normalized Geostationary Projection (NGP) Projection of the underlying MSG LRIT source data Important parameters for MSG satellite: Satellite height above ellipsoid: m Central meridian: 0º Documented on EUMETSAT's LRIT and HRIT Global Specification

37 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: LSA SAF Product Coordinate System LSA SAF products hold coordinate system information in the attributes of the root group But HDF5 files have no standard way of defining coordinate systems It's hard to automatically import LSA SAF products into a Geographical Information System (GIS) software AND have the data correctly georeferenced

38 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Visualization of LSA SAF products HDFView is nice for a quick inspection Some GIS tools can open HDF5 directly BUT... GIS tools are nice for visualization, analysis map making, etc. Most GIS tools cannot recognize LSA SAF coordinate information The files will not be projected It won't be possible to overlay different data UNLESS …

39 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: LSASAF Georeferencer Tool A graphical tool for georeferencing LSA SAF products Still in-development, but already usable Open development Source code hosted on a public repository https://github.com/ricardogsilva/LSASAF-georeferencer Uses open source tools Python, GDAL, Proj, Qt4 Multi-platform

40 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Convert LSASAF HDF5 files to GeoTiff GeoTiff is a Tiff based interchange format for georeferenced raster imagery Reproject from the NGP projection to other coordinate systems Apply scaling factor and missing value information to the output file Output GeoTiff files are suitable for further processing in GIS software Process multiple files simultaneously Also has a CLI interface, useful for scripting LSASAF Georeferencer Tool

41 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: LSASAF Georeferencer Tool Graphical Window usage example 1.Select the file(s) to process 2.Load file information 3.Choose the dataset(s) to be processed 4.Choose the output projection 5.Choose the output directory 6.Process the files

42 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: LSASAF Georeferencer Tool Command line usage example # display usage information -h # process all the LST files in ~/input_dir -x -o ~/my_output_dir ~/input_dir/*LST*

43 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Guess who is developing the LSA SAF Georeferencer Tool? Some other dude... Ricardo Silva (me) :) Contributions, bug reports, etc. are very much welcome!

44 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Programming tools – Python Python(x, y) A (free) scientific and engineering development software for numerical computations, data analysis data visualization Bundles Python and a big set of extra modules in a single package Includes modules for manipulating HDF5 files

45 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Programming tools – Python Some useful python modules for working with LSA SAF products: NumPy The fundamental package needed for scientific computing with Python Provides an N-dimensional array object and specialized data types and functions Scipy A library built to work with NumPy arrays Provides efficient numerical routines for interpolation, statistics, linear algebra, etc. Matplotlib 2D plotting library which produces publication quality figures PyTables Package for managing HDF5 files Designed to efficiently and easily cope with extremely large amounts of data Gdal A translator library for raster geospatial data formats. Has read/write functionality for many raster formats (HDF4/5, GeoTiff, GRIB, netCDF, etc.) Allows assigning georeferencing information

46 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Programming tools – Python python >>> import tables >>> import numpy as np >>> import matplotlib.pyplot as plt >>> product = tables.openFile('path_to_HDF5_file') # list attributes of the root group >>> product.root._v_attrs # list attributes of the LST dataset >>> product.root.LST.attrs # import the LST dataset into a 32bit float numpy array >>> lst = np.array(product.root.LST, dtype=np.float32) # convert missing values to nan and apply scaling factor >>> lst[lst == product.root.LST.attrs['MISSING_VALUE']] = np.nan >>> lst = lst / product.root.LST.attrs['SCALING_FACTOR'] # plot the data >>> plt.imshow(lst, interpolation='nearest') >>> plt.colorbar() >>> plt.show() Example interactive session for manipulating a LST product

47 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: Programming tools – Python The script's output:

48 Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa Lisboa – Portugal Tel.: (351) Fax: (351) URL: END of session 2 Thank you for attending! Enjoy the rest of the LSA SAF week


Carregar ppt "Ministério da Ciência, Tecnologia e Ensino Superior Instituto de Meteorologia, I. P. Rua C – Aeroporto de Lisboa 1749-077 Lisboa – Portugal Tel.: (351)"

Apresentações semelhantes


Anúncios Google