diff --git a/CHANGELOG.md b/CHANGELOG.md index db03be450..e4567609b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,13 @@ - Only versions 2.0.0 and newer can be upgraded to this version. For older versions, please upgrade to 2.0.0 first. ### Migrations and checks #### Schema migrations +- [#1185](https://github.com/LayerManager/layman/issues/1185) Add new text column `file_path` in `publications` table in prime DB schema. Add constraint that `file_path` can be non-null only when `geodata_type` is `raster`. #### Data migrations ### Changes - [#1168](https://github.com/LayerManager/layman/issues/1168) Extend [PATCH Workspace Layer](doc/rest.md#patch-workspace-layer) with ability of appending data to existing time-series layer. - When publishing a layer or map to Micka via CSW, Layman sends the creating user (Layman username) in the SOAP request header (`CreateUser`), so the metadata record in Micka is associated with the user who created the publication. +- [#1185](https://github.com/LayerManager/layman/issues/1185) POST Workspace [Layers](doc/rest.md#post-workspace-layers) supports import of raster layers from an existing server-side directory via the file_path parameter, including ImageMosaic timeseries layers. +- [#1185](https://github.com/LayerManager/layman/issues/1185)[GET Workspace Layer](doc/rest.md#get-workspace-layer) returns `file_path` key for raster layers published using this parameter. ## v2.3.0 2025-12-02 diff --git a/doc/rest.md b/doc/rest.md index a67b77e73..6fc48c33f 100644 --- a/doc/rest.md +++ b/doc/rest.md @@ -107,6 +107,7 @@ Processing chain consists of few steps: - for vector layers import vector file (if sent) to PostgreSQL database as new table into workspace schema - files with invalid byte sequence are first converted to GeoJSON, then cleaned with iconv, and finally imported to database. - for raster layers normalize and compress raster file to GeoTIFF with overviews (pyramids); NoData values are normalized as transparent +- if `file_path` parameter is used raster files are used directly from GeoServer data directory without normalization - save bounding box into PostgreSQL - for vector layers publish the vector table as new layer (feature type) within appropriate WFS workspace of GeoServer - for vector layers @@ -115,7 +116,7 @@ Processing chain consists of few steps: - for layers with QML style: - create QGS file on QGIS server filesystem with appropriate style - publish the layer on GeoServer through WMS cascade from QGIS server -- for raster layers publish normalized GeoTIFF as new layer (coverage) on GeoServer WMS workspace +- for raster layers publish GeoTIFF as new layer (coverage) on GeoServer WMS workspace (normalized if `file` parameter is used, original if `file_path` parameter is used) - generate thumbnail image - publish metadata record to Micka (it's public if and only if read access is set to EVERYONE; the creating user is sent as CreateUser in the CSW SOAP request) - update thumbnail of each [map](models.md#map) that points to this layer @@ -140,7 +141,7 @@ Body parameters: - used if specified, otherwise generated - it's meant mostly for testing purposes - *file*, file(s) or file name(s) - - exactly one of `file` or `external_table_uri` must be set + - exactly one of `file`, `file_path`, or `external_table_uri` must be set - one of following options is expected: - GeoJSON file - ShapeFile files (at least three files: .shp, .shx, .dbf) @@ -162,8 +163,24 @@ Body parameters: - if published file has empty bounding box (i.e. no features), its bounding box on WMS/WFS endpoint is set to the whole World - attribute names are [laundered](https://gdal.org/en/stable/drivers/vector/pg.html#layer-creation-options) to be safely stored in DB - if QML style is used in this request, it must list all attributes contained in given data file +- *file_path*, string + - exactly one of `file`, `file_path`, or `external_table_uri` must be set + - relative path to a directory that already exists on the server + - the path must be relative to the root of the GeoServer data directory + - the referenced directory must be physically located inside the GeoServer data directory + - the directory must contain at least one GeoTIFF file (with extension `.tif` or `.tiff`) + + - for raster layers: + - supported only for GeoTIFF files (`.tif` or `.tiff` extension) + - raster files are not normalized when using `file_path` parameter + - this may result in different styling behavior compared to layers published via `file` parameter + - may point to a directory containing a single raster file (published as a single coverage) + - may point to a directory containing multiple raster files: + - if `time_regex` parameter is provided, files are treated as a time series and published as an ImageMosaic + - if `time_regex` parameter is not provided and directory contains multiple raster files, an error is raised + - *external_table_uri*, string - - exactly one of `file` or `external_table_uri` must be set + - exactly one of `file`, `file_path`, or `external_table_uri` must be set - [connection URI](https://www.postgresql.org/docs/15/libpq-connect.html#id-1.7.3.8.3.6) is required, usual format is `postgresql://:@:/?schema=&table=&geo_column=` - `host` part and query parameters `schema` and `table` are mandatory - URI scheme is required to be `postgresql` @@ -329,6 +346,7 @@ JSON object with following structure: - *status*: Status information about publishing style. See [GET Workspace Layer](#get-workspace-layer) **wms** property for meaning. - *error*: If status is FAILURE, this may contain error object. - **original_data_source**: String. Either `file` if layer was published from file, or `database_table` if layer was published from external database table +- **file_path**: String. Available only for raster layers published using `file_path` parameter. Relative path to the directory containing raster files, relative to the root of the GeoServer data directory. - *metadata* - *identifier*: String. Identifier of metadata record in CSW instance. - *record_url*: String. URL of metadata record accessible by web browser, probably with some editing capabilities. diff --git a/src/layman/common/prime_db_schema/publications.py b/src/layman/common/prime_db_schema/publications.py index 76da8f6f7..17c1e2a37 100644 --- a/src/layman/common/prime_db_schema/publications.py +++ b/src/layman/common/prime_db_schema/publications.py @@ -173,6 +173,7 @@ def get_publication_infos_with_metainfo(workspace_name=None, pub_type=None, *, ST_YMAX(p.bbox) as ymax, p.srid as srid, PGP_SYM_DECRYPT(p.external_table_uri, p.uuid::text)::json external_table_uri, + p.file_path, (select rtrim(concat(case when u.id is not null then w.name || ',' end, string_agg(COALESCE(w2.name, r.role_name), ',' ORDER BY COALESCE(w2.name, r.role_name)) || ',', case when p.everyone_can_read then %s || ',' end @@ -308,10 +309,11 @@ def get_publication_infos_with_metainfo(workspace_name=None, pub_type=None, *, 'used_in_maps': layer_maps or [], '_wfs_wms_status': settings.EnumWfsWmsStatus(wfs_wms_status) if wfs_wms_status else None, '_is_public_workspace': is_public_workspace, + 'file_path': file_path, } for id_publication, workspace_name, publication_type, publication_name, title, description, uuid, geodata_type, style_type, image_mosaic, updated_at, created_at, xmin, ymin, xmax, ymax, - srid, external_table_uri, read_users_roles, write_users_roles, map_layers, layer_maps, wfs_wms_status, is_public_workspace, _ + srid, external_table_uri, file_path, read_users_roles, write_users_roles, map_layers, layer_maps, wfs_wms_status, is_public_workspace, _ in values} infos = {key: {**value, @@ -501,8 +503,8 @@ def insert_publication(workspace_name, info): check_publication_info(workspace_name, info) insert_publications_sql = f'''insert into {DB_SCHEMA}.publications as p - (id_workspace, name, title, description, type, uuid, style_type, geodata_type, everyone_can_read, everyone_can_write, updated_at, image_mosaic, external_table_uri, wfs_wms_status) values - (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, current_timestamp, %s, PGP_SYM_ENCRYPT(%s::text, %s::text), %s ) + (id_workspace, name, title, description, type, uuid, style_type, geodata_type, everyone_can_read, everyone_can_write, updated_at, image_mosaic, external_table_uri, file_path, wfs_wms_status) values + (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, current_timestamp, %s, PGP_SYM_ENCRYPT(%s::text, %s::text), %s, %s ) returning id ;''' @@ -527,6 +529,7 @@ def insert_publication(workspace_name, info): info.get("image_mosaic"), external_table_uri, info.get("uuid"), + info.get("file_path"), info.get("wfs_wms_status") ) pub_id = db_util.run_query(insert_publications_sql, data)[0][0] @@ -597,6 +600,7 @@ def update_publication(workspace_name, info, is_part_of_user_delete=False): updated_at = current_timestamp, image_mosaic = coalesce(%s, image_mosaic), external_table_uri = PGP_SYM_ENCRYPT(%s::text, uuid::text), + file_path = coalesce(%s, file_path), geodata_type = coalesce(%s, geodata_type) where id_workspace = %s and name = %s @@ -611,6 +615,7 @@ def update_publication(workspace_name, info, is_part_of_user_delete=False): access_rights_changes['write']['EVERYONE'], info.get("image_mosaic"), external_table_uri, + info.get("file_path"), info.get("geodata_type"), id_workspace, info.get("name"), @@ -671,11 +676,12 @@ def set_bbox(workspace, publication_type, publication, bbox, crs, ): def set_geodata_type(workspace, publication_type, publication, geodata_type, ): query = f'''update {DB_SCHEMA}.publications set - geodata_type = %s + geodata_type = %s, + file_path = CASE WHEN %s = %s THEN NULL ELSE file_path END where type = %s and name = %s and id_workspace = (select w.id from {DB_SCHEMA}.workspaces w where w.name = %s);''' - params = (geodata_type, publication_type, publication, workspace,) + params = (geodata_type, geodata_type, settings.GEODATA_TYPE_UNKNOWN, publication_type, publication, workspace,) db_util.run_statement(query, params) diff --git a/src/layman/layer/__init__.py b/src/layman/layer/__init__.py index ea1fc4359..bc6386112 100644 --- a/src/layman/layer/__init__.py +++ b/src/layman/layer/__init__.py @@ -68,7 +68,7 @@ def get_layer_patch_keys(): ('layman.layer.prime_db_schema.table', InternalSourceTypeDef(info_items=[ 'access_rights', 'name', 'workspace', 'title', 'uuid', 'bounding_box', 'style_type', 'native_crs', 'native_bounding_box', 'geodata_type', 'updated_at', 'id', 'type', 'image_mosaic', 'table_uri', - 'original_data_source', 'wfs_wms_status', 'used_in_maps', 'description', 'created_at', 'is_public_workspace', ]),), + 'original_data_source', 'wfs_wms_status', 'used_in_maps', 'description', 'created_at', 'is_public_workspace', 'file_path', ]),), ('layman.layer.filesystem.input_chunk', InternalSourceTypeDef(info_items=['file', ]),), ('layman.layer.filesystem.input_file', InternalSourceTypeDef(info_items=['file', ]),), ('layman.layer.filesystem.input_style', InternalSourceTypeDef(info_items=[]),), @@ -128,7 +128,7 @@ def get_layer_patch_keys(): settings.GEODATA_TYPE_RASTER: { 'name', 'uuid', 'layman_metadata', 'url', 'title', 'description', 'updated_at', 'wms', 'thumbnail', 'file', 'metadata', 'style', 'access_rights', 'bounding_box', 'native_crs', 'native_bounding_box', 'image_mosaic', - 'original_data_source', 'geodata_type', 'used_in_maps', + 'original_data_source', 'geodata_type', 'used_in_maps', 'file_path', }, settings.GEODATA_TYPE_UNKNOWN: { 'name', 'uuid', 'layman_metadata', 'url', 'title', 'description', 'updated_at', 'wms', 'thumbnail', 'file', 'metadata', diff --git a/src/layman/layer/filesystem/gdal.py b/src/layman/layer/filesystem/gdal.py index f60dd8c96..9c47f8ecb 100644 --- a/src/layman/layer/filesystem/gdal.py +++ b/src/layman/layer/filesystem/gdal.py @@ -21,8 +21,63 @@ def get_layer_info(workspace, layername, *, extra_keys=None): return get_layer_info_by_uuid(publ_uuid, extra_keys=extra_keys) if publ_uuid else {} +def to_gs_path(path): + if path.startswith(settings.GEOSERVER_DATADIR): + return os.path.relpath(path, settings.GEOSERVER_DATADIR) + return path + + +def add_file_extra_keys(file_dict, gdal_path, gdal_paths_for_stats, extra_keys, normalized_gdal_path=None): + if '_file.color_interpretations' in extra_keys: + file_dict['color_interpretations'] = get_color_interpretations(gdal_path) + if '_file.mask_flags' in extra_keys: + file_dict['mask_flags'] = get_mask_flags(gdal_path) + norm_file_dict = {} + if '_file.normalized_file.stats' in extra_keys: + norm_file_dict['stats'] = get_file_list_statistics(gdal_paths_for_stats) + normalized_path = normalized_gdal_path + if normalized_path is None: + normalized_path = gdal_paths_for_stats[0] if isinstance(gdal_paths_for_stats, list) and len(gdal_paths_for_stats) > 0 else gdal_path + if '_file.normalized_file.mask_flags' in extra_keys: + norm_file_dict['mask_flags'] = get_mask_flags(normalized_path) + if '_file.normalized_file.color_interpretations' in extra_keys: + norm_file_dict['color_interpretations'] = get_color_interpretations(normalized_path) + if '_file.normalized_file.nodata_value' in extra_keys: + norm_file_dict['nodata_value'] = get_nodata_value(normalized_path) + if norm_file_dict: + file_dict['normalized_file'] = norm_file_dict + + +def get_file_path_layer_info(file_path_info_list, extra_keys): + file_path_info = file_path_info_list[0] + gdal_path = file_path_info['gdal'] + paths_dict = { + os.path.basename(info['gdal']): { + 'normalized_absolute': info['absolute'], + 'normalized_geoserver': to_gs_path(info['gdal']), + } + for info in file_path_info_list + } + + result = { + '_file': { + 'paths': paths_dict, + }, + } + file_dict = result['_file'] + gdal_paths_for_stats = [info['gdal'] for info in file_path_info_list] + normalized_gdal_path = gdal_paths_for_stats[0] if gdal_paths_for_stats else gdal_path + add_file_extra_keys(file_dict, gdal_path, gdal_paths_for_stats, extra_keys, normalized_gdal_path=normalized_gdal_path) + return result + + def get_layer_info_by_uuid(publ_uuid, *, extra_keys=None): extra_keys = extra_keys or [] + file_path_info_list = input_file.get_file_path_info(publ_uuid) + + if file_path_info_list: + return get_file_path_layer_info(file_path_info_list, extra_keys) + gdal_paths = get_normalized_raster_layer_main_filepaths(publ_uuid) gs_directory = get_normalized_raster_layer_dir(publ_uuid, geoserver=True) result = {} @@ -43,25 +98,8 @@ def get_layer_info_by_uuid(publ_uuid, *, extra_keys=None): input_file_info = input_file.get_layer_info_by_uuid(publ_uuid) result['_file']['file_type'] = input_file_info['_file']['file_type'] input_file_gdal_path = next(iter(input_file_info['_file']['paths'].values()))['gdal'] - if '_file.color_interpretations' in extra_keys: - file_dict['color_interpretations'] = get_color_interpretations(input_file_gdal_path) - if '_file.mask_flags' in extra_keys: - file_dict['mask_flags'] = get_mask_flags(input_file_gdal_path) - norm_file_dict = {} - if '_file.normalized_file.stats' in extra_keys: - stats = get_file_list_statistics(gdal_paths) - norm_file_dict['stats'] = stats - if '_file.normalized_file.mask_flags' in extra_keys: - mask_flags = get_mask_flags(gdal_paths[0]) - norm_file_dict['mask_flags'] = mask_flags - if '_file.normalized_file.color_interpretations' in extra_keys: - color_interpretations = get_color_interpretations(gdal_paths[0]) - norm_file_dict['color_interpretations'] = color_interpretations - if '_file.normalized_file.nodata_value' in extra_keys: - nodata_value = get_nodata_value(gdal_paths[0]) - norm_file_dict['nodata_value'] = nodata_value - if norm_file_dict: - file_dict['normalized_file'] = norm_file_dict + normalized_gdal_path = gdal_paths[0] if len(gdal_paths) > 0 else input_file_gdal_path + add_file_extra_keys(file_dict, input_file_gdal_path, gdal_paths, extra_keys, normalized_gdal_path=normalized_gdal_path) return result @@ -461,14 +499,28 @@ def get_bbox_from_files(filepaths): return result +def get_layer_filepaths(publ_uuid): + file_path_info_list = input_file.get_file_path_info(publ_uuid) + if file_path_info_list: + return [info['gdal'] for info in file_path_info_list] + return get_normalized_raster_layer_main_filepaths(publ_uuid) + + +def get_layer_filepath(publ_uuid): + file_path_info_list = input_file.get_file_path_info(publ_uuid) + if file_path_info_list: + return file_path_info_list[0]['gdal'] + return get_normalized_raster_layer_main_filepaths(publ_uuid)[0] + + def get_bbox(publ_uuid): - filepaths = get_normalized_raster_layer_main_filepaths(publ_uuid) + filepaths = get_layer_filepaths(publ_uuid) result = get_bbox_from_files(filepaths) return result def get_crs(publ_uuid): - filepath = get_normalized_raster_layer_main_filepaths(publ_uuid)[0] + filepath = get_layer_filepath(publ_uuid) data = open_raster_file(filepath, gdalconst.GA_ReadOnly) spatial_reference = osr.SpatialReference(wkt=data.GetProjection()) auth_name = spatial_reference.GetAttrValue('AUTHORITY') @@ -478,7 +530,7 @@ def get_crs(publ_uuid): def get_normalized_ground_sample_distance_in_m(publ_uuid, *, bbox_size): - filepath = get_normalized_raster_layer_main_filepaths(publ_uuid)[0] + filepath = get_layer_filepath(publ_uuid) raster_size = get_raster_size(filepath) pixel_size = [bbox_size / raster_size[idx] for (idx, bbox_size) in enumerate(bbox_size)] distance_value = sum(pixel_size) / len(pixel_size) diff --git a/src/layman/layer/filesystem/input_file.py b/src/layman/layer/filesystem/input_file.py index 0b6a5b494..fcebe18f2 100644 --- a/src/layman/layer/filesystem/input_file.py +++ b/src/layman/layer/filesystem/input_file.py @@ -11,8 +11,9 @@ from layman import settings, patch_mode from layman.common import empty_method, empty_method_returns_dict from layman.common.filesystem import input_file as common +from layman.common.prime_db_schema import publications as pubs_util from . import util, gdal as fs_gdal -from .. import LAYER_TYPE +from .. import LAYER_TYPE, util as layer_util from ..layer_class import Layer from ...util import get_publication_uuid @@ -62,35 +63,78 @@ def get_layer_info(workspace, layername): return get_layer_info_by_uuid(publ_uuid) if publ_uuid else {} -def get_layer_info_by_uuid(publ_uuid): - input_files = get_layer_input_files(publ_uuid) +def is_file_path_layer(publ_uuid): + if not publ_uuid: + return False + infos = pubs_util.get_publication_infos(uuid=publ_uuid) + if not infos: + return False + info = list(infos.values())[0] + return info.get('file_path') is not None - if input_files.saved_paths: - # input_files.raw_or_archived_main_file_path is None if user sent ZIP file by chunks without main file inside - main_file_path = input_files.raw_or_archived_main_file_path or input_files.saved_paths[0] - rel_main_filepath = os.path.relpath(main_file_path, settings.LAYMAN_DATA_DIR) - main_files = input_files.raw_or_archived_main_file_paths or input_files.saved_paths - file_type = get_file_type(rel_main_filepath) - result = { - 'file': { - 'paths': [os.path.relpath(filepath, settings.LAYMAN_DATA_DIR) for filepath in main_files], - }, + +def get_file_path_info(publ_uuid): + if not publ_uuid: + return None + infos = pubs_util.get_publication_infos(uuid=publ_uuid) + if not infos: + return None + info = list(infos.values())[0] + file_path_relative = info.get('file_path') + if not file_path_relative: + return None + + abs_path = os.path.join(settings.GEOSERVER_DATADIR, file_path_relative) + + if not os.path.isdir(abs_path): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Path is not a directory', + 'expected': 'Relative path to directory containing raster files', + 'found': file_path_relative, + }) + + tifs = layer_util.get_geotiff_files(abs_path) + if not tifs: + return None + return [{'absolute': tif, 'gdal': tif, 'file_path': file_path_relative} for tif in tifs] + + +def get_layer_info_by_uuid(publ_uuid): + file_path_info_list = get_file_path_info(publ_uuid) + if file_path_info_list: + return { + 'file': {'paths': [os.path.relpath(info['gdal'], settings.GEOSERVER_DATADIR) for info in file_path_info_list]}, '_file': { - 'file_type': file_type, + 'file_type': get_file_type(file_path_info_list[0]['gdal']), 'paths': { - slugify_timeseries_filename(os.path.splitext(os.path.basename(main_file))[0]) if len(main_files) > 1 else publ_uuid: - { - 'absolute': main_file, - 'gdal': main_file if input_files.archive_type is None - else settings.COMPRESSED_FILE_EXTENSIONS[input_files.archive_type] + main_file, - } - for main_file in main_files + os.path.basename(info['gdal']): {'absolute': info['absolute'], 'gdal': info['gdal']} + for info in file_path_info_list }, }, } - else: - result = {} - return result + + input_files = get_layer_input_files(publ_uuid) + if not input_files.saved_paths: + return {} + + main = input_files.raw_or_archived_main_file_path or input_files.saved_paths[0] + main_files = input_files.raw_or_archived_main_file_paths or input_files.saved_paths + + return { + 'file': {'paths': [os.path.relpath(p, settings.LAYMAN_DATA_DIR) for p in main_files]}, + '_file': { + 'file_type': get_file_type(os.path.relpath(main, settings.LAYMAN_DATA_DIR)), + 'paths': { + slugify_timeseries_filename(os.path.splitext(os.path.basename(p))[0]) if len(main_files) > 1 else publ_uuid: + { + 'absolute': p, + 'gdal': p if input_files.archive_type is None else settings.COMPRESSED_FILE_EXTENSIONS[input_files.archive_type] + p + } + for p in main_files + }, + }, + } def get_all_main_file_names(filenames): diff --git a/src/layman/layer/filesystem/tasks.py b/src/layman/layer/filesystem/tasks.py index f6e1976fc..befc21731 100644 --- a/src/layman/layer/filesystem/tasks.py +++ b/src/layman/layer/filesystem/tasks.py @@ -69,7 +69,10 @@ def refresh_input_chunk(self, workspace, layername, *, uuid, check_crs=True, ove publ_info = layman_util.get_publication_info(workspace, LAYER_TYPE, layername, context={'keys': ['file']}) main_filepaths = list(path['gdal'] for path in publ_info['_file']['paths'].values()) - input_file.check_main_files(main_filepaths, check_crs=check_crs, overview_resampling=overview_resampling) + + is_file_path = input_file.is_file_path_layer(uuid) + if not is_file_path: + input_file.check_main_files(main_filepaths, check_crs=check_crs, overview_resampling=overview_resampling) file_type = input_file.get_file_type(input_files.raw_or_archived_main_file_path) if enable_more_main_files and file_type == settings.GEODATA_TYPE_VECTOR: @@ -116,6 +119,10 @@ def finish_gdal_process(process): if file_type != settings.GEODATA_TYPE_RASTER: return + is_file_path = input_file.is_file_path_layer(uuid) + if is_file_path: + return + gdal.ensure_normalized_raster_layer_dir(uuid) if self.is_aborted(): diff --git a/src/layman/layer/geoserver/tasks.py b/src/layman/layer/geoserver/tasks.py index f9f0cb4ed..8e3ddce6b 100644 --- a/src/layman/layer/geoserver/tasks.py +++ b/src/layman/layer/geoserver/tasks.py @@ -98,9 +98,14 @@ def refresh_wms( source_file_or_dir = gs_file_path else: coverage_store_name = wms.get_image_mosaic_store_name(uuid=layer.uuid) - source_file_or_dir = os.path.dirname(gs_file_path) file_path = file_paths['normalized_absolute'] - dir_path = os.path.dirname(file_path) + gs_file_path_rel = file_paths['normalized_geoserver'] + if os.path.isdir(file_path): + source_file_or_dir = gs_file_path_rel + dir_path = file_path + else: + source_file_or_dir = os.path.dirname(gs_file_path_rel) + dir_path = os.path.dirname(file_path) is_append = existing_input_file_names is not None and len(existing_input_file_names) > 0 if is_append: diff --git a/src/layman/layer/prime_db_schema/table.py b/src/layman/layer/prime_db_schema/table.py index 995c3abe4..7ac61403c 100644 --- a/src/layman/layer/prime_db_schema/table.py +++ b/src/layman/layer/prime_db_schema/table.py @@ -85,6 +85,7 @@ def post_layer(workspace, image_mosaic, external_table_uri, style_type=None, + file_path=None, ): db_info = {"name": layername, "title": title, @@ -97,6 +98,7 @@ def post_layer(workspace, 'style_type': style_type.code if style_type else None, 'image_mosaic': image_mosaic, 'external_table_uri': external_table_uri, + 'file_path': file_path, 'wfs_wms_status': settings.EnumWfsWmsStatus.PREPARING.value, } pubs_util.insert_publication(workspace, db_info) diff --git a/src/layman/layer/rest_workspace_layers.py b/src/layman/layer/rest_workspace_layers.py index 15ce30675..53977a5c1 100644 --- a/src/layman/layer/rest_workspace_layers.py +++ b/src/layman/layer/rest_workspace_layers.py @@ -1,3 +1,4 @@ +import os from flask import Blueprint, jsonify, request, g from flask import current_app as app @@ -75,57 +76,59 @@ def post(workspace): }}) check_crs = crs_id is None + file_path = request.form.get('file_path', '').strip() + # EXTERNAL_TABLE_URI external_table_uri_str = request.form.get('external_table_uri', '') - if not input_files and not external_table_uri_str: + if not input_files and not external_table_uri_str and not file_path: raise LaymanError(1, { - 'parameters': ['file', 'external_table_uri'], - 'message': 'Both `file` and `external_table_uri` parameters are empty', + 'parameters': ['file', 'external_table_uri', 'file_path'], + 'message': 'All parameters `file`, `external_table_uri`, and `file_path` are empty', 'expected': 'One of the parameters is filled.', }) - if input_files and external_table_uri_str: + filled_params = [] + if input_files: + filled_params.append('file') + if external_table_uri_str: + filled_params.append('external_table_uri') + if file_path: + filled_params.append('file_path') + if len(filled_params) > 1: raise LaymanError(48, { - 'parameters': ['file', 'external_table_uri'], - 'message': 'Both `file` and `external_table_uri` parameters are filled', + 'parameters': filled_params, + 'message': f'Multiple parameters are filled: {", ".join(filled_params)}', 'expected': 'Only one of the parameters is fulfilled.', 'found': { - 'file': input_files.raw_paths, - 'external_table_uri': external_table_uri_str, + 'file': input_files.raw_paths if input_files else None, + 'external_table_uri': external_table_uri_str if external_table_uri_str else None, + 'file_path': file_path if file_path else None, }}) external_table_uri = util.parse_and_validate_external_table_uri_str(external_table_uri_str) if external_table_uri_str else None + time_regex = request.form.get('time_regex') or None + time_regex_format = request.form.get('time_regex_format') or None + + file_path_relative, file_path_absolute, _, _ = util.validate_and_process_file_path(file_path, check_crs=check_crs) + if file_path_relative: + file_path = file_path_absolute + util.validate_file_path_requires_time_regex(file_path_absolute, time_regex) + + util.validate_time_regex(time_regex, time_regex_format) + slugified_time_regex = input_file.slugify_timeseries_filename_pattern(time_regex) if time_regex else None + slugified_time_regex_format = input_file.slugify_timeseries_filename_pattern(time_regex_format) if time_regex_format else None # NAME unsafe_layername = request.form.get('name', '') if len(unsafe_layername) == 0: - unsafe_layername = input_file.get_unsafe_layername(input_files) if input_files else external_table_uri.table + if file_path: + unsafe_layername = os.path.basename(file_path) + else: + unsafe_layername = input_file.get_unsafe_layername(input_files) if input_files else external_table_uri.table layername = util.to_safe_layer_name(unsafe_layername) util.check_layername(layername) info = layman_util.get_publication_info(workspace, LAYER_TYPE, layername) if info: raise LaymanError(17, {'layername': layername}) - # Timeseries regex - time_regex = request.form.get('time_regex') or None - slugified_time_regex = input_file.slugify_timeseries_filename_pattern(time_regex) if time_regex else None - if time_regex: - try: - import re - re.compile(time_regex) - except re.error as exp: - raise LaymanError(2, {'parameter': 'time_regex', - 'expected': 'Regular expression', - }) from exp - time_regex_format = request.form.get('time_regex_format') or None - if time_regex_format and not time_regex: - raise LaymanError(48, { - 'parameters': ['time_regex_format'], - 'message': 'Parameter `time_regex_format` needs also parameter `time_regex`.', - 'expected': 'Image mosaic regex in `time_regex` parameter or empty `time_regex_format` parameter.', - 'found': { - 'time_regex_format': time_regex_format, - }}) - slugified_time_regex_format = input_file.slugify_timeseries_filename_pattern(time_regex_format) if time_regex_format else None - name_normalized_tif_by_layer = time_regex is None name_input_file_by_layer = time_regex is None or input_files.is_one_archive enable_more_main_files = time_regex is not None @@ -141,7 +144,19 @@ def post(workspace): enable_more_main_files=enable_more_main_files, time_regex=time_regex, slugified_time_regex=slugified_time_regex, name_input_file_by_layer=name_input_file_by_layer) - geodata_type = input_file.get_file_type(input_files.raw_or_archived_main_file_path) if not external_table_uri else settings.GEODATA_TYPE_VECTOR + if file_path: + geodata_type = settings.GEODATA_TYPE_RASTER + if not crs_id: + tif_files = util.get_geotiff_files(file_path) + crs_id = input_file.get_raster_crs_id(tif_files[0]) if tif_files else None + if not crs_id: + raise LaymanError(4, {'found': None, 'supported_values': settings.INPUT_SRS_LIST}) + elif input_files: + geodata_type = input_file.get_file_type(input_files.raw_or_archived_main_file_path) + elif external_table_uri: + geodata_type = settings.GEODATA_TYPE_VECTOR + else: + geodata_type = settings.GEODATA_TYPE_UNKNOWN # TITLE if len(request.form.get('title', '')) > 0: @@ -190,6 +205,7 @@ def post(workspace): 'name_input_file_by_layer': name_input_file_by_layer, 'enable_more_main_files': enable_more_main_files, 'external_table_uri': external_table_uri, + 'file_path': file_path_relative if file_path else None, 'original_data_source': settings.EnumOriginalDataSource.TABLE.value if external_table_uri else settings.EnumOriginalDataSource.FILE.value, } diff --git a/src/layman/layer/util.py b/src/layman/layer/util.py index 4f04cc886..2d1cdf36c 100644 --- a/src/layman/layer/util.py +++ b/src/layman/layer/util.py @@ -1,8 +1,9 @@ from functools import wraps, partial from urllib import parse +import os +import glob import re import logging -import os import shutil import psycopg2 @@ -86,6 +87,7 @@ def clear_publication_info(layer_info, file_type): clear_info = common_clear_publication_info(layer_info) if file_type != settings.GEODATA_TYPE_RASTER: clear_info.pop('image_mosaic') + clear_info.pop('file_path') return clear_info @@ -251,6 +253,115 @@ def layer_info_to_metadata_properties(info): return result +def get_geotiff_files(directory): + files = [] + for ext in settings.FILE_PATH_MAIN_FILE_EXTENSIONS: + files.extend(glob.glob(os.path.join(directory, f'*{ext}'))) + files.extend(glob.glob(os.path.join(directory, f'*{ext.upper()}'))) + return files + + +def validate_and_process_file_path(file_path, *, check_crs=True): + from .filesystem import input_file + + if not file_path: + return None, None, None, None + + if os.path.isabs(file_path): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Absolute path is not allowed', + 'expected': 'Relative path to directory (relative to GEOSERVER_DATADIR)', + 'found': file_path, + }) + + file_path_absolute = os.path.join(settings.GEOSERVER_DATADIR, file_path) + file_path_absolute = os.path.abspath(file_path_absolute) + geoserver_datadir_abs = os.path.abspath(settings.GEOSERVER_DATADIR) + if not file_path_absolute.startswith(geoserver_datadir_abs + os.sep): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Path is outside GeoServer data directory', + 'expected': 'Relative path to directory inside GeoServer data directory', + 'found': file_path, + }) + + if not os.path.exists(file_path_absolute): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Directory does not exist', + 'expected': 'Relative path to existing directory on server', + 'found': file_path, + }) + + if not os.path.isdir(file_path_absolute): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Path is not a directory', + 'expected': 'Relative path to existing directory containing raster files', + 'found': file_path, + }) + + if not ( + os.access(file_path_absolute, os.R_OK) + and os.access(file_path_absolute, os.X_OK) + ): + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Directory is not readable', + 'expected': 'Readable directory containing raster files', + 'found': file_path, + }) + + tif_files = get_geotiff_files(file_path_absolute) + if not tif_files: + raise LaymanError(2, { + 'parameter': 'file_path', + 'message': 'Directory does not contain any raster files', + 'expected': 'Directory containing at least one GeoTIFF file (.tif or .tiff)', + 'found': file_path, + }) + + if check_crs: + for tif_file in tif_files: + input_file.check_raster_layer_crs(tif_file) + + return file_path, file_path_absolute, None, None + + +def validate_file_path_requires_time_regex(file_path_absolute, time_regex): + tif_files = get_geotiff_files(file_path_absolute) + if len(tif_files) > 1 and not time_regex: + file_path_relative = os.path.relpath(file_path_absolute, settings.GEOSERVER_DATADIR) + raise LaymanError(48, { + 'parameters': ['file_path', 'time_regex'], + 'message': 'Directory contains multiple raster files, but time_regex is not provided', + 'expected': 'Provide time_regex parameter for image mosaic when directory contains multiple raster files', + 'found': { + 'file_path': file_path_relative, + 'raster_files_count': len(tif_files), + }, + }) + + +def validate_time_regex(time_regex, time_regex_format): + if time_regex: + try: + re.compile(time_regex) + except re.error as exp: + raise LaymanError(2, {'parameter': 'time_regex', + 'expected': 'Regular expression', + }) from exp + if time_regex_format and not time_regex: + raise LaymanError(48, { + 'parameters': ['time_regex_format'], + 'message': 'Parameter `time_regex_format` needs also parameter `time_regex`.', + 'expected': 'Image mosaic regex in `time_regex` parameter or empty `time_regex_format` parameter.', + 'found': { + 'time_regex_format': time_regex_format, + }}) + + def get_metadata_comparison(publication: Layer): layman_info = get_complete_layer_info(publication.workspace, publication.name) layman_props = layer_info_to_metadata_properties(layman_info) diff --git a/src/layman/map/util.py b/src/layman/map/util.py index 6c0cba725..21d19b2b4 100644 --- a/src/layman/map/util.py +++ b/src/layman/map/util.py @@ -150,6 +150,7 @@ def delete_map(map: Map, kwargs=None, *, x_forwarded_items=None): def clear_publication_info(layer_info): clear_info = common_clear_publication_info(layer_info) clear_info.pop('image_mosaic') + clear_info.pop('file_path') return clear_info diff --git a/src/layman/upgrade/__init__.py b/src/layman/upgrade/__init__.py index bd813611f..ba34f2607 100644 --- a/src/layman/upgrade/__init__.py +++ b/src/layman/upgrade/__init__.py @@ -2,7 +2,7 @@ from db import util as db_util from layman.upgrade import upgrade_v1_8, upgrade_v1_9, upgrade_v1_10, upgrade_v1_12, upgrade_v1_16, upgrade_v1_17, upgrade_v1_18, \ - upgrade_v1_20, upgrade_v1_21, upgrade_v1_22, upgrade_v1_23, upgrade_v2_0 + upgrade_v1_20, upgrade_v1_21, upgrade_v1_22, upgrade_v1_23, upgrade_v2_0, upgrade_v2_4 from layman import settings from . import consts @@ -22,7 +22,9 @@ MIGRATIONS = { consts.MIGRATION_TYPE_SCHEMA: [ - ((2, 4, 0), [lambda: logger.info("2.4.0 schema - no structural changes"), ]), + ((2, 4, 0), [ + upgrade_v2_4.adjust_db_for_file_path, + ]), ], consts.MIGRATION_TYPE_DATA: [ ((2, 4, 0), [lambda: logger.info("2.4.0 data - no data changes"), ]), diff --git a/src/layman/upgrade/upgrade_v2_4.py b/src/layman/upgrade/upgrade_v2_4.py new file mode 100644 index 000000000..8b0998853 --- /dev/null +++ b/src/layman/upgrade/upgrade_v2_4.py @@ -0,0 +1,19 @@ +import logging + +from db import util as db_util +from layman import settings + +logger = logging.getLogger(__name__) +DB_SCHEMA = settings.LAYMAN_PRIME_SCHEMA + + +def adjust_db_for_file_path(): + logger.info(f' Alter DB prime schema for file_path') + + statement = f''' + ALTER TABLE {DB_SCHEMA}.publications ADD COLUMN IF NOT EXISTS file_path text;''' + db_util.run_statement(statement) + + statement = f'alter table {DB_SCHEMA}.publications add constraint file_path_with_geodata_type_check CHECK ' \ + f'(file_path IS NULL OR geodata_type = %s);' + db_util.run_statement(statement, (settings.GEODATA_TYPE_RASTER,)) diff --git a/src/layman_settings.py b/src/layman_settings.py index 3dee52c93..5d52e34e0 100644 --- a/src/layman_settings.py +++ b/src/layman_settings.py @@ -40,6 +40,8 @@ class EnumWfsWmsStatus(Enum): '.jpeg': GEODATA_TYPE_RASTER, } +FILE_PATH_MAIN_FILE_EXTENSIONS = ['.tif', '.tiff'] + # Files are opened with dedicated tools for each format, so adding new extension is not sufficient for new compress format to start working COMPRESSED_FILE_EXTENSIONS = { '.zip': '/vsizip/', diff --git a/test_tools/process_client.py b/test_tools/process_client.py index c93e9680f..9a6ec840a 100644 --- a/test_tools/process_client.py +++ b/test_tools/process_client.py @@ -368,6 +368,7 @@ def publish_workspace_publication(publication_type, file_paths=None, file_path_pattern=None, external_table_uri=None, + file_path=None, headers=None, actor_name=None, access_rights=None, @@ -404,7 +405,7 @@ def publish_workspace_publication(publication_type, assert not (check_response_fn and raise_if_not_complete) assert not (file_paths and file_path_pattern) - file_paths = [publication_type_def.source_path] if file_paths is None and external_table_uri is None and not map_layers else file_paths + file_paths = [publication_type_def.source_path] if file_paths is None and external_table_uri is None and file_path is None and not map_layers else file_paths if style_file or with_chunks or compress or compress_settings or overview_resampling: assert publication_type == LAYER_TYPE @@ -446,11 +447,13 @@ def publish_workspace_publication(publication_type, if not do_not_post_name: data['name'] = name data['title'] = title + if file_path: + data['file_path'] = file_path if file_paths: if not with_chunks: - for file_path in file_paths: - assert os.path.isfile(file_path), file_path - files = [('file', (os.path.basename(fp), stack.enter_context(open(fp, 'rb')))) for fp in file_paths] + for path in file_paths: + assert os.path.isfile(path), path + files = [('file', (os.path.basename(path), stack.enter_context(open(path, 'rb')))) for path in file_paths] else: data['file'] = [os.path.basename(file) for file in file_paths] if style_file: diff --git a/tests/asserts/final/publication/internal.py b/tests/asserts/final/publication/internal.py index 52ba2e013..0be9b7e82 100644 --- a/tests/asserts/final/publication/internal.py +++ b/tests/asserts/final/publication/internal.py @@ -44,6 +44,8 @@ def source_internal_keys_are_subset_of_source_sibling_keys(workspace, publ_type, internal_keys = [key[1:] for key in info if key.startswith('_')] if publ_type == MAP_TYPE and source_name == 'layman.map.prime_db_schema.table': internal_keys.remove('style_type') + if 'file_path' in internal_keys: + internal_keys.remove('file_path') assert set(internal_keys) <= all_sibling_keys, \ f'internal_keys={set(internal_keys)}, all_sibling_keys={all_sibling_keys}, key={key}, info={info}' @@ -119,6 +121,8 @@ def all_keys_assigned_to_source(workspace, publ_type, name): info_keys = {key[1:] if key.startswith('_') else key for key in info} if publ_type == MAP_TYPE: info_keys.remove('style_type') + if 'file_path' in info_keys: + info_keys.remove('file_path') assert info_keys.issubset(source_keys), f'missing={info_keys.difference(source_keys)} ,info_keys={info_keys}, source_keys={source_keys}' @@ -141,8 +145,16 @@ def correct_values_in_detail(workspace, publ_type, name, *, exp_publication_deta publ_type_detail = (settings.GEODATA_TYPE_VECTOR, 'sld') with app.app_context(): pub_info = layman_util.get_publication_info(workspace, publ_type, name) + uuid = pub_info["uuid"] + is_file_path_layer = False + file_path_input_file_info = None + file_path_gdal_info = None + if publ_type == process_client.LAYER_TYPE: + if input_file.is_file_path_layer(uuid): + is_file_path_layer = True + file_path_input_file_info = input_file.get_layer_info_by_uuid(uuid) + file_path_gdal_info = gdal.get_layer_info_by_uuid(uuid) publ_type_dir = util.get_directory_name_from_publ_type(publ_type) - uuid = pub_info["uuid"] expected_detail = { 'name': name, '_workspace': workspace, @@ -160,6 +172,7 @@ def correct_values_in_detail(workspace, publ_type, name, *, exp_publication_deta 'access_rights': {'read': ['EVERYONE'], 'write': ['EVERYONE']}, 'image_mosaic': False, '_is_public_workspace': True, + 'file_path': None, } if publ_type == process_client.LAYER_TYPE: geodata_type = publ_type_detail[0] @@ -260,7 +273,22 @@ def correct_values_in_detail(workspace, publ_type, name, *, exp_publication_deta '_file': {'file_type': 'raster'}, '_table_uri': None, }) - if file_extension: + if is_file_path_layer: + file_paths = file_path_input_file_info.get('_file', {}).get('paths', {}) + gdal_paths = file_path_gdal_info.get('_file', {}).get('paths', {}) + combined_paths = {} + for key in file_paths: + combined_paths[key] = {**file_paths[key], **gdal_paths.get(key, {})} + util.recursive_dict_update(expected_detail, + { + 'file': file_path_input_file_info.get('file', {}), + '_file': { + **expected_detail.get('_file', {}), + 'paths': combined_paths, + }, + 'file_path': pub_info.get('file_path'), + }) + elif file_extension: util.recursive_dict_update(expected_detail, { '_file': { @@ -348,10 +376,12 @@ def nodata_preserved_in_normalized_raster(workspace, publ_type, name): publ_info = layman_util.get_publication_info(workspace, publ_type, name, {'keys': ['geodata_type', 'file']}) geodata_type = publ_info['geodata_type'] if geodata_type == settings.GEODATA_TYPE_RASTER: + is_file_path = publ_info.get('original_data_source') == settings.EnumOriginalDataSource.FILE.value and publ_info.get('file', {}).get('paths', []) for file_paths in publ_info['_file']['paths'].values(): gdal_path = file_paths['gdal'] input_nodata_value = gdal.get_nodata_value(gdal_path) - normalized_nodata_value = gdal.get_nodata_value(file_paths['normalized_absolute']) + normalized_path = gdal_path if is_file_path else file_paths['normalized_absolute'] + normalized_nodata_value = gdal.get_nodata_value(normalized_path) assert normalized_nodata_value == pytest.approx(input_nodata_value, 0.000000001) @@ -360,9 +390,10 @@ def stats_preserved_in_normalized_raster(workspace, publ_type, name): publ_info = layman_util.get_publication_info(workspace, publ_type, name, {'keys': ['geodata_type', 'file']}) geodata_type = publ_info['geodata_type'] if geodata_type == settings.GEODATA_TYPE_RASTER: + is_file_path = publ_info.get('original_data_source') == settings.EnumOriginalDataSource.FILE.value and publ_info.get('file', {}).get('paths', []) for file_paths in publ_info['_file']['paths'].values(): gdal_path = file_paths['gdal'] - normalized_path = file_paths['normalized_absolute'] + normalized_path = gdal_path if is_file_path else file_paths['normalized_absolute'] input_stats = gdal.get_statistics(gdal_path) driver_name = gdal.get_driver_short_name(gdal_path) tolerance = 0.000000001 if driver_name != 'JPEG' else 0.1 @@ -380,9 +411,10 @@ def size_and_position_preserved_in_normalized_raster(workspace, publ_type, name) publ_info = layman_util.get_publication_info(workspace, publ_type, name, {'keys': ['geodata_type', 'file']}) geodata_type = publ_info['geodata_type'] if geodata_type == settings.GEODATA_TYPE_RASTER: + is_file_path = publ_info.get('original_data_source') == settings.EnumOriginalDataSource.FILE.value and publ_info.get('file', {}).get('paths', []) for file_paths in publ_info['_file']['paths'].values(): gdal_path = file_paths['gdal'] - normalized_path = file_paths['normalized_absolute'] + normalized_path = gdal_path if is_file_path else file_paths['normalized_absolute'] input_raster_size = gdal.get_raster_size(gdal_path) normalized_raster_size = gdal.get_raster_size(normalized_path) diff --git a/tests/asserts/final/publication/internal_rest.py b/tests/asserts/final/publication/internal_rest.py index 00f25296c..86cca2860 100644 --- a/tests/asserts/final/publication/internal_rest.py +++ b/tests/asserts/final/publication/internal_rest.py @@ -18,5 +18,7 @@ def same_values_in_internal_and_rest(workspace, publ_type, name, rest_publicatio if 'image_mosaic' not in rest_publication_detail: publ_info.pop('image_mosaic') + if 'file_path' not in rest_publication_detail: + publ_info.pop('file_path') assert publ_info == rest_publication_detail diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.dbf b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.dbf new file mode 100644 index 000000000..14716051e Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.dbf differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.fix b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.fix new file mode 100644 index 000000000..52c6bac18 Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.fix differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.prj b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.prj new file mode 100644 index 000000000..d7447b2e8 --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.prj @@ -0,0 +1 @@ +PROJCS["WGS 84 / UTM zone 33N", GEOGCS["WGS 84", DATUM["World Geodetic System 1984", SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH], AUTHORITY["EPSG","4326"]], PROJECTION["Transverse_Mercator", AUTHORITY["EPSG","9807"]], PARAMETER["central_meridian", 15.0], PARAMETER["latitude_of_origin", 0.0], PARAMETER["scale_factor", 0.9996], PARAMETER["false_easting", 500000.0], PARAMETER["false_northing", 0.0], UNIT["m", 1.0], AXIS["Easting", EAST], AXIS["Northing", NORTH], AUTHORITY["EPSG","32633"]] \ No newline at end of file diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.properties b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.properties new file mode 100644 index 000000000..b256087df --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.properties @@ -0,0 +1,18 @@ +#-Automagically created from GeoTools- +#Tue Jan 27 15:39:50 UTC 2026 +TimeAttribute=ingestion +ExpandToRGB=false +TypeName=4d2ee21d-f7d9-4f16-a191-f15637c94c96 +SuggestedFormat=org.geotools.gce.geotiff.GeoTiffFormat +Name=4d2ee21d-f7d9-4f16-a191-f15637c94c96 +SuggestedSPI=it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi +LevelsNum=3 +PathType=RELATIVE +Heterogeneous=false +Caching=false +HeterogeneousCRS=false +LocationAttribute=location +Levels=10.0,10.0 20.0,20.0 40.0,40.0 +CheckAuxiliaryMetadata=false +MosaicCRS=EPSG\:32633 +ElevationAttribute=elevation diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.qix b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.qix new file mode 100644 index 000000000..05467aea6 Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.qix differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shp b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shp new file mode 100644 index 000000000..403a38dee Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shp differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shx b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shx new file mode 100644 index 000000000..eb0d3ba1c Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/4d2ee21d-f7d9-4f16-a191-f15637c94c96.shx differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif new file mode 100644 index 000000000..f0f6e02a3 Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif.aux.xml b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif.aux.xml new file mode 100644 index 000000000..b77190fcf --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/S2A_MSIL2A_20220316T100031.0.tif.aux.xml @@ -0,0 +1,29 @@ + + + + 255 + 102.4319462963 + 41 + 20.743259819823 + 100 + + + + + 255 + 94.710711111111 + 49 + 16.432475323946 + 100 + + + + + 255 + 85.819131481481 + 50 + 14.299599434474 + 100 + + + diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/indexer.properties b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/indexer.properties new file mode 100644 index 000000000..20f0f6be4 --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/indexer.properties @@ -0,0 +1,5 @@ +TimeAttribute=ingestion +ElevationAttribute=elevation +Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date,elevation:Integer +PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion) + diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/sample_image.dat b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/sample_image.dat new file mode 100644 index 000000000..3b1a782ac Binary files /dev/null and b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/sample_image.dat differ diff --git a/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/timeregex.properties b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/timeregex.properties new file mode 100644 index 000000000..183d08e0b --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/layers/4d2ee21d-f7d9-4f16-a191-f15637c94c96/timeregex.properties @@ -0,0 +1 @@ +regex=[0-9]{8} diff --git a/tests/dynamic_data/publications/layer_local_path/test_file_path.py b/tests/dynamic_data/publications/layer_local_path/test_file_path.py new file mode 100644 index 000000000..c9afcc4a8 --- /dev/null +++ b/tests/dynamic_data/publications/layer_local_path/test_file_path.py @@ -0,0 +1,356 @@ +import os +import shutil +import glob +import pytest + +from layman import LaymanError, settings, app, util as layman_util +from test_tools import process_client +from tests.asserts.final import publication as asserts_publ +from tests.asserts.final.publication import util as assert_util +from tests.dynamic_data import base_test +from tests.dynamic_data.publications import common_publications +from tests import Publication4Test + +pytest_generate_tests = base_test.pytest_generate_tests +DIRECTORY = os.path.dirname(os.path.abspath(__file__)) +WORKSPACE = "test_file_path_ws" + + +TEST_UUID_MOSAIC = '4d2ee21d-f7d9-4f16-a191-f15637c94c96' +TEST_UUID_SINGLE = 'a1b2c3d4-e5f6-7890-abcd-ef1234567890' + + +@pytest.fixture(scope='class') +def prepare_file_path_data(): + normalized_raster_data_dir = settings.LAYMAN_NORMALIZED_RASTER_DATA_DIR + assert os.path.exists(normalized_raster_data_dir), \ + f"Normalized raster data directory not found: {normalized_raster_data_dir}." + + layers_dir = os.path.join(normalized_raster_data_dir, 'layers') + target_dirs = [] + + source_dir_mosaic = os.path.join(DIRECTORY, 'layers', TEST_UUID_MOSAIC) + + target_dir_mosaic = os.path.join(layers_dir, TEST_UUID_MOSAIC) + if os.path.exists(target_dir_mosaic): + shutil.rmtree(target_dir_mosaic) + os.makedirs(layers_dir, exist_ok=True) + assert os.path.exists(source_dir_mosaic), f"Source directory does not exist: {source_dir_mosaic}" + shutil.copytree(source_dir_mosaic, target_dir_mosaic) + tif_files = glob.glob(os.path.join(target_dir_mosaic, '*.tif')) + assert tif_files, f"No .tif files found in target directory: {target_dir_mosaic}." + timeregex_file = os.path.join(target_dir_mosaic, 'timeregex.properties') + assert os.path.exists(timeregex_file), f"timeregex.properties not found in target directory: {target_dir_mosaic}." + target_dirs.append(target_dir_mosaic) + + target_dir_single = os.path.join(layers_dir, TEST_UUID_SINGLE) + if os.path.exists(target_dir_single): + shutil.rmtree(target_dir_single) + + base_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(DIRECTORY)))) + sample_file = os.path.join(base_dir, 'sample', 'layman.layer', 'sample_tif_rgb.tif') + assert os.path.exists(sample_file), f"Sample raster file not found. This is a test setup error." + + os.makedirs(target_dir_single, exist_ok=True) + target_file = os.path.join(target_dir_single, 'raster.tif') + shutil.copy2(sample_file, target_file) + target_dirs.append(target_dir_single) + + yield + + for target_dir in target_dirs: + if os.path.exists(target_dir): + try: + shutil.rmtree(target_dir, ignore_errors=True) + except Exception: + pass + + +def generate_test_cases(): + normalized_raster_data_dir_name = settings.LAYMAN_NORMALIZED_RASTER_DATA_DIR_NAME + test_cases = [] + + file_path_relative_mosaic = os.path.join( + normalized_raster_data_dir_name, + 'layers', + TEST_UUID_MOSAIC + ) + publication_mosaic = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_mosaic', + ) + test_case_mosaic = base_test.TestCaseType( + key='file_path_mosaic', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_mosaic, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': file_path_relative_mosaic, + 'time_regex': '[0-9]{8}', + }, + params={ + 'exp_info': { + 'exp_publication_detail': { + 'geodata_type': 'raster', + 'image_mosaic': True, + }, + 'publ_type_detail': ('raster', 'sld'), + }, + 'exp_thumbnail': 'test_tools/data/thumbnail/raster_layer_tif.png', + }, + ) + test_cases.append(test_case_mosaic) + + file_path_relative_single = os.path.join( + normalized_raster_data_dir_name, + 'layers', + TEST_UUID_SINGLE + ) + publication_single = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_single', + ) + test_case_single = base_test.TestCaseType( + key='file_path_single', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_single, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': file_path_relative_single, + }, + params={ + 'exp_info': { + 'exp_publication_detail': { + 'geodata_type': 'raster', + 'image_mosaic': False, + }, + 'publ_type_detail': ('raster', 'sld'), + }, + 'exp_thumbnail': 'test_tools/data/thumbnail/raster_layer_tif.png', + }, + ) + test_cases.append(test_case_single) + + return test_cases + + +@pytest.mark.usefixtures('prepare_file_path_data') +class TestFilePath(base_test.TestSingleRestPublication): + workspace = WORKSPACE + publication_type = process_client.LAYER_TYPE + rest_parametrization = [] + test_cases = generate_test_cases() + + def test_file_path(self, layer, rest_args, params): + self.post_publication(layer, args=rest_args) + + assert_util.is_publication_valid_and_complete(layer) + + with app.app_context(): + pub_info = layman_util.get_publication_info(layer.workspace, layer.type, layer.name) + if params['exp_info']['exp_publication_detail'].get('image_mosaic'): + assert 'wms' in pub_info and 'time' in pub_info['wms'], \ + "Timeseries layer (image_mosaic=True) must expose WMS time dimension" + assert 'bounding_box' in pub_info, "Layer must have bounding_box" + assert 'native_bounding_box' in pub_info, "Layer must have native_bounding_box" + assert 'native_crs' in pub_info, "Layer must have native_crs" + assert 'file_path' in pub_info, "Layer must have file_path key in GET response" + assert pub_info['file_path'] == rest_args['file_path'], \ + f"file_path in response ({pub_info['file_path']}) must match requested file_path ({rest_args['file_path']})" + + asserts_publ.internal.correct_values_in_detail( + layer.workspace, layer.type, layer.name, + full_comparison=False, + **params['exp_info'] + ) + + asserts_publ.internal.thumbnail_equals( + layer.workspace, layer.type, layer.name, + exp_thumbnail=params['exp_thumbnail'], + max_diffs=100000 + ) + + +def generate_negative_test_cases(): + normalized_raster_data_dir_name = settings.LAYMAN_NORMALIZED_RASTER_DATA_DIR_NAME + test_cases = [] + + publication_abs = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_absolute', + ) + test_case_abs = base_test.TestCaseType( + key='file_path_absolute', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_abs, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': '/absolute/path/to/directory', + }, + params={ + 'should_succeed': False, + 'expected_error_code': 2, + 'expected_error_param': 'file_path', + }, + ) + test_cases.append(test_case_abs) + + publication_nonexistent = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_nonexistent', + ) + test_case_nonexistent = base_test.TestCaseType( + key='file_path_nonexistent', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_nonexistent, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': os.path.join(normalized_raster_data_dir_name, 'layers', 'nonexistent_uuid'), + }, + params={ + 'should_succeed': False, + 'expected_error_code': 2, + 'expected_error_param': 'file_path', + }, + ) + test_cases.append(test_case_nonexistent) + + publication_file = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_is_file', + ) + test_case_file = base_test.TestCaseType( + key='file_path_is_file', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_file, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': os.path.join(normalized_raster_data_dir_name, 'layers', TEST_UUID_SINGLE, 'raster.tif'), + }, + params={ + 'should_succeed': False, + 'expected_error_code': 2, + 'expected_error_param': 'file_path', + }, + ) + test_cases.append(test_case_file) + + publication_no_tif = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_no_tif', + ) + test_case_no_tif = base_test.TestCaseType( + key='file_path_no_tif', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_no_tif, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': os.path.join(normalized_raster_data_dir_name, 'layers', 'empty_dir'), + }, + params={ + 'should_succeed': False, + 'expected_error_code': 2, + 'expected_error_param': 'file_path', + }, + ) + test_cases.append(test_case_no_tif) + + publication_no_regex = Publication4Test( + type=process_client.LAYER_TYPE, + workspace=WORKSPACE, + name='test_file_path_no_regex', + ) + test_case_no_regex = base_test.TestCaseType( + key='file_path_no_regex', + type=base_test.EnumTestTypes.MANDATORY, + publication=publication_no_regex, + rest_method=base_test.RestMethod.POST, + rest_args={ + 'file_path': os.path.join(normalized_raster_data_dir_name, 'layers', 'multi_no_regex'), + }, + params={ + 'should_succeed': False, + 'expected_error_code': 48, + 'expected_error_params': ['file_path', 'time_regex'], + }, + ) + test_cases.append(test_case_no_regex) + + return test_cases + + +@pytest.fixture(scope='class') +def prepare_negative_test_data(): + normalized_raster_data_dir = settings.LAYMAN_NORMALIZED_RASTER_DATA_DIR + assert os.path.exists(normalized_raster_data_dir), \ + f"Normalized raster data directory not found: {normalized_raster_data_dir}." + + layers_dir = os.path.join(normalized_raster_data_dir, 'layers') + target_dirs = [] + + empty_dir = os.path.join(layers_dir, 'empty_dir') + if os.path.exists(empty_dir): + shutil.rmtree(empty_dir) + os.makedirs(empty_dir, exist_ok=True) + target_dirs.append(empty_dir) + + multi_no_regex_dir = os.path.join(layers_dir, 'multi_no_regex') + if os.path.exists(multi_no_regex_dir): + shutil.rmtree(multi_no_regex_dir) + os.makedirs(multi_no_regex_dir, exist_ok=True) + + base_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(DIRECTORY)))) + sample_file = os.path.join(base_dir, 'sample', 'layman.layer', 'sample_tif_rgb.tif') + assert os.path.exists(sample_file), f"Sample raster file not found: {sample_file}" + shutil.copy2(sample_file, os.path.join(multi_no_regex_dir, 'raster1.tif')) + shutil.copy2(sample_file, os.path.join(multi_no_regex_dir, 'raster2.tif')) + target_dirs.append(multi_no_regex_dir) + + yield + + for target_dir in target_dirs: + if os.path.exists(target_dir): + try: + shutil.rmtree(target_dir, ignore_errors=True) + except Exception: + pass + + +@pytest.mark.usefixtures('prepare_negative_test_data') +class TestFilePathNegative(base_test.TestSingleRestPublication): + workspace = WORKSPACE + publication_type = process_client.LAYER_TYPE + rest_parametrization = [] + test_cases = generate_negative_test_cases() + + def test_file_path_negative(self, layer, rest_args, params): + should_succeed = params.get('should_succeed', True) + + assert not should_succeed, "Negative test case should have should_succeed=False" + + expected_error_code = params.get('expected_error_code') + expected_error_param = params.get('expected_error_param') + expected_error_params = params.get('expected_error_params') + + with pytest.raises(LaymanError) as exc_info: + self.post_publication(layer, args=rest_args) + + error = exc_info.value + assert error.code == expected_error_code, \ + f"Expected error code {expected_error_code}, got {error.code}. Error: {error.data}" + assert error.http_code == 400, \ + f"Expected HTTP code 400, got {error.http_code}" + + if expected_error_param: + assert error.data.get('parameter') == expected_error_param, \ + f"Expected parameter '{expected_error_param}', got '{error.data.get('parameter')}'" + + if expected_error_params: + assert error.data.get('parameters') == expected_error_params, \ + f"Expected parameters {expected_error_params}, got {error.data.get('parameters')}" diff --git a/tests/dynamic_data/publications/wrong_input/wrong_input_test.py b/tests/dynamic_data/publications/wrong_input/wrong_input_test.py index 0c65d12ab..de08df950 100644 --- a/tests/dynamic_data/publications/wrong_input/wrong_input_test.py +++ b/tests/dynamic_data/publications/wrong_input/wrong_input_test.py @@ -1186,8 +1186,8 @@ class Key(Enum): 'code': 1, 'message': 'Missing parameter', 'data': { - 'parameters': ['file', 'external_table_uri'], - 'message': 'Both `file` and `external_table_uri` parameters are empty', + 'parameters': ['file', 'external_table_uri', 'file_path'], + 'message': 'All parameters `file`, `external_table_uri`, and `file_path` are empty', 'expected': 'One of the parameters is filled.', }, }, @@ -1219,7 +1219,18 @@ class Key(Enum): }, Key.MANDATORY_CASES: None, Key.RUN_ONLY_CASES: frozenset([RestMethod, WithChunksDomain, CompressDomain.FALSE]), - Key.SPECIFIC_CASES: {}, + Key.SPECIFIC_CASES: { + frozenset([RestMethod.POST, WithChunksDomain, CompressDomain.FALSE]): { + Key.EXPECTED_EXCEPTION: { + 'data': { + 'message': 'Multiple parameters are filled: file, external_table_uri', + 'found': { + 'file_path': None, + }, + }, + }, + }, + }, }, 'partial_external_table_uri': { Key.PUBLICATION_TYPE: process_client.LAYER_TYPE,