Loading data

autk-db can load data from multiple formats and sources. Load methods store data as named tables in DuckDB. Those table names must be unique since they are used to identify tables in queries, joins, updates, and retrieval methods.

When a loading method is called, it ingests data into DuckDB and returns the created table metadata. To retrieve stored data, one of the retrieving data methods must be used.

OpenStreetMap

autk-db can fetch OpenStreetMap (OSM) data from the Overpass API or parse static .pbf extracts obtained from websites such as Geofabrik and SliceOSM.

Using the Overpass API

To directly fetch from the public Overpass API and load OpenStreetMap data into DuckDB tables, autk-db provides the loadOsm method. The most important parameters are:

1. queryArea — Defines the geographic region of interest. The region definition is broken into two parts: the geocodeArea and a list of administrative areas areas. geocodeArea is used to define the data search scope and avoid naming ambiguities when querying the Overpass API. areas must identify OpenStreetMap boundary relations whose member ways can be reconstructed into a closed polygon. For best results, use exact OSM boundary relation names rather than informal place names.

2. autoLoadLayers — List of data layers to automatically extract from raw OSM data. The valid osm layer values in Autark are buildings, roads, surface, parks, and water. The optional coordinateFormat specifies the source CRS of the OSM coordinates before they are transformed into the workspace CRS.

3. outputTableName — Optional parameter used to define the base name for the produced tables. Each automatically loaded layer is stored as {outputTableName}_{layer}. It defaults to table_osm. For example, if layers: ['surface', 'roads'], the resulting tables are table_osm_surface, and table_osm_roads.

Overpass API limits

• Fetching large areas is slow and may fail. Be aware that the public Overpass API servers can reject queries when they're busy or out of slots. Keep areas small and use specific geocodeArea + areas for the best results.

• autk-db provides the onProgress callback that may be used to track the loading status.

Using static .pbf files

Instead of querying the Overpass API, you can load OSM data from a local or remote .osm.pbf file. PBF extracts are available from Geofabrik and SliceOSM.

To load from a PBF file, provide the pbfFileUrl parameter to the loadOsm function. All other parameters must be defined as in the Overpass API use case.

PBF loading times

• The .pbf loader scans the file in three stages: first it identifies the regions informed in queryArea, then it computes the bounding box of these regions and, lastly, it collects the OSM features inside these areas.

• Very large files may also take long to process, but the process runs etirely in the browser and no API limits apply. To reduce the loading time, crop the .pbf file first using Osmium as a pre-processing step: osmium extract --strategy=smart -b <minLon>,<minLat>,<maxLon>,<maxLat> <input.osm.pbf> -o <output.osm.pbf>.

List of loadOsm parameters

Option	Type	Description
outputTableName	string	Base table name.
queryArea geocodeAreaareas	stringstring[]	Geocode scope.Boundary names.
autoLoadLayers coordinateFormatlayers	stringLayerType[]	Source CRS.Layer names.
pbfFileUrl	string	Optional PBF URL.
forceRefresh	boolean	Bypass cache.
workspace	string	Workspace name.
onProgress	function	Progress callback.

Load OSM first when combining layer sources

If you plan to load OSM and additional layers in the same workspace, you must load OSM first. By doing so, the osm data bounding box and the surface layer geometry will be used to filter and clip the additional layers to make sure all data span the same area (see workspace).

GeoJSON

loadGeojson loads a GeoJSON FeatureCollection from a URL or an in-memory object and stores it as a named layer. The only required parameter is outputTableName.

By default, the input coordinates are expected to be in latitude/longitude, that is, it uses the EPSG:4326 system. If the loaded GeoJSON uses a different coordinates system,its coodinate system must be provided using the coordinateFormat attribute.

Also, you must use layerType define the type of the loaded layer. If no type is provided, it will be authomatically inference performed by autk-db.

List of loadGeojson parameters

Option	Type	Description
geojsonFileUrl	string	GeoJSON file URL.
geojsonObject	FeatureCollection	In-memory GeoJSON.
outputTableName	string	Output table name.
coordinateFormat	string	Source CRS.
layerType	LayerType	Override inferred layer type.
boundingBox	BoundingBox	Optional clipping bounds.
workspace	string	Workspace name.

GeoTIFF

loadGeoTiff loads raster data from a URL or an ArrayBuffer and stores it as a raster table in DuckDB. The only required parameter is outputTableName.

By default, the input raster is expected to use EPSG:4326. If the GeoTIFF uses a different coordinate system, provide it through coordinateFormat. For large rasters, reduce maxPixels to avoid loading too many pixels into browser memory.

Try changing the previous example

Modify the previous code sample to explore more of autk-db. For example, try setting maxPixels or use a different outputTableName.

List of loadGeoTiff parameters

Option	Type	Description
geotiffFileUrl	string	GeoTIFF file URL.
geotiffArrayBuffer	ArrayBuffer	In-memory GeoTIFF data.
outputTableName	string	Output table name.
coordinateFormat	string	Source CRS.
maxPixels	number	Pixel limit.
workspace	string	Workspace name.

CSV

loadCsv loads tabular data from a CSV file or an in-memory matrix and stores it as a table in DuckDB. The only required parameter is outputTableName. If the CSV contains spatial information, provide geometryColumns so autk-db can create geometries during import.

By default, geometryColumns: true expects Latitude and Longitude columns in EPSG:4326. For columns with different names or with WKT geometry, provide them using the geometryColumns object. For tab-separated files, set delimiter: '\t'.

List of loadCsv parameters

Option	Type	Description
csvFileUrl	string	CSV file URL.
csvObject	unknown[][]	In-memory CSV data.
outputTableName	string	Output table name.
delimiter	string	Field separator.
geometryColumns truelatColumnNamelongColumnNamewktColumnNamecoordinateFormat	truestringstringstringstring	Default lat/lng mapping.Latitude column.Longitude column.WKT column.Source CRS.
workspace	string	Workspace name.