Loading data

autk-db can load data from multiple formats and sources. Load methods store data as named tables in DuckDB. Those table names must be unique since they are used to identify tables in queries, joins, updates, and retrieval methods.

When a loading method is called, it ingests data into DuckDB and returns the created table metadata. To retrieve stored data, one of the retrieving data methods must be used.

OpenStreetMap

autk-db can fetch OpenStreetMap (OSM) data from the Overpass API or parse static .pbf extracts obtained from websites such as Geofabrik and SliceOSM.

Using the Overpass API

To directly fetch from the public Overpass API and load OpenStreetMap data into DuckDB tables, autk-db provides the loadOsm method. The most important parameters are:

1. queryArea — Defines the geographic region of interest in one of two ways:

   • Named area — provide geocodeArea and a list of administrative boundary names in areas. geocodeArea defines the search scope and avoids naming ambiguities; use exact OSM boundary relation names rather than informal place names.
   • Bounding box — provide bbox: [west, south, east, north] in WGS84 (EPSG:4326). All four values must be finite geographic coordinates, with west < east and south < north; antimeridian-crossing boxes are not supported.

2. autoLoadLayers — List of data layers to automatically extract from raw OSM data. The valid osm layer values in Autark are buildings, roads, surface, parks, and water. The optional coordinateFormat specifies the source CRS of the OSM coordinates before they are transformed into the workspace CRS. In v4, surface is always built as the workspace clipping mask, even when it is omitted from layers; include it when you also want it returned as a public layer.

3. outputTableName — Optional parameter used to define the base name for the produced tables. Each automatically loaded layer is stored as {outputTableName}_{layer}. It defaults to table_osm. For example, if layers: ['surface', 'roads'], the resulting tables are table_osm_surface, and table_osm_roads.

Overpass API limits

• Fetching large areas is slow and may fail. Be aware that the public Overpass API servers can reject queries when they're busy or out of slots. Keep areas small and use specific geocodeArea + areas for the best results.

• autk-db provides the onProgress callback that may be used to track the loading status.

Surface filtering in v4.1.1

All roads, including bbox loads, are cropped to the mandatory surface. Bridge and over-water segments outside that surface may be removed. Building parts are never cut into new footprints: parts not fully covered by the surface are discarded, and retained geometry/property indices are updated together.

For named areas, coastline classification uses the full geocodeArea bounding box, including neighboring coastlines. PBF extracts must include the complete geocode boundary and coastline coverage; the loader does not fetch missing data. When classification succeeds, requested water layers can include sea context within the selected areas' bounding box, marked with properties.generated = 'coastline'. Missing or invalid coastline data never implies sea.

Loading custom OSM tag sets

With Overpass, tagSets can load custom layers beside the standard OSM themes. Each OsmTagSet declares exactly one geometry family (points, polylines, or polygons) and produces {outputTableName}_{name}_{type}. Filters in one set are combined with OR.

await db.loadOsm({
  queryArea: { bbox: [-74.019, 40.700, -74.003, 40.714] },
  autoLoadLayers: { layers: [] },
  tagSets: [{
    name: 'cafes',
    type: 'points',
    tags: [{ key: 'amenity', value: 'cafe' }],
  }],
});

const cafes = await db.getLayer('table_osm_cafes_points', { osmElements: true });

tagSets are not supported with pbfFileUrl; use the Overpass workflow for these custom layers.

Using static .pbf files

Instead of querying the Overpass API, you can load OSM data from a local or remote .osm.pbf file. PBF extracts are available from Geofabrik and SliceOSM.

To load from a PBF file, provide the pbfFileUrl parameter to the loadOsm function. All other parameters must be defined as in the Overpass API use case.

PBF loading times

• The .pbf loader scans the file in three stages: first it identifies the regions informed in queryArea, then it computes the bounding box of these regions and, lastly, it collects the OSM features inside these areas.

• Very large files may also take long to process, but the process runs etirely in the browser and no API limits apply. To reduce the loading time, crop the .pbf file first using Osmium as a pre-processing step: osmium extract --strategy=smart -b <minLon>,<minLat>,<maxLon>,<maxLat> <input.osm.pbf> -o <output.osm.pbf>.

List of loadOsm parameters

Option	Type	Description
outputTableName	string	Base table name.
queryArea geocodeAreaareas	stringstring[]	Named-area geocode scope.Boundary names, or use a WGS84 bbox.
autoLoadLayers coordinateFormatlayers	stringLayerType[]	Source CRS.Layer names.
pbfFileUrl	string	Optional PBF URL.
forceRefresh	boolean	Bypass cache.
workspace	string	Workspace name.
onProgress	function	Progress callback.

Load OSM first when combining layer sources

If you plan to load OSM and additional layers in the same workspace, you must load OSM first. By doing so, the osm data bounding box and the surface layer geometry will be used to filter and clip the additional layers to make sure all data span the same area (see workspace).

JSON tables

loadJson() imports an array of records into DuckDB. Use it for API responses, event lists, or other tabular JSON that is not a GeoJSON FeatureCollection.

Provide either jsonFileUrl or jsonObject, not both. Run this self-contained example to import spatial sensor records from memory and plain event rows from a local Blob URL. In your application, replace the Blob URL with your own JSON endpoint.

Without geometryColumns, the result is a plain table: inspect it with getTable(). With geometry columns, it can also be exported with getLayer() for mapping.

Option	Purpose
jsonFileUrl / jsonObject	JSON array URL or in-memory array of records.
outputTableName	Required output table name.
geometryColumns	Omit for a plain table; use true for Latitude/Longitude in EPSG:4326, or an object for custom fields.
workspace	Optional destination workspace.

For custom coordinate fields, use geometryColumns: { latColumnName: 'lat', longColumnName: 'lon', coordinateFormat: 'EPSG:4326' }. For WKT, use { wktColumnName: 'shape', coordinateFormat: 'EPSG:4326' }. Coordinates are transformed into the workspace CRS.

Use loadGeojson() instead when the input already contains GeoJSON features and geometries. See LoadJsonParams for the full contract.

GeoJSON

loadGeojson loads a GeoJSON FeatureCollection from a URL or an in-memory object and stores it as a named layer. The only required parameter is outputTableName.

By default, the input coordinates are expected to be in latitude/longitude, that is, it uses the EPSG:4326 system. If the loaded GeoJSON uses a different coordinates system,its coodinate system must be provided using the coordinateFormat attribute.

Also, you must use layerType define the type of the loaded layer. If no type is provided, it will be authomatically inference performed by autk-db.

List of loadGeojson parameters

Option	Type	Description
geojsonFileUrl	string	GeoJSON file URL.
geojsonObject	FeatureCollection	In-memory GeoJSON.
outputTableName	string	Output table name.
coordinateFormat	string	Source CRS.
layerType	LayerType	Override inferred layer type.
boundingBox	BoundingBox	Optional clipping bounds.
workspace	string	Workspace name.

GeoTIFF

loadGeoTiff loads raster data from a URL or an ArrayBuffer and stores it as a raster table in DuckDB. The only required parameter is outputTableName.

By default, the input raster is expected to use EPSG:4326. If the GeoTIFF uses a different coordinate system, provide it through coordinateFormat. For large rasters, reduce maxPixels to avoid loading too many pixels into browser memory.

Valid zero values are preserved when no NoData value is declared. Negative grid-axis directions are supported; rotated or sheared grids are rejected.

Try changing the previous example

Modify the previous code sample to explore more of autk-db. For example, try setting maxPixels or use a different outputTableName.

List of loadGeoTiff parameters

Option	Type	Description
geotiffFileUrl	string	GeoTIFF file URL.
geotiffArrayBuffer	ArrayBuffer	In-memory GeoTIFF data.
outputTableName	string	Output table name.
coordinateFormat	string	Source CRS.
maxPixels	number	Pixel limit.
workspace	string	Workspace name.

CSV

loadCsv loads tabular data from a CSV file or an in-memory matrix and stores it as a table in DuckDB. The only required parameter is outputTableName. If the CSV contains spatial information, provide geometryColumns so autk-db can create geometries during import.

By default, geometryColumns: true expects Latitude and Longitude columns in EPSG:4326. For columns with different names or with WKT geometry, provide them using the geometryColumns object. For tab-separated files, set delimiter: '\t'.

List of loadCsv parameters

Option	Type	Description
csvFileUrl	string	CSV file URL.
csvObject	unknown[][]	In-memory CSV data.
outputTableName	string	Output table name.
delimiter	string	Field separator.
geometryColumns truelatColumnNamelongColumnNamewktColumnNamecoordinateFormat	truestringstringstringstring	Default lat/lng mapping.Latitude column.Longitude column.WKT column.Source CRS.
workspace	string	Workspace name.