Streaming Load API
The Streaming Load API is used to read data from your local files and load it into Databend.
Creating an API Request
To create a request with the Streaming Load API, follow the format below:
curl -H "insert_sql:<value>" -F "upload=@<file_location>" [-F "upload=@<file_location>"] -XPUT http://<user_name>:[password]@<http_handler_host>:<http_handler_port>/v1/streaming_load
The parameter insert_sql is required and must include an INSERT statement as well as the FILE_FORMAT parameter that specifies the file formats. For details about FILE_FORMAT, see Input & Output File Formats.
| Parameter | Values | Supported Formats | Examples |
|---|---|---|---|
| insert_sql | [INSERT_statement] + [FILE_FORMAT] | All | -H "insert_sql: insert into ontime file_format = (type = CSV skip_header = 1 compression = 'bz2')" |
Alternatives to Streaming Load API
The COPY INTO command enables you to load data from files using insecure protocols, such as HTTP. This simplifies the data loading in some specific scenarios, for example, Databend is installed on-premises with MinIO. In such cases, you can load data from local files with the COPY INTO command.
Example:
COPY INTO ontime200 FROM 'fs://<file_path>/ontime_200.csv' FILE_FORMAT = (type = CSV field_delimiter = ',' record_delimiter = '\n' skip_header = 1);
To do so, you must add the setting allow_insecure to the configuration file databend-query.toml as indicated below and set it to true:
...
[storage]
# fs | s3 | azblob | obs
type = "fs"
allow_insecure = true
...
For security reasons, Databend does NOT recommend insecure protocols for data loading. Use them for tests only. DO NOT set allow_insecure to true in any production environment.