This document gives a short description of the APIs that Energydata.dk makes available to the user.
If you are looking for a general introduction to the Energydata.dk platform, please refer to the user manual.
The platform has three APIs: Datastream, MQTT and Batch Upload. They are described in the following sections.
Please note, that you need a token to access data through the APIs. The token serves as userID & password to verify that you are allowed to access the data. Please note, that you can get a general introduction to the API in the user manual in the API section.
Please report any error in this document or suggestions for improvement to info@energydata.dk
http:get:: /api/v1/datastreams/values
Retrieve values for one or more properties.
The properties which have their values retrieved depend on the query parameter ids. This should be a comma-separated list of datastream IDs.
The resource can be used to retrieve either the latest values, or values which correspond to a timespan.
Please note that this resource does not return JSON for successful requests. In these cases, response data is streamed back to the client as CSV. But you should still specify Accept: application/json in the header to ensure errors are returned to the client as JSON.
Parameters for the API are:
GET /api/v1/datastreams/values?ids=65603,65607,65863&latest=true HTTP/1.1
Host: admin.energydata.dk
Accept: application/json
Authorization: Bearer <your-token>
HTTP/1.1 200 OK
Content-Description: File Transfer
Content-Disposition: attachment; filename=data.csv
Content-Type: text/csv; charset=UTF-8
Transfer-Encoding: chunked
65603,1547043555943,25.04
65607,1547043557698,25.12
65863,1544089165888,24
HTTP/1.1 200 OK
Content-Description: File Transfer
Content-Disposition: attachment; filename=data.csv
Content-Type: text/csv; charset=UTF-8
Transfer-Encoding: chunked
65607,1517443389806,25.12
65607,1517443449746,25.12
65607,1517443509696,25.12
65607,1517443569646,25.12
65607,1517443629586,25.12
65607,1517443689537,25.12
65863,1517443749495,25.12
65863,1517443809416,25.12
65863,1517443869366,24.04
65863,1517443929309,24.04
65863,1517443989256,24.04
65863,1517444049233,24.04
65863,1517444109146,24.04
65603,1517443389806,25.12
65603,1517443449746,25.12
65603,1517443509696,25.12
65603,1517443569646,25.12
65603,1517443629586,25.12
65603,1517443689537,25.12
Energydata.dk provides a real-time publishing and subscription API. This API serves two purposes:
The API is built on top of the MQTT protocol, version 3.1.1.
This section describes how to connect to the MQTT broker, how MQTT topics relate to datasets defined in the energydata.dk and specifies the format and content of messages exchanged on these topics.
The MQTT broker is available at mqtts.energydata.dk, and is open for MQTT connections on port 8883.
Before you can connect to the broker, you must create a token in Energydata.dk, with the mqtt pubsub permission as well permission to use the licenses that gives you access to the datasets you want.
When connecting you must use the token as the username.
An example in Python (requires paho-mqtt <https://pypi.python.org/pypi/paho-mqtt>_):
An example using the mosquitto_sub command-line client (bundled with Mosquitto: <https://mosquitto.org/>_):
In Energydata.dk time series are defined as datastreams and grouped into a dataset.
To send data to Energydata.dk using MQTT, you must create a dataset with a topic prefix and a datastream within that dataset with a topic suffix. The topic prefix is one string of alphanumerics, the suffix can be several alphanumeric strings separated by /. So if the prefix is my-topic-prefix and the suffix is my/topic/suffix , the complete MQTT topic of that datastream is my-topic-prefix/my/topic/suffix.
This topic will be used on the MQTT broker, to identify the datastream.
Examples with Mosquitto:
MQTT messages can be formatted a one of two types:
Single value messages contain a timestamp and value, as described in the table below. The datastream associated with the message is inferred from the MQTT topic.
Key | Type | Required | Description | Example |
---|---|---|---|---|
timestamp | Integer | Yes | Number of milliseconds since 1970-01-01 (Unix epoch) | 1521797973469 |
value | String, Double or Integer | Yes | Value, encoded either as String, Double or Integer, depending on property definition in DMS | 14.47 |
Example message:
{
"timestamp": 1521797973469,
"value": 14.47
}
Multi value messages are used in situations where you need to publish several different but related values. They will be persisted with the same timestamp in the same dataset, on topics specified that follows.
Multi value messages are published with the topic prefix as the topic of the mqtt message, for example my-topic-prefix. The payload has the following form:
All message payloads must be serialized as JSON.
When publishing to Energydata.dk using MQTT you should use Quality of Service (QoS) 1 to ensure that messages are delivered. This is especially important if you publish with a high throughput, as overload protection mechanisms can discard your messages. Why and how is described in detail below.
When the Energydata.dk MQTT broker receives a publish message from your client, the message is added to a queue of incoming messages. This queue is per-client, and can contain up to 1000 messages. Messages in the queue are dequeued serially in a FIFO manner. When dequeued, the MQTT broker checks that the client is authorized for publishing messages on the given topic.
If authorization succeeds, the message is forwarded for storage in Energydata.dk, and to any clients subscribed to the given topic. If the message was published with QoS 1 or 2, the MQTT broker sends the acknowledgement after authorization succeeds.
If authorization does not succeed, the client is immediately disconnected and any messages in the incoming message queue are discarded.
If the MQTT broker is not able to keep up with the client, the incoming message queue will eventually be full. When the queue is full, the MQTT broker discards any messages it receives from the client.
In case you need to upload large amount of data either historical or non-time critical data it is beneficial to use the batch upload methods. There is a batch upload procedure which either can be called as HTTP command using “curl” – this is intended for manual upload, or it can be imbedded in a Python script.
To import data from csv-files you need to:
There is a python helper class that makes this easier, see the next section.
In all the http requests to http://admin.energydata.dk/api/v1/import below you need to add an authorization header and an accept header. The authorization must be a bearer token, where the token is a batch import data token, that the user can create in the api token section of the energydata.dk ui.
Here is an example of the headers, with a fictional token:
Authorization: Bearer bzkMFuZgTzsuOldud98fJ5RkRHBI4Uoprdub1R"
Accept: application/json
The file must be formatted as CSV according to RFC4180 RFC4180. Additionally, if the file contains non-ascii characters, the file should be encoded as UTF-8.
The first row of the CSV file must contain headers. The header of a given column identifies the datastream to which the column’s data must be imported. The header can either be the 1 2 topic associated with the given datastream, e.g. my/topic or the datastream ID, e.g. 19323. The first column of the first row is ignored.
Every row after the header contains a timestamp in the first column. The rest of the columns contain the values applicable for the given datastream at the given timestamp.
The timestamps must be formatted as YYYY-MM-DD[T]HH:mm:ss.SSS[Z], e.g. 2021-05-01T10:12:44.432Z for May 1st 2023, 10:12:44.432 UTC . The timestamp must be in UTC timezone.
The type of the data in a column must match the datatype of the datastream to which the column belongs. If a datastream is an integer, any values for that datastream must be integers as well.
If a field is left empty, the handling depends on the datatype of the corresponding datastream. If the datastream is of type string, an empty string will be imported. If the datastream is of type integer or double, no value will be imported.
Below is an example file. Note that the first datastream (117217) is of type string, the second datastream (my/topic) is of type double and the last datastream (119221) is of type integer.
;117217;my/topic;119221
2021-03-10T20:24:30.139Z;a_string;23.4121;-10
2021-03-10T20:24:31.144Z;"another string";999888777.121;0
2021-03-10T20:24:32.161Z;a third string;-1.33e-16;45
2021-03-10T20:24:33.186Z;;54.1;11
2021-03-10T20:24:34.201Z;a-fourth-string;;45
To create an import, post an import name to https://admin.energydata.dk/api/v1/import, with the import as an url parameter.
Show and delete
Get the upload url with the id returned in the creation above:
curl -H "Authorization: Bearer
zGefiozgtCOfW70jnydNkTF35WOctQUnEMGAYKpi" -H "Accept: application/json"
https://admin.energydata.dk/api/v1/import/1340/upload_url
# Response example
# {
# "upload_url":
"https://s3.energydata.dk/import/inbox/example-user/b23sdfb23b163-4dfc-84c8-87ded19fdse1a.csv?X-Amz-ContentSha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&XAmz-Credential=edk-s3root%2F20230327%2Fdefault%2Fs3%2Faws4_request&X-AmzDate=20230327T083455Z&X-Amz-SignedHeaders=host&X-AmzExpires=7200&X-AmzSignature=16697645e0f009c791e62f237dfd80e74d9ddfda83feb8801sd5 af186c81e0c0b"
}
Upload the file using curl, the python helper script in or see the minio documentation for alternatives.
In the following curl example the file-path used is batch-import-example.csv
To validate the file format and data, send a PUT to https://admin.energydata.dk/api/v1/import/1340/validate, in the following example the import id 1340 is used
curl -H "Authorization: Bearer
bzkMFuZgTzsuOldud98fJ5RkRHBI4Uoprdub1R" -H "Accept:
application/json" -X PUT
https://admin.energydata.dk/api/v1/import/1340/validate"
A correct request will just accept the task. To see the status of the validation you need to post a GET request as specified in section 2.1. On the returned import will be a status and any errors.
To ingest the file format and data, send a PUT to https://admin.energydata.dk/api/v1/import/1340/ingest, in the following example the import id 1340 is used
curl -H "Authorization: Bearer
bzkMFuZgTzsuOldud98fJ5RkRHBI4Uoprdub1R" -H "Accept:
application/json" -X PUT
https://admin.energydata.dk/api/v1/import/1340/ingest"
A correct request will just accept the task. To see the status of the ingestion you need to post a GET request as specified in section 2.1. On the returned import will be a status and any errors.
This is the documentation for a class made to make importing into Energydata.dk easier from Python. The code can be found in this repository:
https://git.elektro.dtu.dk/energydatadk/batch-import-python-api
EnergyDataImport
class EnergyDataImport()
A simple class to make batch importing to Energydata.dk easier from Python
This class handles building a csv file in the correct format to be used for importing to Energydata.dk via the batch API. The batch API itself is documented seperately (see previous section). This class can handle everything from creating the propper csv to uploading, validating and ingesting the file. You can check the ‘example_import.py’ script for an example of how to use this class.
__init__(upload_filename: str, properties: List, sftp_username: str, sftp_password: str, energydata_api_token: str, overwrite: bool = False, tmp_dir: str = '/tmp/energydata_batch_uploud', autoclean_tmp_files: bool = True)
Constructor for class. Create an instance of EnergydataImport for each batch of data you want to upload. An instance can not be reused between multiple imports.
Arguments:
add_values(time: datetime, values: List)
Call while in open state to keep adding values to be batch imported.
Arguments:
Raises:
upload(print_progress=True)
When all values have been added using add_values method, this method can be called to upload the generated csv file to the SFTP server, and create a import job with the energydata.dk API. Once called, state of the job will transition from open to uploading, uploaded and finally to stored once the job is created.
Arguments:
validate(block=True, print_progress=True)
When in stored state, this method can be called to trigger validation of the data to be imported. This will happen by calling the corresponding validate endpoint on energydata.dk. State will transition from stored to validating and finally to ready.
Arguments:
ingest(block=True, print_progress=True)
When in ready state, this method can be called to trigger ingestion of the data to be imported. This will happen by calling the corresponding ingest endpoint on energydata.dk. State will transition from ready to ingesting and finally to done.
Arguments:
refresh_status()
This method can be used to manually refresh and sync the internal state of the import job with the state in energydata.dk. If for instance validate is called with block=False, this method must be used to refresh the state before ingest can be called, once validation has completed on the server.
Returns:
EnergyDataDK is continuously developed in collaboration with our partners.
EnergyDataDK was originally supported by EUDP and was completed in August 2022. EnergyDataDK has subsequently been further developed and expanded with, among other things, data from Bornholm with support from the EU. EnergyDataDK now functions as an independent data platform and hub for digital solutions.
EnergyDataDK is part of DTU’s PowerLabDK and is a digital hub for testing and developing digital energy solutions.
If you have any questions, please feel free to contact us at EnergydataDK@dtu.dk