a digitalization layer for the physical laboratories at PowerLab.
Please note, since EnergyDataDK is continuously provided with additional datasets, the images in this document may be dissimilar from what you see on your own screen.
Please report any error in this document or suggestions for improvement to EnergyDataDK@dtu.dk
| Term | Definition |
|---|---|
| Administrator (group) | The administrator is a groupmember with full privileges, e.g., the administrator can add users to, or remove users from a group. They can additionally grant license to a dataset, thereby making it accessible to the group. The administrator can only grant access to data sets that the administrator has ownership of. |
| Datastream | Data is organized in EnergyDataDK in datasets and datastreams. A datastream is a source of data, for example from a sensor, a measurement device, or something similar. All observations from the source are a timestamp (when the measurement was performed) and the measurement itself, i.e., the “value”. All timestamps in EnergyDataDK are in UTC time. A new observation with a duplicate timestamp will overwrite the existing observation. Please note that the measurement device provides a timestamp for the measurement. Each data stream shall be described by a number of mandatory “tags” (meta data) in order the qualify the data. Additional non-mandatory tags may be added. |
| Dataset | A dataset is a collection of datastreams which are related in some way. The owner of the data can grant users access to a dataset, thereby granting access to all the datastreams within the dataset. |
| Group | A collection of users which share the same privileges, i.e., which datasets they can access and what privileges they have (read or write access).Any user who owns (administrates) a dataset can create a group. |
| License | read or write access to a dataset is controlled by a license associated to the dataset for the group. |
| Member (group) | Membership of a group grants access to the datasets linked to the group |
| Owner | The owner of a dataset has privileges to maintain the dataset and is usually the dataset's creator. An owner can appoint additional owners. Appointed owners share identical privileges as the appointer. |
| Read | A privilege which enables viewing a datastream in the preview window, exporting the data to a CSV file, or fetching the data via an API. |
| Tag | A metadata field for a datastream which describes or qualifies the datastream. |
| Topic | A unique label for a datastream. The topic is used when inserting or requesting data for a data stream via an API. |
| Write | A privilege which enables adding data to a datastream via an API |
Many of the datasets available in EnergyDataDK are described in the document data in EnergyDataDK.
When you sign up to EnergyDataDK you'll be granted access to all public (open) data in EnergyDataDK. Access to other datasets in EnergyDataDK must be granted by one of the owners of the dataset. Refer to the description of the dataset to see the owner(s) and to understand how to request access to the data. For many datasets you can request access by clicking the padlock on one of the datastreams in the dataset.
EnergyDataDK is available at this address: https://admin.energydata.dk/login
When you navigate to this address you'll be presented with the login page shown in the image below (Figure 1).
If you're a DTU user, i.e., employed or student at DTU, please select the DTU login tab. If you're logging in for the first time, you must read and accept the Terms of Service and Privacy Policy. Confirm by checking the “I agree…” box.
After logging in you'll be presented with a welcome page as shown in Figure 3. To the right you'll be presented with graphical tiles, each representing available datasets in EnergyDataDK.
Please note that empty datasets with no defined datastreams are dimmed.
The left side of the screen has a search bar and a filtering form. In the search bar you can enter a search string to search for specific datasets. The search engine will search for a match in any part of the metadata and the description of available datasets. The search engine works like other search engines so you can enter e.g. “SGU” (Smart Grid Unit) and the search will find all data set that contains data streams with the search string "SGU" in its metadata or description. The result will be displayed as number of datastreams which match the search string, see Figure 4.
In the above example the dataset “EnergyLab Nordhavn” holds 64 datastreams which match our query, the other datasets don't contain any streams matching the query as indicated by the “No datastreams found” message, and the dimmed tiles.
The search can be tailored by operators, e.g., you can search for “SGU” and exclude those that match “Rønne” by entering the following search string: “SGU -Rønne”. The search result will exclude datastreams which match “Rønne”, see Figure 5
The search now finds 48 datastreams.
Be aware that the search engine uses a fuzzy search algorithm. This means that it will find matches that are close to your search string but not necessarily an exact match, e.g., if you search for “SGU frequency” the results will include “SGU fuse”, because the words “frequency” and “fuse” are sufficiently similar.
In the Filter on tags section you can apply filters to your search, see Figure 6. Only datasets that include streams which match the filter and search string will be presented.
In the geo tags section you can enter a geographic location. The text field is a drop down presenting a list of all available geo tags. Available “geo tags” are all the declared “geo tags” when all data streams in EnergyDataDK are defined. The same principle applies to other tags.
When you select a dataset, the frame around the tile will turn blue and to the left you'll be provided with a description of the data, see Figure 7.
The search button will automatically select all datastreams in the dataset, these streams will be shown in the datastream window, see Figure 8
If there's a padlock icon to the left of a datastream you can't access the data stream. To gain access you must contact the dataset owner. You can do this by clicking on the padlock icon.
A form will appear, see Figure 9. Completing the form will send an access request via email to the data owner, who can then choose to grant you access.
If you select a stream among those you have access to, a preview of the data is displayed in the preview window, see Figure 10.
In the preview window a graph will be presented for the
metrics for the specified time range (one week by default). You can (de)select metrics by clicking on the corresponding icons. The time range can be adjusted by the glider below the graph or by entering a range in the time range field to adjust the start- and end date, see Figure 11
Please observe that:
When signing up to EnergyDataDK you'll be granted access to the publicly available datasets. Even among the datasets which are publicly available there will be some licensee’s conditions that you will have to comply with. You'll find these licensee’s conditions in the description part for the data set, see Figure 12. It's your responsibility that these conditions are complied with.
Other available datasets will be shown as tiles, see Figure 3. If you're not granted access to a dataset, you'll only be able to read the dataset description. Here you can also see who you need to reach out to to be granted access.
All datasets to which you have read privileges can be exported i.e., you can download the complete dataset or a subset. Read access is indicated by a tick box in the datastream window, see Figure 8. A padlock icon indicates that access is restricted. Exporting a dataset results in a link to a CSV file, which can be downloaded. The link is sent in a mail to the email address listed in your profile. You can select a stream to be included in the export by checking the box to the left of the data stream, see Figure 13.
In the staged tab you can see which datastreams have been selected for export. This can be beneficial if the dataset contains many streams, as in this example where there are 29 streams included.
In the export tab the export is configured and initiated, see Figure 14
By default the export file will have the following naming format: “Energydata export dd-mm-yyyy HH-MM-SS”, i.e. date and time for the export. However, you can enter any name according to suit your preferences.
The export file is a CSV file which can be imported into a different application. The file contains a column for each stream selected for the export plus a column that contains the timestamp for the observation. Each column heading is named according to this standard:
<data stream name>|<data stream topic>|<data stream property ID> You can manage your own datasets and create new datasets for your projects. Select Setting in the dropdown menu the in the upper right corner and choose the Datasets option, see Figure 15.
You'll be presented with all the datasets you have ownership of, i.e., also datasets others have created and granted you ownership to, see Figure 16 if in tile mode.
or as in Figure 17 if in list mode
You switch between tile and list mode using the icons in the upper right corner.
Be mindful as you're describing your data set, clear and concise descriptions ensure others can read about and understand the content of the dataset. Besides, it may serve you well in the future, if you need to revisit your dataset later. You can add a picture illustrating the content to help you easily identify your dataset. This picture will be shown in the date set tile window, see Figure 3.
The MQTT prefix is a prefix for all data stream topics. The prefix must be unique and identify the data set.
In the next step you must complete the form, see Figure 21.
You'll be notified by email about the outcome of the upload.
You can access your dataset via one of these methods:
To do so, you'll need a token, this serves as a authentication method to verify that you're allowed to access the dataset.
To create a token, select API Tokens in the Settings menu in Figure 24
An API token creation form as shown in Figure 25 should appear. Assign a name for the new token, this is used when managing your token in the system. Choose a name which is associated with the intended usage of the token.
Select the type of token (personal or deployment), note their different usage:
DEPLOY TOKENS
Deploy tokens are explicitly allowed to do certain operations on specific datasets, and should be used when deploying devices and similar, out in the world. If these devices are compromised, the token has very limited access. You can link any of the licenses you have via group memberships to the token
PERSONAL ACCESS TOKENS
These tokens carry all the same rights as the user issuing them. They should be used only on the user’s own computer for local development.
⚠ WARNING: A compromised token can be used to access everything you can.
For deployment tokens select for which of the APIs the token will be used, multiple selections are allowed. Finally, select which combination of permission, dataset, and group the token will be used for, see Figure 26. You can use the search field to find the dataset, see Figure 26. A personal token grants you access to all the datasets you have access to and for all APIs, therefore you shouldn't make a selection in Figure 26.
If you select View on the token line, you can inspect its details, see Figure 28
Finally, you can delete tokens. When you delete a token, it is no longer valid and any API call using it will fail. You'll get the same result if license associated with the token is revoked. When you grant a similar license again in the future, the token won't be associated with a new license. A new token must be created for the new group-dataset-license relationship.
To manage one of your datasets, mouse over either the dataset tile if in tile mode (Figure 16), or the dataset list if in list mode (Figure 17). A window similar to the one in Figure 29 will appear when a dataset is selected.
Upon selecting Edit dataset details, you'll access a form where you can update the name, the description for the dataset, and the image, see Figure 30
From the top menu bar (Figure 21) there are 4 tabs: Details, Owners, Licenses, and Publish. Details is selected by default.
General information about the dataset is displayed in the top section; when it was created, the number of datastreams, etc.
Datastreams are listed in the lower section.
When you hover over a datastream tag, a pencil icon will appear, see Figure 31, clicking it allows you to edit the tag. A bin icon is displayed for optional tags, which allows for deletion of the tag.
To allow others to benefit from your data, or to collaborate, you can share your data with other users of EnergyDataDK. You can do this by granting access to your datasets. Access rights are assigned to a group, all its members share the same privileges. A dataset can be linked to a single-, or multiple groups. This is illustrated in Figure 32.
Please note that a user can be a member of several groups and several of these groups can have access to the same dataset. Therefore, if you need to revoke access to a dataset for a specific user be sure to examine all groups that are associated with the dataset.
The list shows all groups that you are either a member of, or an administrator for.
Create a new group by pressing the + GROUP button and complete the form in Figure 34
When you press the CREATE button a new view appears as shown in Figure 35.
From this view (Figure 35) you can grant a license to a group (see 8.4.2) or to individual members (see 8.4.3)
To assign a license to a group, navigate to the group details form (see Figure 35) and press the +LICENSE button. The form in Figure 36 will appear. In this form select the privilege level in the Permission field. If you want the license to have an expiry date, you can set it in the Expiration Date field, the Terms field is for future use. Link the license to a dataset from the list.
Repeat the procedure if you need to assign more than one permission or grant permissions to additional datasets.
Press the + ADD MEMBER button, a new view appears as shown in Figure 38
Enter the new group member's email address, and select their role. You can select "Member" or "Administrator". A member can access the data the group is has access to, an administrator can also manage the group.
You can set an expiration date; this can be useful if you want to automatically let the user's membership expire, for example when a project ends.
You can publish a dataset. When it is published, all users on EnergyDataDK can see the dataset and it will show up in search queries.
When a request is made, the energy data-administrator will receive an email with details about the request, and if there is an adequate description and sufficient metadata, the request can be approved. If it is rejected, a justification can be provided.
When the PUBLISH DATASET button is pressed a permission level must be selected, as shown in figure 40. From here you can send a publishing request by pressing the SEND PUBLISH REQUEST button.
When a request is made, the EnergyDataDK data-administrator will receive the request. If there is an adequate description and sufficient metadata, the request will be approved. If it is rejected, an explanation will be provided.
You can supervise your datastreams in your datasets if data is published via the MQTT publish service. If you have one or more scripts feeding data to your datasets via the MQTT publish service you can set up a supervision that will monitor if data is received within a custom defined frequency and if the received values are within a set range. If the data feed doesn't comply with your settings, an alarm will be raised and you'll be notified by email.
The create a supervision select the Alarms option in the Settings menu, see Figure 39.
A new view with two tabs appears; ALARM LOG and ALARMS. In the Alarm log view all raised alarms are listed with their corresponding details, see Figure 40
On the Alarms tab you can find a list of all existing alarms. Additional alarms can be defined using the + ALARM button, see Figure 41
Complete the form, see Figure 42
When defining the new supervision note:
By dividing a dataset, you'll create a new dataset with the given name and MQTT topic prefix. The datastreams you select will be tranferred to the new dataset.
The datastreams which are moved into the new dataset will be assigned a new MQTT topic prefix. This means any programs which publish data to these datastreams using MQTT, must be updated to use the new topic. However, it will still be possible to publish data to the old topics during the grace period.
The moved datastreams are also assigned a new datastream ID. Thus, any programs that interact with the HTTP API (either import or export), which use datastream IDs as identifiers must also be updated. Remember to create new alarm rules for the migrated datastreams and the new dataset, as the existing alarm rules will not be migrated.
The Divide dataset option is available from the Dataset menu, as in figure 29. The option is highlighted in Figure 43.
From here, a form appears as shown in figure 44.
To log out from the system go to the Settings dropdown menu and select Log out and your session will be closed and you will be presented with the log in page.