a digitalisation layer for the physical laboratories in PowerLab.
Note, that energydata.dk is constantly evolving with new data sets so the “screen dumps” in the figures in this document will most likely be outdated.
Please report any error in this document or suggestions for improvement to
info@energydata.dk
| Term | Meaning |
|---|---|
| Administrator (group) |
The administrator is a group (member) user with additional privileges, e.g., the administrator can add new users to a group (making the user a member of the group), remove members from the group or grant license to a data set and by this make it accessible for the group. The administrator can only grant access to data sets that the administrator owns (has ownership of). |
| Data set | A data set is a collection of data streams which are related in some way. As user you cna by an owner of the data set be granted access to a data set by which you have access to all the data streams within the data set. |
| Data stream | Data is organized in energydata.dk in data sets and data streams. A data stream is like channel where data from a sensor, measurement device or similar is received. All observation at the channel is a tuple with a time stamp for when the observation was measured and the measurement itself, i.e., the “value”. All timestamps in energydata.dk is in UTC time. A new observation with same timestamp will overwrite an existing observation. Note it is the measurement device that set a time stamp for the observation. Each data stream shall be described by a number of mandatory “tags” (meta data) the qualify the data. Additional tags can be added. |
| Group | A collection of users is called a group, they all share the same privileges, i.e., which data sets they can access and what privileges they have for the data set (read / write access). Every user that owns (administrate) a data set can create a group |
| License | Read or write access to a data set is controlled by a license associated to the data set for the group. |
| Member (group) | A user can be member of a group and by that membership have access to the data sets linked to the group |
| Owner | The owner of a data set is the user that has privileges to maintain the data set and normally also the user, that created the data set. An owner can appoint other users to be owner. Appointed owners share the same privileges as the appointer. |
| Read | A privilege that makes it possible to view a data stream in the preview window, export the data to a csv file or access the data via an API (for reading) |
| Tag | Field in the meta data for a data stream that describes / qualify the data stream |
| Topic | A unique label for a data stream. The topic is used when insert or requesting data for a data stream via an API. |
| Write | A privilege that makes it possible to insert data to a data stream via an API |
Many of the data sets available in energydata.dk is described in the document Data in energydata.dk.
At signup to energydata.dk you will be granted access to all public (open) data in energydata.dk. Access to other data sets in energydata.dk must be granted by one of the owners of the data set. Refer to the description of the data set to see the owner(s) and to understand how to request access to the data. For many datasets you simply request it by clicking the padlock on one of the datastreams in the dataset
After log in you will be presented for a welcome page as shown in Figure 3. To the right you will see small graphical tiles representing each of the available data set in energydata.dk. Note, that data sets with no defined data streams are dimmed.
In the left part there is a search bar and a filtering form. In the search bar you can enter a search string for searching a data set. The search engine will search for a match in any part of the meta data and description of the available data sets. The search engine works like other search engines so you can enter e.g. “SGU” (Smart Grid Unit) and the search will find all data set that contains data streams with the search string (SGU) in its meta data or description. The result will be displayed as number of data streams that match the search string, see Figure 4.
In the above example the data set “EnergyLab Nordhavn” holds 64 data streams that match SGU, the other data sets do not contain any streams indicated by the “No data streams found” and the tiles are dimmed.
The search can be tailored by operators, e.g., you can search for SGU and exclude those that match “Rønne” by entering a search string as “SGU -Rønne”. The search result will exclude SGU in Rønne, see Figure 5
The search now finds 48 data streams.
Be aware that the search engine uses a fuzzy search algorithm meaning that it will find matches that are close to your search string but not necessarily an exact match, e.g., if you search for SGU frequency it will also show results for SGU fuse, because the words “frequency” and “fuse” are sufficient close to each other to make a “hit”.
In the “Filter on tags” part you can apply filters to your search, see Figure 6 and only the data sets including streams that match the filter and search string will be presented.
In the “geo tags" section you can enter a geographic location. The text field is a drop down presenting a list of all available geo tags. Available “geo tags” are all the declared “geo tags” when all data streams in energydata.dk are defined. The same principle applies to other “tags”.
When you select a data set, the frame around the tile will be blue and to the left you will see a description of the data, see Figure 7.
The search button will automatically select all data streams in the data set and these streams will be shown in the data stream window, see Figure 8
If there is a “padlock” symbol to the left of a data stream you are not granted access to the data stream. For getting access you must contact the data set owner. You do this by “left clicking” on the padlock. A form appears, see Figure 9, and you can request by fulfilling the form. The data owner will receive a mail by the system and decide to grant you access.
If you have access to the data streams and you select a stream, a preview of the data is shown in the preview window, see Figure 10.
In the preview window you will see a plot for the:
metrics for the default time range (one week). You can select / de-select the metrics by “clicking” on the icons. The time range can be adjusted by the glider below the plot or by selecting the “time range” box and adjust the start and end date, see Figure 11
Please observe that:
At signup to energydata.dk you will be granted access to the public available data sets. Even the data sets are public available there will be some licensee’s conditions that you will have to comply with. You will find these licensee’s conditions in the description part for the data set, see Figure 12.
You are responsible for respecting these conditions.
Other available data sets will be shown as tiles, see Figure 3. If you are not granted access to the data set, you will only be able to read the data set description. From here it shall also appear who you must contact to be granted access.
All data sets where you have a read access granted can be exported i.e., you can download the complete data set or a subset. Read access is indicated by a tick box in the data stream window, see Figure 8. A “padlock” indicates no access. The export of a data set results in a link to a csv file, that can be downloaded. The link is sent in a mail to the email address in your profile. You select a stream to be included in the export by check-marking the box to the left of the data stream, see Figure 13.
In the “staged” tab you can see which data streams that has been selected for export. This can be beneficial if the data set contains many streams, as in this example where there are 29 streams.
In the “export” tab the export is configured and initiated, see Figure 14
Per default the export file is named “Energydata export dd-mm-yyyy HH-MM-SS”, i.e. date and time for the export but you can select another name according to your preferences, e.g., the data set that is exported.
The export file is a csv file that can be imported into whatever application you prefer. The file contains a column for each stream selected for the export plus a column that contains the time stamp for the observation. Each column heading is named according to this:
<data stream name>|<data stream topic>|<data stream property ID>
The “topic” and “property ID” are useful to know and needed if you want to use the data APIs.
You can manage your own data sets and create new data sets for your projects. Select the “Setting” in the dropdown menu the in the upper right corner and choose the “Datasets” option, see Figure 15.
Then you will be presented for all the data sets you have ownership of, i.e., also data sets others have created and granted you ownership to, see Figure 16 if in tile mode.
or as in Figure 17 if in list mode
You switch between tile and list mode by clicking the icons in the upper right corner.
Spend some time describing your data set – this way others can read about and understand the content of the data set and you may yourself need it for later reference, if you must go back to your data set later. You can add a picture illustrating the content. This picture will be shown in the date set tile window, see Figure 3.
The MQTT prefix is a prefix for all data stream topics. The prefix must be unique and identify the data set.
Then you must complete the form, see Figure 21, that appear when pressing the button.
You will be notified by mail about the outcome of the upload.
You can insert data to your data set (data stream) by using one of API’s, see 8.2.
You can access your data set via one of these methods:
To do so, you need a token, the token serves as userID & password to verify that you are allowed to access the data set.
To create a token, select the “API Tokens” in the “Setting menu” in Figure 24
An API token creation formular as in Figure 25 appears. Assign a name for new token, the name is used when managing your token in the system, choose a name that is associated with usage of the token.
Select the type of token (personal or deployment), note their different usage:
DEPLOY TOKENS
Deploy tokens are explicitly allowed to do certain operations on specific datasets, and should be used when deploying devices and similar, out in the world. If these devices are compromised, the token has very limited access. You can link any of the licenses you have via group memberships to the token
PERSONAL ACCESS TOKENS
These tokens carry all the same rights as the user issuing them. They should be used only on the user’s own computer for local development. WARNING: If compromised this token can do be used to access everything you can.
For deployment tokens select for which of the API's the token will be used, multiple selections are allowed and finally select which combination of permission, data set and group, the token will be used for see Figure 26. You can use the search box to find the data set, see Figure 26. A personal token grant you access to all the data sets where you have access and for all API, therefore you shall do no selection in Figure 26.
The tokens you have created are listed in the “Manage API Tokens” list which you find below the API create formular, see Figure 27
If you select “View” for a token you can see the details for that token, see Figure 28
Further you can delete the token. If you delete a token, it is no longer valid and the usage in the APIs will fail. The same will happen if you revoke the license associated with the token. If you later grant a similar license again, the token will not be associated with a new license, you must create a new token for the new “group – data set – license” relationship.
You can find a description of the API's and how to use them in the API description
To manage one of your data sets, mouse over either the data set tile if in tile mode (Figure 16) or the data set list if in list mode (Figure 17). Select a data set and a window as in Figure 29 will appear.
Selecting “Edit dataset details”, you will access a form where you can update the name, description for the data set and the image, see Figure 30
From the top menu bar (Figure 21) there are 4 tabs: “Details”, “Owners”, “Licenses” and “Publish”. “Details” are by default selected.
In the top form there is general information about the data set; when it was created, number of data streams etc. In the lower form all the data streams are listed.
When you mouse over a data stream tag a pencil will appear for the data stream, see Figure 31, and you can edit the tag. For optional tags a bin icon is shown and the tag can be deleted.
For others to benefit from your data or to collaborate, you can share your data with other users of energydata.dk. You can do this by granting access to your data sets. Access rights are assigned to a “group” and all members of the group share the same privileges. A data set is linked to one or more groups. This is illustrated in Figure 32.
Please note that a user can be a member of several groups and several of these groups can have access to the same data set, so if you need to revoke access to a data set for a user be sure to examine all groups linked to the data set.
The list shows all the groups that you are either members of or administrator for.
Create a new group by pressing the “+ GROUP” button and complete the form in Figure 34
When you press the ”create” button a new form appear as shown in Figure 35.
From this form (Figure 35) you can grant a license to a group (see 8.4.2) or members (see 8.4.3)
To assign a license to a group go to the group details form (see Figure 35) and press the “+license” button. The form in Figure 36 will appear. In this form select the “permission” and eventually an expiration date, the “Terms” is for future use. Link the license to a data set from the list. Repeat the procedure if you need to assign more than one permission or grant it to additional data sets.
Press the “+ ADD MEMBER” button and (another 😊) new form appears as shown in Figure 38
Enter the new group member (user) email address, the role he/she shall take. Here you can select (ordinary) member or administrator. A member can access the data the group is granted access to, an administrator can further also manage (edit) the group.
You can set an expiration date; this can be useful if you want to automatically revoke the membership e.g. when the project ends.
You can publish a data set. When it is published, all users on EDDK can see that the dataset exists and it will be searchable.
When a request is made, the energy data administrator will receive the request, and if there is sufficient description and metadata, the request will be approved. If it is rejected, an explanation will be provided.
Press the PUBLISH DATASET button and you will have to choose the publishing permission, as shown in figure 40. From here you can send a publishing request.
When a request is made, the EDDK data-administrator will receive the request, and if there is sufficient description and metadata, the request will be approved. If it is rejected, an explanation will be provided.
You can supervise your data streams in your data sets if data is published via the MQTT publish service. If you have one or more scripts feeding data to your data sets via the MQTT publish you can set up a supervision that will supervise if data is received within a frequency that you define and the received values are within a range you define. If the data feed fails to comply with your settings an alarm will be raised and you will be notified by mail.
The create a supervision select the “alarms” option in the setting menu, see Figure 39.
New window with two tabs appears; “Alarm log” and “Alarms”. In the “Alarm log” all raised alarms are listed with information about the raised alarms, see Figure 40
Selecting the “alarm” tab, all the defined supervisions (alarms) is listed and it is possible to define a new alarm by selecting the “+ ALARM” button, see Figure 41
Complete the form that appears, see Figure 42
When defining the new supervision note:
Select frequency if you need to supervise that data is received and threshold if you need to supervise the received values
Can either be for a stream or data set
By dividing a dataset, you will create a new dataset with the given name, MQTT topic prefix. The datastreams you select will be moved into the new dataset.
The datastreams which are moved into the new dataset will be assigned a new MQTT topic prefix. This means any programs which publish data to these datastreams using MQTT, must be updated to use the new topic. However, it will still be possible to publish data to the old topics during the grace period.
The moved datastreams are also assigned a new datastream ID. Thus, any programs that interact with the HTTP API (either import or export), which use datastream IDs as identifiers must also be updated. Remember to create new alarm rules for the migrated datastreams and the new dataset, as the existing alarm rules will not be migrated.
The "Divide dataset" option is available from the dataset menu, as in figure 29. The option is highlighted in Figure 43.
From here, a form appears as shown in figure 44.
To log out from the system go to the “Setting” drop down menu and select “Log out” and your session will be closed and you will be presented for the log on page.
EnergyDataDK is continuously developed in collaboration with our partners.
EnergyDataDK was originally supported by EUDP and was completed in August 2022. EnergyDataDK has subsequently been further developed and expanded with, among other things, data from Bornholm with support from the EU. EnergyDataDK now functions as an independent data platform and hub for digital solutions.
EnergyDataDK is part of DTU’s PowerLabDK and is a digital hub for testing and developing digital energy solutions.
If you have any questions, please feel free to contact us at EnergydataDK@dtu.dk
