Table of contents

Energydata.dk User Manual

1. Introduction

 
This user manual aims at giving you a quick introduction for how to use the energydata.dk.
 
Energydata.dk is a data logging and data warehouse solution for collecting and sharing energy data, and it is

a digitalisation layer for the physical laboratories in PowerLab.

Note, that energydata.dk is constantly evolving with new data sets so the “screen dumps” in the figures in this document will most likely be outdated.

Please report any error in this document or suggestions for improvement to
info@energydata.dk

2. Terminology

Some terms are useful to know when reading this guide and using energydata.dk. The terms are listed in Table 1
Term Meaning
Administrator
(group)
The administrator is a group (member) user with additional privileges, e.g., the administrator can add new users to a group (making the user a member of the group), remove members from the group or grant license to a data set and by this make it accessible for the group. The administrator can only grant access to data sets that the administrator owns (has ownership of).
Data set A data set is a collection of data streams which are related in some way. As user you cna by an owner of the data set be granted access to a data set by which you have access to all the data streams within the data set.
Data stream Data is organized in energydata.dk in data sets and data streams. A data stream is like channel where data from a sensor, measurement device or similar is received. All observation at the channel is a tuple with a time stamp for when the observation was measured and the measurement itself, i.e., the “value”. All timestamps in energydata.dk is in UTC time. A new observation with same timestamp will overwrite an existing observation. Note it is the measurement device that set a time stamp for the observation. Each data stream shall be described by a number of mandatory “tags” (meta data) the qualify the data. Additional tags can be added.
Group A collection of users is called a group, they all share the same privileges, i.e., which data sets they can access and what privileges they have for the data set (read / write access). Every user that owns (administrate) a data set can create a group
License Read or write access to a data set is controlled by a license associated to the data set for the group.
Member (group)    A user can be member of a group and by that membership have access to the data sets linked to the group
Owner The owner of a data set is the user that has privileges to maintain the data set and normally also the user, that created the data set. An owner can appoint other users to be owner. Appointed owners share the same privileges as the appointer.
Read A privilege that makes it possible to view a data stream in the preview window, export the data to a csv file or access the data via an API (for reading)
Tag Field in the meta data for a data stream that describes / qualify the data stream
Topic A unique label for a data stream. The topic is used when insert or requesting data for a data stream via an API.
Write A privilege that makes it possible to insert data to a data stream via an API

3. Data in energydata.dk

Many of the data sets available in energydata.dk is described in the document Data in energydata.dk.

At signup to energydata.dk you will be granted access to all public (open) data in energydata.dk. Access to other data sets in energydata.dk must be granted by one of the owners of the data set. Refer to the description of the data set to see the owner(s) and to understand how to request access to the data. For many datasets you simply request it by clicking the padlock on one of the datastreams in the dataset

4. Signup and login

Energydata.dk is available at this address: https://admin.energydata.dk/login. When
you go to this address you will see the login page shown in Figure 1.
 
If you are DTU user, i.e., employed or student at DTU please select the “DTU login” – at the first log in, you must read and accept the “Terms of Service” and “Privacy Policy” and confirm this by checkmark the “checkmark” box.
Figure 1 Sign up and log in
Figure 1: Signup and login
If you are from outside DTU then go to the “Sign up” tab at first log in and complete the form shown in Figure 2, the e-mail address is the address where to reports, alarms etc will be sent to. 
 
Later go to the “Energydata login” tab and log in with the email and password selected at sign up.
Figure 2 Sign up formular
Figure 2: Signup formular

5. Find a data set

After log in you will be presented for a welcome page as shown in Figure 3. To the right you will see small graphical tiles representing each of the available data set in energydata.dk. Note, that data sets with no defined data streams are dimmed. 

Figure 3 Start page - data set tile
Figure 3 - Start page - data set tile

5.1 Search bar

In the left part there is a search bar and a filtering form. In the search bar you can enter a search string for searching a data set. The search engine will search for a match in any part of the meta data and description of the available data sets. The search engine works like other search engines so you can enter e.g. “SGU” (Smart Grid Unit) and the search will find all data set that contains data streams with the search string (SGU) in its meta data or description. The result will be displayed as number of data streams that match the search string, see Figure 4.

Figure 4 Search result simple search
Figure 4 - Search result simple search

In the above example the data set “EnergyLab Nordhavn” holds 64 data streams that match SGU, the other data sets do not contain any streams indicated by the “No data streams found” and the tiles are dimmed.  

The search can be tailored by operators, e.g., you can search for SGU and exclude those that match “Rønne” by entering a search string as “SGU -Rønne”. The search result will exclude SGU in Rønne, see Figure 5 

Figure 5 Tailored Search
Figure 5 - Tailored search

The search now finds 48 data streams.  

Be aware that the search engine uses a fuzzy search algorithm meaning that it will find matches that are close to your search string but not necessarily an exact match, e.g., if you search for SGU frequency it will also show results for SGU fuse, because the words “frequency” and “fuse” are sufficient close to each other to make a “hit”.  

In the “Filter on tags” part you can apply filters to your search, see Figure 6 and only the data sets including streams that match the filter and search string will be presented. 

Figure 6 Search Filter
Figure 6 - Search Filter

In the “geo tags" section you can enter a geographic location. The text field is a drop down presenting a list of all available geo tags. Available “geo tags” are all the declared “geo tags” when all data streams in energydata.dk are defined. The same principle applies to other “tags”. 

5.2 Data description

When you select a data set, the frame around the tile will be blue and to the left you will see a description of the data, see Figure 7.

Figure 7 Data set description
Figure 7 - Data set description

5.3 Data streams

The search button will automatically select all data streams in the data set and these streams will be shown in the data stream window, see Figure 8 

Figure 8 Data stream window
Figure 8 - Data stream window

If there is a “padlock” symbol to the left of a data stream you are not granted access to the data stream. For getting access you must contact the data set owner. You do this by “left clicking” on the padlock. A form appears, see Figure 9, and you can request by fulfilling the form. The data owner will receive a mail by the system and decide to grant you access. 

Figure 9 - Request form a data access
Figure 9 - Request for data set access

If you have access to the data streams and you select a stream, a preview of the data is shown in the preview window, see Figure 10.

Figure 10 Data preview
Figure 10 - Data preview

In the preview window you will see a plot for the:

metrics for the default time range (one week). You can select / de-select the metrics by “clicking” on the icons. The time range can be adjusted by the glider below the plot or by selecting the “time range” box and adjust the start and end date, see Figure 11

Figure 11 Adjusting start and stop dates
Figure 11 - Adjusting start and stop dates

Please observe that:

6. Access a data set

At signup to energydata.dk you will be granted access to the public available data sets. Even the data sets are public available there will be some licensee’s conditions that you will have to comply with. You will find these licensee’s conditions in the description part for the data set, see Figure 12

You are responsible for respecting these conditions. 

Other available data sets will be shown as tiles, see Figure 3. If you are not granted access to the data set, you will only be able to read the data set description. From here it shall also appear who you must contact to be granted access. 

Figure 12 License Conditions
Figure 12 - License Conditions

7. Data export

All data sets where you have a read access granted can be exported i.e., you can download the complete data set or a subset. Read access is indicated by a tick box in the data stream window, see Figure 8. A “padlock” indicates no access. The export of a data set results in a link to a csv file, that can be downloaded. The link is sent in a mail to the email address in your profile. You select a stream to be included in the export by check-marking the box to the left of the data stream, see Figure 13.

Figure 13 Data export
Figure 13 - Data stream selection for export

In the “staged” tab you can see which data streams that has been selected for export. This can be beneficial if the data set contains many streams, as in this example where there are 29 streams.

In the “export” tab the export is configured and initiated, see Figure 14 

Figure 14 Data stream export
Figure 14 - Data stream export

Per default the export file is named “Energydata export dd-mm-yyyy HH-MM-SS”, i.e. date and time for the export but you can select another name according to your preferences, e.g., the data set that is exported.  

The export file is a csv file that can be imported into whatever application you prefer. The file contains a column for each stream selected for the export plus a column that contains the time stamp for the observation. Each column heading is named according to this:

<data stream name>|<data stream topic>|<data stream property ID>

The “topic” and “property ID” are useful to know and needed if you want to use the data APIs. 

8. Your own data

You can manage your own data sets and create new data sets for your projects. Select the “Setting” in the dropdown menu the in the upper right corner and choose the “Datasets” option, see Figure 15.

Figure 15 Setting menu
Figure 15 - Setting menu

Then you will be presented for all the data sets you have ownership of, i.e., also data sets others have created and granted you ownership to, see Figure 16 if in tile mode.

Figure 16 - My data sets, tile mode

or as in Figure 17 if in list mode

Figure 17 - My data sets, list mode

You switch between tile and list mode by clicking the icons in the upper right corner.

8.1 How to create a new data set

To create a new data set, select the +DATASET button in the upper right corner (Figure 17) and complete the formular that appears, see Figure 18.

Figure 18 Creating a data set
Figure 18 - Creating a data set

Spend some time describing your data set – this way others can read about and understand the content of the data set and you may yourself need it for later reference, if you must go back to your data set later. You can add a picture illustrating the content. This picture will be shown in the date set tile window, see Figure 3

The MQTT prefix is a prefix for all data stream topics. The prefix must be unique and identify the data set. 

8.1.1 Add a data stream to a data set

The “+ New datastream” button in Figure 19 allows you to add a new data stream to the data set; type the data stream name and press “Add”, see Figure 20 

Figure 19 Add first data stream
Figure 19 - Add first data stream
Figure 20 Adding a new data stream
Figure 20 - Adding a new data stream

Then you must complete the form, see Figure 21, that appear when pressing the button.

Figure 21 - Data stream definition

The “upload” button, see Figure 22, allows you to upload a data stream definition file where you can add several new data streams by importing a data stream definition file. A guidance for the file format is available, see Figure 23

Figure 22 Data stream definition upload
Figure 22 - Data stream definition upload
Figure 23 Data stream definition upload
Figure 23 - File format guidance

 You will be notified by mail about the outcome of the upload. 

8.1.2 Inserting data to a data set

You can insert data to your data set (data stream) by using one of API’s, see 8.2.

8.2 Access your data set via an API

You can access your data set via one of these methods: 

To do so, you need a token, the token serves as userID & password to verify that you are allowed to access the data set. 

8.2.1 Create and manage API tokens

To create a token, select the “API Tokens” in the “Setting menu” in Figure 24 

Figure 24 Setting - API token
Figure 24 - Setting: API Tokens

An API token creation formular as in Figure 25 appears. Assign a name for new token, the name is used when managing your token in the system, choose a name that is associated with usage of the token. 

Figure 25 - API token formular

Select the type of token (personal or deployment), note their different usage: 

DEPLOY TOKENS

Deploy tokens are explicitly allowed to do certain operations on specific datasets, and should be used when deploying devices and similar, out in the world. If these devices are compromised, the token has very limited access. You can link any of the licenses you have via group memberships to the token

PERSONAL ACCESS TOKENS

These tokens carry all the same rights as the user issuing them. They should be used only on the user’s own computer for local development. WARNING: If compromised this token can do be used to access everything you can.  

For deployment tokens select for which of the API's the token will be used, multiple selections are allowed and finally select which combination of permission, data set and group, the token will be used for see Figure 26. You can use the search box to find the data set, see Figure 26. A personal token grant you access to all the data sets where you have access and for all API, therefore you shall do no selection in Figure 26

Figure 26 - Create an API token

The tokens you have created are listed in the “Manage API Tokens” list which you find below the API create formular, see Figure 27 

Figure 27 - Manage token list

If you select “View” for a token you can see the details for that token, see Figure 28 

Figure 28 Token details
Figure 28 - Token details

Further you can delete the token. If you delete a token, it is no longer valid and the usage in the APIs will fail. The same will happen if you revoke the license associated with the token. If you later grant a similar license again, the token will not be associated with a new license, you must create a new token for the new “group – data set – license” relationship.

8.2.2 API documentation

You can find a description of the API's and how to use them in the API description

8.3 How to manage an existing data set

To manage one of your data sets, mouse over either the data set tile if in tile mode (Figure 16) or the data set list if in list mode (Figure 17). Select a data and a window as in Figure 29 will appear.

Figure 29 - Selecting a data set for managing

Selecting “Edit dataset details”, you will access a form where you can update the name, description for the data set and the image, see Figure 30 

Figure 30 Update data set
Figure 30 - Update data set

From the top menu bar (Figure 21) there are 4 tabs: “Details”, “Owners”, “Licenses” and “Publish”. “Details” are by default selected. 

In the top form there is general information about the data set; when it was created, number of data streams etc. In the lower form all the data streams are listed.

When you mouse over a data stream tag a pencil will appear for the data stream, see Figure 31, and you can edit the tag. For optional tags a bin icon is shown and the tag can be deleted. 

Figure 31 - Selecting a data stream for editing

8.4 Sharing your data

For others to benefit from your data or to collaborate, you can share your data with other users of energy.data.dk. You can do this by granting access to your data sets. Access rights are assigned to a “group” and all members of the group share the same privileges. A data set is linked to one or more groups. This is illustrated in Figure 32.

Figure 32 Group - user - data set relation

Please note that a user can be a member of several groups and several of these groups can have access to the same data set, so if you need to revoke access to a data set for a user be sure to examine all groups linked to the data set. 

8.4.1 Creating a group

Go to the setting menu, see Figure 15 and select the “Group menu”. A form as shown in Figure 33 appears. 

Figure 33 - Group menu

The list shows all the groups that you are either members of or administrator for.  

Create a new group by pressing the “+ GROUP” button and complete the form in Figure 34 

Figure 34 - Create new group

When you press the ”create” button a new form appear as shown in Figure 35.

Figure 35 - Group details

From this form (Figure 35) you can grant a license to a group (see 8.4.2) or members (see 8.4.3)

8.4.2 Assigning a license to a group

To assign a license to a group go to the group details form (see Figure 35) and press the “+license” button. The form in Figure 36 will appear. In this form select the “permission” and eventually an expiration date, the “Terms” is for future use. Link the license to a data set from the list. Repeat the procedure if you need to assign more than one permission or grant it to additional data sets. 

Figure 36 - Linking license to a data set

8.4.3 Adding members to a group

To add members to the group, select the “member” tab in the upper left corner, Figure 35.

Add new form as shown in Figure 37 appears  

Figure 37 - Add member form

Press the “+ ADD MEMBER” button and (another 😊) new form appears as shown in Figure 38

Figure 38 - Member details form

Enter the new group member (user) email address, the role he/she shall take. Here you can select (ordinary) member or administrator. A member can access the data the group is granted access to, an administrator can further also manage (edit) the group. 

You can set an expiration date; this can be useful if you want to automatically revoke the membership e.g. when the project ends. 

8.5 Monitor a data set

You can supervise your data streams in your data sets if data is published via the MQTT publish service. If you have one or more scripts feeding data to your data sets via the MQTT publish you can set up a supervision that will supervise if data is received within a frequency that you define and the received values are within a range you define. If the data feed fails to comply with your settings an alarm will be raised and you will be notified by mail. 

The create a supervision select the “alarms” option in the setting menu, see Figure 39

Figure 39 Alarm setting menu
Figure 39 - Alarm setting menu

New window with two tabs appears; “Alarm log” and “Alarms”. In the “Alarm log” all raised alarms are listed with information about the raised alarms, see Figure 4

Figure 40 - Alarm log

Selecting the “alarm” tab, all the defined supervisions (alarms) is listed and it is possible to define a new alarm by selecting the “+ ALARM” button, see Figure 41

Figure 41 - Create a new supervision

Complete the form that appears, see Figure 42

Figure 42 - Define a new supervision

When defining the new supervision note:

  • Alarm type

Select frequency if you need to supervise that data is received and threshold if you need to supervise the received values

  • Source type

Can either be for a stream or data set

  • Alarm type
Define how frequently you want alarms, set to 0 (hours) if you need a continuously supervision.

9. Log out

To log out from the system go to the “Setting” drop down menu and select “Log out” and your session will be closed and you will be presented for the log on page.