Microcontroller projects

SmartyReader

Reading Luxembourgish smartmeter data from P1 interface

last updated: 04/09/17
Its hard work to write a good documentation. This is a first draft. Thanks for your feedback. It will help to improve this page. If your interested in a pcb (10€) or a basic construction kit (smd already soldered 35€), send a mail.

smarty

Introduction

Because of the EU Energy Efficiency Directive from 2012 the seven gas and electricity DSO's in Luxembourg are replacing there gas and energy meters with smartmeters (named smarty :() til the end of 2019 (2020 for gas).
Besides gas and electricity metering, the system is open for other metering data like water and district heat (M-Bus).
The French group Sagemcom delivers the 305.000 smartmeters. All meters have to be read by one national central system, operated by a common operator. This is an economic group of interest (G.I.E.) of the 7 luxembourgish gas and electricity DSO‘s named Luxmetering G.I.E.
Luxmetering is getting the data from 4 registers for active, reactive, import and export energy (1/4h) and the 3 registers for gas, water & heat (1 h) over PLC. The smartmeter have also alarms and logs for quality of electrical energy supply (voltage, outages,...) and fraud detection, and calendar functions for the 2 external relays (home applications).

The customer wants to get his data and this is possible by reading the blinking LED of the smartmeter. This can be done with the IoT-board (see end op page).
Another possibility is the 10 second data from the smartmeter P1 port (RJ12 connector under the green lid). The P1 Data Output communication protocol and format is specified in the Dutch Smart Meter Requirements v5.0.2 . The solution deployed in Luxembourg includes an additional security layer standard that is conform to the IDIS package 2.0 requirement. The encryption layer is based on DLMS security suite 0 algorithm: AES128-GCM. More information can be found in this document.

P1 port

Hardware

The P1 port connector is a 6 pole RJ12. We get 5V (250mA) on pin 1 (Power GND on pin 6, data GND on pin 3). Data pin 5 sends an inverted serial signal (EIA232) with 8N1. To get this data the data request µline (pin 2) has to be high (5V). Pin 4 is not connected. More details in the Dutch Smart Meter Requirements. As we use a microcontroller working on 3.3V we need a transistor to bring the voltage of the data line to 5V (Q1). The second transistor inverts the serial data to be EIA232 compliant. A voltage divider (R3, R4) brings the voltage down to 3.3V.

2x2N7002

Encryption

As stated the communication on P1 port is encrypted with AES128-GCM (Galois Counter Mode). Each meter has its own 16 byte encryption key. Ask your DSO or Luxmetering for your key.
With the key we need the cypher text, 17 byte Additional Authenticated Data (AAD), a 12 byte Initialization Vector (IV) and a 12 byte GCM Tag.
The AAD is fix: 0x3000112233445566778899AABBCCDDEEFF. The other data is extracted from the serial stream.

GCM1

The Initialization Vector (12 byte) consists of the system title (8 byte) and the frame counter (4 byte). The GCM Tag is found at the end of the stream.

IoT and MQTT

The MQTT-protocol is a publisher/subscriber protocol and it is quite simple to implement this protocol on microcontrollers like the WeMos D1 mini board (ESP8266). The smartmeter data is published by the WeMos board over Wifi.
It is necessary to run a message broker (server) to distribute the data. I use for this mosquitto on a raspberry pi. The WeMos board publishes the data and the same raspberry pi with the broker or another computer subscribes to the data and generates p. ex. a graphic.

For testing and debugging you can use the cool MQTT.FX software. It's a JavaFX based MQTT Client based on Eclipse Paho an very comfortable for testing purpose. Downloads on http://www.mqttfx.org.

mqttfx

An alternative softaware is mqtt-spy.

Circuit and PCB

smartyreader circuit

BOM (basic kit)
1 220Ω SMD 0805 reichelt.de: RND 0805 1 220
1 1kΩ SMD 0805 reichelt.de: RND 0805 1 1,0K
1 10kΩ SMD 0805 reichelt.de: RND 0805 1 10K
1 15kΩ SMD 0805 reichelt.de: RND 0805 1 15K
2 100nF SMD 0805 reichelt.de: X7R 0805 CF 100N
2 2N7002 SOT-23 reichelt.de : 2N 7002 SMD
1 Wemos D1 mini pro www.wemos.cc
1 RJ12 Jack reichelt.de: MEBP 6-6S
1 RJ12 Jack reichelt.de: MEBP 6-6S
2 socket 1x8 straight reichelt.de: MPE 115-1-008
1 pin header reichelt.de: MPE 087-1-002
1 jumper reichelt.de: JUMPER 2,54GL SW
1 PCB www.weigu.lu

The WeMos is sending the data per Wifi. If your Wifi is not reliable and you have the possibility to use ethernet, an W5500 ethernet board can be added. The PCB is also prepared to use an RTC with DS3231 and an WeMos µSD card shield to log the data.


pcb_WeMos1 WeMos_PCB_Front WeMos_PCB_Back

pcb_WeMos2 pcb_WeMos3

smartyreader smartyreader boards smartyreader fully stacked

smd

Arduino Software for the WeMos-board

First install the newest Arduino IDE (1.8.3). To use our ESP8266 (WeMos) we add this line https://github.com/esp8266/Arduino/releases/download/2.4.0-rc1/package_esp8266com_index.json to File-Preferences-Additional_Boards_Manager_URLs:. We need a bigger serial buffer as default and some additional methods not available in release 2.3.0 (stable). This may change in the future so check https://github.com/esp8266/Arduino.

To install the manager go to Tools-Board:-Boards_Manager..., select the Manager and click install. No chose under Tools-Board: (you have to scroll) WeMos_D1_R2_&_mini.

We need a Crypto library to decode the AES128-GCM and a MQTT library to publish our data. Download the Crypto library at https://github.com/rweather/arduinolibs and the MQTT library at https://github.com/knolleary/pubsubclient (green field Clone_or_download-Download_ZIP). Install the library (Sketch-Include_Library-Add_.ZIP_Library...)

Unzip the file arduinolibs-master and copy the Crypto folder (in folder libraries) to your ~/Arduino/libraries/ folder.

In our Arduino Sketch we have to change the Wifi SSID and Password and our Key for the smartmeter. We use a static IP address and we can choose our mac address. The broker has also a static address. Change the parameters as needed:

    /* smartyreader.ino
      read p1 port from luxemburgish smartmeter,
      decode and publish with MQTT over Wifi
      using ESP8266 (Wemos D1 mini pro)
      weigu.lu
    */

    #include <ESP8266WiFi.h>
    #include <PubSubClient.h>
    #include <Crypto.h>
    #include <AES.h>
    #include <GCM.h>
    #include <string.h>

    #define MAX_LINE_LENGTH 1024

    byte mac[] = {  0xDE, 0xAA, 0xAA, 0xBE, 0xFE, 0xEE };
    IPAddress wemos_ip   (192,168,178,100); //static IP
    IPAddress dns_ip     (192,168,178,1);
    IPAddress gateway_ip (192,168,178,1);
    IPAddress subnet_mask(255,255,255,0);

    WiFiClient espClient;
    PubSubClient client(espClient);

    const char ssid[]     = "mywifi";
    const char password[] = "thisismypassword";

    const char *mqtt_server = "192.168.178.101";
    const int  mqttPort     = 1883;
    const char *clientId    = "smartmeters_p1";
    const char *topic       = "basement/smarty";
    const char *topic_sm1   = "basement/smarty1_p1";

    //Keys for SAG10307000xxxxx
    uint8_t key_SM1[] = {0x3B, 0x9C, 0xDB, 0x8C, 0xE3, 0xFD, 0xB7, 0x02,
                         0x16, 0x35, 0xFF, 0x6F, 0xB0, 0x2E, 0xE1, 0xDF};

Python software to get the data

A python (python3) script is used to get our smartmeter MQTT data from the broker. We use the paho.mqtt.client library which can be installed with pip to subscribe to our topic. The data is saved in a file (/data) and the old day-files are archieved (/data_archive). The script also generates a png-file with gnuplot that is displayed on an internal homepage and sent to an email address (full code in Downloads).

To to so we can use the same raspberry pi witch holds the broker.

MQTT client

You have to adjust possibly the broker IP address and your smartmeter id (these are the last 3 digits of your smarty id (ex: "345" for SAG1030700012345) ).

    #!/usr/bin/python3
    #
    # Name:         smartyreader.py
    # Purpose:      Client to get MQTT data from Mosquitto
    # Author:       weigu.lu
    # Date:         8/17
    #
    ...
    import paho.mqtt.client as mqtt
    ...
    clientID   = "getsmarty_p1"
    brokerIP   = "192.168.178.101"
    brokerPort = 1883
    topic      = "basement/smarty1"
    sm_id      = "345"

    # Callback that is executed when the client receives a CONNACK response from the server.
    def onConnect(client, userdata, flags, rc):
       print("Connected with result code " + str(rc))
       mqttc.subscribe(topic, 0)  # Subscribe to the topics (topic name, QoS)

    # Callback that is executed when we disconnect from the broker.
    def onDisconnect(client, userdata, message):
       print("Disconnected from the broker.")

    # Callback that is executed when subscribing to a topic
    def onSubscribe(client, userdata, mid, granted_qos):
       print('Subscribed on topic.')

    # Callback that is executed when unsubscribing to a topic
    def onUnsubscribe(client, userdata, mid, granted_qos):
       print('Unsubscribed on topic.')   

    # Callback that is executed when a message is received.
    def onMessage(client, userdata, message):
        # Subscribing in on_connect() means that if we lose the connection and
        # reconnect then subscriptions will be renewed.
        global sm, smp, sme, sm_mn #p power, e energy, mn at midnight
        now = datetime.now()
        now_time = now.time()
        if now_time >= time(23,59,00) and now_time <= time(23,59,59):
            sm_mn=ioj["p1"].rstrip("*kWh") # change to "c1" for consumption
        ftime = strftime("%Y_%m_%d", localtime())
        ftime3 = strftime("%d.%m.%y %H:%M:%S", localtime())
        io=message.payload.decode("utf-8");
        try:  
            ioj=json.loads(io)
        except:
            ioj={"dt":"error"}
        print(ioj)
        temp = ioj["dt"]
        if (temp[0]!='e') and (temp[0]!='c'):
            sm_new=ioj["p1"].rstrip("*kWh") # change to "c1" for consumption
            if sm!="0":
                smp = str(round((float(sm_new)-float(sm))*60000.0,3))
            sm=sm_new    
            sme=str(float(sm)-float(sm_mn))
            if sme[0]=='-':
                sme="0"
            try:    
                f = open (sm_p1_datafile1, 'r')
            except IOError:
                print("error reading file "+sm_p1_datafile1)
            lineList = f.readlines()            #read all lines
            f.close()
            try:    
                f = open (sm_p1_datafile1, 'a')
            except IOError:
                print ("Cannot create or find file: " + sm_p1_datafile1)
            try:    
                f2 = open (sm_p1_datafile2+ftime+'.min', 'a')
            except IOError:
                print ("Cannot create or find file: " + sm_p1_datafile2)
            if (len(lineList)) == 1:
                sm_p1_data = ' '.join((ftime3,sm,sme,smp))
                sm_p1_data = sm_p1_data + '\n'                                
            else:                        
                line = lineList[len(lineList)-1]    #get the last line
                lline =shlex.split(line)            #convert string (space seperated items) to list            
                sm_p1_data = ' '.join((ftime3,sm,sme,smp))
                sm_p1_data = sm_p1_data + '\n'
            print (sm_p1_data,end='')
            f.write(sm_p1_data)
            f2.write(sm_p1_data)
            f.close()
            f2.close()
        else:
            print("loop not executed (error or connect message)")

    ...
    # Main
    mqttc = mqtt.Client(client_id=clientID, clean_session=True) # create client
    mqttc.on_connect      = onConnect   # define the callback functions
    mqttc.on_disconnect   = onDisconnect
    mqttc.on_subscribe    = onSubscribe
    mqttc.on_unsubscribe  = onUnsubscribe
    mqttc.on_message      = onMessage
    mqttc.connect(brokerIP, brokerPort, keepalive=60, bind_address="") # connect to broker
    mqttc.loop_start() # start loop to process callbacks! (new thread!)

    sm ="0"
    smp = "0"
    sme = "0"
    sm_mn = "0"

    try:
        while True:  
            now = datetime.now()
            now_time = now.time()
            ...

    except KeyboardInterrupt:
        print("Keyboard interrupt by user")
        mqttc.loop_stop() # clean up
        mqttc.disconnect()

Internal homepage with daily graphic

Static IP address

To access the raspberry pi we will set a static IP address. Use the editor nano to append the following to the file /etc/dhcpcd.conf.

    # Custom static IP address for eth0.
    interface eth0
    static ip_address=192.168.1.67
    static routers=192.168.1.1
    static domain_name_servers=192.168.1.1 8.8.8.8

    # Custom static IP address for wlan0.
    interface wlan0
    static ip_address=192.168.1.69
    static routers=192.168.1.1
    static domain_name_servers=192.168.1.1 8.8.8.8
    cd /etc
    sudo nano dhcpcd.conf

Save with CTRL+O and exit with CTRL+X.

Setting up the webserver Lighttpd on the raspi

Lighttpd is an efficienct high performance webserver. It has a small memory footprint and an effective management of the cpu-load compared to other web-servers. Naturally you can use annother webserver, but possibly you have to adjust the path to your webbage.

    sudo apt update
    sudo apt upgrade
    sudo apt install lighttpd    

Test if the webserver is running by typing the ip address of your raspi in the url field of your browser.

The html files are in in /var/www/html. Copy the following html code (filename: index.html) to /var/www/html:

    <!DOCTYPE html>
    <head>
      <title>Smarty P1</title>
    </head>
    <body>
    <h1>Smarty Data</h1>
    <p><img src="sm_p1_daily.png" alt="smarty data"></p>
    </body>
    </html>

Also create an empty directory named /png in /var/www/html.

    sudo mkdir /var/www/html/png

Using gnuplot for graphics

    sudo apt install gnuplot    

To test gnuplot you can use the following command:

    cd /home/pi/smarty/gp
    gnuplot sm_p1.gp   

The sm_p1.gpis created by our python script from a template file. This template file is found in /smarty/gp (it is contained in the file smartyreader.zip). Here is the code that generates the gp file:

    def sm_create_gp_file():
        """ The function prepares the gp file for plotting with gnuplot. First the
        old gp file is deleted. Then it uses the xx_gp_template.gp file in
        ~/../gp and replaces the keywords between the % sign by creating
        a new gp (xx.gp) file."""
        ftime2 = strftime("%d.%m.%y", localtime())
        Title = ftime2
        XFormat = '"%H:%M"'
        XTics = "60*60" #seconds
        Begin = ftime2 +" 00:00:01"
        End = ftime2 +" 23:59:59"
        Output = png_dir + "sm_p1_" + ftime + ".png"
        Input = sm_p1_datafile1
        try:
            os.remove(sm_p1_gnupfile2)
        except OSError:
            pass
        try:
            gf1 = open (sm_p1_gnupfile1,'r')
        except IOError:
            print ("Cannot find file: " + sm_p1_gnupfile1)
        try:
            gf2 = open (sm_p1_gnupfile2,'a')
        except IOError:
            print ("Cannot find file: " + sm_p1_gnupfile2)    
        gline1 = gf1.readline()
        while gline1 != "":
            if "%TITLE%" in gline1:
                gline1 = gline1.replace("%TITLE%",Title)
            if "%XFORMAT%" in gline1:
                gline1 = gline1.replace("%XFORMAT%",XFormat)
            if "%XTICS%" in gline1:
                gline1 = gline1.replace("%XTICS%",XTics)
            if "%BEGIN%" in gline1:
                gline1 = gline1.replace("%BEGIN%",Begin)
            if "%END%" in gline1:
                gline1 = gline1.replace("%END%",End)
            if "%OUTPUT%" in gline1:
                gline1 = gline1.replace("%OUTPUT%",Output)
            if "%INPUT%" in gline1:
                gline1 = gline1.replace("%INPUT%",Input)
            gf2.write(gline1)
            gline1 = gf1.readline()
        gf1.close()
        gf2.close()

Here the result with gnuplot:

gnuplot

Sending mails

First you have to install ssmtp:

    sudo apt-get install ssmtp      # needed
    sudo apt-get install mailutils  # not mandatory
    sudo apt-get install mpack      # for attachments

With your editor, set up the defaults for SSMTP in /etc/ssmtp/ssmtp.conf. Edit the fields:

    root=my@mail.adr
    mailhub=smtp.xxx.xx:587
    hostname=localhost
    rewriteDomain=xxx.com
    FromLineOverride=YES
    AuthUser=youruserid
    AuthPass=xxxxxxxxxxxx
    UseSTARTTLS=YES

Test your mail with:

    echo "Hello world email body" | mail -s "Test Subject" my@mail.adr

The python script will send the daily graphic per mail at 1 pm in the morning.

Start your script automatically with cron

If you want to start the python script automatically at reboot, add the following line to your /etc/crontab file.

    @reboot pi python3 /home/pi/smarty/smartyreader.py >> /home/pi/smarty/smartyreader_log.txt 2>&1

The output of the python script is redirected to a text-file, for debugging. To log the cron jobs uncomment cron in the file /etc/rsyslog.conf. You will find the log file in /var/log/cron.log. Here is a helpfull link if you have trouble with your cron job.

Downloads

SmartyReader⁵

A second board with a Teensy 3.6 microcontroller was created to read 5 smartmeter and serial data from my rainwater tank (Teensy 3.6 has 6!! serial ports).


smartyreader fully stacked smartyreader smartyreader boards

The housing is designed with blender and uses 4 strong magnets with 6 mm diameter (stl and blender files in Downloads).

Circuit and PCB

smartyreader5 circuit

The PCB is one sided.


pcb_WeMos1 WeMos_PCB_Front

Downloads