Microcontroller projects
SmartyReader
Reading Luxembourgish smartmeter data from P1 interface
last updated: 16/04/20 (created 16/08/2017)
!!!New!!! CREOS changed data stream: failures, sags, swells and current per phase no longer available!
!!!New!!! Use the SmartyReader with an ESP32. Look at the end of the page.
New software and alternative software now on github:
The software (smartyreader.ino) was enhanced (thanks to Bob (fisch.lu)). Sam Grimee helps to improve the software of this project and created a repo on github with an alternative soft: https://github.com/sgrimee/smarty-reader (Thanks to Sam :)).
Basic construction kit::
If your interested in a bare PCB (5€) or a basic construction kit (SMD already soldered, Wemos, Jumper and cable included (see picture below) 30€), send a mail.
An 1000 µF Elko is missing on the picture with the kits.
Introduction
Because of the EU Energy Efficiency Directive from 2012 the gas and electricity Distribution System Operators (DSO) in Luxembourg replaced there gas and energy meters with smartmeters (named smarty :(). Besides gas and electricity metering, the system is open for other metering data like water and district heat (M-Bus).
The French group Sagemcom delivered the smartmeters. All meters have to be read by one national central system, operated by a common operator. This is an economic group of interest (G.I.E.) of the 7 Luxembourgian gas and electricity DSO‘s named Luxmetering G.I.E.
Luxmetering is getting the data from 4 registers for active, reactive, import and export energy (1/4h) and the 3 registers for gas, water & heat (1 h) over Power Line Communication (PLC). The smartmeters have also alarms and logs for quality of electrical energy supply (voltage, outages,...) and fraud detection, and calendar functions for the 2 external relays (home applications).
The customer wants to get his data and this is possible by reading the blinking LED of the smartmeter. This can be done with the IoT-board. Another possibility is the 10 second data from the smartmeter P1 port (RJ12 connector under the green lid). The P1 data output communication protocol and format is specified in the Dutch Smart Meter Requirements v5.0.2 . The solution deployed in Luxembourg includes an additional security layer standard that is conform to the IDIS package 2.0 requirement. The encryption layer is based on DLMS security suite 0 algorithm: AES128-GCM. More information can be found in this document.
P1 port
P1 hardware
The P1 port connector is a 6 pole RJ12. We get 5 V (250 mA) on pin 1 (Power GND on pin 6, data GND on pin 3). Data pin 5 sends an inverted serial signal (EIA232) with 8N1. To get this data the data request line (pin 2) has to be high (5 V). Pin 4 is not connected. More details in the Dutch Smart Meter Requirements. As we use a microcontroller working on 3.3 V we need a transistor to bring the voltage of the data line to 5 V (Q1). The second transistor inverts the serial data to be EIA232 compliant. A voltage divider (R3, R4) brings the voltage down to 3.3 V.
Encryption
As stated the communication on P1 port is encrypted with AES128-GCM (Galois Counter Mode). Each meter has its own 16 byte encryption key. Ask your DSO or Luxmetering for your key.
With the key we need the cypher text, 17 byte Additional Authenticated Data (AAD), a 12 byte Initialization Vector (IV) and a 12 byte GCM Tag.
The AAD is fix: 0x3000112233445566778899AABBCCDDEEFF. The other data is extracted from the serial stream.
The Initialization Vector (12 byte) consists of the system title (8 byte) and the frame counter (4 byte). The GCM Tag is found at the end of the stream.
More information on AES128-GCM can be found on http://weigu.lu/tutorials/sensors2bus/04_encryption/index.html.
IoT and MQTT
The MQTT-protocol is a publisher/subscriber protocol and it is quite simple to implement this protocol on microcontrollers like the LOLIN D1 mini Pro board (ESP8266). The smartmeter data is published by the LOLIN board over Wifi. It is necessary to run a message server (broker) to distribute the data. I use for this mosquitto on a raspberry pi. The LOLIN board publishes the data and the same raspberry pi with the broker or another computer subscribes to the data and generates p. ex. a graphic.
For testing and debugging you can use the cool MQTT.FX software. It's a JavaFX based MQTT Client based on Eclipse Paho an very comfortable for testing purpose. Downloads on http://www.mqttfx.org.
An alternative software is mqtt-spy.
More information on MQTT can be found on http://weigu.lu/tutorials/sensors2bus/06_mqtt/index.html.
Circuit and PCB
BOM (basic kit)
| 1 | 220 SMD 0805 | reichelt.de: RND 0805 1 220 |
| 1 | 1k Ω SMD 0805 | reichelt.de: RND 0805 1 1,0K |
| 1 | 10k Ω SMD 0805 | reichelt.de: RND 0805 1 10K |
| 1 | 15k Ω SMD 0805 | reichelt.de: RND 0805 1 15K |
| 1 | 100 nF SMD 0805 | reichelt.de: X7R 0805 CF 100N |
| 1 | 1000 µF/6,3 V ELKO | reichelt.de: RAD LXZ 6,3/1K0 |
| 2 | 2N7002 SOT-23 | reichelt.de : 2N 7002 SMD |
| 1 | LOLIN D1 mini pro | www.wemos.cc |
| 1 | RJ12 Jack | reichelt.de: MEBP 6-6S |
| 2 | socket 1x8 straight | reichelt.de: MPE 115-1-008 |
| 1 | pin header | reichelt.de: MPE 087-1-002 |
| 1 | jumper | reichelt.de: JUMPER 2,54GL SW |
| 1 | PCB | www.weigu.lu |
| 1 | Western cable, 2x connectors, 6-pin | reichelt.de: WK 6-6 2,5M |
The LOLIN (Wemos) is sending the data over WiFi. If your metal control cabinet shields to much the signal, it is possible to connect an external antenna to the Lolin D1 mini pro by changing a 0 Ω resistor (look here).
If your Wifi is not reliable and you have the possibility to use ethernet, an W5100 Funduino ethernet board can be added. The PCB is also prepared to use an RTC with DS3231 and an LOLIN µSD card shield to log the data.
In the second version of the board I omitted the 100 µF capacitors, because the boards all worked fine without the capacitor. My new ammeter currentRanger from LowPowerLab helped to exactly measure the currents and I saw that while WiFi is used, there are many peaks drawing a current from 400 mA up to 800 mA! depending of the board. Even with same boards there could be big differences. An external antenna reduced the current, so I think that not all antennas are well matched. As the peaks are very short, the internal power supply of the smarty (specified up to 250 mA) has no problem to deliver the current, but I guess it is better to add a capacitor to the board. With a 1000 µF capacitor, the spikes come down to 300 mA. The ESP32 and also the new (green) Lolin have sometimes problems without that capacitor, so add it to the circuit! In the middle of the picture (time axes) the capacitor was added (100 mV correspond to 100 mA)!
Testing the hardware
If you think the hardware is not working correctly, you can test it with a voltmeter. Without LOLIN (Wemos) board (jumper must be connected), the enable pin gets 5V and data is sent every 10 seconds.
You can see a change in voltage (3.1 V to 2.5 V) every 10 s (jumper connected, no board) measuring between pin 2 (GND) and pin 7 (RX) on the board header (second and 7th pin on the right side). On one of my voltmeter I don't see it in the displayed numbers but on the bar graph that reacts quicker. This can of course be observed much better with an oscilloscope, where the serial data line changes between 3 V and GND for about 70 ms.
Another possibility is to connect a TTL/USB adapter (RX to RX pin 7 (which is TX from the Smartymeter) and GND to GND, jumper set, no board), and check the data stream with a terminal program (e.g. cleverterm) with 115200 bit/s (8 data bit, no parity and 1 stop bit).
If you don't see a change there is a possibility that the hardware is not working. To be sure you can measure directly on the P1 cable. Connect pin 2 (Enable) to pin 1 (5 V) and measure between pin 5 (TX) and pin 6 (GND). As the signal is inverted the Voltmeter shows 0 V and reacts every 10 s. To be totally sure use an oscilloscope.
If you get no signal on the cable, test another cable. If there is still no signal, the Smartymeter does not send a signal. Ask your DSO for support.
Arduino Software for the LOLIN/WEMOS D1 mini board
First install the newest Arduino IDE (1.8.8). To use our ESP8266 LOLIN/WEMOS we add this line http://arduino.esp8266.com/stable/package_esp8266com_index.json to File-Preferences-Additional_Boards_Manager_URLs:.
To install the manager go to Tools > Board: > Boards Manager..., select the Manager and click install. No chose under Tools > Board: (you have to scroll) LOLIN/WEMOS D1 mini Pro.
We need a Crypto library to decode the AES128-GCM and a MQTT library to publish our data. Go to Tools > Manage Libraries.... Type in the Search field Crypto. Click on Crypto by Rhys Wheatherley and install it. Search for mqtt pubsub and click on PubSubClient by Nick O'Leary. Install the library.
In our Arduino sketch we have 4 "switches" to comment or uncomment.
+ SECURE is recommended to get a secure MQTT connection with your MQTT server. You need to define a user and a password.
+ DEBUG outputs messages on 'Serial1(D4 on LOLIN/WEMOS, SD3 on MHEtLive) and is only needed when changing the software.
+STATICgets you a fixed IP address in your network. Provide IP and gateway IP.
+ESP32MK` let's you use the MHEtLive ESP32MiniKit instead of LOLIN/WEMOS D1 mini pro (look at the text on the end of this page.).
Further you have to provide a Wifi SSID and password, the mqttserver IP, the mqtttopic and your key for the smartmeter.
// Comment or uncomment the following lines suiting your needs
#define SECURE // if you want a secure MQTT connection (recommended!!)
//#define DEBUG // if debugging requested
//#define STATIC // if static IP needed (no DHCP)
//#define ESP32MK // if MH ET LIVE ESP32MiniKit instead of LOLIN D1 mini pro
...
// wifi and network settings
const char *ssid = "mywifi";
const char *password = "mypass";
#ifdef STATIC
IPAddress network_static_IP (192,168,178,14); //static IP
IPAddress network_subnet_mask (255,255,255,0);
IPAddress network_gateway (192,168,178,1);
#endif // ifdef STATIC
// MQTT settings
#ifdef SECURE
const short mqttPort = 8883; // TLS=8883
const char *mqtt_user = "me";
const char *mqtt_pass = "myMqttPass12!";
WiFiClientSecure espClient;
#else
const short mqttPort = 1883; // clear text = 1883
WiFiClient espClient;
#endif
PubSubClient client(espClient);
const char *mqtt_server = "192.168.178.160";
const char *mqtt_client_Id = "smarty_lam1_p1";
const char *mqtt_topic = "lamsmarty";
const char *smartyreader_hostname = "SmartyReader";
...
//Keys for SAG10307000xxxxx
byte key_SM1[] = {0x3B, 0x9C, 0xDB, 0x8C, 0xE3, 0xFD, 0xB7, 0x02,
0x16, 0x35, 0xFF, 0x6F, 0xB0, 0x2E, 0xE1, 0xDF};
The code is in the Downloads section at the end of the page.
To program the board, you have to take it out of the socket (the transistor on RXD prevents proper programming).
The software allows debugging and output of the data over serial1 on Pin D4. For more info see: http://weigu.lu/microcontroller/tips_tricks/esp8266_tips_tricks
Python software to get the data
A python (python3) script is used to get our smartmeter MQTT data from the broker. We use the paho.mqtt.client library which can be installed with pip to subscribe to our topic.
sudo pip3 install paho-mqtt
The data is saved in a file (/data) and the old day-files are archieved (/data_archive). The script also generates a png-file with gnuplot that is displayed on an internal homepage and sent to an email address (full code in Downloads).
To to so we can use the same raspberry pi witch holds the broker.
MQTT client
You have to adjust possibly the broker IP address and your smartmeter id (these are the last 3 digits of your smarty id (ex: "345" for SAG1030700012345) ).
#!/usr/bin/python3
#
# Name: smartyreader.py
# Purpose: Client to get MQTT data from Mosquitto
# Author: weigu.lu
# Date: 8/17
#
...
import paho.mqtt.client as mqtt
...
clientID = "getsmarty_p1"
brokerIP = "192.168.178.101"
brokerPort = 1883
topic = "basement/smarty1"
sm_id = "345"
# Callback that is executed when the client receives a CONNACK response from the server.
def onConnect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
mqttc.subscribe(topic, 0) # Subscribe to the topics (topic name, QoS)
# Callback that is executed when we disconnect from the broker.
def onDisconnect(client, userdata, message):
print("Disconnected from the broker.")
# Callback that is executed when subscribing to a topic
def onSubscribe(client, userdata, mid, granted_qos):
print('Subscribed on topic.')
# Callback that is executed when unsubscribing to a topic
def onUnsubscribe(client, userdata, mid, granted_qos):
print('Unsubscribed on topic.')
# Callback that is executed when a message is received.
def onMessage(client, userdata, message):
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
global sm, smp, sme, sm_mn #p power, e energy, mn at midnight
now = datetime.now()
now_time = now.time()
if now_time >= time(23,59,00) and now_time <= time(23,59,59):
sm_mn=ioj["c1"].rstrip("*kWh") # change to "p1" for production
ftime = strftime("%Y_%m_%d", localtime())
ftime3 = strftime("%d.%m.%y %H:%M:%S", localtime())
io=message.payload.decode("utf-8");
try:
ioj=json.loads(io)
except:
ioj={"dt":"error"}
print(ioj)
temp = ioj["dt"]
if (temp[0]!='e') and (temp[0]!='c'):
sm_new=ioj["c1"].rstrip("*kWh") # change to "p1" for production
if sm!="0":
smp = str(round((float(sm_new)-float(sm))*60000.0,3))
sm=sm_new
sme=str(float(sm)-float(sm_mn))
if sme[0]=='-':
sme="0"
try:
f = open (sm_p1_datafile1, 'r')
except IOError:
print("error reading file "+sm_p1_datafile1)
lineList = f.readlines() #read all lines
f.close()
try:
f = open (sm_p1_datafile1, 'a')
except IOError:
print ("Cannot create or find file: " + sm_p1_datafile1)
try:
f2 = open (sm_p1_datafile2+ftime+'.min', 'a')
except IOError:
print ("Cannot create or find file: " + sm_p1_datafile2)
if (len(lineList)) == 1:
sm_p1_data = ' '.join((ftime3,sm,sme,smp))
sm_p1_data = sm_p1_data + '\n'
else:
line = lineList[len(lineList)-1] #get the last line
lline =shlex.split(line) #convert string (space seperated items) to list
sm_p1_data = ' '.join((ftime3,sm,sme,smp))
sm_p1_data = sm_p1_data + '\n'
print (sm_p1_data,end='')
f.write(sm_p1_data)
f2.write(sm_p1_data)
f.close()
f2.close()
else:
print("loop not executed (error or connect message)")
...
# Main
mqttc = mqtt.Client(client_id=clientID, clean_session=True) # create client
mqttc.on_connect = onConnect # define the callback functions
mqttc.on_disconnect = onDisconnect
mqttc.on_subscribe = onSubscribe
mqttc.on_unsubscribe = onUnsubscribe
mqttc.on_message = onMessage
mqttc.connect(brokerIP, brokerPort, keepalive=60, bind_address="") # connect to broker
mqttc.loop_start() # start loop to process callbacks! (new thread!)
sm ="0"
smp = "0"
sme = "0"
sm_mn = "0"
try:
while True:
now = datetime.now()
now_time = now.time()
...
except KeyboardInterrupt:
print("Keyboard interrupt by user")
mqttc.loop_stop() # clean up
mqttc.disconnect()
Internal homepage with daily graphic
Static IP address
To access the raspberry pi we will set a static IP address.
Use the editor nano to append the following to the file /etc/dhcpcd.conf.
# Custom static IP address for eth0.
interface eth0
static ip_address=192.168.1.67
static routers=192.168.1.1
static domain_name_servers=192.168.1.1 8.8.8.8
# Custom static IP address for wlan0.
interface wlan0
static ip_address=192.168.1.69
static routers=192.168.1.1
static domain_name_servers=192.168.1.1 8.8.8.8
cd /etc
sudo nano dhcpcd.conf
Save with CTRL+O and exit with CTRL+X.
Setting up the webserver Lighttpd on the raspi
Lighttpd is an efficient high performance web server. It has a small memory footprint and an effective management of the cpu-load compared to other web-servers. Naturally you can use another web server, but possibly you have to adjust the path to your web page.
sudo apt update
sudo apt upgrade
sudo apt install lighttpd
Test if the web server is running by typing the ip address of your raspi in the url field of your browser.
The html files are in in /var/www/html. Copy the following html code (filename: index.html) to /var/www/html:
<!DOCTYPE html>
<head>
<title>Smarty P1</title>
</head>
<body>
<h1>Smarty Data</h1>
<p><img src="png/sm_p1_daily.png" alt="smarty data"></p>
</body>
</html>
Also create an empty directory named /png in /var/www/html.
sudo mkdir /var/www/html/png
Using gnuplot for graphics
sudo apt install gnuplot
To test gnuplot you can use the following command:
cd /home/pi/smarty/gp
gnuplot sm_p1.gp
The sm_p1.gp is created by our python script from a template file. This template file is found in /smarty/gp (it is contained in the file smartyreader.zip). Here is the code that generates the gp file:
def sm_create_gp_file():
""" The function prepares the gp file for plotting with gnuplot. First the
old gp file is deleted. Then it uses the xx_gp_template.gp file in
~/../gp and replaces the keywords between the % sign by creating
a new gp (xx.gp) file."""
ftime2 = strftime("%d.%m.%y", localtime())
Title = ftime2
XFormat = '"%H:%M"'
XTics = "60*60" #seconds
Begin = ftime2 +" 00:00:01"
End = ftime2 +" 23:59:59"
Output = png_dir + "sm_p1_" + ftime + ".png"
Input = sm_p1_datafile1
try:
os.remove(sm_p1_gnupfile2)
except OSError:
pass
try:
gf1 = open (sm_p1_gnupfile1,'r')
except IOError:
print ("Cannot find file: " + sm_p1_gnupfile1)
try:
gf2 = open (sm_p1_gnupfile2,'a')
except IOError:
print ("Cannot find file: " + sm_p1_gnupfile2)
gline1 = gf1.readline()
while gline1 != "":
if "%TITLE%" in gline1:
gline1 = gline1.replace("%TITLE%",Title)
if "%XFORMAT%" in gline1:
gline1 = gline1.replace("%XFORMAT%",XFormat)
if "%XTICS%" in gline1:
gline1 = gline1.replace("%XTICS%",XTics)
if "%BEGIN%" in gline1:
gline1 = gline1.replace("%BEGIN%",Begin)
if "%END%" in gline1:
gline1 = gline1.replace("%END%",End)
if "%OUTPUT%" in gline1:
gline1 = gline1.replace("%OUTPUT%",Output)
if "%INPUT%" in gline1:
gline1 = gline1.replace("%INPUT%",Input)
gf2.write(gline1)
gline1 = gf1.readline()
gf1.close()
gf2.close()
Here the result with gnuplot:
Sending mails
First you have to install ssmtp:
sudo apt-get install ssmtp # needed
sudo apt-get install mailutils # not mandatory
sudo apt-get install mpack # for attachments
With your editor, set up the defaults for SSMTP in /etc/ssmtp/ssmtp.conf. Edit the fields:
root=my@mail.adr
mailhub=smtp.xxx.xx:587
hostname=localhost
rewriteDomain=www.xxx.com
FromLineOverride=YES
AuthUser=youruserid
AuthPass=xxxxxxxxxxxx
UseSTARTTLS=YES
Test your mail with:
echo "Hello world email body" | mail -s "Test Subject" my@mail.adr
The python script will send the daily graphic per mail at 1 pm in the morning.
Start your script automatically with cron
If you want to start the python script automatically at reboot, add the following line to your /etc/crontab file.
@reboot root python3 /home/pi/smarty/smartyreader.py >> /home/pi/smarty/smartyreader_log.txt 2>&1
The output of the python script is redirected to a text-file, for debugging. To log the cron jobs uncomment cron in the file /etc/rsyslog.conf. You will find the log file in /var/log/cron.log.
Here is a helpful link if you have trouble with your cron job.
SmartyReader with ESP32
We have problems with ESP8266 WIFI in school! The ESP32 has no problems, so here is a SmartyReader with ESP32.
The MH ET LIVE ESP32MiniKit board is almost pin compatible with the LOLIN (wemos) D1 mini pro. But I had problems to get it work, because there is an error in the pinout sheet on internet. RxD ant TxD are interchanged and not compatible with D1 mini pro!
Fortunately ESP32 has multiplexing features, and so pins can be changed in code. This can be done with the begin command SR_Serial.begin(115200,SERIAL_8N1, 1, 3). With this command we define GPIO pin 1 for RxD1 and pin 3 for TxD1.
A second change is the capacitor (1000µF/10V, even better 4700µF/10V) that was soldered to the board to the 5 V header. The ESP32 draws a higher current using WiFi for short time slices.
If you want to debug the code, Arduino Serial1 is on SD3 (u1TxD, GPIO10).
Here are the changes in the code:
// uncomment if MH ET LIVE ESP32MiniKit instead of LOLIN/WEMOS D1 mini pro
#define ESP32MK
...
#ifdef ESP32MK
#include <WiFi.h> // ESP32 MH ET LIVE ESP32MiniKit
#else
#include <ESP8266WiFi.h> // ESP8266 LOLIN/WEMOS D1 mini pro
#endif // ifdef ESP32MK
...
#ifdef ESP32MK
const byte DATA_REQUEST_SM = 17; //active Low! 17 for MH ET LIVE ESP32MiniKit
#else
const byte DATA_REQUEST_SM = D3; //active Low! D3 for LOLIN/WEMOS D1 mini pro
#endif // ifdef ESP32MK
void setup() {
...
#ifdef ESP32MK
SR_Serial.begin(115200,SERIAL_8N1, 1, 3); // change reversed pins of ESP32
#else
SR_Serial.begin(115200); // Hardware serial connected to smarty
#endif //ESP32MK
...
}
void setup_wifi() {
...
WiFi.begin(ssid, password);
#ifdef ESP32
WiFi.setHostname(smartyreader_hostname);
#else
WiFi.hostname(smartyreader_hostname);
#endif // ifdef ESP32MK
...
}
A new file to download will follow in the next days.
Downloads
- Arduino ino file for µC board
- Python code and directory structure (unzip to your home directory (/home/pi))
- Gnuplot template file
- HTML file
- Kicad files
SmartyReader⁵
A second board with a Teensy 3.6 microcontroller was created to read 5 smartmeter and serial data from my rainwater tank (Teensy 3.6 has 6!! serial ports). Look here for more information
