Ad

Saturday, December 7, 2019

Reverse engineering the ZMAi-90 DIN rail meter/switch and integrating with Hass.io using Tasmota - Part 2

I finished the first post with  a tone of optimism, in spite of not being quite there yet. But this time I'm bringing the complete story, with something which hopefully can be a useful takeaway for most users.

Initial analysis of the MCU communication

After figuring out what kind of communication was going on between the ESP8266 and the Vangotech V9821 chip (the specialized MCU which does all the metering functions - and a bit more which I will go in detail afterwards), I got a bit puzzled with the output and its consistency. I first connected a known AC current source through the shunt mounted in the relay's output rail, and in the middle of a stream of garbage, some values seemed consistent with the current I was putting and  being shown in the device's display.


I still cracked my mind at trying to figure out a pattern (I felt as if I was trying to incarnate John Nash while looking for patterns in seemingly chaotic data), and trying to prove assumptions such as the last byte being a checksum. But nothing fruitful came out of that first iteration.



In this first attempt to eavesdrop and decode the communication between the two MCUs that composed this device, I was using the UART feature of a Pickit2, which supposedly had the advantage of not disrupting the impedance of the serial lines.

Pondering on the possibility that I was collecting the serial data incorrectly, I went for a different approach, and instead of using the Pickit, I tried using the regular UART of a standard USB to serial adaptor, and put a 4.7 kOhm resistor between its RX pin and the Vangotech chip TX pin.


With this, the result was certainly more consistent, with message lengths that were regular. This led me to assume that I was collecting the messages correctly, and the commands that the ESP was sending were probably the correct ones. All these captures were done at 9600 bps 8/N/1 settings. I found that at that speed, the 0xFE01 preamble would appear in the messages, and coincidentally or not, the serial protocol from another metering chip, the V9261F, would also have its messages start with the same header sequence:



As such, holding the confidence acquired from that observation, I decided to accept the risk, and use the tuya-convert tool (https://github.com/ct-Open-Source/tuya-convert) to  try to flash Tasmota via OTA (I knew that with the slave MCU always connected to the ESP8266, flashing Tasmota via the serial port would probably not work). The flashing worked, and I managed to retain a backup of the original firmware, through the same tool.

Switching from the original MCU firmware to Tasmota


Already in Tasmota, I setup the hardware serial port (Serial TX and Serial RX in GPIO1 and GPIO 3 pins respectively), and tried to send the same command message that the original firmware would send, but nothing happened. No response.

Rolling back...


I thought of several possibilities, including that some sort of authentication procedure could be happening between the ESP and the Vangotech chip, prior to the actual commands, or that some form of signalling via different GPIO pins could be happening.

That is when I took the backup of the original Tuya firmware and reverted to it. To do this however, I had to forcefully flash via the serial port. I first tried to just connect the pins to GPIO1 and 3 from the ESP chip, but soon figured out that the only immediate option was to remove the R1 and R14 resistors during the flashing, and at the end put these back in place:



Sucess was not immediate, however. The first time I flashed the original firmware, it would enter a boot loop. After some support from the tuya-convert community, it was found that the problem was in the flashing tool recognizing an incorrect flash size for this chip (2 MB instead of 1MB, which is the actual size of the flash). The selection of an incorrect flash size apparently affects the location where the firmware tries to find RF calibration data during the boot process:

https://github.com/ct-Open-Source/tuya-convert/issues/419

As such, flashed it again, by forcing the real size in esptool (1 MB):

esptool.py --port COM3 --baud 74880 --after no_reset write_flash --flash_size 1MB --flash_mode dio 0x00000 tuya_backup.bin --erase-all

And this time it worked like a charm.

Resuming communication analysis in a more proper way


With the Tuya firmware running, I picked up the oscilloscope and started analysing the signals from other GPIO pins. A brief 18 us pulse would indeed occur in the GPIO15 pin during bootup, but apart from that, nothing. With the exception of the GPIO2 which puzzled me initially, because it appeared to send a burst of data during bootup.

When I tried to capture the data, I could only capture garbage. But switching to the ESP8266 bootloader baudrate of 74880 bps, I could see that this was just the debug output from the firmware:

OS SDK ver: 2.0.0(e8c5810) compiled @ Jan 25 2019 14:26:04
รบ[notice]wf_sdk_adpt.c:888 country code:CN,SC:1,NC:13
[notice]user_main.c:310 SDK version:2.0.0(e8c5810)
[notice]user_main.c:314 fireware info name:esp_v9801_1plug_zhimei version:1.0.2
[notice]user_main.c:317 tuya sdk compiled at Jul 25 2019 20:39:58
[notice]user_main.c:319 BV:5.46 PV:2.2 LPV:3.3
reset reason: 4
epc1=0x00000000, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000,depc=0x00000000
mode : null
[notice]gw_intf.c:405 Authorization success
mode : sta(bc:dd:c2:xx:xx:xx)
add if0
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
[notice]mqtt_client.c:576 gw wifi stat is:5
state: 3 -> 5 (10)
add 0
aid 6
pm open phy_2,type:2 0 0
cnt 

connected with xxxxx, channel 11
dhcp client start...
ip:192.168.1.42,mask:255.255.255.0,gw:192.168.1.1
[notice]mqtt_client.c:576 gw wifi stat is:5
[notice]mqtt_client.c:607 DNS START 11-27 00:11:05
[notice]mqtt_client.c:634 DNS END 11-27 00:11:05
[notice]mqtt_client.c:640 MQTT CONN START 11-27 00:11:05
[notice]mqtt_client.c:663 MQTT CONN END 11-27 00:11:05
[notice]mqtt_client.c:758 mqtt connect success
[notice]smart_wf_frame.c:3800 firmware self detect upgrade start...
[notice]smart_wf_frame.c:3810 get fw ug info op_ret:2

(As a matter of fact, this second UART is frequently used in the ESP8266 platform for debug logging, as it only features a TX pin)

That is when I had the "brilliant" idea of using the oscilloscope to compare the command as sent by the Tuya firmware, with the command sent by the computer (until this point I was always assuming I had the correct serial port parameters). These were clearly different waveforms, in spite of the minimum pulse length being the same in both cases. Played a bit with the settings on the computer side, until I found a setting which was "bang on" with what Tuya was sending: 9600 8/E/1:


This was like the eureka moment, because it fitted the puzzle and explained the inconsistencies I was having, and potentially why the device was not answering to the command.

With the 2 serial pin resistors still removed from the board, hooked the computer UART to the V9821 serial port, and sent the command originally captured from the ESP:

FE010F080000001C

As expected, the device would respond with a byte sequence that would now make much more sense:

FE010800000000000000000000000000000000000000000000000000000000001000001B

And the checksum documented by Vangotech, once applied in these messages, (i.e. performing the sum of all the bytes except the last, doing the bitwise NOT on the result, and finally adding 0x33), gave the same result.

Just before switching back to Tasmota, and after an accident with the C20 capacitor: I had a wire soldered there, for resetting the Vangotech chip, while doing some experiments. The wire detached taking the solder pad and the capacitor with it.  I noticed that with the Vangotech chip not running because of the reset pin, the relay would also not switch. This made me think that unlike the initial expectation, the Vangotech chip had to have a role in switching the relay. I first thought it was just a GPIO pin from the ESP chip, connected to some driver chip for the relay itself.

With some more investigation and following the traces, I found that there is indeed a driver chip, a Chipnorth CN8023, which receives signals from two pins from the Vangotech V9821. The later, on the other hand, receives a signal from GPIO12 on the ESP chip: when GPIO12 switches from high to low, the V9821 sends a pulse to relay1 pin of the CN8023 chip. This causes the relay to switch off. When the GPIO12 switches from low to high, the V9821 sends a pulse to relay2 pin of the CN8023. This causes the relay to switch on. This slightly complex approach allows the ESP firmware to send a signal as if it would be a regular relay, abstracting the nuances of controlling a bistable relay.

Rolling forward to Tasmota


While investigating on the Tasmota side, I found that (to make things harder :) ) it had no support for changing any serial port parameters other than the baudrate. Given this constraint, I talked with a few people related to the project, via the Discordapp, to get a pulse on what would be the status of this limitation, and if there would be some kind of relevant constraint for not supporting it longer ago.

Given that there was no previous feature request, or any pending Pull Request covering this feature, I decided to try to contribute myself, forking from the project and doing the changes myself.

There was (and is) indeed a challenge in adding a feature to change these settings in the software UART, due to the lack of available iRAM in the current implementation.

But regarding the hardware UART this is just a parameter that has to be set during the initialization of the device.

As such I did the change, tested on a NodeMCU, and after everything was tuned and working, flashed it to the ZMAi-90 meter. After some tests in all of the supported serial port modes, opened the Pull Request:


Theo Arends (the main developer of the Tasmota project) promptly responded to the contribution, making the changes in order to cover all of the 24 possible serial port settings (in my implementation I only covered four of those - I was being conservative to not use more bits from what was an apparently already full flash memory erase block). Theo obviously much more knowledgable of what to change, allocated a spare byte in this block of flash, and dedicated it for the serial configuration.

First success

With Tasmota now able to send data in the needed 9600 8/E/1 format, it was the moment of truth, of trying to send a command and obtaining the response from the V9821 chip.

Configured the port with my new command SerialConfig, for using 8/E/1:

00:01:59 CMD: SerialConfig
00:01:59 RSL: stat/tasmota/RESULT = {"SerialConfig":3}

And upon sending the command to the V9821, bang:

00:04:06 CMD: SerialSend5 fe010f080000001c
00:04:06 RSL: stat/tasmota/RESULT = {"SerialSend":"Done"}
00:04:07 RSL: tele/tasmota/RESULT = {"SerialReceived":"FE01081400000059230000478604000050000096381100000000004149110000100000F0"}

Full success! Communication between the MCUs was fully working.

In-depth protocol analysis

Now it was time to understand the protocol. It didn't take much time to figure out what the data was: I  first noticed that every pair of digits in the hex string represents actual digits from the values, in the correct order. This means we have BCD (binary coded decimal) encoded data. On the other hand, between pairs of digits, the order is inverted.

I found that each data field contains exactly 4 bytes. Splitting the message by each field, for the sample message we have:

FE0108 14000000 59230000 47860400 00500000 96381100 00000000 41491100 00100000 F0

There are always 8 data fields. According to the Vangotech specification (obtained from the datasheet of a simpler device, the V9261F), the number of fields that we expect to receive must be provided in the 4th byte of the request. In our command, indeed we have  the same value:

fe010f080000001c

I first determined, by comparing with the value shown in the LCD display, that the 2nd field was the voltage. By looking at it:

59230000 

If we invert the order of the bytes (pairs of digits) we obtain:

00002359

Interpreting the value as a number and dividing by 10, we obtain the voltage:

235.9 Volts

Using the same principle for the other fields, I could determine the following:

  • the 1st field corresponds to the consumed energy, in kWh after being divided by 100;
  • the 2nd field is the Voltage, as mentioned above. Needs to be divided by 10;
  • the 3rd field corresponds to the current in Amps. Needs to be divided by 10000;
  • with the 4th field I observed it would output 00500000 with the device powered by the mains supply, but stay at 00000000 if supplied directly by a DC supply. Reading the datasheet it was documented that the mains frequency is measured and placed on a register. Given that the mains frequency in my location is 50 Hz (and has to be a very stable value), there is a great likelihood that this field is actually the mains frequency. For the output in Hz, it needs to be divided by 100;
  • the 3 following fields would only output values on the presence of a load. Given that the values were closely related to the power measurement shown in the LCD, and because there is also indication in the datasheet, I started by assuming that the 5th field would corresponds to the Active Power, the 6th field the Reactive Power (in this example taken during the use of a heater attached to the meter), and the 7th field the Apparent Power. The reasoning was due to the observation that the active power field had a value slightly smaller than the apparent power, and that with some loads - such as a motor, the reactive power field would display a value, while with a mostly resistive load such as the heater, it would be zero. For these three values, to obtain the power in Watts, it needs to be divided by 100;
  • The 8th field I observed it would be 00100000 everytime there would be no load connected, or with a resistive load, but display a different value, such as 68070000 when a motor would be connected. The value would be consistent with the relation between active and apparent power, which corresponds to the power factor. So I got to the conclusion that dividing this value by 10 (after inverting the order of the bytes as previously explained) we obtain the power factor in %.

In sum we have:

FE0108 [E (KWh)] [V (Volts)] [I (Amps)] [f (Hz)] [Pactive (Watts)] [Preactive (Watts)] [Papparent (Watts)] [pf (%)] [Checksum]

where the header FE0108 is 3 bytes long, the [Checksum] is 1 byte long and is calculated as the sum of all of the previous bytes, which is the applied bitwise NOT, followed by the addition of value 0x33. Each of the data fields is 4 bytes long and corresponds to the BCD encoded digits of the measurements. The order of the bytes needs to be reversed prior to parsing.

As you may have noticed by now (especially if you have this device and used the Tuya firmware and app), there are more quantities being measured than the original app actually exposes! For example it is particularly useful to obtain the measure of reactive power, as we can characterize the types of loads and their proportion in the appliances consuming energy in the house. Or by monitoring the frequency, we can setup an alarm, notify the user or take some other relevant action if for some reason this value changes (the frequency is critical for some loads).

With the message format resolved, it was now time to go to the fun part, and integrate with Hass.io. Given that I had not written any driver on the Tasmota side, to parse these message and expose the data as sensors, this work had to be done on Hass.io side.

Before that, one important step would be required: configure Tasmota itself. The sections that follow are potentially useful for the user to replicate, and as such I will describe with the clarity that a tutorial/script requires.

Guide for setting up Tasmota and Hass.io with the ZMAi-90

Setup your device

The first thing you will need to setup is your ZMAi-90 device. In order to flash it, you will need to use either tuya-convert or directly via the serial cable.

1. Using tuya-convert:
  • follow the instructions as described in the tuya-convert project: https://github.com/ct-Open-Source/tuya-convert
  • make sure your tasmota binary is greater or equal to version 7.1.2.2 (where the SerialConfig command was introduced). Replace with the correct binary, in the tuya-convert/files directory;
2. Using the serial cable:
  • temporarily solder a wire betwen the RSTn pin (the point between R30 and C20, as shown in the image below) and GND. This will keep the V9821 MCU from booting up and interfere with the communication on the ESP UART;
  • temporarily solder a wire betwen the TYWE3S module (where the ESP8266 chip is) GPIO0 pin and GND. This will put the ESP chip in programming mode;
  • connect TX, RX and GND to your USB-serial bridge. Make sure it is set for 3.3 Volts (5 Volts TTL will fry the chip);
  • provide 5 Volts and GND to the pins in the "J-V" connector, as shown in the picture:

  • using esptool or similar, flash the chip, making sure the flash size is forced to 1MB:
$ esptool.py --port COM3 --baud 74880 --after no_reset write_flash --flash_size 1MB --flash_mode dio 0x00000 tasmota.bin --erase-all

  • remove the shunt between GPIO0 and GND;
  • remove the shunt between RSTn and GND;
  • mount the components together and power the device.

Configure Tasmota

Besides the common Tasmota aspects which I will not detail here (setup WiFi, MQTT, etc), you will need to:
  • configure the module:
    • in the Tasmota web UI, go to Configuration > Configure Module and select Generic (0) as the module type; click "Save". The module will restart afterwards;
    • go back to the same configuration screen, and cofigure the I/O pins like the following screen. Click "Save" and the module will restart once again:
  • configure the serial port to the correct speed: 
    • in the Tasmota console type: Baudrate 9600
  • configure the serial port to the correct mode: 
    • in the Tasmota console type: SerialConfig 8E1
  • you will need to create a rule to periodically send a command to the V9821, to obtain the measurements:
    • in the Tasmota console type:
      Rule1 on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog SerialSend5 fe010f080000001c; RuleTimer1 10 endon
    • next, enable the rule by typing:
      Rule1 1
With this done, Tasmota will publish every 10 seconds the output from the V9821, into the MQTT topic tele/general-meter-switch/RESULT. For example:

02:59:15 MQT: tele/general-meter-switch/RESULT = {"SerialReceived":"FE01083002000076230000808404000050000028431100000000007253110000100000A6"}

We will need this on Hass.io side.


Configure Hass.io

On Hass.io (Home Assistant) you will need to define the two new types of entities, namely a sensor for each of the 8 measurements, and a switch for the relay. This boils down to adding the following configuration, which covers the parsing (through jinja2) of the BCD encoded data, and the conversion of the values as explained above.

1. Edit the /config/configuration.yaml file, and add the following entries to the "sensor:" section:

  - platform: mqtt
    name: "Mains Consumed Energy"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[6:14] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 100 }}
    unit_of_measurement: 'kWh'    
  - platform: mqtt
    name: "Mains Voltage"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[14:22] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 10 }}
    unit_of_measurement: 'Volts'
  - platform: mqtt
    name: "Mains Current"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[22:30] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 10000 }}
    unit_of_measurement: 'Amps'
  - platform: mqtt
    name: "Mains Frequency"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[30:38] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 100 }}
    unit_of_measurement: 'Hz'
  - platform: mqtt
    name: "Mains Active Power"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[38:46] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 100 }}
    unit_of_measurement: 'Watts'    
  - platform: mqtt
    name: "Mains Reactive Power"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[46:54] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 100 }}
    unit_of_measurement: 'Watts'      
  - platform: mqtt
    name: "Mains Apparent Power"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[54:62] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 100 }}
    unit_of_measurement: 'Watts'
  - platform: mqtt
    name: "Mains Power Factor"
    state_topic: "tele/general-meter-switch/RESULT"
    value_template: >- 
      {% set message = value_json.SerialReceived %}
      {% set payload = message[62:70] %}
      {% set payload_len = (payload | length) %}
      {% set result = namespace(value='') %}
      
      {% for i in range(0, payload_len + 1) | reverse -%}
        {%- if i is divisibleby 2 -%}
          {%- set result.value = result.value + payload[i:i+2] -%}
        {%- endif -%}
      {%- endfor -%}
      
      {{ (result.value|float) / 10 }}
    unit_of_measurement: '%'

You may rename the topic name and the entity name as appropriate for your setup.

2. Edit the "switch:" section, by adding:

  - platform: mqtt
    name: general-meter-switch
    state_topic: 'stat/general-meter-switch/RESULT'
    value_template: '{{ value_json["POWER"] }}'
    command_topic: 'cmnd/general-meter-switch/POWER'
    availability_topic: 'tele/general-meter-switch/LWT'
    qos: 1
    payload_on: 'ON'
    payload_off: 'OFF'
    payload_available: 'Online'
    payload_not_available: 'Offline'
    retain: true 

Again, rename the topic name and the entity name as appropriate for your setup.

3. Restart Hass.io, and you should be done! In the web UI, the new badges should become visible:


Obviously like the sensors in general, the history of values is retained:


And the relay will appear just like any other switch:


Final considerations

This work was quite a bit of a journey, with all the learning that it carried, but also for how pleasing it was with the results that it brought after all of the challenges, and moreover, the satisfaction of sharings results with the community which may prove useful to other users.

There are still potential steps of improvement which I intend to pursue, such as developing the driver on the Tasmota side, or discovering other metering MCU commands such as how to reset the energy counter.

I thank the community, especially the people from the Tasmota and tuya-convert projects, who have provided helpful feedback when needed.

43 comments:

Jarek said...

What’s your guess?
Will it work with who’s meter:

https://a.aliexpress.com/_dtjs2

Creation Factory said...

Hi,

I've looked into this meter before. Given the fact that this is a 3-phase device (the v9821 that is in the ZMAI-90 is a single phase metering chip) and because of the RS-485 bus capability and the type of protocol that is detailed in the product page, it is likely a very different device. It is of course also an interesting target for reverse engineering just as done with the ZMAI-90..

Cheers

Unknown said...

great work!! Thank, I hope you find the way to reset the energy counter!! that would be the cherry on top of the cake. Your blog gave me the missing push to get a Zmai90. flashed and setup following your instruction, the only thing I had to change was the "tele/general-meter-switch/RESULT" part of the mass setup into "general-meter-switch/tele/RESULT"

again MANY THANKS!!!

Rogerio said...


Hello friend, congratulations for the post. I recently purchased a Zmai-90 and it is not registering energy consumption in the Tuya application. It does not register and does not record. I would like to leave it in original firmware, but it would work. My Zmai90 has current firmware 1.0.2 and I found that updated firmware would be 1.0.6. Would the current firmware update (1.0.6) flash? Would you have this file to provide me? You said you backed up yours. I thank you for your attention.

Creation Factory said...

Hello Rogerio, thanks! I'm sorry, I don't have the original firmware v1.0.6. I converted my own test device to Tasmota before any original firmware update could take place (it was in version 1.0.2). Nevertheless (not sure if is true or not), I read that Tuya firmwares can contain device specific data, preventing a firmware image extracted from one device, to work in another device.

Cheers

Rogerio said...

Thanks for the reply friend. Could you give me a hint as to why my device does not register or record energy consumption in the application?

I bought 2 devices and both are having the same problem.

Creation Factory said...

Hello Rogerio, it is difficult to troubleshoot just based on the behaviors you describe. It is possible that you are not provisioning it with the correct Wifi password? Can it be the case that you Wifi access point uses older encryption standards such as WEP? It can be the case that open or unsafe encryption is unsupported by these Tuya firmwares..honestly my experience with stock firmware was short and not incredibly satisfying. Sorry for not being able to help further..

L02R said...

Thanks for all your hard work, and thanks for sharing it for free! Cheers :)

Mark said...

Hello, me again :)

I've noticed in Home Assistant that the switch is very rarely "on" even though it is. I cannot post a picture, but is it either unavailable or mainly OFF. I was off for 24 hours, till I logged into the web console and it changed to ON. Also I am trying to change the friendly name and when I do it resets. Not sure what I have done wrong, but I followed the above to the letter.. :D

00:00:00 CFG: Loaded from flash at F9, Count 35
00:00:00 Project tasmota Tasmota Version 8.1.0(tasmota)-2_6_1
00:00:00 WIF: Connecting to AP1 xxxxx in mode 11N as StudioPower-5893...
00:00:00 RSL: tele/StudioPower/RESULT = {"SerialReceived":"F0"}
00:00:06 WIF: Connected
00:00:06 HTP: Web server active on StudioPower-5893 with IP address 192.168.0.27
10:04:52 MQT: Attempting connection...
10:04:52 MQT: Connected
10:04:52 MQT: tele/StudioPower/LWT = Online (retained)
10:04:52 MQT: cmnd/StudioPower/POWER =
10:04:52 MQT: tele/StudioPower/INFO1 = {"Module":"Generic","Version":"8.1.0(tasmota)","FallbackTopic":"cmnd/DVES_593705_fb/","GroupTopic":"cmnd/tasmotas/"}
10:04:52 MQT: tele/StudioPower/INFO2 = {"WebServerMode":"Admin","Hostname":"StudioPower-5893","IPAddress":"192.168.0.27"}
10:04:52 MQT: tele/StudioPower/INFO3 = {"RestartReason":"Hardware Watchdog"}
10:04:52 MQT: stat/StudioPower/RESULT = {"POWER":"ON"}
10:04:52 MQT: stat/StudioPower/POWER = ON

Creation Factory said...

Hello Mark,

Can you try increasing the log level so that we can have more details? Go to Configuration > Configure Logging > Web log level > 4 More debug.

Could you share the output of the State command? Do you have a solid wifi connection with the device?

You may also want to share this behavior as an issue in the Tasmota project (https://github.com/arendst/Tasmota/issues)

Thank you

Cheers

Mark said...

Hello

I upgraded to 8.2 and the problem went away. Then I realised HA was calculating the formula. After checking the console, I noticed it was reporting the right value but it was missing the quotes... so I downgraded back to 8.1 and everything is fine now and I could change the name.. very odd..

16:25:47 MQT: tele/StudioPower/RESULT = {"SerialReceived":FE01083233000050230000402211000050000004392600521300008241260000100000CF}

notice no quotes on 8.2 and 8.1 below

17:04:20 MQT: tele/StudioPower/RESULT = {"SerialReceived":"FE01085234000094230000871906000050000068831400283100003386140000100000C3"}

Angelo said...

Hi , thank's to your post i was able to do what you wrote but instead of using home assistant, i'm using openhab...could you help me please to find a way to analyze and convert the string from the MQTT message into some ussefull items? thank you very much

Angelo said...

Hi,
how to implement this in openhab instead of home assitant?
Thank you very much

ViktorSWA said...

Sorry for my English. I use Google translate.
Changed the electric meter WDS688.
Version Tasmota 8.2.0.
I need your help.
Here is what Tasmota displays in the console:
11:30:57 MQT: stat / tasmota_C557E6 / RESULT = {"SerialSend": "Done"}
11:30:58 MQT: stat / tasmota_C557E6 / RESULT = {"T1": 10, "T2": 0, "T3": 0, "T4": 0, "T5": 0, "T6": 0, "T7": 0, "T8": 0}
What could be the problem?

Creation Factory said...

Hello Angelo,

I don't use OpenHAB so I can't readily help you with your question. Take a look at the part of my post where I detail the meaning of each field in the hex string and how to parse these. The jinja code may also be useful as a starting point for your setup.

I cannot promise when but I have the ambition of implementing logic on Tasmota side for doing the parsing, and exposing the calculated values as sensors, making the integration simpler and similar to other sensors/meters.

Cheers

Creation Factory said...

Hello Unknown,

Could you send more of your log output?

I would expect to see something like this, if the meter chip is responding:

14:43:23 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
14:43:23 MQT: stat/general-meter-switch/RESULT = {"SerialSend":"Done"}
14:43:23 MQT: stat/general-meter-switch/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
14:43:23 MQT: tele/general-meter-switch/RESULT = {"SerialReceived":"FE01080927090053230000236401000050000088120300364900000487030000080000F2"}

Have you set the serial parameters correctly? I.e.:

Baudrate 9600
SerialConfig 8E1

Cheers

Angelo said...

I included the baudrate command inside the system boot rule, because I have the same issue, when you restart the tasmota, it loses the configuration of the baudrate

Angelo said...

I did it by myself, I created a similar rule inside openhab to split the data and parse them. Thank you

ViktorSWA said...

Hello.
Thanks for the answer.
You write: Could you send more of your log output?
Unfortunately in the logs only what I indicated.
There aren't records of the form:
{"SerialReceived":"FE01080927090053230000236401000050000088120300364900000487030000080000F2"}

As for setting up the serial port: I did everything according to your instructions.
How can I check if the serial port settings have been applied?

ViktorSWA said...

Hello.
Thanks for the answer.
You write: Could you send more of your log output?
Unfortunately in the logs only what I indicated.
There aren't records of the form:
{"SerialReceived":"FE01080927090053230000236401000050000088120300364900000487030000080000F2"}

As for setting up the serial port: I did everything according to your instructions.
How can I check if the serial port settings have been applied?

Creation Factory said...

Hi ViktorSWA,

You can check by issuing the commands in the console:

Baudrate
SerialConfig

Cheers

ViktorSWA said...

Hi Creation Factory,

log from the console:

07:19:40 CMD: Baudrate
07:19:40 MQT: stat/tasmota_C557E6/RESULT = {"Baudrate":9600}
07:19:42 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
07:19:43 MQT: stat/tasmota_C557E6/RESULT = {"SerialSend":"Done"}
07:19:43 MQT: stat/tasmota_C557E6/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
07:19:53 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
07:19:53 MQT: stat/tasmota_C557E6/RESULT = {"SerialSend":"Done"}
07:19:54 MQT: stat/tasmota_C557E6/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
07:20:04 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
07:20:04 MQT: stat/tasmota_C557E6/RESULT = {"SerialSend":"Done"}
07:20:05 MQT: stat/tasmota_C557E6/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
07:20:14 CMD: SerialConfig
07:20:14 MQT: stat/tasmota_C557E6/RESULT = {"SerialConfig":"8E1"}
07:20:15 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
07:20:15 MQT: stat/tasmota_C557E6/RESULT = {"SerialSend":"Done"}
07:20:16 MQT: stat/tasmota_C557E6/RESULT = {"T1":9,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}

As you can see, the serial port settings are correct.
But there is no answer in the form: E1:
{"SerialReceived":"FE01080927090053230000236401000050000088120300364900000487030000080000F2"}
Please help me

Creation Factory said...

Hi ViktorSWA,

It is possible that you have different hardware in your device. Is there a chance you can open your device and share photos of the boards?

It would be particularly relevant to know if you have the same metering chip.

Thank you

Cheers

Creation Factory said...

Regarding the change in format of the SerialReceived JSON output, this was the motivation:

https://github.com/arendst/Tasmota/issues/7506

Cheers

ViktorSWA said...

Hi Creation Factory,
Yes, there is such an opportunity. Tomorrow I will take a photo

And one more question ... the state of the switch is not updated after rebooting the home assistant ... only after changing the state of the switch, the home assistant will receive information about this. How to make sure that the state of the switch is sent to the MKTT topic not only when it changes, but with a certain frequency ...

Creation Factory said...

Fyi regarding the format of the SerialReceived, the regression was fixed:

https://github.com/arendst/Tasmota/commit/caff54da7ccadf437a1d5e4bc72599cdc0f530dc

If you can, please build from development branch and give it a try (currently I cannot take my ZMAi-90 out of service for testing :) ).

Thank you.

Cheers

ViktorSWA said...

Hi Creation Factory,

You write:
"Fyi regarding the format of the SerialReceived, the regression was fixed:
https://github.com/arendst/Tasmota/commit/caff54da7ccadf437a1d5e4bc72599cdc0f530dc"

unfortunately, for me it is incomprehensible and difficult.Can you explain more clearly (I am not very good at it)?

Now I also do not have access to the device (quarantine), but the task of reengineering the electric meter is very interesting, and I hope to defeat it :)

Francis said...

I have the same problem as VictorSWA

16:35:15 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
16:35:15 MQT: stat/electricity1/RESULT = {"SerialSend":"Done"}
16:35:16 MQT: stat/electricity1/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
16:35:26 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
16:35:26 MQT: stat/electricity1/RESULT = {"SerialSend":"Done"}
16:35:27 MQT: stat/electricity1/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}
16:35:28 CMD: Rule1 on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog SerialSend5 fe010f080000001c; RuleTimer1 10 endon
16:35:28 MQT: stat/electricity1/RESULT = {"Rule1":"ON","Once":"OFF","StopOnError":"OFF","Free":396,"Rules":"on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog SerialSend5 fe010f080000001c; RuleTimer1 10 endon"}
16:35:32 MQT: tele/electricity1/STATE = {"Time":"2020-04-04T16:35:32","Uptime":"0T01:15:12","UptimeSec":4512,"Heap":29,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"MqttCount":1,"POWER":"ON","Wifi":{"AP":1,"SSId":"SNOW","BSSId":"10:FE:ED:AF:44:7A","Channel":10,"RSSI":80,"Signal":-60,"LinkCount":1,"Downtime":"0T00:00:06"}}
16:35:37 RUL: RULES#TIMER=1 performs "backlog SerialSend5 fe010f080000001c; RuleTimer1 10"
16:35:37 MQT: stat/electricity1/RESULT = {"SerialSend":"Done"}
16:35:38 MQT: stat/electricity1/RESULT = {"T1":10,"T2":0,"T3":0,"T4":0,"T5":0,"T6":0,"T7":0,"T8":0}

but nothing received on tele/electricity1/RESULT

ViktorSWA said...

Hi, Francis.

I still have not solved my problem. Unfortunately, I am quarantined by the coronovirus.
I hope in the near future I will be able to access my device. I will take a photo of the processor.
Let's hope that dear Creation Factory will help us.

ViktorSWA said...

Hi Creation Factory.

Please see the photo of the chip from my electric meter. (link below).
https://www.dropbox.com/s/h6zu6dqw5og0dq1/2020-04-13%2018.18.03.jpg?dl=0

If there are suggestions what could be the cause of my problem, I will be very happy to receive help from you.

Thanks!

Creation Factory said...

Hi ViktorSWA, thanks. It pretty much the same board and chip as in the ZMAi-90. Have you ever used your the device with the original firmware? Did it work?

If so, by any chance you have a copy of the original firmware? That could be helpful in understanding if the command set is the same of if the manufacturer changed the commands (technically, the V9821 chip is also programmable and quite versatile just like an ordinary microcontroller).

Cheers.

ViktorSWA said...

Hi Creation Factory.

Yes, I tested the device with original firmware. It worked. Unfortunately, the original firmware was not preserved.
I think my V9821 chip has a different firmware and command system. My device has a different name - WDS-688. I have a digital oscilloscope, but, I'm afraid, without the original firmware, this will not help.
Do you have any further suggestions?

Unknown said...

Hi Creation Factory,

Firstly, thanks for an in-depth analysis of the ZMAi-90. It helped get my meter integrated with HA (or hass, hassio, home assistant core or what ever it is called this week).

Just a few issues,
1) I think something has changed in Tasmota 8.2, and is referred to above about the serial received format.
I noticed that I was not getting what I expected via MQTT;
expecting "SerialReceived":"FE0108000000009123000000000000005000000000000000000000000000000010000017"
getting "SerialReceived":FE0108000000009123000000000000005000000000000000000000000000000010000017
Notice the second set of quotes around the data have gone missing.

To solve this I modified your sensor.yaml example
I decided to simplify by removing the JSON decode which is now broken, and go for direct string slicing on the full received data. The changes are simply using 'value' instead of 'value_json.SerialReceived', and a change to the index of characters picked up. I also fixed 'payload_len' at 8 as I couldn't see why we should calculate it every time.

#Example fo picking up voltage reading
- platform: mqtt
   name: "Server Room Mains Voltage"
    state_topic: "tele/unit1/Serverroom-EM/RESULT"
    value_template: >-
        {% set payload = value[32:40] %}
        {% set payload_len = 8 %}
        {% set result = namespace(value='') %}

        {% for i in range(0, payload_len + 1) | reverse -%}
            {%- if i is divisibleby 2 -%}
                {%- set result.value = result.value + payload[i:i+2] -%}
            {%- endif -%}
        {%- endfor -%}

        {{ (result.value|float) / 10 }}
    unit_of_measurement: 'Volts'




2) Another poster has also said above, that the command BaudRate 9600 does not seem to be 'sticky'. I.e. on a reboot, the data messages are not seen over MQTT. On typing Baudrate, you can see that 9600 is apparently set. However, typing Baudrate 9600 then makes the messages flow again.

I haven't been able to fix this yet.

I think all of these are issues in Tasmota 8.2

3) I wondered if I would have a black start issue... In my configuaration, the wireless access point is on the load side of the energy meter. On power cycle, the access point will be off, and no way to access the ZMAi-90 until that AP is powered. However, it appears that the relay is bistable and retains its last known state. I am sure I could engineer a black start problem, buy turning off the ZMAi-90 and then removing its supply. On start up it would still be off, with no live access point with which to control it. In practice this is unlikely. Short term 'ghost switching' can be tollerated as this is supplying a UPS anyway. Long term blackouts where the UPS will have shutdown too, might need some looking at.

Thanks again for a great analysis of the ZMAi-90.

Unknown said...

Just to clarify the fix given by Angelo, which has also now fixed my issue. I think it is also ViktorSWA's issue.

Change the command issued at the console to;
Rule1 on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog Baudrate 9600; SerialSend5 fe010f080000001c; RuleTimer1 10 endon

Note that Baudrate 9600 is added to the sequence.

With Tasmota 8.2 (the only one I have tested with), the serial data does not flow until the Baudrate command is issued after boot (even though it is set to the same value).

My meter now properly communicates after a power cycle.

Thanks again.

Creation Factory said...

Hi all,

Responding in sequence of arrival of the questions:

@ViktorSWA: possibly there are differences in the command set, yes. Regarding it being a WDS688 and not a ZMAi-90, I don't think that is necessarily the cause because people have been reporting success with the conversion of either device.

You could potentially go brute force and vary the 3 header bytes to see if you obtain an answer:

fe010f080000001c

There is however some risk by doing this, as you may discover first some command that could be changing calibration data or something worse.

@Francis do you also have a WDS688? Do you have a backup of the original firmware?

btw, you are also setting the serial port to the correct parity settings? I.e.: {"SerialConfig":"8E1"}. Take into account the comment provided by @Unknown.


@Unknown, regarding the "SerialReceived" with the hex data not delimited by " ", there is a fix for that already in the Tasmota project (is currently in the development branch, should go in the next release).

Regarding the bitrate setting not being sticky I will check if I can reproduce the issue, and in that case if there was a fix already.

Cheers

kvvoff said...

Hello! You did a fantastic job of reverse engineering this device. I reflash it on tasmota 8.1 and it works great. But my question is: is it possible to make the relay remember its previous state? It is very inconvenient when the relay always turns off when power is applied. Regards

kvvoff said...

At the moment, I solved this problem by creating and restoring a scene in HA, but this is not an entirely elegant way.

#
- alias: 'zmai_90_revert_create'
description: '.'
trigger:
- platform: state
entity_id: switch.zmai_90_general_switch
to: 'unavailable'
condition: []
action:
service: scene.create
data:
scene_id: zmai_90_revert
snapshot_entities: 'switch.zmai_90_general_switch'

#
- alias: 'zmai_90_revert_on'
description: '.'
trigger:
- platform: state
entity_id: switch.zmai_90_general_switch
from: 'unavailable'
action:
- service: scene.turn_on
data:
entity_id: scene.zmai_90_revert

Best when the device is self-contained and autonomous

Creation Factory said...

Hi @kvvoff, thanks for your comment. Are you talking to the switch via MQTT?


I've created a post exactly about that topic some time ago:

https://www.creationfactory.co/2019/12/ghost-switching-can-be-bitch-if-your.html

Basically what I believe is happening with you, is not the switch itself that is not remembering the state (the relay inside it is bistable, it actually stays in whatever state you leave it in), but it is related to the mqtt broker and how these are published. The retain settings are tricky because these can cause the behavior you describe: once the device turns on, if it consumes a message that is persisting in the mqtt broker, telling it to turn off, it will turn off all the time.

Even after you disable and configure the retain settings properly as I described in my post, If you still have messages persisting in the MQTT broker, the behavior will happen again. I recommend also deleting the broker database file, for allowing it to start from a clean slate.

Cheers.

kvvoff said...

Thanks for your reply! Yes, I use mqtt. After your comment, I figured out the tasmota and zmai90 saves the state of the relay after a power outage.

Here is what I did:
In ha switch: retain: false
In tasmota console: PowerOnState 3, SwitchRetain 0, ButtonRetain 0, PowerRetain 0
In Tasmota Device Manager: Clear retained

And after that, everything works great, and most importantly - the relay does not click at startup, it remains in the previous state. Thank you very much for your help! I will write about you on my blog

Creation Factory said...

Ok, what's your blog by the way?

Cheers

ViktorSWA said...

Hi all,

Thanks, unknown! After I fulfilled your recommendation (Change the command issued at the console to; Rule1 on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog Baudrate 9600; SerialSend5 fe010f080000001c; RuleTimer1 10 endon),
it all worked for me!
Of course, I had to change the sensor so that the data would be processed correctly.

Question to kvvoff:
You write:
"Here is what I did:
In ha switch: retain: false
In tasmota console: PowerOnState 3, SwitchRetain 0, ButtonRetain 0, PowerRetain 0
In Tasmota Device Manager: Clear retained"
1. Why do all this need to be done?
2. I did not find the following: In Tasmota Device Manager: Clear retained. Where is the device manager?

kvvoff said...

my blog kvvhost.ru (in russian)

@ViktorSWA, you ask:

"1. Why do all this need to be done?
2. I did not find the following: In Tasmota Device Manager: Clear retained. Where is the device manager?"

1. I did this so that the relay retains its status when power is restored.
2. Available here: https://github.com/jziolkowski/tdm/releases

In general, all this is painted by the author of this blog at this link: https://www.creationfactory.co/2019/12/ghost-switching-can-be-bitch-if-your.html

Thank you, @Creation Factory, for this information, friend!


Francis said...

Change the command issued at the console to; Rule1 on System#Boot do RuleTimer1 10 endon on Rules#Timer=1 do backlog Baudrate 9600; SerialSend5 fe010f080000001c; RuleTimer1 10 endon

Did it for me to !