Help request for Gebhard Combined script

by vk2him on

Hi Gebhard,

I've been playing with your excellent mqtt scripts trying to get the combined to work from https://github.com/gebhardm/flmdisplay , but with no luck.

I have the following displayed when starting the node:

  1. www:combined vk2him$ node serve_flmdata.js
  2. Database flm successfully connected
  3. Detected MQTT service on: 192.168.1.73:1883
  4. Create/connect to config table successful...
  5. Create/connect to data table successful...

However when I navigate to localhost:1080 I get the following displayed

  1. Latest MQTT message: no message received

The mysql table is also empty.

I can connect fine with mosquito with no problem as you can see:

  1. www:combined vk2him$ mosquitto_sub -h 192.168.1.73 -p 1883 -v -t /sensor/#
  2. /sensor/c7c90d029e018769b5fb26380c369845/counter [1428230773,924264,"Wh"]
  3. /sensor/c7c90d029e018769b5fb26380c369845/gauge [1428230775,590,"W"]
  4. /sensor/c7c90d029e018769b5fb26380c369845/gauge [1428230776,586,"W"]
  5. /sensor/c7c90d029e018769b5fb26380c369845/gauge [1428230777,596,"W"]
  6. /sensor/c7c90d029e018769b5fb26380c369845/gauge [1428230778,588,"W"]
  7. /sensor/c7c90d029e018769b5fb26380c369845/counter [1428230779,924265,"Wh"]

As well as the latest versions, I have even tried installing an older 0.9 version of socket.io but it fails too:

  1. npm install socket.io@"~0.9.1"

I have tried your other older scripts by using the 0.9 version of socket.io but I always get this:

  1. Latest MQTT message: no message received

So I was wondering if I have done something wrong, or if my FLM needs something else to make it work? I am running Version 2.32

Thanks
Ian

‹ Voltage measurement Obtain FLM configuration via MQTT [solved] ›
gebhardm's picture
gebhardm |

There was an issue with the MQTT version that was "incompatible" with the one I used; I changed that yesterday; try will a "git pull".

vk2him's picture
vk2him |

Sorry, I'm only new to this - how do I do a git pull please?
Thanks

gebhardm's picture
gebhardm |

How did you get the software in the first place?
If you used git http://github.com/gebhardm/flmdisplay then change into the created folder ./flmdisplay and just perform git pull - this updates the received code to the current state; if you downloaded the software from a release as .zip then delete the complete flmdisplay folder rm -r flmdisplay and get it via git again as above described.
By the way, the socket.io was not the problem, but the mqtt module...

vk2him's picture
vk2him |

Thanks for your help, I downloaded the .zip file however I still have the same Latest MQTT message: no message received problem.

I tried also loading older versions of mqtt with same problem.

I'm running mqtt version 1.1.2 - which one do you use?

vk2him's picture
vk2him |

P.S.
Other than read.me changes, I couldn't see any changes to the files you did yesterday?

gebhardm's picture
gebhardm |

Git is a code management system based on continuous commits... Help yourself to read the pretty good documentation...
If not denoted otherwise, all shall work with a current npm install.
I assume that it is your localhost:1080; why so ever there are computers that point this to 127.0.0.1 and not the machine's IP address; thus, try with <machine's IP address>:1080 - in both documentations, on Ubuntu AND RasPi there was never a word of localhost, but http://"local IP address":1080 (but documentation seems for the ... these days - sorry).

vk2him's picture
vk2him |

Thanks anyway - I tried localhost, 127.0.0.1 and even the ip of the system itself - also tried to connect to it from another computer across the lan but always Latest MQTT message: no message received problem.

i'm running it on a Mac Mini running Yosemite.

gebhardm's picture
gebhardm |

By the way, there are two different things:

1) node serve_flmdata.sh & had an issue with the current mqtt module; after fetching the current version of the "combined script" this should resolve in database entries - better use "git" instead of .zip download.
2) http://<ip address>:1080> relies on correct socket assignment, which partly seems an issue when localhost does not point to the machine's IP address, but "home"; as an alternative the corresponding web server advertises itself as "FLM data processor" in Bonjour, so you may also take this endpoint.

So, progress step by step; if the database gets readings the first half is reached...

gebhardm's picture
gebhardm |

As a test, try https://github.com/gebhardm/energyhacks/tree/master/Graph with changing "serve_mqtt.js" on your fluksometer's address

  1. var mqttbroker = "192.168.1.73", mqttport = "1883";

This works on my iMac also with "localhost:1080" - if that doesn't work out of the box then something "deeper" in your machine is not working...

vk2him's picture
vk2him |

No luck with Graph - I just tried on two Macs - when I connect to localhost:1080 it displays the same page without any graphs - it looks just like running graph.html ... Can you please confirm Flukso Version 2.32 is enough to run this? Thanks

vk2him's picture
vk2him |

I wonder too if this is a Yosemite issue - Is your iMac on Yosemite?

gebhardm's picture
gebhardm |

It works also on an FLM v232 (there are no changes on the MQTT handling) and yes, my Mac runs Yosemite - just did a complete renewal of node.js and its modules to v0.10.38-release and plain "npm update" - thus, everything up-to-date.
By the way: What browser do you use? JavaScript switched on in the settings? - just tested on FF 37.0.1 and Safari 8.0.4.

gebhardm's picture
gebhardm |

In the browser right-click:element information >> does the console show errors (except for missing .ico)

gebhardm's picture
gebhardm |

Folder access rights set correctly for finding the js-libraries? (see also the "symbolic link" issue)

vk2him's picture
vk2him |

Thanks for staying with me ... I just reloaded node.js and modules then npm update .... tried in Safari and Chrome ... no problem with finding js-libraries and links etc.

Looking at you code I saw a comment : // gauge length 2 is sent from Arduino sensors (in my case)

I'm not sure if this might be an issue, I have one flukso clamp connected - is this a Arduino sensor and is teh gauge length also 2?

Thanks

gebhardm's picture
gebhardm |

No, this something different wäre I enrich sensor data without time, no changes for FLM...

gebhardm's picture
gebhardm |

  1. mqttclient.on("message", function(topic, payload) {
  2. ...
  3.         io.sockets.emit("mqtt", {
  4.             topic: topic,
  5.             payload: payload.toString()
  6.         });
  7. });

All messages received are passed on to the socket; function handle_sensor just stores the FLM messages and enriches messages without timestamp.
You may put a console.log into this function to see if the mqtt client works at all:
  1. mqttclient.on("message", function(topic, payload) {
  2.     console.log("message:" + topic + payload);
  3. ...

As written, it "should" work as it does on my Mac (or RasPi)...

vk2him's picture
vk2him |

I added the console.log to a few other lines to diagnose further like this

  1.  // log connection
  2.     if (mqttclient) console.log("Connected to " + mqttbroker + ":" + mqttport);
  3.     // handle socketio requests
  4.     io.on("connection", function(socket) {
  5.         // handle subscription request(s)
  6.         socket.on("subscribe", function(data) {
  7.                 console.log("subscribe:" + data);
  8.             mqttclient.subscribe(data.topic);
  9.             console.log("mqttclient:" +data.topic );
  10.         });
  11.     });
  12.     // handle mqtt messages
  13.     mqttclient.on("message", function(topic, payload) {
  14.     console.log("message:" + topic + payload);
  15.         var topicArray = topic.split("/");

This displays the following output with nothing else:

  1. Ians-MacBook-Pro:Graph vk2him$ node serve_mqtt.js
  2. Connected to 192.168.1.73:1883
  3. subscribe:[object Object]
  4. mqttclient:/device/+/config/sensor
  5. subscribe:[object Object]
  6. mqttclient:/sensor/+/gauge

vk2him's picture
vk2him |

i did some more testing ... I installed node.js onto a windows machine and tried the Graph script with the same problem.

I then tried changing the mqttbroker to a IP address that isn't the FLM and it now says it connects to that IP address which is impossible:
 Connected to 192.168.1.173:1883

So somehow these lines of code isn't working for me on Mac and Windows?

  1.  function mqttconnect() {
  2.     mqttclient = mqtt.connect({
  3.         host: mqttbroker,
  4.         port: mqttport
  5.     });
  6.     // log connection
  7.     if (mqttclient) console.log("Connected to " + mqttbroker + ":" + mqttport);
  8.     // handle socketio requests

gebhardm's picture
gebhardm |

As I do nothing special than using available open source stuff, to narrow the source of better switch over to what JP Mens elaborated on MQTT; it maked IMHO no sense to fiddle around with my scripts when the basic stuff is not working. So, reduce to the minimum to get MQTT working on the node.js level - see http://jpmens.net/2015/02/13/mqtt-article-series/ and other stuff on http://jpmens.net

vk2him's picture
vk2him |

Yes you're right - I'll try to get mqtt working by itself - thanks for your help!

gebhardm's picture
gebhardm |

The if (mqttclient) console.log("... is misleading as a real connect is not indicated by a non-null mqttclient but a mqttclient.on("connect", function() {}); event - I changed that in the code and assume that your mqttclient never really connected; what network segments do you use? As the mdns detects it, well, I have no clue what happens...
Thus I would recommend to strip everything down to the minimum (actually the serve_mqtt.js cannot do much less) and take a closer look at the mqtt connect or error?!? (there is also a mqttclient.on("error", function(){}); event)

vk2him's picture
vk2him |

Thanks Gebhard, I'll try those functions to further fault-find. I have a very simple network - an ADSL router/wifi modem with the FLM connected via wireless. the only change I've made to the FLM is to disabled the firewall so I can connect via wireless.

gebhardm's picture
gebhardm |

If your WLAN is in another IP-area than the LAN (for example 192.168.1.x versus 192.168.0.y) then the network mask might prevent a connect from a LAN attached computer if it is set to 255.255.255.0; then it should be 255.255.254.0 to allow reaching n.m.0.x and n.m.1.y - but this is just a guess as even the simplest network installation may have its surprises :-) (when I plug in a WLAN-router "tp-link tl-wr702n" to gain WLAN access for a tablet exactly this prevents reaching my FLM if the router is not "just an access point" to the LAN) - but this is computer "science": Why simple if it can be complicated; we do it not because we have to but we can (I hate theses guys)

vk2him's picture
vk2him |

Thanks - all my devices are on the same sub-net - 192.168.1.xxx - the FLM is .73 and the MacMini is .79 ... I'm still running checks, it's very strange that the mosquito command mosquitto_sub -h 192.168.1.73 -p 1883 -v -t /sensor/# works fine ...

gebhardm's picture
gebhardm |

"Was lange währt wird endlich gut" - I guess I fixed it...
With the new version of the mqtt module the payload format changed to BUFFER; this makes the payload handling a bit more tricky than beforehand and what seemed to have caused problems with a fresh npm install.
At least on my Raspberry Pi after a npm update node all node modules (mqtt, mysql, mdns, socket.io) are on the current release and FLM messages (and Arduino messages) are "still" shown in the graph...
Thus, in contrary to the "never change a running system" you may now check again (hopefully the last time) and enjoy a working combined visualization.
I will never understand why people change public interfaces... (and sorry for any inconveniences)
https://github.com/gebhardm/flmdisplay

vk2him's picture
vk2him |

I am sorry Gebhard, I have updated to your latest combined version and performed again npm install mqtt mdns socket.io mysql but I still get Latest MQTT message: no message received.

This is very strange as your code successfully finds my FLM from the following, so it's nearly working:

  1. www:combined vk2him$ node serve_flmdata.js
  2. Database flm successfully connected
  3. Create/connect to config table successful...
  4. Create/connect to data table successful...
  5. Detected MQTT service on: 192.168.1.73:1883

vk2him's picture
vk2him |

I've done further testing by inserting console.logs at various points in serve_flmdata.js .. all seems OK until this line as it never displays anything:

  1. // handle mqtt messages
  2.     mqttclient.on("message", function(topic, message) {
  3.         var topicArray = topic.split("/");
  4.         var payload = message.toString();
  5.        
  6.          console.log("topicArray payload " + topicArray + payload);

gebhardm's picture
gebhardm |

...so actually there is never received a message... No clue what's going on within the mqtt module...

gebhardm's picture
gebhardm |

Corresponding to your post from 02:06 you didn't even get a connect, so here starts the search:

  1. // log connection
  2. mqttclient.on("connect", function() {
  3.     console.log("Connected to " + mqttbroker + ":" + mqttport);
  4. });