New deployer experience

[toma315]

Get it up and running

Having lately decided that it's time to deploy a self hosted solution for our typical use case (user support, the typical case: download a sharing tool, share your desktop, get help, close the connection, forget it until next problem) and we tried out a few of them.

MeshCentral looked to us like a really great tool: we got it up and running within a few minutes and were quickly able to get our first test machine remote controlled. The great collection of tutorial videos helped us advance further.

We are directly running meshcentral as Node app, no containers, behind a HA proxy. We are packing our "Quick Support" binary by packing the Mesh agent together with .msh into a package built via NSIS. We actually even decided to dive into code signing in order to get less warnings on client side when downloading that "Quick Support" beast. Having our logo and custom texts in the client ... all very cool.

Using a web browser as a client with no need to install some binary is great for us.

Our typical workflow: we publish our "Quick Support" agent on our website. Client calls us for help, we point him to the agent which he downloads, starts and "connects" (usually without installation). The client automatically appears in our Support group within Mesh Central and we can proceed to solving the client's troubles.

However, then we ran into ...

Stability issues

Unfortunately, we had a number of stability issues from the beginning.

What we already tried

Apparently we are not the only ones with such problems, however we found a number of issues on Github for some of them, most of them abandoned and closed, because apparently people gave up without ever solving the problems.

We actually tried:

1. the entirety of AgentPing/AgentPong/BrowserPing/BrowserPong stuff
2. tweaking ha proxy settings so that ha doesn't close connections it believes to be idle
3. exposing our Mesh server directly to the Internet in order to exclude proxy problems
4. using different browsers after we have seen some hints that the black screen problem might even be caused on browser side
5. verifying we haven't a general problem with TCP sessions being closed arbitrarily

Problem 1: agent disappearing

User starts agent, it appears in the Support group, we connect to the desktop, everything works smooth. On an arbitrary moment the connection is lost, the agent is disappearing from the Support group. On the client system the agent looks like being still connected, with "Install" and "Connect" buttons greyed out. We have to ask the user to explicitly disconnect and restart the agent.

This happens often. Most connections sooner or later end up this way, usually within minutes, thus before the work is done.

Note that this happens when the connection is actually actively in use, thus, there's no idling involved, though it also happens if we just leave the agent idle.

Obviously, the agent doesn't care about the connection being down and never tries to just reconnect. We thought about adding our own small monitoring thingy restarting the agent if necessary, but then for confidentiality reasons we don't want to deploy agents starting themselves without user interaction.

Problem 2: desktop session freezes / black screen

While using the desktop sharing the screen arbitrarily freezes and usually won't recover. Disconnecting and reconnecting the desktop connection leaves us with the agent side being notified the desktop is being accessed, but on our side we just get a black screen. In rare cases the sharing somehow recovers after minutes or hours. The command connection is not lost. Sharing files, the terminal connection and the agent console still work. It's just the desktop sharing.

Up to now we haven't found another solution to that problem than to restart the agent.

Conclusion

At the moment our evaluation of MeshCentral hasn't been very successful. While we expected from the one-TCP-connection-only approach much less hassles we are actually experiencing the contrary. The initial connections work like a charm, but unfortunately they then appear to be rather clumsy. The fact that there is no auto recovery of a once degraded connection between agent and server has left us very often with the need to tell users to restart connections again and again.

[si458]

the connect and install have two different meanings

install installs the app as a service in the background and window can restart the service etc when things go wrong
with the connect it only connects TEMPORARILY! so if the internet drops the client disappears, THIS IS EXPECTED!

its the same when you click connect it shows up, then you click disconnect or close the app, the client will disappear

as for the black screen issue, this can happen from time to time again because you are running TEMPORARILY as the user and NOT as the system user

what you should be doing is asking the users to click INSTALL then this runs as a service user and has full access so you should have less issues

also without seeing your config.json its hard to say if its maybe a config issue and you have it setup incorrectly

[toma315]

Thank you very much for your answer.

Config

for completeness, our current config.json (well, anonymized)

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "mesh.obfuscated.invalid",
    "WANonly": true,
    "_LANonly": true,
    "sessionKey": "obfuscated",
    "Port" : 1443,
    "aliasPort": 2443,
    "redirPort": 1080,
    "redirAliasPort": 80,
    "AgentPing": 45,
    "BrowserPong": 45
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true,
      "agentCustomization": {
	"displayName": "Obfuscated Helpdesk",
	"description": "Obfuscated Helpdesk Access",
	"serviceName": "obfuscated-agent",
	"fileName": "obfuscated-agent",
	"image": "logo.png",
	"foregroundColor": "30,30,30",
	"backgroundColor": "235,235,235"
      }
    }
  },
  "letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "info@obfuscated.invalid",
    "names": "mesh.obfuscated.invalid",
    "skipChallengeVerification": true,
    "production": true
  }
}

Believe me we have tried lots of {Agent,Browser}{Ping,Pong} variations. However, I'd still prefer to hear something like "wow, of course you did a silly beginner's mistake, just change setting xxx" over being an "interesting case".

Network issues

Well, we are not talking about some outpost in the Australian desert. Network connections are generally rather stable, here, nothing like disconnects every few minutes.

Think of two of my test scenarios:

1. Connected to a server via RDP, starting myself Meshagent and connecting. Seeing like the agent disappears every now and then and using the same (persistent during days without any necessary reconnects) RDP session to restart it whenever Mesh becomes unusable
2. Connected to a PC via RustDesk, running Meshagent on the same PC. Seeing the Meshagent disconnect every few minutes and having to restart it while the RustDesk connection is up for more than 24h. Both being in "Quick Support" mode with no service installed.

We actually run a myriad of remote services on our infrastructure and we would learn rather quickly if we had some troubles with other network based services dropping connections in short intervals. Our customers would call in if e.g. their terminal server sessions disconnected every few minutes.

install versus connect

The main reasons we use "connect" rather than "install" in the quick support case are (well known):

1. some users don't have administrative rights themselves
2. installing a service and having to uninstall it afterwards causes extra efforts, especially when we talk about users who need quite a lot of guidance even to just download and run the agent
3. installing a service and leaving it behind might cause unpleasant discussions about the question if we keep having access to computers once our job is done

We have successfully operated Teamviewer, Anydesk and RustDesk using the same "don't install anything" approach. None of them had an issue which resulted in frequently dropping connections without recovery mechanism. Actually, it's common practice to restore connections even when the client computer changes its network configuration, e.g. moving from Wifi to wired LAN with reassignment of IP adresses, though none of them does that fully reliably.

Black or frozen screen due to user permissions

Again, we have run a number of other remote support tools like Teamviewer, Anydesk and RustDesk with user rights, during our evaluation phase sometimes even in parallel. While permission issues arise in some cases we haven't had that frequent freeze/black screen issues.

Unfortunately we have had analogue freeze/black screen issues with installed agents, though in this case its much easier to recover since restarting the agent service remotely isn't difficult, as long as the agent is still connected and only desktop connections are affected.

[si458]

how are you accessing the web ui?
as based on your config.json you are telling meshcentral the web ui and agents should talk to https://mesh.obfuscated.invalid:2443
but this seems odd if you also have an HA proxy in the mix as well?
generally you would just use port 80/443 on ha proxy then set the port to say 1234 and aliasport to 443, then tell haproxy to talk to meshcentral on port 1234

but also how is ha proxy setup?
is it doing the tls for meshcentral OR is it just a tcp proxy and meshcentral does the tls?
(the 2nd one looks like u are doing based on config.json)

also maybe try the meshcentral assistant rather than the meshagent, the assistant runs as the user and not as admin so you wont see UAC prompts etc but you might have better look as its written differently to how meshagent is written
as again meshagent is really designed to be as a service with 24/7 access, no faffing around, but you can enable consent so no one access the machine without the end users permission

[toma315]

Mesh Assistant

Ok, that's interesting. I actually haven't realized that the Mesh Assistant does not use the same code base as the agent. I thought it actually was only another wrapper around the agent implementation presenting a different workflow to the user. Together with assistantTypeAgentInvite 2 or 3 it's quite a good thing. I'll give it a try and see if it is more stable in our special case.

Well, customizing colors would be a good thing. Our companies logo is basically blue, and blue on blue doesn't look that great. Form follows function, however.

So, in a few days after running our usual test cases I can report back if connections from the assistant are more stable than those from the agent. At least the first trials didn't end quickly with broken connections.

Ports and Proxies

Since we wanted to make sure it's not the proxy causing the issues we decided to use a non-standard port not going through the proxy. Thus, connections via 2443 as in config.json are not proxied, while 443, which we were originally using does. The ha proxy figures out where connections have to be forwarded using SNI. TLS is done by Mesh.

[toma315]

A few days later ...

Use of agents and assistant

We adopted the suggestions in this thread and are now using the assistant for the "quick support" case without installation, and the "agent" only when we want a system to be managable for a longer time, thus when we actually install the agent. The assistant is configured to be auto-connecting, thus the procedure is

• download and start
• assistant connects
• supporter finds the newly appeared system and connects
• user gets a notification and accepts the incoming request to share his desktop

Disconnects

Regarding disconnects we have had much less issues. Both agent and assistant keep their connections up reliably. We haven't had to ask users to re-start their assistant or agent. However, that doesn't necessarily imply remote access constantly works.

Screen freeze / black screen

We still see a lot of screen freezes and black screens. Usually every few minutes.

Agent

Disconnecting/reconnecting the desktop heals freezing/black screens. Since we have configured our agents to allow connection in without user interaction, this allows us to continue our work though having to disconnect/reconnect frequently is still a nuisance.

Assistant

Since the assistant asks for confirmation from the user we cannot disconnect/reconnect without user interaction, and since the freezes occur frequently enough, we changed our policy to ask the user to stay with the computer so that he can confirm our re-connection attempts. Sometimes, connections seem to recover when left alone for a few minutes.

Patterns

• view only desktop sharing does not seem to suffer from the freeze/black screen issue. Desktop connections kept open during the night were still ok when re-starting work the next day, for instance. However, the same connection may run into a freeze just minutes after we start actively using it.
• since connections seem to recover automatically it's also possible that during idle times they freeze and recover without us noticing
• we are unsure if black/freezed screens also occur if we are just watching, thus if the desktop is actively used from someone else, but we had a few issues that might indicate the answer was yes

[Melo-Professional]

I would like add my 2 cents here:
Try "peers".
Check Logs! Turn logs on "log": "*" (it's very process intensive so be careful)
Decrease even more agentPing to, let's say 20.
Try using a database like Mongo and Postgres.

Other stuff I would try turn on/off:
"agentPort"
"RelayDNS"
"mongoDbBulkOperations"
"WebRTC"
"allowHighQualityDesktop"
"compression"
"agentWsCompression"
"wsCompression"
"DesktopMultiplex"

[toma315]

Thanks a lot. After the latest configuration changes the freeze/black screen issue is gone. I am not sure what actually did the trick. We went through the list of settings Melo-Professional suggested under "other stuff." I'd say DesktopMultiplex might be the hottest candidate(?).

If we find some time we'll try out each option individually in order to determine which one actually made the difference (hoping that it's a single one) and report back.

[Melo-Professional]

That's good news! Glad it helped.

[tschettervictor]

Thanks a lot. After the latest configuration changes the freeze/black screen issue is gone. I am not sure what actually did the trick. We went through the list of settings Melo-Professional suggested under "other stuff." I'd say DesktopMultiplex might be the hottest candidate(?).

If we find some time we'll try out each option individually in order to determine which one actually made the difference (hoping that it's a single one) and report back.

I'd be really curious to know what did it for you. I also suffer from this issue but only when remotely supporting agents outside of our local network. We run the server in Hybrid mode.

Around 10 minutes the screen will just go black. The symptoms seem identical. I'll have to ask them to disconnect and reconnect the agent, while other tools like RustDesk work just fine.

Thanks for your time.

[toma315]

It's surprisingly difficult to nail it to one single cause. "agentPort" seems to be out of the game, and DesktopMultiplex (enabled) top candidate.

Please don't forget to exclude all other usual causes, like session timeouts on firewall/filters or proxies and set agentPing/agentPong if necessary.

We actually had the issue internally and externally, Maybe(!) the fact that your internal connections are not suffering from the issue points rather to some session timeout outside of MeshCentral.

Our new issue is that Microsoft has decided to mark our copy of the assistant a virus, though we actually signed it and it is very clear and well declared what it does and what it is meant for.