How to setup my own server?

[bazooka70]

I have a machine attached to our network, that run a full node Bitcoin Core (both main and testnet3).
Is it possible to install the QBitNinja server on that machine without the need of an extra Azure VM?

Although there are some documentations regarding the issue, I feel lost at how to exactly configure my own server for both main and testnet.

The public QBitNinja server works great, but it is somewhat slow and is mentioned that it is not suitable for production. Can you please help me (and others) ?

I'm not familiar with Azure at all, but willing to learn. (BTW, too bad we can't use QBitNinja server with a local SQL-Server db and should depend on a service we can't control such Azure)

[NicolasDorier]

@bazooka70 you can't depends on a sql server for QBitNinja, it need Azure services.

If you really want a block explorer not coupled with azure and don't mind it is not indexing everything (only specific HDPubKey), you can take a look at https://github.com/dgarage/NBXplorer

Sadly I can't spend time explaining you how to setup.
I mention the component in the readme.

I regret it is not easier to install. I was looking to migrate QBitNinja to .NET Core so I can make a one click install easily, but some component were missing.

I advise you to hire an .NET developer familiar with Azure for a day in order to setup this for you, it should be rather obvious for them. (I sadly don't have time on my side though :( )

[bazooka70]

@NicolasDorier , Believe me I trying very hard to learn your code, and learn about azure. I actually managed to go feather with QBitNinja. I opened azure storage, a bus, and run QBitNinja.Listener.Console.exe -- init on my network machine with a local synced node of Bitcoin Core on it, in the hope that I would not need a VM on azure.

I configured QBitNinja.Listener.Console.exe.config : Azure.AccountName, Azure.Key, ServiceBus (i.e. Endpoint=sb://...)

All the other parameters are default.

When I run it, I can see the tables created on my azure storage (i.e balances, callbacks, chain, etc...) but then it throws an exception:

D:\QBitNinja.Listener.Console>QBitNinja.Listener.Console.exe --init
QBitNinja.Listener Information: 0 : Start initial indexing

Unhandled Exception: System.Net.Sockets.SocketException: No such host is known
   at System.Net.Dns.GetAddrInfo(String name)
   at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
   at NBitcoin.Utils.ParseIpEndpoint(String endpoint, Int32 defaultPort)
   at NBitcoin.Indexer.IndexerConfiguration.ConnectToNode(Boolean isRelay)
   at QBitNinja.InitialIndexer.Run(ChainBase chain) in D:\Projects\QBitNinja-master\QBitNinja\InitialIndexer.cs:line 125
   at QBitNinja.Listener.Console.Program.Main(String[] args) in D:\Projects\QBitNinja-master\QBitNinja.Listener.Console\Program.cs:line 101

So what am I doing wrong?

[bazooka70]

OK, this seem to work now - i.e. something is happening ...

QBitNinja.Listener Information: 0 : Enqueing from 351900 1500 blocks...

I missed the <add key="Node" value="unused" /> key, and changed it to localhost.

I'm not sure what to do next after it finishes indexing...
meaning how to set the QBitNinja client to connect and fetch information from my network machine?
e.g.

var client = new QBitNinjaClient("myip", network);

[NicolasDorier]

so you are good at this point.

It just takes LOTS of time to index everything, and lot's of bandwidth, which is why I advice running the indexer in the same data center than the storage.

If your VM is not in azure it will take forever. (Might take 5 days even inside azure)

[NicolasDorier]

So QBitNinja is divided into two projects

• The listener that you just set up correctly
• The front end

The front end project (QBitNinja project in the solution) is a ASP.NET WEB API which you can easily host as an Azure Web App for example. You can also use your own server of course, this is less critical than the Listener.

The front end settings are the same as the listener.

The listener index all data by asking your full node.
The front end just query the index.

Yeah it is a mess to setup. However, once it is on it is rock solid :)

[bazooka70]

Our basic strategy is to host everything on our servers which can control, this is why I think a local index database (not azure) such as sql-server, mysql, sqlite could be a good idea. But I might consider staying with azure if everything works fine and fast. in any case while indexing the QBitNinja.Listener.Console.exe it trowed an exception while processing these lines:

...
QBitNinja.Listener Information: 0 : Enqueing from 509600 1200 blocks
QBitNinja.Listener Information: 0 : Enqueing from 510800 546 blocks
QBitNinja.Listener Information: 0 : Dequeuing index jobs
Work remaining in the queue : 1400
QBitNinja.Listener Information: 0 : Processing default/blocks- 0-135300
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed

The exception:

Unhandled Exception: Microsoft.ServiceBus.Messaging.MessagingCommunicationException: The request was aborted: The request was canceled. ---> System.ServiceModel.CommunicationException: The request was aborted: The request was canceled. ---> System.IO.IOException: The request was aborted: The request
 was canceled. ---> System.Net.WebException: The request was aborted: The request was canceled.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__45.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.WebSocketConnectAsyncResult.<GetAsyncSteps>b__14_1(WebSocketConnectAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Messaging.Channels.SharedChannel`1.CreateChannelAsyncResult.<GetAsyncSteps>d__6.MoveNext()
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.Start()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Channels.SharedChannel`1.OnEndCreateInstance(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.SingletonManager`1.EndGetInstance(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_1(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass8_1.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory`1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_3(RequestAsyncResult thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Common.AsyncResult`1.End(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.EndRequest(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.RenewLockAsyncResult.<>c.<GetAsyncSteps>b__10_1(RenewLockAsyncResult thisPtr, IAsyncResult a)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.EnumerateSteps(CurrentThreadType state)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageReceiver.OnEndRenewMessageLocks(IAsyncResult result)
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.ReceiveContext.EndRenewLock(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.BrokeredMessage.EndRenewLock(IAsyncResult result)
   at Microsoft.ServiceBus.Messaging.BrokeredMessage.RenewLock()
   at QBitNinja.InitialIndexer.Run(ChainBase chain) in D:\Projects\QBitNinja-master\QBitNinja\InitialIndexer.cs:line 199
   at QBitNinja.Listener.Console.Program.Main(String[] args) in D:\Projects\QBitNinja-master\QBitNinja.Listener.Console\Program.cs:line 101

I run QBitNinja.Listener.Console.exe gain, and after a few minutes the same exception happen.

What could cause this and how to proceed?

[NicolasDorier]

@bazooka70 QBitNinja is too heavily coupled to Azure (many optimization trick only possible on azure), it is impossible to change the backend.

Concerning the exception, it is the first time I see this one. It might come from disconnection with Azure.

You can run on your server, but really, the initial indexing is VERY long and this exception is probably caused by this. At least use a server in azure for the initial indexing, then once it is over migrate on premise.

That said, you are trying NBXplorer, I think that for your case it might be enough and way easier.

[bazooka70]

@NicolasDorier Now this exception is constant every time I run it:

Work remaining in the queue : 1400
QBitNinja.Listener Information: 0 : Processing default/blocks- 0-135300
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed
QBitNinja.Listener Information: 0 : Lock renewed

Only god knows what "Lock renewed" means.
This is exactly the reason I was against azure in the first place. Azure reports no problems and I'm lost at what could be causing this...

That said, you are trying NBXplorer, I think that for your case it might be enough and way easier.

NBXplorer loos very promising to us, but I want a plan B. it has become a bit of a challenge for me to beat azure and install QbitNinja on my server! :)

[NicolasDorier]

Lock renewed is normal.
Basically the init process has been done so one can use multiple listener to index at the same time.

The idea is (only for initial indexing):

A server split the blocks in several batch called work unit and push each work unit in a queue.
Once done, if you have like 5 indexers, each indexer will dequeue a unit of work and process it in parallel.

However, it might happen that an indexer get disconnected (crashed), in which case the unit of work should go back in the queue after a timeout.

The periodic lock renewal is a way for an indexer to prevent the unit of work to go back in the queue, by saying to azure "I am still processing this message".

This stuff happen only during the initial indexing.

Now the exception you get and that I never got is probably caused because the unit of work take very long time to process due to the huge bandwidth and latency to the Azure server. This is why I advise you to run this indexer only for initial indexing directly in an Azure VM located in the same datacenter as your storage.

[bazooka70]

Just a guess, maybe the free trail account on Azure I have now, is limiting requests... dunno :/
I will deferentially try to focus on NBXplorer now, and leave this battle to another day, maybe.

[NicolasDorier]

I don't think this is rate limiting issue. But really you can easily take a look by running the indexer from a VM hosted in Azure.