Season 1 Retrospective

[jamesbayly]

This thread is for season 1 indexers to share information about the resources they used to run their indexer. We're interested answers to the following:

Infrastructure

• What cloud provider did you use (or did you run your own hardware)
• How large was your compute and database instances
• Did you run this in a single VM, or did you run all the services in seperate instances
• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node
• What monitoring solution did you use?

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?
• What improvements can we make to the documentation? Was there anything unclear or missing
• Can you please report any bugs you saw in Frontier - Bug Reporting
• What is your view on the user experience of the applications, what can we improve or add in to the indexer app
• Would you be willing to participate in a UX interview with our designers?

[Validatrium]

Infrastructure

• What cloud provider did you use (or did you run your own hardware)

Hetzner dedicated server

• How large was your compute and database instances

Created VM with 4 CPU Cores, 8 GB RAM, 100GB SSD. It was more than enough. The real usage is about: 2 CPU, 4 GB RAM

• Did you run this in a single VM, or did you run all the services in seperate instances

Single VM

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

CPU: 4 cores (~20% load), RAM: up to 6 GB, up to 10GB disk usage

• What monitoring solution did you use?

Zabbix

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

I've noticed that docker setup have all the ports are available from outside.
I believe this may pose a security risk. If this is not made by purpose, would be a good idea to fix this in future updates, using localhost or docker network for example.

• What improvements can we make to the documentation? Was there anything unclear or missing

very detailed documentation, our tech team member did not have any issues along the road

• Can you please report any bugs you saw in Frontier - Bug Reporting
  Discord - bug #1
  Discord - bug #2

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

overall experience is good, everything neat, intuitive and clear.
we do understand that this is the start of the projects and bugs will occur, although we tried reporting to discord channel as much as we can on a daily interaction basis.
a lot was changed along the way, and we are looking forward to report more bugs we will find along the Season 2, to make smooth start of Genesis SubQuery Network!

• Would you be willing to participate in a UX interview with our designers?

yes, we are willing to participate and provide necessary feedback

[WayneWayner]

This thread is for season 1 indexers to share information about the resources they used to run their indexer. We're interested answers to the following:

Infrastructure

• What cloud provider did you use (or did you run your own hardware)

AWS t3.medium & t3.xlarge

• How large was your compute and database instances

2 CPU 8GB RAM 30GB EBS Storage // 4 CPU 16GB RAM 30GB EBS Storage

• Did you run this in a single VM, or did you run all the services in seperate instances

In a single VM multiple docker container.

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

t3.medium with 2 CPU and 8GB RAM was enough. 
It used during indexing about 55.8% CPU. 
Memory usage was around 4GB RAM

Postgres database is still very small after season 1:

bash-5.1# du -hs /var/lib/postgresql/
74.0M   /var/lib/postgresql/

• What monitoring solution did you use?

AWS Cloud Monitoring 
Prometheus + node exporter
Grafana

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Move the unregister feature to the staking dashboard. Caused some confusion in my case because you don't see the withdraw option on your :8000 endpoint after you unregister the indexer.

• What improvements can we make to the documentation? Was there anything unclear or missing

I personally like it more to have a non docker setup. I prefer using Proxmox with VMs and LXC container. Therefore a setup guide with building from source and using systemd is something which can be added in the future.

Some logging documentation would be nice for people who are new to the space.
 e.g. sudo docker logs coordinator_service -f

• Can you please report any bugs you saw in Frontier - Bug Reporting

- Metamask does not automatically receive network settings - https://discord.com/channels/796198414798028831/958834183105577020/961892942686064700

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

I am not a fan of the feature that indexers cannot receive delegations until they have claimed the rewards. I think I understand the background of the whole thing (that new delegators don't pick up the rewards of the previous delegator), however it means for the indexer that he would have to do a lot of claim transactions. If these are inexpensive, an automated solution would be desirable. Otherwise, it might be good to address the problem at the smart contract level and only allow delegators rewards from the era they delegate from.

• Would you be willing to participate in a UX interview with our designers?

Sure

[decisionBasis]

Infrastructure

What cloud provider did you use (or did you run your own hardware)

• We used spare server on Hetzner

How large was your compute and database instances

• Ryzen 3600, 64Gb RAM, 1Tb NVMe

Did you run this in a single VM, or did you run all the services in seperate instances

• Dedicated

What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

• Less than 1% when indexing, however up to 50% load when self-querying in 1000 concurrent channels.

What monitoring solution did you use?

• Grafana, netstat, htop

Product

What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

• Isolate sensitive ports inside docker, or provide guidelines how to do so

What improvements can we make to the documentation? Was there anything unclear or missing

• While spinning documentation is rather extensive, it lacks polish. Screenshots could be downscaled, text can be better written.

What is your view on the user experience of the applications, what can we improve or add in to the indexer app

• It's important to provide documentation on indexer/delegator dashboard - explain what are the epochs, what are the rewards to be claimed by router, etc.

Would you be willing to participate in a UX interview with our designers?

• Absolutely

[Gribnik2K]

Infrastructure

• What cloud provider did you use (or did you run your own hardware)
  Hetzner dedicated server

• How large was your compute and database instances
  Ubuntu server with Core(TM) i9-9900K CPU @ 3.60GHz (Cores 16), 126 GB RAM, 2000GB NVME-SSD (RAID 0).

• Did you run this in a single VM, or did you run all the services in seperate instances
  Fully hosted on one server

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node
  CPU: 16 cores (~3% load), RAM: up to 7 GB, up to 15GB disk usage

• What monitoring solution did you use?
  Nload, htop

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Ports that are open to the outside are dangerous.
I would like to be able to change the Postgre password.

• What improvements can we make to the documentation?
  I did not have any problems at the start and from the technical side everything was fine. But it's not entirely clear how to see the indexing metrics, plans...

• Can you please report any bugs you saw in Frontier - Bug Reporting https://discord.com/channels/796198414798028831/958834183105577020/967152502145449994
  https://discord.com/channels/796198414798028831/958834183105577020/966458448826998846

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

I really liked the testnet. He was simple, but apparently he didn’t understand staking and requests a little. I will try to gain experience in this matter.

Also on the first day in the "ru" group, I wrote my review immediately after the launch of the node.

https://discord.com/channels/796198414798028831/876583868533198918/962095788849262643
A review that found a lot of positive feedback among other SubQuerry Discord members

• Would you be willing to participate in a UX interview with our designers?
  Of course, it would be interesting to discuss and share your vision a little.

[IALT72]

This thread is for season 1 indexers to share information about the resources they used to run their indexer. We're interested answers to the following:

Infrastructure

*** What cloud provider did you use (or did you run your own hardware)**

AWS t3.medium

*** How large was your compute and database instances**

2 CPU 8GB RAM 30GB

*** Did you run this in a single VM, or did you run all the services in seperate instances**

Single VM

*** What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node**

CPU: Undex indexing 49.41% .
Memory: around 4.21 GB RAM
Disk usage: Up to 10 GB

What monitoring solution did you use?

AWS Cloud Monitoring
Grafana

Product
What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Era and epoch counters
Auto refresh for all data infos on admin page

*** What improvements can we make to the documentation? Was there anything unclear or missing**

It would be great to create tutorial videos. Short and easy.

** Can you please report any bugs you saw in [Frontier - Bug Reporting]**(https://github.com/subquery/subql/discussions/categories/frontier-testnet-bug-reporting)

MM call

** What is your view on the user experience of the applications, what can we improve or add in to the indexer app**

In total it is good experience , we should make it more informative and user-friendly.

Would you be willing to participate in a UX interview with our designers?
Sure

[burgheleaico]

Infrastructure
What cloud provider did you use (or did you run your own hardware)

• AWS

How large was your compute and database instances

• t3.medium/2 CPU 8GB RAM

Did you run this in a single VM, or did you run all the services in seperate instances

• Single VM

What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

• ~50%

What monitoring solution did you use?

• AWS monitoring, alert bot

Product

What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

• The current block is ahead of the present

What improvements can we make to the documentation? Was there anything unclear or missing

• Unclear how Override default plan

Can you please report any bugs you saw in

• Metamask problems: https://discord.com/channels/796198414798028831/958834183105577020/966234735242256414

What is your view on the user experience of the applications, what can we improve or add in to the indexer app

• Dark theme)

Would you be willing to participate in a UX interview with our designers?

• No

My discord: AndrN#5174

[FlyVeryHigh]

• What cloud provider did you use (or did you run your own hardware)

AWS

• How large was your compute and database instances

t3.medium 2CPU, 4GB RAM, 30GB SSD

• Did you run this in a single VM, or did you run all the services in seperate instances

Single VM

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

<25% CPU and <2GB RAM <10GB disk space - without load.
Up to 60% CPU and up to 4GB RAM - indexing and querying

• What monitoring solution did you use?

Mode_exporter + Prometheus + Grafana

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

We need console commands for all the services, not only UI. I suppose there are some of them but not documented.
Prometheus endpoints also would be useful for monitoring.
And as was mentioned open ports should be fixed in docker.

• What improvements can we make to the documentation? Was there anything unclear or missing

Docs for testner were clear. Even too much )) But if you mean the real docs for future indexers than I would like to see the hole way becoming an indexer:

1. Requirements, recommended architecture for running subquery services, risks
2. Services installation (both docker and separate services)
3. Options and configs description
4. Staking, choosing subgraphs to index, plans, etc.
5. Monitoring setup

Now I could only find some docs for services install (without docker) on npmjs.com/package/@subql/... but we need much more.

• Can you please report any bugs you saw in Frontier - Bug Reporting

As everybody I had to add network mannually to metamask and also there was some UI bugs, like claiming was not working. But as I understand they were fixed.

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

It is a good starting point. I am not sure that GUI is good for indexers since it can attract guys without tech skills and I am not sure that it is safe for the network. But all in all I like it since I spend less time being indexer. I would move staking/claiming/etc functions to indexer app on the node if it is possible. I would also like to home some more info about services statistics (processed queryes, load and etc.)

• Would you be willing to participate in a UX interview with our designers?

Yes

[arcicomunteanu1]

Infrastructure

What cloud provider did you use (or did you run your own hardware)?
AWS

How large was your compute and database instances?
t3.medium

Did you run this in a single VM, or did you run all the services in seperate instances?
Single VM

What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node?
Less than half of the resources were used

What monitoring solution did you use?
Prometheus + Grafana

Product

What improvements can we make to the services?
Nothing to advise

What improvements can we make to the documentation? Was there anything unclear or missing
Translation into other languages, as it was with the academy

Can you please report any bugs you saw?
Problems with metamask and change of commissions (metamask window did not appear)

What is your view on the user experience of the applications, what can we improve or add in to the indexer app
Restricting delegation due to uncommitted rewards is inconvenient

Would you be willing to participate in a UX interview with our designers?
No, tnx

Thanks to the team, the testnet was easy, good documentation and responsive colleagues in the discord

[skyman-dev]

Infrastructure

• What cloud provider did you use (or did you run your own hardware)
  Contabo

• How large was your compute and database instances
  8 cpu, 400 gb SSD, 30 gb RAM

• Did you run this in a single VM, or did you run all the services in seperate instances
  single vm using docker

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node
  90% of time used I had ~10% of CPU, 4 gb RAM, 7 gb Disk usage; 10% of time due to "coordinator_db" fail we ran up to 100% usage of CPU. Yo can see my logs: [Link](https://github.com/subquery/subql/files/8550509/logs.txt)

• What monitoring solution did you use?
  InfluxDB + Telegraf + Grafana + htop

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?
  I had an issue wit hsubql/subql-coordinator (see logs), other than that everything works. I had issues with your UI.

• What improvements can we make to the documentation? Was there anything unclear or missing
  1. No Infrastructure requirements
2. I had problem to find this link: https://testnet.thechaindata.com/, my suggestion is to add a map of useful links.
3. I would move node install section out from the "Set Up Indexer VM on AWS", i thinks this process is independent and nothing to do with AWS, especially docker installation.
4. Guide "Using Indexer Admin App to Manage the Services" is clear, except of "Proxy Endpoint", please clarify it with the tooltip, or smth like that.

• Can you please report any bugs you saw in Frontier - Bug Reporting
  I had issues with error messages json null, but they were fixed. Critical errors on Profile page to visualize Project list, now they looks fixed.

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app
  It seems nice to have more details on some of the tabs, buttons and actions. Tooltips and/or other types of descriptions are appreciated.

• Would you be willing to participate in a UX interview with our designers?
  Sure, any time

[pjenviri]

Infrastructure

• What cloud provider did you use (or did you run your own hardware)
  AWS c4.2xlarge
• How large was your compute and database instances
  8 CPU Cores, 15 GB RAM, 256GB EBS.
• Did you run this in a single VM, or did you run all the services in seperate instances
  We run in a dedicated single VM
• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node
  on avg -- CPU: ~10% load on 8 cores, ~40% load on 15 GBRAM, ~12GB disk usage
• What monitoring solution did you use?
  AWS Cloud watch, htop command, Node explorer

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?
  The docker setup allows all the ports from outside connections as default. This might be a security risk that could be improved in the next release.

• What improvements can we make to the documentation? Was there anything unclear or missing
  Having more refined example images/screenshots through the docs would help improve the explanations. Also, adding more details on log records would help.

• Can you please report any bugs you saw in Frontier - Bug Reporting
  Sure, we found an unable to claim reward bug. It has been reported by Gribnik2K: https://github.com/subquery/subql/discussions/909. Another bug we found was a visualization bug on the profile page, but it has been fixed.

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app
  Our experience so far with the app is good. By the way, there is room for improvement. In our opinion, it would be good if you could provide more descriptions of each menu. For example, rather than just showing a 'Connect wallet' label on the _Stake and Delegate_ page and _Plan & Offer_ page, it would be better to have s short/gets to the point description of what feature/action the page is provided.

• Would you be willing to participate in a UX interview with our designers?
  Sorry, no, but we could provide more feedback on this overtime.

[minatofund]

Infrastructure

• What cloud provider did you use (or did you run your own hardware)

AWS t3.medium

• How large was your compute and database instances

2 vCPU, 4GB RAM, 300GB EBS Storage

• Did you run this in a single VM, or did you run all the services in seperate instances

Single VM

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

CPU: Peak 52%, Average 27%
RAM: Peak 77%, Average 43%
Storage: 1.9GB

• What monitoring solution did you use?

AWS Cloud Monitoring
Prometheus + Grafana

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Having an extra authentication procedure for entering admin dashboard can be super helpful.

• What improvements can we make to the documentation? Was there anything unclear or missing

I would like to see the request/cli commands to interact with indexer settings, although using UI is great.

• Can you please report any bugs you saw in Frontier - Bug Reporting

We did not face that.

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

The overall experience is good. However, it is better to have more features on projects curation, so the indexers can make the decide on whether index the project or not based on that.

• Would you be willing to participate in a UX interview with our designers?

Not for now, but we would like to do it after phase 2, so we can gather more ideas about it.

[heistlad]

• What cloud provider did you use (or did you run your own hardware)

I was running two separate nodes simultaneously on aws and hetzner!

• How large was your compute and database instances

On aws. I was using t3a.large with 128 GB and on Hetzner I was using 8 Core CPU with 64GM RAM and 512 GB SSD.

• Did you run this in a single VM, or did you run all the services in seperate instances

Single VM!

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

AWS: I could not ssh into it or I could not even access admin panel but Indexer kept running. I could see the status in explorer that it's running fine. I was also able to stake/unstake and change commission but server was entirely inaccessible due to load.
Hetzner: ~10% load, ~10GB disk usage. No real issues faced during entire testnet on hetzner. It was smooth sailing.

• What monitoring solution did you use?

AWS Monitoring + Prometheus + Grafana + Glances

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Error messages could be more precise and clear. Most of the time, errors reported were not clear to me. I had to find my way out using good old trial and error!
Fix the open port access in docker as mentioned by others.

• What improvements can we make to the documentation? Was there anything unclear or missing

I had no trouble

• Can you please report any bugs you saw in Frontier - Bug Reporting

Sure!

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app

I think It was thoughtfully laid out! I like the simplicity of it

• Would you be willing to participate in a UX interview with our designers?

Sure! Any time...

[hitesh23k]

Hey! I'm from team DappLooker. We're interested answers to the following:

Infrastructure

What cloud provider did you use (or did you run your own hardware)

We used Hetzner Dedicated Server

How large was your compute and database instances

1TB Storage, 32GB RAM, AMD Ryzen™ 7 3700X Octa-Core

Did you run this in a single VM, or did you run all the services in seperate instances

Running on single VM and multiple docker containers

What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node

10 - 20% resource was used, not observe any significant load

What monitoring solution did you use?

grafana + prometheus + node exporter

Product

What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?

Inbuild docker support for monitoring and alerting. Add support to update Postgres password.

What improvements can we make to the documentation? Was there anything unclear or missing.

Documentation can be more detailed. people don't know steps of most of things like how can we change commission rate or how can we get a service agreement.

Can you please report any bugs you saw in Frontier - Bug Reporting

I was not able to stake/unstake. There was also a bug, commission is not reflecting even after approving metamask transaction, In some case metamask confirmation dialogue does not open.

What is your view on the user experience of the applications, what can we improve or add in to the indexer app

User Experience is nice. State change from one state to another state is taking lot of time.
Ex. If user click on Start-Indexing button, to display Stop-Indexing state on button is taking time.

Would you be willing to participate in a UX interview with our designers?

Yes, we will love to participate in UX interview.

[DyckGerman]

Hey! I'm from Bitscale team. Sorry for the delay, we provided our feedback early in Discord and now moved it to the Github.

Infrastructure

• What cloud provider did you use (or did you run your own hardware)
  Hetzner

• How large was your compute and database instances
  We used CPX21 server with 3 vCPU, 4GB RAM, 80 GB SSD Storage

• Did you run this in a single VM, or did you run all the services in separate instances
  Single VM

• What was your CPU, Memory and storage usage over season 1, did you observe any weird behaviour with your node
  The average performance metrics were as follows: LA 0.36 on 3 vCPU VM, Memory usage - 1,84GB, Storage - 2.2GB. Highest load was during the start of the Season at 8-9 of April

• What monitoring solution did you use?
  Hetzner Cloud Monitoring
Node exporter + Prometheus + Grafana
Alert manager with E-mail and Telegram alerting

Product

• What improvements can we make to the @subql/node, @subql/query, @subql/indexer, @subql/subql-coordinator, or @subql/subql-indexer-proxy services?
  Add support to update Postgres password. As was mentioned open ports should be fixed in docker. Also time to time there are different errors in logs without precise description. It is absolutely unclear how they affect performance. So it will be useful to create database with different logs warnings and errors

• What improvements can we make to the documentation? Was there anything unclear or missing
  No infrastructure requirements

• Can you please report any bugs you saw in Frontier - Bug Reporting
  • The «Remove» feature doesn’t work for both default and specific plans, it shows the error «invalid BigNumber string» fixed
  • A minor mistake in the pop-up window “Create a Plan”. The template has got a field “Price” meaning “a day”, but I guess, the field should be called “Period” fixed
  • A similar mistake can be found at the pop-up window when pressing the «Remove» bottom. Period is 86400 days. Though, I suppose, there should be seconds instead of days fixed
  • Also, we’ve faced an error while trying to purchase a plan. Despite a successful transaction confirmation, the pop-up window was still showing the loading purchase button along with a network error.

• What is your view on the user experience of the applications, what can we improve or add in to the indexer app
  I think, it’d be great to add a «Network» tab that would include all network parameters. For example, Graph dashboard (https://thegraph.com/explorer/network)
  «Stake & Delegate» tab
  The common sheet in the Delegate section consists has pretty poor useful information about Indexers (there’s no info about remaining capacity and which projects are indexed). Identification is possible only by address, not moniker. And there is also no sort function.
  «Plan & Offer» tab
  • I think it’ll be useful to add to the plan template such parameters as «expiration period» and «max number of Service Agreements per plan». It’ll provide more flexibility for indexers to set different price terms for a specific bunch of requests. Btw, why do indexers need to create several default plans? Because customers would prefer anyway a cheaper offer with the same other parameters.
• I suppose it’ll be more user-friendly if the aggregated info about all active plans for a specific project would be in a separate tab, not in the «Indexers» section. It’ll allow customers to compare and filter plans by price, latency, and uptime.

• Would you be willing to participate in a UX interview with our designers?
  Not for now

Kind Regards,
Bitscale Lab team