[Adminsysters] [mur.at #9912] server disconnected

Fri Jan 21 10:12:19 CET 2022

On 21/01/22 00:30, info at ooooo.be via RT wrote:
> Hi
> and we don't see this peak
> :/ - i checked with nload and nethogs - but maybe we need btter
> minitoring.+ we are working on it *
> xm

hem, yeh,

we need something that collect and store datanetwork, I think that from 
load we see only the current values and the total.

I see that Ralph is using Munin, but maybe for us could be better use 
Prometheus.

I have a bit of experience on it and I use for monitoring/testing the 
antennas in a mesh network.

Hem.. but after.. to debug.. I'm not so expert, my approach, before 
enter in the world of logs... could be monitoring and keep up a service 
for time. To understand which service is producing the issue, and after 
read the logs.

Thanks for the rate-limit, so we don't produce too noise in your network 
capability and in the same time the services are up and we can 
debug/reinstall/whatever..

:) We have a working session today later.. 17:30 cest

Ralph, can you explain better this frase:

<<restricting traffic at our edge router was a few hops too late, obviously.>>

I have completly not Idea of your network, so... are you saing that we 
saturate the LAN, so the trick bandwidth limit that you added, is not 
enought to live in peace together. And if I understand well... you are 
sayng that is better if we put soon  a limit to our network use.

for us is ok.. it's only to understand the priority. thanks for all!

ignifugo

hugs ignifugo

>
>
> On 20/01/2022 22:57, Ralph Wozelka via RT wrote:
>> and this screeny
>>
>> On 20.01.22 22:23, Ralph Wozelka wrote:
>>> FYI, adele is maxing out its port limit hard!
>>> Your services cannot be reached from the outside since 30 minutes.
>>>
>>> see scrshots attached
>>>
>>> cheers,
>>> ralph
>>>
>>> On 20.01.22 21:38, Ralph Wozelka via RT wrote:
>>>> Dear adminsysters,
>>>>
>>>> around 8pm CET our core switch went into saturation again.
>>>>
>>>> this time we put a rate-limit on right onto the network port restricting to
>>>> 30Mbps in/out.
>>>> this should do the trick finally.
>>>> restricting traffic at our edge router was a few hops too late, obviously.
>>>>
>>>> see screenshot of traffic incoming from the core network at our edge route
>>>> v767-fe0-r1ko.mur.at
>>>>
>>>> let us know how adele performs on your end! :-)
>>>> cheers,
>>>> ralph
>>>>
>>>>
>>>>
>>>> On Thu Jan 20 12:48:15 2022,info at ooooo.be  wrote:
>>>>> FIY # we did a full backup again to check the restricted rate limit we
>>>>> put, we expect the next backups to be incremental and much swifter.
>>>>>
>>>>> And we soon try to formulate a eeply.
>>>>> Thanks already for the support.
>>>>>
>>>>> XmOn Jan 19, 2022 9:40 PM, Djamil Vardag via RT<amt at mur.at>  wrote:
>>>>>> Hey you all
>>>>>>> Hey ralph
>>>>>>> Hey mur.at
>>>>>>>
>>>>>>> We are studying to put some bandwidth limit and monitor so we can
>>>>>>> troubleshoot better the machine. But where to put this limit? in
>>>>> our
>>>>>>> side or in your gateway? And how much ? Like during the night we
>>>>> can use
>>>>>>> the entire bandwidth for the server upload and download..
>>>>>>> and during the day (8:00-22:00) Adele can use... eg. 30MB/s for
>>>>> upload
>>>>>> You can use 20 Mbit up/down, that would be ok for now. We have to
>>>>> discuss that
>>>>>> internally.
>>>>>>> and 10MB/s for download ? Or what do you suggest ? We need again
>>>>> access
>>>>>>> again - port 22 incoming is not enough to troubleshoot to check
>>>>> our borg
>>>>>>> backup script - for that we need for sure 22 outgoing port.
>>>>>> Ok, for now we have tried to implement a iptables rate limit. You
>>>>> should have
>>>>>> full TCP access to adele again, UDP is still blocked.
>>>>>> It would be ok for us if you use the available bandwidth as long as
>>>>> our
>>>>>> services are still working fluidly, but that isn't that easy to
>>>>> implement.
>>>>>>> It can also be our peertube instance. We disabled the webtorrent.
>>>>> We
>>>>>>> understand that peertube needs specific requirements -
>>>>>>> https://docs.joinpeertube.org/install-any-os?id=installation
>>>>>>> No adsl - you have a 100M/Bit symmetric fiber. Is that dsl? Do you
>>>>>> This is a 100/100 MBit fiber, no DSL, the bottleneck is the fiber
>>>>> and one
>>>>>> 100MBit switch wich will get overwhelmed if there are to many
>>>>> packets.
>>>>>>> suggest that we can run our peertube instance on your
>>>>> infrastructure?
>>>>>>> Our users are very limited and moderated. Also we don't have a lot
>>>>> of
>>>>>>> material on it. It is rather an experimental instance.
>>>>>> Ok, i guess this would't be a problem for our connection. Webtorrent
>>>>> suggests
>>>>>> that it uses less bandwidth on our end.
>>>>>>> We do understand your decision but for the moment we our server
>>>>> back
>>>>>>> running.
>>>>>> Sure, we can understand that and sorry that we had to be so drastic,
>>>>> but we
>>>>>> didn't have a choice as all of mur.at wasn't working well and we are
>>>>> loosing
>>>>>> our face in front of our members and financers.
>>>>>>> We want to install a monitoring software like netdata -
>>>>> promotheus.
>>>>>> Seems like a good idea
>>>>>>> Or do you have any suggestions to tackle the bandwidth problem? We
>>>>> have
>>>>>>> currently a funded project running until May so an online presence
>>>>> +
>>>>>>> visibility is importantand urgent- even while we are learning.
>>>>>> The problem is that we don't have any QOS on our switches enabled
>>>>> yet, to
>>>>>> tackle rate limitting there. We need to learn and implement that
>>>>> first.
>>>>>> We tried establishing a 10Mbit link to your machine, but it didn't
>>>>> like that.
>>>>>> The reasoning being that you could still use the machine, and we
>>>>> would have a
>>>>>> pseudo rate limit.
>>>>>> You mentioned a daily backup, maybe this is the culprit. As i said
>>>>> your machine
>>>>>> made traffic of 8TB in one week, that is just too much.
>>>>>> There are some websites and services of us that are also funded and
>>>>> we could
>>>>>> get a problem with renewing these services, which could cost us
>>>>> thousends in
>>>>>> the future.
>>>>>>
>>>>>>> Can you also add this emailsystercloud at grrlz.net  to the mur.at
>>>>>>> mailinglist
>>>>>> we would suggest to putsystercloud at grrlz.net  as a forwarding
>>>>> address of donna
>>>>>> (donna at mur.at), who already is on all the lists.
>>>>>>> xm
>
>
> _______________________________________________
> Adminsysters mailing list
> Adminsysters at lists.genderchangers.org
> https://lists.genderchangers.org/mailman/listinfo/adminsysters
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.genderchangers.org/pipermail/adminsysters/attachments/20220121/799a0d90/attachment.html>