OSArch digital infrastructure

This discussion was created from comments split from: Organisation of meetups.

Comments

  • I agree with @Jesusbill and @duncan that BBB is a fine platform for video chat which could be used for osarch monthly meetups. The clear Presenter role makes it more fit for this purpuse than jitsi, imho. For my last office I looked into various solutions, but mostly from the angle of data protection, privacy and GDPR compliance. For that reason I had not looked into recording capabilities yet. It seems though when looking at:
    https://docs.bigbluebutton.org/dev/recording.html
    that the streams recorded are pretty exhausting, enabling many ways for composition later on. It might be though, that the features are only available if you rent/run a server.
    The pricing I have seen (EU companies with EU-servers, to be out of reach of GDPR-colliding Cloud-Act) was around 30-40€ per month. For this price we could also get a server at Hetzner from their special occasions section (last I saw was: unlimited traffic 1gb connection, AMD quadcore 16gb ram , iirc) I need to check again how that page address was. If we went the route of our own server a bunch of other possibilities are available of course. (nextcloud, speckle or things like that (ideally run in docker containers to have them cleanly separated and easily de/activate-able)). If it is a Debian based system I am happy to also help with admin tasks.

    Jesusbill
  • Sorry @Moult , just read your comment now. So we do already have a server at Hetzner? Great! (-:
    I remember there are BBB docker images, which we theoretically could spin up for a test run, not to "pollute" the system.

  • edited March 2021

    @fbpyr it would be so great if you could work with @Moult on things like this, getting more than one person helping with server admin is really important. Especially if we want to host more than just our own websites... and I hope we can host all sorts of things as time goes by.

    We can also start offering free services to members.

  • @Moult I could help with sysadmin stuff as well. I've managed servers for a long time and have experience in creating reproducible setups (infrastructure as code using Ansible or docker-compose). Would be happy to contribute :).

    Jesusbillduncan
  • @fbpyr @tetov if you could help @Moult on sysadmin it would be great! too many things on his plate at the moment ...
    So would it be feasible to have a BBB instance on the current server? I think it would worth the try. At the moment we are about 30 people in the meetups, we can considered 50 or something as a requirement and figure out what resources would be needed for that, if anyone has any knowledge on this. Dion, what are the specs of the actual server?

  • edited March 2021

    Gday all! The OSArch web infra currently is all hosted and managed by me. This is a bad thing! It means I have to do all the work, as well as OSArch's online presence having a bus factor of 1.

    I notice there are some guys joining who have some web experience who may be interested in volunteering to be an OSArch sysadmin! In the interests of transparency I'm starting this thread to outline the (shared) responsibilities and procedure for anyone interested. Pinging @tetov and @denissoto .

    The stack is currently a Gentoo Linux dedicated server that I own by Hetzner. It has logical volumes to expand partitions when needed. The domain name is actually owned by @dimitar but the Bind DNS server is hosted on the same server, with an Nginx web server speaking to PHP via PHP-FPM via ports (not sockets). MySQL is running, and also a PHPMyAdmin interface if thats your thing. Osarch is a single user on the server with their own homedir. Entire server backups are via rsync to a ZFS file store provided by rsync.net. ZFS snapshots provide timed backups. A cronjob runs a mysqldump to a file. Sopel is run in tmux for the IRC chat bot. SSL is via certbot. Wiki is mediawiki. Forums is vanilla. Osarch main site is WordPress, and learn.osarch.org is bespoke php with minor composer dependencies. Blenderbim.org is bespoke static generated pages via bash.

    What you won't find is dockerised, cheffed, or puppetry stuff. This dedicated runs as my personal server, and I tend to play with the setup :) not saying it has to stay this way, of course!

    Obviously it makes no sense to hand out SSH key access without some logical trust building and working together first (say, adding translation module to the wiki). So I propose these steps:

    1. Say hello if you're interested :)
    2. Run through a backup and recovery exercise together via a screen share. This'll make sure we both speak the same language.
    3. Do a simple task with me watching
    4. Agree on some responsibilities, who touches what, and what is hands off vs what we should discuss first
    5. Ssh key share party!

    Responsibilities include updating the OS, renewing certs, updating the webapps (mediawiki, vanilla, wordpress), massaging the webapps (customisations, plugins), debugging if things break, and being someone who can resurrect osarch if a bus hits me.

    Thoughts?

    CGR
  • edited March 2021

    I’m in!
    I put some contact details on my wiki page and I'll start idling in the IRC this afternoon. Contact me and we can set up a time.

    duncanMoultJesusbillCyril
  • It'd be great if someone could add some info to the supporters page listing who has rights to what. I've done this somewhere for who has access to social media accounts, but we also have other things we need to keep track of. Makes it easier for everyone to know who they can ask for help - and supports transparency.

    CyrilJanF
  • From https://wiki.osarch.org/index.php?title=OSArch_Supporters I've now linked to https://wiki.osarch.org/index.php?title=OSArch_Digital_Infrastructure where I've tried to list the info I can find about who has elevated rights to what. Please just check it's correct. It's not because we desperately need to maintain that page, it's just that we do need to keep track of who to contact and keep an eye on who has what and keep the beloved bus factor safe.

    tetovJesusbill
  • @denissoto @tetov @Moult any movement here?

    Can you all help me collect links to website statistics here: https://wiki.osarch.org/index.php?title=User:Duncan/Notes#Online_presence_stats ?

    We need to know more about who and where our site is being visited and linked to. Our own stats are good, but knowing what our presense is out there is extremely valuable. Does anyone have an objection to dropping some html into the subdomains so we can see who links to them? (why is that even necessary) We could use Google Search Console but an alternative to Google should always be considered. I've never heard of any of these alternatives: https://alternativeto.net/software/google-webmaster-tools/

    @dimitar is the osarch domain on auto renewal? I'd sleep better knowing it was. We should find a way to have a backup if you for some reason don't renew it. I don't know what the options are for a domain. Same goes of course for the server.

  • Haven't yet found the time to connect with @tetov.

    Those two stats pages are merely a parsing of the server logs. No other user tracking is done.

    I've heard Matomo is an ethical alternative to Google Analytics. I haven't tried it personally. Thoughts, @tetov?

    duncan
  • edited April 2021

    @duncan @Moult I was just about to link to it. They seem to offer free solutions for open source/non profit. (We can of course also self host)

    I’m very anti-tracking, but from what I’ve read matomo seems to handle privacy well.

    Edit: We should definitely self host, we don’t need the extra features.

    duncan
  • @dimitar is the osarch domain on auto renewal? I'd sleep better knowing it was. We should find a way to have a backup if you for some reason don't renew it. I don't know what the options are for a domain. Same goes of course for the server.

    Yes, it's on auto renew

  • edited April 2021

    @Moult just curious how your costs add up compared to something more automated like digitalocean? And on that note, not sure if there is yet, but we should have a way to donate to osarch via paypal/patreon/etc incl. corporate sponsorships eventually, probably going to your account for the time being

    duncan
  • @dimitar not sure what you mean by "more automated" like digitalocean? Do you mean dedicated vs VPSes? Or dedicated vs cloud? In general, dedicated is always cheaper if you're fully utilising the server at a more or less predictable level. Cloud is only cheaper if you're scaling up or down.

    @duncan is the man to speak to about creating a legal entity representing OSArch. See https://wiki.osarch.org/index.php?title=Organization and https://community.osarch.org/discussion/182/organizational-structure-and-governance-of-osarch - this would form the way for corporate sponsorships.

  • I clearly don't know enough about different solutions, just a little experiment with discourse and digitalocean once, so I will leave it to the experts ;)

  • @Moult said:
    Gday all! The OSArch web infra currently is all hosted and managed by me. This is a bad thing! It means I have to do all the work, as well as OSArch's online presence having a bus factor of 1.

    I notice there are some guys joining who have some web experience who may be interested in volunteering to be an OSArch sysadmin! In the interests of transparency I'm starting this thread to outline the (shared) responsibilities and procedure for anyone interested. Pinging @tetov and @denissoto .

    The stack is currently a Gentoo Linux dedicated server that I own by Hetzner. It has logical volumes to expand partitions when needed. The domain name is actually owned by @dimitar but the Bind DNS server is hosted on the same server, with an Nginx web server speaking to PHP via PHP-FPM via ports (not sockets). MySQL is running, and also a PHPMyAdmin interface if thats your thing. Osarch is a single user on the server with their own homedir. Entire server backups are via rsync to a ZFS file store provided by rsync.net. ZFS snapshots provide timed backups. A cronjob runs a mysqldump to a file. Sopel is run in tmux for the IRC chat bot. SSL is via certbot. Wiki is mediawiki. Forums is vanilla. Osarch main site is WordPress, and learn.osarch.org is bespoke php with minor composer dependencies. Blenderbim.org is bespoke static generated pages via bash.

    What you won't find is dockerised, cheffed, or puppetry stuff. This dedicated runs as my personal server, and I tend to play with the setup :) not saying it has to stay this way, of course!

    Obviously it makes no sense to hand out SSH key access without some logical trust building and working together first (say, adding translation module to the wiki). So I propose these steps:

    1. Say hello if you're interested :)
    2. Run through a backup and recovery exercise together via a screen share. This'll make sure we both speak the same language.
    3. Do a simple task with me watching
    4. Agree on some responsibilities, who touches what, and what is hands off vs what we should discuss first
    5. Ssh key share party!

    Responsibilities include updating the OS, renewing certs, updating the webapps (mediawiki, vanilla, wordpress), massaging the webapps (customisations, plugins), debugging if things break, and being someone who can resurrect osarch if a bus hits me.

    Thoughts?

    @duncan said:
    @denissoto @tetov @Moult any movement here?

    Can you all help me collect links to website statistics here: https://wiki.osarch.org/index.php?title=User:Duncan/Notes#Online_presence_stats ?

    We need to know more about who and where our site is being visited and linked to. Our own stats are good, but knowing what our presense is out there is extremely valuable. Does anyone have an objection to dropping some html into the subdomains so we can see who links to them? (why is that even necessary) We could use Google Search Console but an alternative to Google should always be considered. I've never heard of any of these alternatives: https://alternativeto.net/software/google-webmaster-tools/

    @dimitar is the osarch domain on auto renewal? I'd sleep better knowing it was. We should find a way to have a backup if you for some reason don't renew it. I don't know what the options are for a domain. Same goes of course for the server.

    Hello everyone! Sorry for the delay, I was really busy during last months. I'm afraid I don't have the enough time to dedicate as sysadmin. Sorry for that, I don't want to commit myself in something that I can't really attend right now.

  • No worries @denissoto ! Just a head's up that @tetov is now on board and already has done a few awesome tweaks (loving the new favicons!). This raises the bus factor to 2. In summary:

    • @tetov is now an admin on all webby apps, and has server access and database access
    • @tetov does not have root access to the web server, so server upgrades is still dependant on me alone
    • If I get hit by a bus, @tetov should have all the data he needs to run the server, and migrate to new servers.
    • If I get hit by a bus and the server dies too, @tetov does not have access to the backups, so we need to solve a solution where backups are distributed.
    • If @tetov is an Autodesk sleeper agent and deletes everything, I can still restore from the backups :D
    Jesusbillkrande
  • edited April 2021

    @Moult hej dude - you could just keep a better look out for buses ... seems to me like that would be much easier.
    I've updated https://wiki.osarch.org/index.php?title=OSArch_Digital_Infrastructure#osarch.org

  • If tetov is a sleeper agent AND drives a bus then I reckon we have a problem :D
    Jokes aside a "thank you" to @tetov for stepping in decisively in the infrastructure management

    tetovMoultduncan
  • I try to not fall a sleep while driving buses!

    Happy to help out! :)

    dimitarJesusbillbasweinMoultCadGiruduncan
  • Anyone have experience with this group: https://fosshost.org/
    Maybe they could be a good place to have a whole bunch of services hosted. Then we might be able to get started on having our own BBB and Nextcloud ...
    It would also mean we are supporting the FLOSS infrastructure in general.

Sign In or Register to comment.